{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
]
}
]
}
val j = os.read(os.pwd/RelPath("src/main/scala/metadata.json"))
val jsonData = ujson.read(j)
ujson.Value$InvalidData: Expected ujson.Arr (data: {"dataflows":[{"name":"test","sources":[{"name":"person_inputs","path":"/data/input/events/person/*","format":"JSON"}],"transformations":[{"name":"validation","type":"validate_fields","params":{"input":"person_inputs","validations":[{"field":"office","validations":["notEmpty"]},{"field":"age","validations":["notNull"]}]}},{"name":"ok_with_date","type":"add_fields","params":{"input":"validation_ok","addFields":[{"name":"dt","function":"current_timestamp"}]}}],"sinks":[{"input":"ok_with_date","name":"raw-ok","paths":["/data/output/events/person"],"format":"JSON","saveMode":"OVERWRITE"},{"input":"validation_ko","name":"raw-ko","paths":["/data/output/discards/person"],"format":"JSON","saveMode":"OVERWRITE"}]}]})
match may not be exhaustive. It would fail on the following input: (x: Int forSome x not in 0)
Hi! :wave:
Does anyone want to review this PR I opened to fix the CI (it wasn't running the tests): com-lihaoyi/upickle#359 ?
Thanks!
Then we can trigger a release @htmldoug @objektwerks
upickle
to extract my case classs schema. The visitor that uPickle provide is great. I would like to have a visitor on the type if you know what I mean:case class Foo(..)
macroRW
@lihaoyi I've been going through your wonderful tutorial on Visitor pattern yet again. I want to modify it slightly to include a cache based early stop for recursive Data structure. Think of an ADT which is self referencing. With the pattern in the tutorial, there is no way for visitor to communicate back to dispatcher to stop going further.
So I came up with this approach, could you please comment on what you think about it:
trait Visitor[T] {
def visitPrimitive(tpe: Type): T
def visitOption(innerVisited: T): T
def visitArrayType(innerVisited: T): T
def visitMapType(keyVisited: T, valueVisited: T): T
def visitStructType(tpe: Type): StructVisitor[T]
def stopValue: Option[T] = None // If defined, we stop visiting and simply return this value
}
trait StructVisitor[T] {
def visitFieldKey(name: String): Visitor[T]
def visitFiledType(name: String, visitedValue: T): Unit
def done: T
def stopValue: Option[T] = None // If defined, we stop visiting and simply return this value
}
This is a visitor to visit Type
from scala.reflect
java.lang.IncompatibleClassChangeError: class ilsop.models.RiskDecompositionSearchResult$$anon$10$$anon$11 has interface upickle.core.Types$CaseR$CaseObjectContext as super class