{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
]
}
]
}
which is in a file called "metadata.json"
and I read just by doing:
val j = os.read(os.pwd/RelPath("src/main/scala/metadata.json"))
val jsonData = ujson.read(j)
however, when I try
jsonData(0)
I get
ujson.Value$InvalidData: Expected ujson.Arr (data: {"dataflows":[{"name":"test","sources":[{"name":"person_inputs","path":"/data/input/events/person/*","format":"JSON"}],"transformations":[{"name":"validation","type":"validate_fields","params":{"input":"person_inputs","validations":[{"field":"office","validations":["notEmpty"]},{"field":"age","validations":["notNull"]}]}},{"name":"ok_with_date","type":"add_fields","params":{"input":"validation_ok","addFields":[{"name":"dt","function":"current_timestamp"}]}}],"sinks":[{"input":"ok_with_date","name":"raw-ok","paths":["/data/output/events/person"],"format":"JSON","saveMode":"OVERWRITE"},{"input":"validation_ko","name":"raw-ko","paths":["/data/output/discards/person"],"format":"JSON","saveMode":"OVERWRITE"}]}]})
Any help would be greatly appreciated. Thanks!!
Next best would be to either convert to Value and count the fields or write a Visitor.Delegate to wrap the Reader and count them that way.
can you minimize a repro and open a ticket
match may not be exhaustive. It would fail on the following input: (x: Int forSome x not in 0)
Currently I use an @nowarn annotation to ignore all such errors. Is there an expected fix to this in a forthcoming version of Scala and/or uPickle? Or is this with us for a long time to come?
Should be fixed in https://github.com/com-lihaoyi/upickle/compare/1.4.0...master. Just needs a release.
Hi! :wave:
Does anyone want to review this PR I opened to fix the CI (it wasn't running the tests): com-lihaoyi/upickle#359 ?
Thanks!
Then we can trigger a release @htmldoug @objektwerks
@objektwerks @htmldoug UPickle 1.4.1 is now going to Maven Central: https://github.com/com-lihaoyi/upickle/runs/3514651632
I'm trying to use
upickle to extract my case classs schema. The visitor that uPickle provide is great. I would like to have a visitor on the type if you know what I mean:- I have user defined
case class Foo(..) - I want a JSON deserializer for that class. That's easy, I can just use
macroRW - I also want to codegen Foo into a Python class. For that, it would be nice if I could re-use any of uPickle's machinery to have a Visitor on the schema of Foo. That way I'll be able to generate the Python code string easily
i could imagine coming up with some way to store a lazily instantiated type description on each readwriter, so others can use it, but that's not what happens now
@lihaoyi I've been going through your wonderful tutorial on Visitor pattern yet again. I want to modify it slightly to include a cache based early stop for recursive Data structure. Think of an ADT which is self referencing. With the pattern in the tutorial, there is no way for visitor to communicate back to dispatcher to stop going further.
So I came up with this approach, could you please comment on what you think about it:
trait Visitor[T] {
def visitPrimitive(tpe: Type): T
def visitOption(innerVisited: T): T
def visitArrayType(innerVisited: T): T
def visitMapType(keyVisited: T, valueVisited: T): T
def visitStructType(tpe: Type): StructVisitor[T]
def stopValue: Option[T] = None // If defined, we stop visiting and simply return this value
}
trait StructVisitor[T] {
def visitFieldKey(name: String): Visitor[T]
def visitFiledType(name: String, visitedValue: T): Unit
def done: T
def stopValue: Option[T] = None // If defined, we stop visiting and simply return this value
}This is a visitor to visit Type from scala.reflect
Not 100% happy with it, so was wondering if you can share some thought on how to improve it.
I always find visitors pretty finnicky to think about, so I always end up with a lot of "shake it around until it compiles, shake it some more until tests pass" kind of workflows
