lihaoyi/upickle

MrMuppet-cut

@MrMuppet-cut

This is the string:

MrMuppet-cut

@MrMuppet-cut

{
  "dataflows": [
    {
      "name": "test",
      "sources": [
       {
          "name": "person_inputs",
          "path": "/data/input/events/person/*",
          "format": "JSON"
        }
      ],
      "transformations": [
        {
          "name": "validation",
          "type": "validate_fields",
          "params": {
            "input": "person_inputs",
            "validations": [
              {
                "field": "office",
                "validations": [
                 "notEmpty"
                ]
              },
              {
                "field": "age",
                "validations": [
                  "notNull"
                ]
              }
            ]
          }
        },
        {
          "name": "ok_with_date",
          "type": "add_fields",
          "params": {
            "input": "validation_ok",
            "addFields": [
              {
                "name": "dt",
                "function": "current_timestamp"

              }
            ]
          }
        }
      ],
      "sinks": [
        {
          "input": "ok_with_date",
          "name": "raw-ok",
          "paths": [
            "/data/output/events/person"
          ],
          "format": "JSON",
          "saveMode": "OVERWRITE"
        },
        {
          "input": "validation_ko",
          "name": "raw-ko",
          "paths": [
            "/data/output/discards/person"
          ],
          "format": "JSON",
          "saveMode": "OVERWRITE"
        }
      ]
    }
  ]
}

which is in a file called "metadata.json"

and I read just by doing:

val j = os.read(os.pwd/RelPath("src/main/scala/metadata.json"))
val jsonData = ujson.read(j)

however, when I try

jsonData(0)

I get

ujson.Value$InvalidData: Expected ujson.Arr (data: {"dataflows":[{"name":"test","sources":[{"name":"person_inputs","path":"/data/input/events/person/*","format":"JSON"}],"transformations":[{"name":"validation","type":"validate_fields","params":{"input":"person_inputs","validations":[{"field":"office","validations":["notEmpty"]},{"field":"age","validations":["notNull"]}]}},{"name":"ok_with_date","type":"add_fields","params":{"input":"validation_ok","addFields":[{"name":"dt","function":"current_timestamp"}]}}],"sinks":[{"input":"ok_with_date","name":"raw-ok","paths":["/data/output/events/person"],"format":"JSON","saveMode":"OVERWRITE"},{"input":"validation_ko","name":"raw-ko","paths":["/data/output/discards/person"],"format":"JSON","saveMode":"OVERWRITE"}]}]})

MrMuppet-cut

@MrMuppet-cut

Any idea why this happens?

Any help would be greatly appreciated. Thanks!!

Cheng Wang

@polarker

Hi, is there a way to fail on excess JSON fields ? This feature is useful when working with optional fields

Doug Roper

@htmldoug

Not easily. The most straightforward way probably requires modifying the macro.

Next best would be to either convert to Value and count the fields or write a Visitor.Delegate to wrap the Reader and count them that way.

Igor Romanov

@pharod

I have in my case class:

@key("stage_name") stageName: Option[String] = None,

and when this property is missing from JSON I am getting an error message:

Caused by: upickle.core.AbortException: missing keys in dictionary: stage_name, state_metadata at index ...

What is wrong?

Igor Romanov

@pharod

@lihaoyi It seems it works ok only when I'm not using @key annotation for mapping. Looks like a bug in uPickle or a limitation?

Li Haoyi

@lihaoyi

huh seems like a bug

can you minimize a repro and open a ticket

Igor Romanov

@pharod

@lihaoyi I've filed issue com-lihaoyi/upickle#360

Objektwerks

@objektwerks

Using Scala 2.13.6 and uPickle 1.4.0, I assume everyone is familiar with this macroRW error:

match may not be exhaustive. It would fail on the following input: (x: Int forSome x not in 0)

Currently I use an @nowarn annotation to ignore all such errors. Is there an expected fix to this in a forthcoming version of Scala and/or uPickle? Or is this with us for a long time to come?

Doug Roper

@htmldoug

Should be fixed in https://github.com/com-lihaoyi/upickle/compare/1.4.0...master. Just needs a release.

Objektwerks

@objektwerks

@htmldoug Thanks for the update, Doug!

Lorenzo Gabriele

@lolgab

Hi! :wave:
Does anyone want to review this PR I opened to fix the CI (it wasn't running the tests): com-lihaoyi/upickle#359 ?

Thanks!

Then we can trigger a release @htmldoug @objektwerks

Objektwerks

@objektwerks

@lolgab Thanks for the update, Lorenzo. Looking forward to the new release.:)

Lorenzo Gabriele

@lolgab

@objektwerks @htmldoug UPickle 1.4.1 is now going to Maven Central: https://github.com/com-lihaoyi/upickle/runs/3514651632

Objektwerks

@objektwerks

@lolgab Awesome!

Objektwerks

@objektwerks

Just upgraded to upickle 1.4.1 - and no more @nowarn annotation required! Thanks, team uPickle!

Lorenzo Gabriele

@lolgab

Thank you to @htmldoug who made the fix! :)
Happy having fixed the CI so now we can finally merge PRs confidently!

Siddhant Sanyam

@siddhant3s

I'm trying to use upickle to extract my case classs schema. The visitor that uPickle provide is great. I would like to have a visitor on the type if you know what I mean:

I have user defined case class Foo(..)
I want a JSON deserializer for that class. That's easy, I can just use macroRW
I also want to codegen Foo into a Python class. For that, it would be nice if I could re-use any of uPickle's machinery to have a Visitor on the schema of Foo. That way I'll be able to generate the Python code string easily

Li Haoyi

@lihaoyi

i dont thinkvthe visitor is helpful for codegen. Visitor is called on the runtime data structure, not on the definition structure. Only macroRW has that; you could try forking it to generate the code directly or to generate metadata you can use for the codegen

Siddhant Sanyam

@siddhant3s

Do you think it's possible to use the Reader[T] generated by macroRW[T] to something like that? I mean I can write my own macro, but I was hoping if I can re-use something from your macro so that I can be assured that 2. and 3. are in sync.

Li Haoyi

@lihaoyi

no, the macro generated read/writers explicitly avoid generating inspectable data structures to avoid performance overhead. It makes them fast, but it means you cannot really repurpose them for anything else

i could imagine coming up with some way to store a lazily instantiated type description on each readwriter, so others can use it, but that's not what happens now

Siddhant Sanyam

@siddhant3s

Thanks for confirming. I'll handwrite it in that case!

Siddhant Sanyam

@siddhant3s

@lihaoyi I've been going through your wonderful tutorial on Visitor pattern yet again. I want to modify it slightly to include a cache based early stop for recursive Data structure. Think of an ADT which is self referencing. With the pattern in the tutorial, there is no way for visitor to communicate back to dispatcher to stop going further.
So I came up with this approach, could you please comment on what you think about it:

    trait Visitor[T] {
        def visitPrimitive(tpe: Type): T
        def visitOption(innerVisited: T): T
        def visitArrayType(innerVisited: T): T
        def visitMapType(keyVisited: T, valueVisited: T): T
        def visitStructType(tpe: Type): StructVisitor[T]
        def stopValue: Option[T] = None // If defined, we stop visiting and simply return this value
    }
    trait StructVisitor[T] {
        def visitFieldKey(name: String): Visitor[T]
        def visitFiledType(name: String, visitedValue: T): Unit
        def done: T
        def stopValue: Option[T] = None  // If defined, we stop visiting and simply return this value
    }

This is a visitor to visit Type from scala.reflect

Not 100% happy with it, so was wondering if you can share some thought on how to improve it.

Li Haoyi

@lihaoyi

I don't have any real insight here, depends on how well it fits your use case

I always find visitors pretty finnicky to think about, so I always end up with a lot of "shake it around until it compiles, shake it some more until tests pass" kind of workflows

Yeitijem

@Yeitijem

Hi is there some documentation about using upickle in conjunction with scala 3? And if yes, where can I find it?

Simon Parten

@Quafadas

WOuld anyone have a hint on this error when using a library with a upickle dependancy from ammonite?

java.lang.IncompatibleClassChangeError: class ilsop.models.RiskDecompositionSearchResult$$anon$10$$anon$11 has interface upickle.core.Types$CaseR$CaseObjectContext as super class

Lorenzo Gabriele

@lolgab

@Quafadas It can be that you're using a library that depends on a different version of upickle that is not binary compatible. I think you can use ammonite without the com-lihaoyi libraries, so you can use the version of upickle you want.

Simon Parten

@Quafadas

Ah, are you saying the com-lihayoi libraries are imported by default... and I should disable that default import so I can use my own version? That sounds plausible... brb... thanks!

novakov-alexey

@novakov-alexey

Hi. Does upickle work with Scala 3? I need to parse JSON text into a list of case classes.

novakov-alexey

@novakov-alexey

ok..... it works fine in Scala 3 project, for example:

object TaskInstance:
  given rw: SnakePickle.ReadWriter[TaskInstance] = SnakePickle.macroRW

Lorenzo Gabriele

@lolgab

Yes, it does @novakov-alexey
Some Scala 3 specific features are not supported well yet, like converting a Scala 3 enum to JSON, but the shared Scala 2 / Scala 3 works almost identically.
Almost because I remember there were some bugs open, maybe they now are all solved.

novakov-alexey

@novakov-alexey

@lolgab thanks for reply. Is there some workaround for reading JSON into Scala 3 enums?

I am wondering which JSON library supports Scala 3 enums today :-)

Where communities thrive

People

Repo info

Activity