lihaoyi/upickle

Thomas Droxler

@tdroxler

I found in this gitter a quickfix to have

implicit class JsonExt(val json: ujson.Value) extends AnyVal {
    def dropNullValues: Option[ujson.Value] = {
   ...

but it sad to have to retraverse the all Value to drop the null values

Andriy Plokhotnyuk

@plokhotnyuk

@tdroxler Do you need solution for Writer[Option[T]]?

Thomas Droxler

@tdroxler

nope, I already use the one proposed in the examples, that serialize to null instead of [] I also use some default value = None, so it serialize as nothing. My problem is that I have a Writer for a case class that I can't modify, I thus can't put None as default value and so it serialize as null while I want to drop them.

Doron Somech

@somdoron

Question about error handling, what is the best way to fail deserialization, should we just throw Abort? What is the best way to catch deserialization error, is it catching AbortException?

Sakib Hadžiavdić

@sake92

is it possible to skip a key when writing?

for example when it's value is None

Doron Somech

@somdoron

@sake92 yes, by using a default value, upickle will drop the field if the value eqauls the default value.

Sakib Hadžiavdić

@sake92

@somdoron IDK why libraries do that

it makes no sense at all

Thomas Droxler

@tdroxler

hello, I'm hitting an issue on CI (github action) when testing on Windows, due to the infamous \r thing. Here's below a small example I did and the error I got on CI:

val raw = s"""
  {
    "test" : "yop",
    "hola" : 1
  }"""
val json = read[ujson.Value](raw)
ujson.ParseException: expected json value got "\r" at index 0

Did anyone already hit that issue?

Thomas Droxler

@tdroxler

I could do the following dirty hack in my custom upickle object https://github.com/alephium/alephium/blob/master/json/src/main/scala/org/alephium/json/Json.scala#L29-L39 but this bug sounds like something that could be tackled in upickle's core no?

jodersky

@jodersky

Does this also happen outside of the CI environment? I can't remember the details, but I've run into strange locale issues in some CI environments in the past (I'm not saying that this isn't a bug in upickle, but it's good to narrow down potential causes).

Piotr Buszka

@pbuszka

is there a way to have two $type keys map to the same case class in sealed trait serialization. Use case is migration of $type keys which contain package name and when package name changes it breaks existing serialization which is already in the DB. The problem is to migrate Foo, A, B to a new package barpkg but be able to still deserialize A with both $type = "foopkg.A" and $type = "barpkg.A", serialization should go always be with $type = "barpkg.A"

package foopkg

object Foo {
  implicit val rw = macroRW[Foo]
}
sealed trait Foo
case class A(a: Int) extends Foo
case class B(b: int) extends Foo

Li Haoyi

@lihaoyi

I'm not sure we support that, but worst come to worst you can use upickle.default.readwriter[ujson.Value].bimap[Foo] to write this serializer/deserializer manually

Jean-Jerome

@jeanjerome

Hi! I'm dealing with upickle.core.Abort: expected sequence got dictionarywhen upickle parses a JSON list which has no bracket when it contains one item.

How to handle that? Anybody managed such case??

I've tried custom pickers with no luck...

Jean-Jerome

@jeanjerome

I've tried the simplest to handle valid JSON list with one item only (that is a dictionary not a list):

  case class Item(prefix: String, name: String, href: String)
  implicit val ItemRW = upickle.default.macroRW[Item]

  case class MultiItem(service: List[Item])
  implicit val MultiItem RW = upickle.default.macroRW[MultiItem]
  case class OneItem(service: Item)
  implicit val OneItemRW = upickle.default.macroRW[OneItem]

  case class Content(services: Either[MultiItem, OneItem], title: String)
  implicit val ContentRW = upickle.default.macroRW[Content]

And still get [error] upickle.core.Abort: expected sequence got dictionary

If this library can't handle simply valid JSON, I don't see its interest ??

Piotr Buszka

@pbuszka

@lihaoyi thx for the hint, looks like a viable path for my problem.

Simon Parten

@Quafadas

@jeanjerome I had this the other day, the library can do it, but it sometimes get's it's error messaging wrong was my conclusion. Have you tried adding types to the implicit RW and making sure they are in companion objects?

Jean-Jerome

@jeanjerome

@Quafadas Thanks for your response. I finally tried traversing JSON as explain in https://www.lihaoyi.com/post/HowtoworkwithJSONinScala.html#traversing-json

Siddhant Sanyam

@siddhant3s

case class Foo(bar: String) { 
  val baz = "inner"
  def dbaz = "definner"
}

Is there any away to support getting baz and dbaz serialized along with Foo(bar)?

Li Haoyi

@lihaoyi

you'd have to write a custom serializer using upickle.readwriter[ujson.Value].bimap[Foo](..., ...)

Siddhant Sanyam

@siddhant3s

Okay. Thanks I'll give it a try.

MrMuppet-cut

@MrMuppet-cut

Hi! I am trying to read a json string using ujson:

This is the string:

MrMuppet-cut

@MrMuppet-cut

{
  "dataflows": [
    {
      "name": "test",
      "sources": [
       {
          "name": "person_inputs",
          "path": "/data/input/events/person/*",
          "format": "JSON"
        }
      ],
      "transformations": [
        {
          "name": "validation",
          "type": "validate_fields",
          "params": {
            "input": "person_inputs",
            "validations": [
              {
                "field": "office",
                "validations": [
                 "notEmpty"
                ]
              },
              {
                "field": "age",
                "validations": [
                  "notNull"
                ]
              }
            ]
          }
        },
        {
          "name": "ok_with_date",
          "type": "add_fields",
          "params": {
            "input": "validation_ok",
            "addFields": [
              {
                "name": "dt",
                "function": "current_timestamp"

              }
            ]
          }
        }
      ],
      "sinks": [
        {
          "input": "ok_with_date",
          "name": "raw-ok",
          "paths": [
            "/data/output/events/person"
          ],
          "format": "JSON",
          "saveMode": "OVERWRITE"
        },
        {
          "input": "validation_ko",
          "name": "raw-ko",
          "paths": [
            "/data/output/discards/person"
          ],
          "format": "JSON",
          "saveMode": "OVERWRITE"
        }
      ]
    }
  ]
}

which is in a file called "metadata.json"

and I read just by doing:

val j = os.read(os.pwd/RelPath("src/main/scala/metadata.json"))
val jsonData = ujson.read(j)

however, when I try

jsonData(0)

I get

ujson.Value$InvalidData: Expected ujson.Arr (data: {"dataflows":[{"name":"test","sources":[{"name":"person_inputs","path":"/data/input/events/person/*","format":"JSON"}],"transformations":[{"name":"validation","type":"validate_fields","params":{"input":"person_inputs","validations":[{"field":"office","validations":["notEmpty"]},{"field":"age","validations":["notNull"]}]}},{"name":"ok_with_date","type":"add_fields","params":{"input":"validation_ok","addFields":[{"name":"dt","function":"current_timestamp"}]}}],"sinks":[{"input":"ok_with_date","name":"raw-ok","paths":["/data/output/events/person"],"format":"JSON","saveMode":"OVERWRITE"},{"input":"validation_ko","name":"raw-ko","paths":["/data/output/discards/person"],"format":"JSON","saveMode":"OVERWRITE"}]}]})

MrMuppet-cut

@MrMuppet-cut

Any idea why this happens?

Any help would be greatly appreciated. Thanks!!

Cheng Wang

@polarker

Hi, is there a way to fail on excess JSON fields ? This feature is useful when working with optional fields

Doug Roper

@htmldoug

Not easily. The most straightforward way probably requires modifying the macro.

Next best would be to either convert to Value and count the fields or write a Visitor.Delegate to wrap the Reader and count them that way.

Igor Romanov

@pharod

I have in my case class:

@key("stage_name") stageName: Option[String] = None,

and when this property is missing from JSON I am getting an error message:

Caused by: upickle.core.AbortException: missing keys in dictionary: stage_name, state_metadata at index ...

What is wrong?

Igor Romanov

@pharod

@lihaoyi It seems it works ok only when I'm not using @key annotation for mapping. Looks like a bug in uPickle or a limitation?

Li Haoyi

@lihaoyi

huh seems like a bug

can you minimize a repro and open a ticket

Igor Romanov

@pharod

@lihaoyi I've filed issue com-lihaoyi/upickle#360

Objektwerks

@objektwerks

Using Scala 2.13.6 and uPickle 1.4.0, I assume everyone is familiar with this macroRW error:

match may not be exhaustive. It would fail on the following input: (x: Int forSome x not in 0)

Currently I use an @nowarn annotation to ignore all such errors. Is there an expected fix to this in a forthcoming version of Scala and/or uPickle? Or is this with us for a long time to come?

Doug Roper

@htmldoug

Should be fixed in https://github.com/com-lihaoyi/upickle/compare/1.4.0...master. Just needs a release.

Where communities thrive

People

Repo info

Activity