lihaoyi/upickle

Alan Burlison

@alanbur

I'm trying to write a custom pickler to handle java.util.Date by converting to/from an ISO8601 date string. What I have is this:

case class JSONDate(date: Date) extends AnyVal
object JSONDate {
  implicit def readWrite: ReadWriter[JSONDate] = readwriter[ujson.Value].bimap[JSONDate](
    (date: JSONDate) => DateTimeFormatter.ISO_INSTANT.format(date.date.toInstant),
    (json: ujson.Value) => JSONDate(Date.from(OffsetDateTime.parse(json.str).toInstant))
  )
}

But whilst this works:

val d: JSONDate = JSONDate(comp.getTimeCreated)
write(d)

This doesn't:

write(ujson.Obj(
  "Name"    -> "myName",
  "Created" -> d
 ))

For some reason the implicit isn't being found in the ujson.Obj case. Any suggestions?

Li Haoyi

@lihaoyi

we currently don't autoconvert non trivial stuff into ujson.Values, you need to manually call upickle.default.writeJs

we could conceivably do such automatic conversions, but we don't

Alan Burlison

@alanbur

OK thanks, just wanted to check I wasn't doing something dumb :-)

Alan Burlison

@alanbur

The release notes for 0.5.1 say:

ujson.write now takes an optional `sortKeys` flag, if you want the JSON dictionaries to rendered in a standardized order

But that doesn't seem to exist any more. How do I ensure that Object keys are sorted? This is to make it easier to compare versions of a JSON file in a diff tool...

Lorenzo Gabriele

@lolgab

@alanbur It might be that this is now the default behaviour, since ujson.Obj is backed by a LinkedHashMap.

Do you have a reproduction that changes the orders of fields?

Li Haoyi

@lihaoyi

i think the sorting feature disappeared when we started doing streaming serialization

maybe it could be added back as a feature on ujson.Value, since that has a concrete data structure to sort. When serializing case classes, we don't have one

Lorenzo Gabriele

@lolgab

Oh, I misread the question, I thought the requested sorting was the one of the input. Nevermind.

Lorenzo Gabriele

@lolgab

You can sort a ujson.Obj by keys using:

def sort(obj: ujson.Obj): ujson.Obj = {
  obj.obj.toSeq.sortBy(_._1)
}

Alan Burlison

@alanbur

I am building a JSON object where I can't guarantee the insertion order is sorted by key, but I would like it to be sorted by key when the JSON is output with write(). The vanished sortBy flag looked like exactly what I wanted.

Alan Burlison

@alanbur

I see ujson.Obj uses a LinkedHashMap so it will preserve insertion order. Rather than sorting into a new Obj when the contents have all been added I could do a two-step process but it's still a bit clunky as it requires a Map copy:

val sortedMap = mutable.SortedMap[String, ujson.Value]()
addSomeStuffWithUnpredictableKeyOrdering(sortedMap)
val obj = ujson.Obj.from(sortedMap)

What would be nice is if Obj could be configured to use either mutable.LinkedHashMap as at present, or mutable.SortedMap - then you could choose to either preserve insertion order, or have ordering by key.

Lorenzo Gabriele

@lolgab

@alanbur It doesn't seem a bad idea. ujson.Obj could hold a reference of a mutable.Map[String, ujson.Value] instead of a mutable.LinkedHashMap[String, ujson.Value].
Then the default implementation would continue to use mutable.LinkedHashMap[String, ujson.Value] and you could pass a different type of map by doing:

val myMap = mutable.SortedMap[String, ujson.Value]()
val obj = new ujson.Obj(myMap)

But having behavioral changes depending on what type of Map you pass as a constructor parameter isn't great either.

Li Haoyi

@lihaoyi

Yeah i dont think there's a better way than copying the map. You can't really sort linked hash maps in place

Alan Burlison

@alanbur

@lolgab well, you shouldn't really be depending on map key ordering in the first place I suppose, but you already get that with the current implementation ;-)

I'd settle for being able to specify ordering during write, again I know you shouldn't do that with JSON but it does make it easier to process with existing diff tools...

Li Haoyi

@lihaoyi

you could mayne write a custom Visitor that can sort the output, but that would basically involve buffering everything up and sorting before sending things downstream, so kind of the same thing. Except instead of going from 1 copy of the data to 2, it is going from 0 copies of the data to 1

basically every time the visitor sees an object, it buffers everything up as ujson.IndexedValues, sorts them, and then serializes them downstream

Alan Burlison

@alanbur

For now I'll just go with the sorting-after-building approach, the Visitor approach looks like it would need digging down into the ujson/upickle source to understand how to do it.

Tobias Roeser

@lefou

Is there any version information available? Documentation lists 1.4.3 as last change. What changed in 1.6.0? Scala-steward opened a PR in Mill, but all CI jobs fail.

Li Haoyi

@lihaoyi

I forgot to push the readme changes, let me do that now

Tobias Roeser

@lefou

:+1:

Li Haoyi

@lihaoyi

it more or less just includes com-lihaoyi/upickle@a4247cd

Li Haoyi

@lihaoyi

looks like the removal of the old (and buggy) implicit readwriter for case objects caused the breakage, lemme add explicit readwriters for those and see if that fixes the build

Tobias Roeser

@lefou

It's worth to mention in the changelog

Li Haoyi

@lihaoyi

yeah I'll add it

Joe

@j-chimienti

how would I traverse a Obj, and return an Obj? Normally in scala i would do .toMap

val updatedData : ujson.Arr = props.obj
  .map {
  case (name, p) =>
    p("type") match {
      case a: ujson.Arr =>
        ujson.Obj(name -> ujson.Obj("type" -> "string"))
      case o => ujson.Obj(name -> p)
    }
}

Igal Tabachnik

@hmemcpy

Hey. I just upgraded our codebase to the new 2.0. We have some custom implementations of Visitors, and it seems I'm now getting an NPE from visitKeyValue.

It now goes to the new implementation of visitObject, and does this:

      def visitKeyValue(s: Any): Unit = {
          lastNested.visitValue(s.asInstanceOf[T], lastKeyIndex)
      }

However, nobody set lastNested, so it's now null... Should I not use this call, and/or use visitKey first?

Li Haoyi

@lihaoyi

I'm not sure I can give a good answer without looking at your codebase, but the next best thing is to ser what the visitors in the upickle repo do and follow suite

that's basically what I do haha

Igal Tabachnik

@hmemcpy

haha :) Thanks, will do that

I think if I take it through the jsonable visitor they should just work

Well, it's a problem for another day. I'll take the rest of this weekend off!

Thanks again :)

Igal Tabachnik

@hmemcpy

Welp, I kinda figured it out what it was. I had to keep using true for jsonableKeys everywhere (ended up just overriding isJsonDictKey to true in our custom Writer) but then I was met with double-quotation in the keys!

All the keys were e.g. "id""

Anyway, I fixed that too by calling .visitKey(-1) everywhere before .visitKeyValue... this seems to solve it :)

(isn't mutation awesome :p)

Ryan Broman

@broman

Is uPickle scala3 compatible? I tried some of the getting started code and it looks like read is part of the legacy package but is still in the getting started guide?

case class Thing(myFieldA: Int, myFieldB: String)
object Thing{
  implicit val rw: RW[Thing] = macroRW
}
case class Big(i: Int, b: Boolean, str: String, c: Char, t: Thing)
object Big{
  implicit val rw: RW[Big] = macroRW
}

object TestClient {
  def main(args: Array[String]): Unit = {
    read[Thing]("""{"myFieldA":1,"myFieldB":"gg"}""")

  }

}

Gives me an error and suggests to import upickle.legacy.read

Then when I do, I get this error ```

no implicit argument of type upickle.legacy.Reader[Thing] was found for an implicit parameter of method read in trait Api. 
I found:  upickle.legacy.given_Reader_T[T](/* missing */summon[deriving.Mirror.Of[T]])  
But no implicit values were found that match type deriving.Mirror.Of[T].

Lorenzo Gabriele

@lolgab

@broman Yes uPickle is compatible with Scala 3.
import upickle.legacy._ was introduced lately to reproduce the old serialization/deserialization format.
import upickle.default._ offers the same API but with a new format

Objektwerks

@objektwerks

I recently ran my Scala json library performance benchmarks and noticed a significant slowdown in uPickle performance. See Results section at https://github.com/objektwerks/scala3.json. The most recent test uses Scala 3 and uPickle 2.0.0. Has anyone else noticed this performance change?

Lorenzo Gabriele

@lolgab

@objektwerks If you switched from Scala 2 to Scala 3, it is known that the Scala 3 implementation is slower than the Scala 2 one.

Li Haoyi

@lihaoyi

yeah there's an open issue. Someone should fix it. I don't know Scala 3 so I cant

Objektwerks

@objektwerks

@lihaoyi @lolgab Okay, guys, thanks for the update! That said, circe, jsoniter and zio-json performance is consistent across Scala 2 and 3. Umm.... :)

Where communities thrive

People

Repo info

Activity