Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Alexey Novakov
    @novakov-alexey
    ok..... it works fine in Scala 3 project, for example:
    object TaskInstance:
      given rw: SnakePickle.ReadWriter[TaskInstance] = SnakePickle.macroRW
    Lorenzo Gabriele
    @lolgab
    Yes, it does @novakov-alexey
    Some Scala 3 specific features are not supported well yet, like converting a Scala 3 enum to JSON, but the shared Scala 2 / Scala 3 works almost identically.
    Almost because I remember there were some bugs open, maybe they now are all solved.
    Alexey Novakov
    @novakov-alexey
    @lolgab thanks for reply. Is there some workaround for reading JSON into Scala 3 enums?
    I am wondering which JSON library supports Scala 3 enums today :-)
    Simon Parten
    @Quafadas
    Does upickle support validation of a json document vs a schema? I couldn't see anything... is anyone aware of a good solution to this in scala / java. I think I'm googling the wrong questions somehow...
    1 reply
    Stephen Link
    @BoopBoopBeepBoop

    I was looking at one of the open bugs for com-lihaoyi/upickle#356 and think I'm struggling a bit to figure out exactly where my understanding is breaking down. I've never used macros before, so that might be part of it.

    The way it appears to me that things are happening right now is that the Sum types for the enumeration are being derived correctly when we go to write the values, but the implicit classtag that is carried through is the classtag for the enumeration supertype.

    package upickle
    
    import scala.language.implicitConversions
    import utest._
    
    import upickle.default._
    
    object EnumTests extends TestSuite {
    
      enum SimpleEnum derives ReadWriter:
        case A, B
    
      val tests = Tests {
        test("simple") {
          test("enum write") {
            val parsed = write(SimpleEnum.B)
            val expected = """{"$type":"upickle.EnumTests.SimpleEnum.B"}"""
            assert(parsed == expected)
          }
        }
      }
    }

    I added some printlns to the code temporarily, but see the line beggining with Instance Check

    ----------------------- Running Tests upickle.EnumTests -----------------------
    >>>>> SUM READER TRAVERSING [upickle.EnumTests$SimpleEnum$@1b73be9f]
    >>>>> PRODUCT READER []
    >>>>> PRODUCT READER []
    >>>>> SUM WRITER TRAVERSING [upickle.EnumTests$SimpleEnum$@1b73be9f]
    CASE CLASS WRITER defaults[]
    >>>>> PRODUCT WRITER FULL CLASS NAME: upickle.EnumTests.SimpleEnum.A
    CASE CLASS WRITER defaults[]
    >>>>> PRODUCT WRITER FULL CLASS NAME: upickle.EnumTests.SimpleEnum.B
    Instance Check upickle.EnumTests.SimpleEnum.A [B <: upickle.EnumTests$SimpleEnum]: true
    RETURNING (upickle.EnumTests.SimpleEnum.A,upickle.implicits.CaseClassWriterPiece$CaseClassWriter@42721fe) for input B
    >>>>> PRODUCT WRITER []
    >>>>> PRODUCT WRITER []
    X upickle.EnumTests.simple.enum write 30ms 
      utest.AssertionError: parsed == expected
      parsed: String = {"$type":"upickle.EnumTests.SimpleEnum.A"}
      expected: String = {"$type":"upickle.EnumTests.SimpleEnum.B"}
        utest.asserts.Asserts$.assertImpl(Asserts.scala:30)
        upickle.EnumTests$.$init$$$anonfun$3$$anonfun$2$$anonfun$2(EnumTests.scala:13)
    1 targets failed
    upickle.jvm[3.0.2].test.test 1 tests failed: 
      upickle.EnumTests upickle.EnumTests.simple.enum write
    That Instance Check line comes from TaggedReader.Leaf.findWriter, which I've modified like this (just enough to get some debugging information):
    class Leaf[T](c: ClassTag[_], tag: String, r: CaseW[T]) extends TaggedWriter[T]{
      def findWriter(v: Any) = {
        val isInstance = c.runtimeClass.isInstance(v)
        println(s"Instance Check $tag [$v <: ${c.runtimeClass.getName}]: ${isInstance}")
        if (isInstance) (tag -> r)
        else null
      }
    }
    Stephen Link
    @BoopBoopBeepBoop

    so, the way that this appears to me, although I'm very much a novice here, is that val parsed = write(SimpleEnum.B) is resulting in the usage of a classtag for SimpleEnum itself? And not SimpleEnum.B.

    Thoughts?

    Stephen Link
    @BoopBoopBeepBoop

    Ah, a simpler answer. The macros are more specific than the classtag. So even though the macro is giving the correct type, the classtag that is implicitly carried is not specific to the enum instance

    CLASS NAME: upickle.EnumTests.SimpleEnum.B Classtag: upickle.EnumTests$SimpleEnum

    Ryan Tomczik
    @Tomczik76
    Hey everyone I'm new to upickle, I'm looking for a way to generate a set of all the paths in a JSON object without using an intermediate AST. It looks like upickle might be a good option using a Visitor. The TraceVisitor looks similar to what I need. Is there a way to use it out of the box to generate all the paths in a JSON object?
    Ryan Tomczik
    @Tomczik76
    Nevermind, I implemented a custom one
    Alan Burlison
    @alanbur

    I'm trying to write a custom pickler to handle java.util.Date by converting to/from an ISO8601 date string. What I have is this:

    case class JSONDate(date: Date) extends AnyVal
    object JSONDate {
      implicit def readWrite: ReadWriter[JSONDate] = readwriter[ujson.Value].bimap[JSONDate](
        (date: JSONDate) => DateTimeFormatter.ISO_INSTANT.format(date.date.toInstant),
        (json: ujson.Value) => JSONDate(Date.from(OffsetDateTime.parse(json.str).toInstant))
      )
    }

    But whilst this works:

    val d: JSONDate = JSONDate(comp.getTimeCreated)
    write(d)

    This doesn't:

    write(ujson.Obj(
      "Name"    -> "myName",
      "Created" -> d
     ))

    For some reason the implicit isn't being found in the ujson.Obj case. Any suggestions?

    Li Haoyi
    @lihaoyi
    we currently don't autoconvert non trivial stuff into ujson.Values, you need to manually call upickle.default.writeJs
    we could conceivably do such automatic conversions, but we don't
    Alan Burlison
    @alanbur
    OK thanks, just wanted to check I wasn't doing something dumb :-)
    Alan Burlison
    @alanbur

    The release notes for 0.5.1 say:

    ujson.write now takes an optional `sortKeys` flag, if you want the JSON dictionaries to rendered in a standardized order

    But that doesn't seem to exist any more. How do I ensure that Object keys are sorted? This is to make it easier to compare versions of a JSON file in a diff tool...

    Lorenzo Gabriele
    @lolgab
    @alanbur It might be that this is now the default behaviour, since ujson.Obj is backed by a LinkedHashMap.
    Do you have a reproduction that changes the orders of fields?
    Li Haoyi
    @lihaoyi
    i think the sorting feature disappeared when we started doing streaming serialization
    maybe it could be added back as a feature on ujson.Value, since that has a concrete data structure to sort. When serializing case classes, we don't have one
    Lorenzo Gabriele
    @lolgab
    Oh, I misread the question, I thought the requested sorting was the one of the input. Nevermind.
    Lorenzo Gabriele
    @lolgab
    You can sort a ujson.Obj by keys using:
    def sort(obj: ujson.Obj): ujson.Obj = {
      obj.obj.toSeq.sortBy(_._1)
    }
    Alan Burlison
    @alanbur
    I am building a JSON object where I can't guarantee the insertion order is sorted by key, but I would like it to be sorted by key when the JSON is output with write(). The vanished sortBy flag looked like exactly what I wanted.
    Alan Burlison
    @alanbur

    I see ujson.Obj uses a LinkedHashMap so it will preserve insertion order. Rather than sorting into a new Obj when the contents have all been added I could do a two-step process but it's still a bit clunky as it requires a Map copy:

    val sortedMap = mutable.SortedMap[String, ujson.Value]()
    addSomeStuffWithUnpredictableKeyOrdering(sortedMap)
    val obj = ujson.Obj.from(sortedMap)

    What would be nice is if Obj could be configured to use either mutable.LinkedHashMap as at present, or mutable.SortedMap - then you could choose to either preserve insertion order, or have ordering by key.

    Lorenzo Gabriele
    @lolgab
    @alanbur It doesn't seem a bad idea. ujson.Obj could hold a reference of a mutable.Map[String, ujson.Value] instead of a mutable.LinkedHashMap[String, ujson.Value].
    Then the default implementation would continue to use mutable.LinkedHashMap[String, ujson.Value] and you could pass a different type of map by doing:
    val myMap = mutable.SortedMap[String, ujson.Value]()
    val obj = new ujson.Obj(myMap)
    But having behavioral changes depending on what type of Map you pass as a constructor parameter isn't great either.
    Li Haoyi
    @lihaoyi
    Yeah i dont think there's a better way than copying the map. You can't really sort linked hash maps in place
    Alan Burlison
    @alanbur

    @lolgab well, you shouldn't really be depending on map key ordering in the first place I suppose, but you already get that with the current implementation ;-)

    I'd settle for being able to specify ordering during write, again I know you shouldn't do that with JSON but it does make it easier to process with existing diff tools...

    Li Haoyi
    @lihaoyi
    you could mayne write a custom Visitor that can sort the output, but that would basically involve buffering everything up and sorting before sending things downstream, so kind of the same thing. Except instead of going from 1 copy of the data to 2, it is going from 0 copies of the data to 1
    basically every time the visitor sees an object, it buffers everything up as ujson.IndexedValues, sorts them, and then serializes them downstream
    Alan Burlison
    @alanbur
    For now I'll just go with the sorting-after-building approach, the Visitor approach looks like it would need digging down into the ujson/upickle source to understand how to do it.
    Tobias Roeser
    @lefou
    Is there any version information available? Documentation lists 1.4.3 as last change. What changed in 1.6.0? Scala-steward opened a PR in Mill, but all CI jobs fail.
    Li Haoyi
    @lihaoyi
    I forgot to push the readme changes, let me do that now
    Tobias Roeser
    @lefou
    :+1:
    Li Haoyi
    @lihaoyi
    it more or less just includes com-lihaoyi/upickle@a4247cd
    Li Haoyi
    @lihaoyi
    looks like the removal of the old (and buggy) implicit readwriter for case objects caused the breakage, lemme add explicit readwriters for those and see if that fixes the build
    Tobias Roeser
    @lefou
    It's worth to mention in the changelog
    Li Haoyi
    @lihaoyi
    yeah I'll add it
    Joe
    @j-chimienti
    how would I traverse a Obj, and return an Obj? Normally in scala i would do .toMap
    val updatedData : ujson.Arr = props.obj
      .map {
      case (name, p) =>
        p("type") match {
          case a: ujson.Arr =>
            ujson.Obj(name -> ujson.Obj("type" -> "string"))
          case o => ujson.Obj(name -> p)
        }
    }
    Igal Tabachnik
    @hmemcpy
    Hey. I just upgraded our codebase to the new 2.0. We have some custom implementations of Visitors, and it seems I'm now getting an NPE from visitKeyValue.
    It now goes to the new implementation of visitObject, and does this:
          def visitKeyValue(s: Any): Unit = {
              lastNested.visitValue(s.asInstanceOf[T], lastKeyIndex)
          }
    However, nobody set lastNested, so it's now null... Should I not use this call, and/or use visitKey first?
    Li Haoyi
    @lihaoyi
    I'm not sure I can give a good answer without looking at your codebase, but the next best thing is to ser what the visitors in the upickle repo do and follow suite
    that's basically what I do haha
    Igal Tabachnik
    @hmemcpy
    haha :) Thanks, will do that
    I think if I take it through the jsonable visitor they should just work
    Well, it's a problem for another day. I'll take the rest of this weekend off!
    Thanks again :)
    Igal Tabachnik
    @hmemcpy
    Welp, I kinda figured it out what it was. I had to keep using true for jsonableKeys everywhere (ended up just overriding isJsonDictKey to true in our custom Writer) but then I was met with double-quotation in the keys!
    All the keys were e.g. "id""