lihaoyi/upickle

Li Haoyi

@lihaoyi

there should be like 5 of them i think

UUIDWriter, BigIntWriter, BigDecimalWriter, SymbolWriter

Michel Charpentier

@charpov

I found myself writing something like that:

        val buffer = new ByteArrayOutputStream()
        val writer = new OutputStreamWriter(buffer, UTF_8)
        writeTo(object, writer)
        writer.close()
        // use buffer.size to get the number of bytes
        // use buffer.writeTo(...) to send them

(I need bytes and content-length for an http response.)
Two questions:

Am I overthinking this, i.e., just build and waste the string to get the bytes?
Should there be a method in the API to get the bytes from the serialization directly as bytes instead of a string?

Andriy Plokhotnyuk

@plokhotnyuk

@charpov AFAIK write(object).getBytes(java.nio.charset.StandardCharsets.UTF_8) is the fastest serialization to byte arrays with uPickle. Please let me know if you will discover a faster way.

Michel Charpentier

@charpov

Since there are methods in uPickle to send characters to a writer without allocating the string in memory, I was hoping that there'd something similar to send bytes to a stream. The way I wrote it shouldn't create the string, but I have no idea if it ends up being faster or slower.

Andriy Plokhotnyuk

@plokhotnyuk

@charpov I'm comparing with transform(object).to(ujson.BytesRenderer()).toBytes which serializes ~1.5x times slower than that line above

mvillafuertem

@mvillafuertem

Hi, how can i convert the ujson to map[String, Any]? Thanks!

Li Haoyi

@lihaoyi

you cannot

best you can us Map[String, ujson.Value]

@charpov you can wrap your OutputStream in a OutputStreamWriter, or you can use upickle.default.stream(foo).writeBytesTo(stream)

Michel Charpentier

@charpov

The problem is I need the number of bytes first (for a content-length http header).

Li Haoyi

@lihaoyi

yeah in that case you need a byte array

it cant know how big the serialized blob is ahead of time

Doug Roper

@htmldoug

@plokhotnyuk I noticed that too, which drove me to swap the scala json parser/generator with jackson-core in the stable fork I'm using for work. (We don't use scalajs, but do use SMILE, BSON, YAML.) My benchmarks clocked weePickle ~5x faster than uPickle over T => json Bytes.

It seemed like java.io.OutputStreamWriter allocates a thick 8 KB buffer which hurts small inputs, and maybe it doesn't like all the small unbuffered write(int) calls from BaseRenderer. I played with another buffer-less UTF8 Writer, but it hurt peak perf over large inputs, so I abandoned that too.

If you were interested in having https://github.com/rallyhealth/weePickle/tree/v1.0.0 in your fantastic benchmark suite, I'd love to see how much slower I am than jsoniter-scala. :)

Doug Roper

@htmldoug

@charpov In play, I've been using Ok.chunked(source: Source[ByteString, _]) with the jackson async parser from #252. If you use HTTP/1.1 Transfer-Encoding: chunked, you don't need the Content-Length up-front. You can also do the whole thing without blocking threads. Definitely matters for memory usage, GC pressure, and TTFB if you're dealing with large http entities.

bmeesters

@bmeesters

Hello all, I was wondering if there is something like ujson.read that does not throw exceptions on invalid Json? Or should I just use some wrapper code that automatically catches this error and turns it into an Either or something similar.

Doug Roper

@htmldoug

Pretty much everything in uPickle throws. Gotta try-catch, yes.

Yeitijem

@Yeitijem

Hello all, I'm using uPickle on an case class which extends a sealed trait. Is there a way to suppress the "$type" field. The type is /needs to be handled higher in the object hierarchy.

Damian Reeves

@DamianReeves

Hi, how would I define the ReadWriter for a type that looks like this:

sealed abstract class TypeAstNode[+Annotation]
object TypeAstNode {

   // Can I leverage the macroRW or does this need to be fully manual???
   implicit def readWriter[A](implicit readWriter[A]) = ???

   case class Variable[A](name:String, annotation:A) extends TypeAstNode[A]
   case class Unit[A](annotation:A) extends TypeAstNode[A]
}

Sequencer

@sequencer

Is possible to a method like this?

val someMap = Map[String, ujson.Readable]()
def readSomeMap[T](key: String) = upickle.default.read[T](someMap(key))

when I define function like this, a error occurred:

could not find implicit value for evidence parameter of type upickle.default.Reader[T]

Damian Reeves

@DamianReeves

I’m having issues using the reader/writer with a generic tagged union

sealed abstract class AccessControlled[A] extends Product with Serializable

object AccessControlled {

  case class Public[A] private (value: A) extends AccessControlled[A]
  object Public {
    implicit def readWriter[A: ReadWriter]: RW[Public[A]] =
      readwriter[ujson.Value].bimap[Public[A]](
        (ac: Public[A]) =>
          ujson
            .Obj("$type" -> ujson.Str("public"), "value" -> writeJs(ac.value)),
        (json: ujson.Value) => {
          json("$type") match {
            case ujson.Str("public") =>
              val value =
                read[A](json("value"))
              Public(value)
            case _ => throw new Abort("expected the tag to be public")
          }
        }
      )
  }

  case class Private[A] private (value: A) extends AccessControlled[A]

  object Private {
    implicit def readWriter[A: ReadWriter]: RW[Private[A]] =
      readwriter[ujson.Value].bimap[Private[A]](
        (ac: Private[A]) =>
          ujson
            .Obj("$type" -> ujson.Str("private"), "value" -> writeJs(ac.value)),
        (json: ujson.Value) => {
          val value = read[A](json("value"))
          Private(value)
        }
      )
  }

  implicit def readWriter[A: ReadWriter]: RW[AccessControlled[A]] = RW.merge(
    Public.readWriter[A],
    Private.readWriter[A]
  )
}

I’m getting this error:

        An unchecked error was produced.
        java.lang.ClassCastException: upickle.core.Types$ReadWriter$$anon$1 cannot be cast to upickle.core.Types$TaggedReadWriter
                at upickle.core.Types.upickle$core$Types$$scanChildren(Types.scala:241)
                at upickle.core.Types$TaggedReadWriter$Node.findReader(Types.scala:296)
                at upickle.AttributeTagged$$anon$4.visitValue(Api.scala:233)
                at ujson.Parser.rparse(Parser.scala:429)
                at ujson.Parser.parse(Parser.scala:347)
                at ujson.Parser.parse(Parser.scala:342)
                at ujson.Parser.parse(Parser.scala:104)
                at ujson.StringParser$.transform(StringParser.scala:29)
                at ujson.StringParser$.transform(StringParser.scala:28)
                at ujson.Readable$fromTransformer.transform(Readable.scala:13)
                at upickle.Api.read(Api.scala:36)
                at upickle.Api.read$(Api.scala:36)
                at upickle.default$.read(Api.scala:124)

Damian Reeves

@DamianReeves

Nevermind… decided my structure was simple enough just to hand encode and decode using ujson

Damian Reeves

@DamianReeves

Is there a way to parse JSON from a string that only uses ujson. I have a project where I’d like to have only the ujson dependency and not the upickle dependency… i.e. I’m using ujson’s JSON AST as my JSON AST.

Damian Reeves

@DamianReeves

Also has any thought been given to having accessors that don’t throw but instead return an Option or `Either:

val foo = read[ujson.Value](“{}”)
foo(“name”) //This throws & there seems to be no way to check ahead of time if “name” exists

Li Haoyi

@lihaoyi

ujson.read(...)

Damian Reeves

@DamianReeves

Thanks.

What about the field test, is there anything supporting that?

Damian Reeves

@DamianReeves

Hmm… seems I can do example.obj.isDefinedAt

Damian Reeves

@DamianReeves

So for others who may be curious, this works:

val jsonValue = json.read(“{}”)
val field = “name”

jsonValue match {
   case value @ ujson.Obj(fields) if fields.contains(field) => ???
    case _ => ???   
 }

Vlad Patryshev

@vpatryshev

Sorry for interrupting; is it true that to use ujson with scala 2.11 I'll only have version 0.7 of upickle? (Just tried 1.0.0, does not seem to compile, but I did not go deep yet).

Li Haoyi

@lihaoyi

yes that’s right

Vlad Patryshev

@vpatryshev

Ok, thanks! Hope it will work for us (at salesforce)

Li Haoyi

@lihaoyi

if necessary you can probably back-publish the recent versions back to scala 2.11 yourself

it’s not hard, just work that I don’t want to do :p

the source code should mostly just work

Vlad Patryshev

@vpatryshev

Yes, I guess so. I'll check. Thank you!

Shane

@Shailpat3Shane_twitter

Is there way to merge ujson.obj

{
  "a": {
    "x": [
      "y"
    ]
  },
  "b": "null",
  "c": "null",
  "d": "null",
  "e": "null",
  "f": "[]"
}

{

  "g": "null",
  "h": "null",
  "i": "null",
  "j": "null",
}

=> {
  "a": {
    "x": [
      "y"
    ]
  },
  "b": "null",
  "c": "null",
  "d": "null",
  "e": "null",
  "f": "[]"
  "g": "null",
  "h": "null",
  "i": "null",
  "j": "null",
}

And another question is there way to remove null properties

Shane

@Shailpat3Shane_twitter

I figured out this works ujson.Obj(....).value + + ujson.Obj(...).value

Ashwin Raman

@ashwin-unifi

Does upickle allow us to create a specific substitute for $type when using hierarchies?

Li Haoyi

@lihaoyi

yeah

search ckr trait Attribute in the github repo

there’s a trait that starts with the work that uou can extend to override the type firld name

Alexis Hernandez

@AlexITC

Hi, I can't seem to find two things, a way to define just a writer as I don't need to parse json, and, an option to drop null values while generating the json, any hints?

I think I found the writer, looks like I needed to ask first, writer[ujson.Value].comap[T] {...}

Alexis Hernandez

@AlexITC

I ended up writing a method to handle that but it would be nice to have such rendering option:

  implicit class JsonExt(val json: ujson.Value) extends AnyVal {
    def dropNullValues: Option[ujson.Value] = {
      json match {
        case ujson.Null => None
        case obj: ujson.Obj =>
          val newValues = obj.value.toList.flatMap {
            case (key, value) =>
              value.dropNullValues.map(key -> _)
          }
          if (newValues.isEmpty) None
          else Some(ujson.Obj.from(newValues))

        case arr: ujson.Arr =>
          val newValues = arr.value.toList.flatMap { value =>
            value.dropNullValues
          }
          if (newValues.isEmpty) None
          else Some(ujson.Arr.from(newValues))

        case x => Some(x)
      }
    }
  }

Where communities thrive

People

Repo info

Activity