Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Li Haoyi
    @lihaoyi
    there should be like 5 of them i think
    UUIDWriter, BigIntWriter, BigDecimalWriter, SymbolWriter
    Michel Charpentier
    @charpov

    I found myself writing something like that:

            val buffer = new ByteArrayOutputStream()
            val writer = new OutputStreamWriter(buffer, UTF_8)
            writeTo(object, writer)
            writer.close()
            // use buffer.size to get the number of bytes
            // use buffer.writeTo(...) to send them

    (I need bytes and content-length for an http response.)
    Two questions:

    • Am I overthinking this, i.e., just build and waste the string to get the bytes?
    • Should there be a method in the API to get the bytes from the serialization directly as bytes instead of a string?
    Andriy Plokhotnyuk
    @plokhotnyuk
    @charpov AFAIK write(object).getBytes(java.nio.charset.StandardCharsets.UTF_8) is the fastest serialization to byte arrays with uPickle. Please let me know if you will discover a faster way.
    Michel Charpentier
    @charpov
    Since there are methods in uPickle to send characters to a writer without allocating the string in memory, I was hoping that there'd something similar to send bytes to a stream. The way I wrote it shouldn't create the string, but I have no idea if it ends up being faster or slower.
    Andriy Plokhotnyuk
    @plokhotnyuk
    @charpov I'm comparing with transform(object).to(ujson.BytesRenderer()).toBytes which serializes ~1.5x times slower than that line above
    mvillafuertem
    @mvillafuertem
    Hi, how can i convert the ujson to map[String, Any]? Thanks!
    Li Haoyi
    @lihaoyi
    you cannot
    best you can us Map[String, ujson.Value]
    @charpov you can wrap your OutputStream in a OutputStreamWriter, or you can use upickle.default.stream(foo).writeBytesTo(stream)
    Michel Charpentier
    @charpov
    The problem is I need the number of bytes first (for a content-length http header).
    Li Haoyi
    @lihaoyi
    yeah in that case you need a byte array
    it cant know how big the serialized blob is ahead of time
    Doug Roper
    @htmldoug

    @plokhotnyuk I noticed that too, which drove me to swap the scala json parser/generator with jackson-core in the stable fork I'm using for work. (We don't use scalajs, but do use SMILE, BSON, YAML.) My benchmarks clocked weePickle ~5x faster than uPickle over T => json Bytes.

    It seemed like java.io.OutputStreamWriter allocates a thick 8 KB buffer which hurts small inputs, and maybe it doesn't like all the small unbuffered write(int) calls from BaseRenderer. I played with another buffer-less UTF8 Writer, but it hurt peak perf over large inputs, so I abandoned that too.

    If you were interested in having https://github.com/rallyhealth/weePickle/tree/v1.0.0 in your fantastic benchmark suite, I'd love to see how much slower I am than jsoniter-scala. :)

    Doug Roper
    @htmldoug
    @charpov In play, I've been using Ok.chunked(source: Source[ByteString, _]) with the jackson async parser from #252. If you use HTTP/1.1 Transfer-Encoding: chunked, you don't need the Content-Length up-front. You can also do the whole thing without blocking threads. Definitely matters for memory usage, GC pressure, and TTFB if you're dealing with large http entities.
    bmeesters
    @bmeesters
    Hello all, I was wondering if there is something like ujson.read that does not throw exceptions on invalid Json? Or should I just use some wrapper code that automatically catches this error and turns it into an Either or something similar.
    Doug Roper
    @htmldoug
    Pretty much everything in uPickle throws. Gotta try-catch, yes.
    Yeitijem
    @Yeitijem
    Hello all, I'm using uPickle on an case class which extends a sealed trait. Is there a way to suppress the "$type" field. The type is /needs to be handled higher in the object hierarchy.
    Damian Reeves
    @DamianReeves
    Hi, how would I define the ReadWriter for a type that looks like this:
    sealed abstract class TypeAstNode[+Annotation]
    object TypeAstNode {
    
       // Can I leverage the macroRW or does this need to be fully manual???
       implicit def readWriter[A](implicit readWriter[A]) = ???
    
       case class Variable[A](name:String, annotation:A) extends TypeAstNode[A]
       case class Unit[A](annotation:A) extends TypeAstNode[A]
    }
    Sequencer
    @sequencer
    Is possible to a method like this?
    val someMap = Map[String, ujson.Readable]()
    def readSomeMap[T](key: String) = upickle.default.read[T](someMap(key))
    when I define function like this, a error occurred:
    could not find implicit value for evidence parameter of type upickle.default.Reader[T]
    Damian Reeves
    @DamianReeves
    I’m having issues using the reader/writer with a generic tagged union
    sealed abstract class AccessControlled[A] extends Product with Serializable
    
    object AccessControlled {
    
      case class Public[A] private (value: A) extends AccessControlled[A]
      object Public {
        implicit def readWriter[A: ReadWriter]: RW[Public[A]] =
          readwriter[ujson.Value].bimap[Public[A]](
            (ac: Public[A]) =>
              ujson
                .Obj("$type" -> ujson.Str("public"), "value" -> writeJs(ac.value)),
            (json: ujson.Value) => {
              json("$type") match {
                case ujson.Str("public") =>
                  val value =
                    read[A](json("value"))
                  Public(value)
                case _ => throw new Abort("expected the tag to be public")
              }
            }
          )
      }
    
      case class Private[A] private (value: A) extends AccessControlled[A]
    
      object Private {
        implicit def readWriter[A: ReadWriter]: RW[Private[A]] =
          readwriter[ujson.Value].bimap[Private[A]](
            (ac: Private[A]) =>
              ujson
                .Obj("$type" -> ujson.Str("private"), "value" -> writeJs(ac.value)),
            (json: ujson.Value) => {
              val value = read[A](json("value"))
              Private(value)
            }
          )
      }
    
      implicit def readWriter[A: ReadWriter]: RW[AccessControlled[A]] = RW.merge(
        Public.readWriter[A],
        Private.readWriter[A]
      )
    }
    I’m getting this error:
            An unchecked error was produced.
            java.lang.ClassCastException: upickle.core.Types$ReadWriter$$anon$1 cannot be cast to upickle.core.Types$TaggedReadWriter
                    at upickle.core.Types.upickle$core$Types$$scanChildren(Types.scala:241)
                    at upickle.core.Types$TaggedReadWriter$Node.findReader(Types.scala:296)
                    at upickle.AttributeTagged$$anon$4.visitValue(Api.scala:233)
                    at ujson.Parser.rparse(Parser.scala:429)
                    at ujson.Parser.parse(Parser.scala:347)
                    at ujson.Parser.parse(Parser.scala:342)
                    at ujson.Parser.parse(Parser.scala:104)
                    at ujson.StringParser$.transform(StringParser.scala:29)
                    at ujson.StringParser$.transform(StringParser.scala:28)
                    at ujson.Readable$fromTransformer.transform(Readable.scala:13)
                    at upickle.Api.read(Api.scala:36)
                    at upickle.Api.read$(Api.scala:36)
                    at upickle.default$.read(Api.scala:124)
    Damian Reeves
    @DamianReeves
    Nevermind… decided my structure was simple enough just to hand encode and decode using ujson
    Damian Reeves
    @DamianReeves
    Is there a way to parse JSON from a string that only uses ujson. I have a project where I’d like to have only the ujson dependency and not the upickle dependency… i.e. I’m using ujson’s JSON AST as my JSON AST.
    Damian Reeves
    @DamianReeves
    Also has any thought been given to having accessors that don’t throw but instead return an Option or `Either:
    val foo = read[ujson.Value](“{}”)
    foo(“name”) //This throws & there seems to be no way to check ahead of time if “name” exists
    Li Haoyi
    @lihaoyi
    ujson.read(...)
    Damian Reeves
    @DamianReeves
    Thanks.
    What about the field test, is there anything supporting that?
    Damian Reeves
    @DamianReeves
    Hmm… seems I can do example.obj.isDefinedAt
    Damian Reeves
    @DamianReeves
    So for others who may be curious, this works:
    val jsonValue = json.read(“{}”)
    val field = “name”
    
    jsonValue match {
       case value @ ujson.Obj(fields) if fields.contains(field) => ???
        case _ => ???   
     }
    Vlad Patryshev
    @vpatryshev
    Sorry for interrupting; is it true that to use ujson with scala 2.11 I'll only have version 0.7 of upickle? (Just tried 1.0.0, does not seem to compile, but I did not go deep yet).
    Li Haoyi
    @lihaoyi
    yes that’s right
    Vlad Patryshev
    @vpatryshev
    Ok, thanks! Hope it will work for us (at salesforce)
    Li Haoyi
    @lihaoyi
    if necessary you can probably back-publish the recent versions back to scala 2.11 yourself
    it’s not hard, just work that I don’t want to do :p
    the source code should mostly just work
    Vlad Patryshev
    @vpatryshev
    Yes, I guess so. I'll check. Thank you!
    Shane
    @Shailpat3Shane_twitter
    Is there way to merge ujson.obj
    {
      "a": {
        "x": [
          "y"
        ]
      },
      "b": "null",
      "c": "null",
      "d": "null",
      "e": "null",
      "f": "[]"
    }
    
    {
    
      "g": "null",
      "h": "null",
      "i": "null",
      "j": "null",
    }
    
    => {
      "a": {
        "x": [
          "y"
        ]
      },
      "b": "null",
      "c": "null",
      "d": "null",
      "e": "null",
      "f": "[]"
      "g": "null",
      "h": "null",
      "i": "null",
      "j": "null",
    }
    And another question is there way to remove null properties
    Shane
    @Shailpat3Shane_twitter
    I figured out this works ujson.Obj(....).value + + ujson.Obj(...).value
    Ashwin Raman
    @ashwin-unifi
    Does upickle allow us to create a specific substitute for $type when using hierarchies?
    Li Haoyi
    @lihaoyi
    yeah
    search ckr trait Attribute in the github repo
    there’s a trait that starts with the work that uou can extend to override the type firld name
    Alexis Hernandez
    @AlexITC
    Hi, I can't seem to find two things, a way to define just a writer as I don't need to parse json, and, an option to drop null values while generating the json, any hints?
    I think I found the writer, looks like I needed to ask first, writer[ujson.Value].comap[T] {...}
    Alexis Hernandez
    @AlexITC
    I ended up writing a method to handle that but it would be nice to have such rendering option:
      implicit class JsonExt(val json: ujson.Value) extends AnyVal {
        def dropNullValues: Option[ujson.Value] = {
          json match {
            case ujson.Null => None
            case obj: ujson.Obj =>
              val newValues = obj.value.toList.flatMap {
                case (key, value) =>
                  value.dropNullValues.map(key -> _)
              }
              if (newValues.isEmpty) None
              else Some(ujson.Obj.from(newValues))
    
            case arr: ujson.Arr =>
              val newValues = arr.value.toList.flatMap { value =>
                value.dropNullValues
              }
              if (newValues.isEmpty) None
              else Some(ujson.Arr.from(newValues))
    
            case x => Some(x)
          }
        }
      }