Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 14:12

    plokhotnyuk on gh-pages

    Update results from JVMs for v2… (compare)

  • 07:31

    plokhotnyuk on master

    Update README.md (compare)

  • 07:30

    plokhotnyuk on master

    Update README.md (compare)

  • 05:51

    plokhotnyuk on master

    Update README.md (compare)

  • Aug 06 12:34

    plokhotnyuk on master

    Close #927 by fixing bit set re… Clear cache of dependencies (compare)

  • Aug 06 09:44

    plokhotnyuk on master

    Close #927 by fixing bit set re… (compare)

  • Aug 06 09:18

    plokhotnyuk on master

    Clear cache of dependencies (compare)

  • Aug 06 06:35

    plokhotnyuk on fix-benchmark-for-jackson-module-scala

    (compare)

  • Aug 06 06:35
    plokhotnyuk closed #927
  • Aug 06 06:35
    plokhotnyuk closed #928
  • Aug 06 06:35

    plokhotnyuk on master

    Close #927 by fixing bit set re… (compare)

  • Aug 06 06:34
    plokhotnyuk commented #927
  • Aug 06 06:34
    plokhotnyuk commented #927
  • Aug 06 06:32
    plokhotnyuk synchronize #928
  • Aug 06 06:32

    plokhotnyuk on fix-benchmark-for-jackson-module-scala

    Close #927 by fixing bit set re… (compare)

  • Aug 06 06:16
    plokhotnyuk opened #928
  • Aug 06 06:16

    plokhotnyuk on fix-benchmark-for-jackson-module-scala

    Close #927 by fixing bit set re… (compare)

  • Aug 06 06:08
    plokhotnyuk labeled #927
  • Aug 06 04:18

    plokhotnyuk on master

    Fix `ArraySeqOfBooleansReading`… (compare)

  • Aug 05 22:10
    pjfanning edited #927
Timur
@Timur83176832_twitter
this is the JsonObject codec:
implicit val jsonObjectCodec: JsonValueCodec[JsonObject] = new JsonValueCodec[JsonObject] {
    import io.circe.generic.encoding.ReprAsObjectEncoder.deriveReprAsObjectEncoder

    override def decodeValue(in: JsonReader, default: JsonObject): JsonObject = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        JsonObject.empty
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromBoolean(false).asObject.getOrElse(JsonObject.empty)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        JsonObject.empty
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong()).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromDoubleOrString(in.readDouble()).asObject.getOrElse(JsonObject.empty)
      } else if (b == '{') {
        val obj = scala.collection.mutable.Map[String, CirceJson]()
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), circeJsonCodec.decodeValue(in, default.asJson)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        JsonObject.fromIterable(obj)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += circeJsonCodec.decodeValue(in, CirceJson.fromFields(default.toMap)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }

        CirceJson.fromValues(arr).asObject.getOrElse(JsonObject.empty)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: JsonObject, out: JsonWriter): Unit =
      mapCodec.encodeValue(x.toMap, out)

    override def nullValue: JsonObject = JsonObject.empty
  }
this is circeJson:
implicit val circeJsonCodec: JsonValueCodec[CirceJson] = new JsonValueCodec[CirceJson] {
    import scala.jdk.CollectionConverters._
    override def decodeValue(in: JsonReader, default: CirceJson): CirceJson = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        CirceJson.fromString(in.readString(null))
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true)
        else CirceJson.fromBoolean(false)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        CirceJson.Null
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong())
        else CirceJson.fromDoubleOrString(in.readDouble())
      } else if (b == '{') {
        val obj = new java.util.LinkedHashMap[String, CirceJson]
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), decodeValue(in, default)) while (in.isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        CirceJson.fromFields(obj.asScala)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += decodeValue(in, default) while (in.isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }
        CirceJson.fromValues(arr.toList)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: CirceJson, out: JsonWriter): Unit =
      x.fold(
        out.writeNull(),
        jsonBoolean => out.writeVal(jsonBoolean),
        jsonNumber => {
          val longNumber = jsonNumber.toLong
          out.writeVal(if (longNumber.isDefined) longNumber.get else jsonNumber.toDouble)
        },
        jsonString => out.writeVal(jsonString),
        jsonArray => {
          out.writeArrayStart()
          jsonArray.foreach(v => encodeValue(v, out))
          out.writeArrayEnd()
        },
        jsonObject => {
          out.writeObjectStart()
          jsonObject.toIterable.foreach {
            case (k, v) =>
              out.writeKey(k)
              encodeValue(v, out)
          }
          out.writeObjectEnd()
        }
      )

    override def nullValue: CirceJson = CirceJson.Null
  }
its probably not ideal,I was toying with it today just to make my project work and the tests pass,so tweaking will be done later
sure @plokhotnyuk i will probably be able to do it tomorrow .My goal is for the Text case class to be with the value of the Either without types/value fields
Andriy Plokhotnyuk
@plokhotnyuk
If you will parse untrusted input then beware that Scala's maps are vulnerable under DoS attacks that exploit hash code collisions: scala/bug#11203
Timur
@Timur83176832_twitter
cool,didnt know! @plokhotnyuk but mostly its just logs
Timur
@Timur83176832_twitter
{
    "text": {
        "processing_breakdown": {
            "fetching_messages": {
                "num": 1,
                "sum": 2
            }
        }
    }
}
vs
{
    "text": {
        "Js": {
            "j": {
                "Left": {
                    "value": {
                        "processing_breakdown": {
                            "fetching_messages": {
                                "num": 1.0,
                                "sum": 2
                            }
                        }
                    }
                }
            }
        }
    }
}
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter I've released v2.7.3 with the issue fix. It is available on the Maven Central already: https://repo1.maven.org/maven2/com/github/plokhotnyuk/jsoniter-scala/jsoniter-scala-macros_2.13/2.7.3/
Timur
@Timur83176832_twitter
@plokhotnyuk great,thanks alot! Will use it
timothy
@timothyklim
Hello! Thanks for the best library for json serialization!
I have a question about ability to disable emit of DiscriminatorFieldName for ADT's. Is that possible? Because many serializers do not like unknown fields and will breaks easily. For my case it's easy to design class structure as ADT and then clients have schema to parse fields. The schema hides implementation design and do not emit extra fields.
Without this ability I have a workaround with custom JsonCodec for each ADT class to avoid extra field.
Andriy Plokhotnyuk
@plokhotnyuk
@TimothyKlim Hi, Timothy! Thanks for your feedback! Currently the no ADT tags option is not supported for derivation of codecs. The main reason is non-safe implementation of parsing for general case. plokhotnyuk/jsoniter-scala#435 is an issue where possible solutions were discussed.
Glen Marchesani
@fizzy33

I am converting from an AST based json library (well a few actually) I am looking for a recommendation. I have messaging middleware where effectively there are two layers one layer has the metadata and another layer handles the message bodies like so

case class Message(
  context: MessageContext,
  sender: NodeId,
  destination: NodeId,
  methodName: MethodName,
  event: StreamEvent,
  headers: Map[String,String],
  body: JsonStr,
)

So JsonStr is really any valid json. In the AST json world I would use the AST type. In jsoniter world that doesn't exist. So I have this

    object JsonStr {
      implicit val codec: JsonValueCodec[JsonStr] =
        new JsonValueCodec[JsonStr] {

          override def decodeValue(in: JsonReader, default: JsonStr): JsonStr =
            JsonStr(Chunk.array(in.readRawValAsBytes))

          override def encodeValue(x: JsonStr, out: JsonWriter): Unit =
            out.writeRawVal(x.rawBytes.toArray) // ??? optimize me we have at least one if not more extra array copies

          override def nullValue: JsonStr = ???
        }

      val Null = JsonStr(Chunk.array("null".getBytes))
      val EmptyObj = JsonStr(Chunk.array("{}".getBytes))
      val EmptyArr = JsonStr(Chunk.array("[]".getBytes))

    }
    case class JsonStr(rawBytes: Chunk[Byte])
So this use case with AST we parse once to AST and then in another layer process Message.body into the underlying type expected.
The use case with jsoniter it looks like I will have to parse the json twice. Once to get JsonStr and another to parse JsonStr into the type expected.
Wondering if anyone has thoughts on how best to do this with JsonIter
second question is any interest in PR with some tweaks to JsonReader and JsonWriter to support JsonStr working via a zero copy ?
Glen Marchesani
@fizzy33
third question is a ponder related to the first, is this double parsing going to ruin the performance gains of using jsonite?
any thoughts are appreciated
Andriy Plokhotnyuk
@plokhotnyuk
@fizzy33 Hi, Glen! Thanks for reaching out here! Usually reading and writing of raw bytes used for speeding up of JSON message transformation when those raw chunks go from the input to the output untouched, or for deferred parsing when raw chunks are re-parsed to case classes on demand in case of some rare conditions. I need to understand your challenge in general to propose the most efficient solution. As an example, if the body is schema-less and should be parsed always fully to some AST then you can do that without intermediate raw chunks using some custom codec. Dijon and ply-json-jsoniter libraries have such kind of codecs for their ASTs.
Andriy Plokhotnyuk
@plokhotnyuk
BTW, if it is the Chunk type is from the fs2 library then it will be quite inefficient to use it with Byte type due lack of specialization for primitives.
Glen Marchesani
@fizzy33
well something Chunk-like... wouldn't want to add fs2 as a dep and roger on the innefciencies without specialization
Well my use case is there is a general messaging layer to pass messages between processes which the Message class is the handler. Then at the lower level each class gets the Body and then turns it into whatever it needs to process. So for example most of the messages are some form of api call so one such call is just a case class e.g.
   case class QueryRequest(query: String, batching: Int)

  // the response
  case class QueryResponse(rows: fs2.Stream[F, Chunk[Row]], metadata: QueryMetadata)
Message and QueryRequest | QueryResponse are distinct layers so one cannot marshal them into instances at the same time.
Glen Marchesani
@fizzy33
fwiw I did some prelim testing on the JsonStr works well enough for now
I get better performance all around and probly can tweak it a bit later
Andriy Plokhotnyuk
@plokhotnyuk
Just wondering why Array[Byte] cannot be used directly but safely like in the RawVal type here instead of using intermediate Chunk[Byte]?
Glen Marchesani
@fizzy33
Array[Byte] can be used directly
the issue is I have offsets
so not in my example out.writeRawVal(x.rawBytes.toArray)
ideally if I could have
def writeRawVal(array: Array[Byte], offset: Int, len: Int)
then I can achieve zero copy (well one copy) on writes
on the read side if there was a read that gave back the same triplet as in (array: Array[Byte], offset: Int, len: Int) then I can get zero copy on my side
Andriy Plokhotnyuk
@plokhotnyuk
Do you mean def readRawValAsBytes(array: Array[Byte], offset: Int): Int = ??? // returns the length of a read chunk?
Could be the java.nio.ByteBuffer type a better option with adding following methods: def writeRawVal(bytes: java.nio.ByteBuffer): Unit and def readRawValAsByteBuffer(bytes: java.nio.ByteBuffer): Unit?
Glen Marchesani
@fizzy33
definitely java.nio.ByteBuffer would work quite well
In the internals I am in I have little choice to be squeemish about mutation ;-)
one layer up it is all pretty clean and immutable
well lets say at the low layer it is immutable when everyone plays nicely together :-)
seems like some extends AnyVal type around an Array[Byte] to give you the data of the array but not have it be mutable would be really nice. I wonder if someone has done the work on such a thing
and thanks for sharing your thoughts on this.
Marcin Szałomski
@baldram

Currently I'm going to check if it will be possible to minimize derivation configuration by adding the support for Enumeratum in some additional sub-project or even in the jsoniter-scala-macros sub-project with an optionally scoped dependency on Enumeratum.
While EnumEntry from Enumeratum is supported [...]

@plokhotnyuk Hi Andriy! Have you had a chance to evaluate creating the Enumeratum support related jsoniter-scala-macros sub-project? Is it something in your current work scope and doable based on the existing work from the past (https://github.com/lloydmeta/enumeratum/pull/176/files)?

Andriy Plokhotnyuk
@plokhotnyuk
@baldram Hi, Marcin! Thank for your question and support! I think that support of value enums of Enumeratum can be added with test-only scoped dependency on Enumeratum. Just need to add the CodecMakerConfig.useEnumeratumEnumValueId flag that turns on using of value from value fields as ids for parsing and serialization of objects which extends sealed traits as it was done for Scala enums here: plokhotnyuk/jsoniter-scala@f37dde6 Also, it can bring maximum performance because Enumeratum API will not be used at all.
Andriy Plokhotnyuk
@plokhotnyuk
I've added tests for Enumeratum enums: plokhotnyuk/jsoniter-scala@73ce8aa
Marcin Szałomski
@baldram
Oh, wow! What an answer :) Just seeing commits. Thank you.
This will be a part of 2.8.2 I guess. Is it already scheduled? I would try out a new feature.
Marcin Szałomski
@baldram

@plokhotnyuk I have one more question Andriy. Is it possible to add to generated code a SuppressWarnings? Maybe as an option?
The macro causes a lot of Wartremover errors. So we need to ignore it like below.

@SuppressWarnings(Array("org.wartremover.warts.All"))

The problem is that it doesn't make sense to create a single file, let's say Codecs.scala, put all codecs there, and add wartremoverExcluded for the whole file from build.sbt.
In this case, we would have to import codecs manually, losing the advantage of implicit codec.
If we have a codec inside a companion object, this imports automatically and also, other features relying on this codec are working automatically.

So everywhere in code we add this linter ignore for JsonCodecMaker.make. This is in each place we use Jsoniter.

object SomeFooBarEvent {
  @SuppressWarnings(Array("org.wartremover.warts.All")) // silencing generated code false-positive issues
  implicit val codec: JsonValueCodec[SomeFooBarEvent] = JsonCodecMaker.make
}

final case class SomeFooBarEvent(
  metadata: FooBarMetadata,
  request: FooBarRequest
)

Maybe such a SuppressWarnings could be already generated or with some option?
Or there is another solution?

Andriy Plokhotnyuk
@plokhotnyuk
@baldram Could you please share the code and commands to reproduce your warnings? Probably they worth to be fixed immediately in the macro that generate implementations of codecs. In the v2.7.1 release I fixed the "match may not be exhaustive" warning and it is possible that others are easy to fix too.
Marcin Szałomski
@baldram
The above code is enough. This already causes warnings if there is no @SuppressWarnings. Any use of JsonCodecMaker.make for any hello world case class caused tons of errors, when used with Wartremover wartremoverErrors ++= Warts.all.
Then enough is to dosbt clean compile from the console and that's it. Shall I provide some example warnings?
Marcin Szałomski
@baldram

I'm not sure it's easy to fix those warnings as "null" is a thing which is prohibited or lack of triple equals and many things.
Here is example:

object FooBar {
  implicit val codec: JsonValueCodec[FooBar] = JsonCodecMaker.make
}
final case class FooBar(foo: Int, bar: Option[String])

...causes:

[warn] /mnt/foobar/model/FooBar.scala:11:63: [wartremover:Equals] != is disabled - use =/= or equivalent instead
[warn]   implicit val codec: JsonValueCodec[FooBar] = JsonCodecMaker.make