Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 30 13:59
    atais commented #564
  • Sep 30 11:08

    plokhotnyuk on master

    Update docs (compare)

  • Sep 30 11:03

    plokhotnyuk on master

    Clean up of tests (compare)

  • Sep 30 08:58

    plokhotnyuk on master

    Clean up of the build script (compare)

  • Sep 29 12:05

    plokhotnyuk on gh-pages

    Update benchmark results from b… (compare)

  • Sep 29 07:46

    plokhotnyuk on master

    Remove redundant line feeds (compare)

  • Sep 29 06:52

    plokhotnyuk on master

    Add codec derivation tests for … (compare)

  • Sep 29 06:51

    plokhotnyuk on master

    Add codec derivation tests for … (compare)

  • Sep 28 13:36

    plokhotnyuk on master

    Update enumeratum to 1.7.0 (compare)

  • Sep 28 11:57

    plokhotnyuk on gh-pages

    Update the browser benchmark pa… (compare)

  • Sep 28 08:14

    plokhotnyuk on master

    Do not check for string index o… (compare)

  • Sep 28 08:07

    plokhotnyuk on master

    Update smithy4s-json to 0.16.2 (compare)

  • Sep 26 08:24
    plokhotnyuk commented #920
  • Sep 26 08:16

    plokhotnyuk on master

    Code clean up (compare)

  • Sep 26 07:42

    plokhotnyuk on master

    Clean up of benchmarks for circe Update jackson-module-scala, ja… (compare)

  • Sep 26 05:27

    plokhotnyuk on gh-pages

    Update benchmark results from J… (compare)

  • Sep 24 05:41
    plokhotnyuk commented #943
  • Sep 23 14:12

    plokhotnyuk on master

    Faster derivation of codecs for… (compare)

  • Sep 22 11:45
    plokhotnyuk closed #947
  • Sep 22 11:45

    plokhotnyuk on master

    Close #947 by removing of redun… (compare)

yankun1992
@yankun1992
{
    "request_id": "",
    "code": 0,
    "msg": "",
    "data": {
        "fields": [
            "exchange",
            "cal_date",
            "is_open",
            "pretrade_date"
        ],
        "items": [
            [
                "SSE",
                "19901219",
                1,
                null
            ],
            [
                "SSE",
                "19941222",
                1,
                "19941221"
            ],
            [
                "SSE",
                "19941223",
                1,
                "19941222"
            ]
        ],
        "has_more": false
    }
}
and i want to parse it to
case class TushareResponse(
                            request_id: String,
                            code: Int,
                            msg: String,
                            data: Data
                          )
case class Data(
                   fields: Array[String],
                   items: Array[Array[Option[String]]],
                   has_more: Boolean
                 )
Andriy Plokhotnyuk
@plokhotnyuk
Below is an implementation of a custom codec for Option[String] of items` elements:
        implicit val optStringCodec: JsonValueCodec[Option[String]] = new JsonValueCodec[Option[String]] {
          override def decodeValue(in: JsonReader, default: Option[String]): Option[String] = {
            val b = in.nextToken()
            if (b == 'n') {
              in.readNullOrError(None, "expected `null` value")
              None
            } else if (b == '"') {
              in.rollbackToken()
              Some(in.readString(null))
            } else if (b == 't' || b == 'f') {
              in.rollbackToken()
              Some(in.readBoolean().toString)
            } else if ((b >= '0' && b <= '9') || b == '-') {
              in.rollbackToken()
              Some(in.readBigDecimal(null).toString)
            } else in.decodeError("unexpected value")
          }

          override def encodeValue(x: Option[String], out: JsonWriter): Unit =
            x.fold(out.writeNull())(out.writeVal)

          override def nullValue: Option[String] = None
        }
Full code please see here
The expected output:
{
    "request_id": "",
    "code": 0,
    "msg": "",
    "data": {
        "fields": [
            "exchange",
            "cal_date",
            "is_open",
            "pretrade_date"
        ],
        "items": [
            [
                "SSE",
                "19901219",
                "1",
                null
            ],
            [
                "SSE",
                "19941222",
                "1",
                "19941221"
            ],
            [
                "SSE",
                "19941223",
                "1",
                "19941222"
            ]
        ],
        "has_more": false
    }
}
yankun1992
@yankun1992
Aha, I got it. Thank you ever much.
By the way, this is a really cool library! Thank you again!
Chetan Mehrotra
@chetanmeh

By the way, this is a really cool library! Thank you again!

+1 To that!

Timur
@Timur83176832_twitter
Hello. I am creating some codecs with JsonCodecMaker.makeWithoutDiscriminator and i can see in my output that my values look like
computerName":{"value":"tier2-logsingest-775446dbc8-m64bg"}
each field contains a value with that contains the value itself.Is there any way to remove the value field all together? @plokhotnyuk
Thank you!
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter Hello, Timur! Could you please share the related part of your data structures? The easiest way to unwrap single field values is to extend AnyVal like here. Another option is writing of some custom codec.
Timur
@Timur83176832_twitter
@plokhotnyuk I added extends AnyVal to everywhere,not it fails with a different error.
The case class is :
case class ReputationJson(
  ip: IPAddress,
  reliability: Option[Reliability],
  activity: Option[Array[Activity]],
  risk: Option[Risk],
  country: Option[Country],
  countryCode: Option[CountryCode],
  city: Option[City],
  coordinates: Option[Coordinates],
  activityRank: Option[Array[ActivityRank]],
  score: Score,
  scoreName: ScoreName
)
overloaded method copyOf with alternatives:
  (x$1: Array[Boolean],x$2: Int)Array[Boolean] <and>
  (x$1: Array[Double],x$2: Int)Array[Double] <and>
  (x$1: Array[Float],x$2: Int)Array[Float] <and>
  (x$1: Array[Char],x$2: Int)Array[Char] <and>
  (x$1: Array[Long],x$2: Int)Array[Long] <and>
  (x$1: Array[Int],x$2: Int)Array[Int] <and>
  (x$1: Array[Short],x$2: Int)Array[Short] <and>
  (x$1: Array[Byte],x$2: Int)Array[Byte] <and>
  [T, U](x$1: Array[U with Object], x$2: Int, x$3: Class[_ <: Array[T with Object]]): Array[T with Object] <and>
  [T](x$1: Array[T with Object], x$2: Int): Array[T with Object]
 cannot be applied to (Array[Activity], Int)
  implicit val codec: JsonValueCodec[ReputationJson] = JsonCodecMaker.make(CodecMakerConfig.withDiscriminatorFieldName(None))
Timur
@Timur83176832_twitter
@plokhotnyuk looks like its fine once i changed Array to List
Andriy Plokhotnyuk
@plokhotnyuk
Do Activity and ActivityRank types extend AnyVal? It looks like an error in the jsoniter-scala-macros module. java.util.Arrays.copyOf is used in the implementation of codecs that are generated by the make macro. I will try to reproduce and fix it. .
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter I've reproduced it in a simple test and created an issue on behalf of you: plokhotnyuk/jsoniter-scala#700
Timur
@Timur83176832_twitter
Yeah they do.cool :)
Also,I have a class with a field of either[JsonObject, List[Json]] of circe, It looks like i managed to create a custom codec from the examples in your github, but when they are deserialized it does show something like "text":{"Js":{"j":{"type":"Left","value":{" , can i paste my hierarchy for assistance? Thanks alot!
Same thing with Option[Either[Long, String]]
the reason for using circe is some manipulation we are doing on fields,and refactoring the project would be a big effort
Andriy Plokhotnyuk
@plokhotnyuk
For circe classes a custom codec like this could be a suitable solution but don't forget to test it carefully. If you need unwraping for Either values then there are examples of custom codecs for Either here and here.
Timur
@Timur83176832_twitter
Ok i will try to use the examples.Thanks alot!! @plokhotnyuk
Andriy Plokhotnyuk
@plokhotnyuk
BTW, if you need to work with some schema-less JSON then you can use the dijon library instead of circe. Or if those parts of JSON don't need to be parsed and just serialized as is then the RawVal types and a corresponding custom codec like here can be used for that.
Timur
@Timur83176832_twitter
Ok ill look into it as well.Thanks.The either example are ok but my eithers areEither[JsonObject,List[Json]](of circe)) ,i am guessing its not that straightforward implementation
Andriy Plokhotnyuk
@plokhotnyuk
Could you please create some small isolated project with tests for some expected inputs and outputs for Either[JsonObject,List[Json]]. I will fork and replace circe by jsoniter-scala in it.
Timur
@Timur83176832_twitter
case class Text(value: JsonOrString) extends AnyVal
sealed abstract class JsonOrString
  case class Str(s: String) extends JsonOrString
  case class Js(j: Either[JsonObject, List[CirceJson]]) extends JsonOrString
This is the hierarchy.Ideally i'd like the text to be serialized/deserialized as : "text":"{..." without types or value keys.
I have created custom decoders encoders for JsonObject and for CirceJson
this is the JsonObject codec:
implicit val jsonObjectCodec: JsonValueCodec[JsonObject] = new JsonValueCodec[JsonObject] {
    import io.circe.generic.encoding.ReprAsObjectEncoder.deriveReprAsObjectEncoder

    override def decodeValue(in: JsonReader, default: JsonObject): JsonObject = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        JsonObject.empty
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromBoolean(false).asObject.getOrElse(JsonObject.empty)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        JsonObject.empty
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong()).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromDoubleOrString(in.readDouble()).asObject.getOrElse(JsonObject.empty)
      } else if (b == '{') {
        val obj = scala.collection.mutable.Map[String, CirceJson]()
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), circeJsonCodec.decodeValue(in, default.asJson)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        JsonObject.fromIterable(obj)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += circeJsonCodec.decodeValue(in, CirceJson.fromFields(default.toMap)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }

        CirceJson.fromValues(arr).asObject.getOrElse(JsonObject.empty)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: JsonObject, out: JsonWriter): Unit =
      mapCodec.encodeValue(x.toMap, out)

    override def nullValue: JsonObject = JsonObject.empty
  }
this is circeJson:
implicit val circeJsonCodec: JsonValueCodec[CirceJson] = new JsonValueCodec[CirceJson] {
    import scala.jdk.CollectionConverters._
    override def decodeValue(in: JsonReader, default: CirceJson): CirceJson = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        CirceJson.fromString(in.readString(null))
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true)
        else CirceJson.fromBoolean(false)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        CirceJson.Null
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong())
        else CirceJson.fromDoubleOrString(in.readDouble())
      } else if (b == '{') {
        val obj = new java.util.LinkedHashMap[String, CirceJson]
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), decodeValue(in, default)) while (in.isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        CirceJson.fromFields(obj.asScala)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += decodeValue(in, default) while (in.isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }
        CirceJson.fromValues(arr.toList)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: CirceJson, out: JsonWriter): Unit =
      x.fold(
        out.writeNull(),
        jsonBoolean => out.writeVal(jsonBoolean),
        jsonNumber => {
          val longNumber = jsonNumber.toLong
          out.writeVal(if (longNumber.isDefined) longNumber.get else jsonNumber.toDouble)
        },
        jsonString => out.writeVal(jsonString),
        jsonArray => {
          out.writeArrayStart()
          jsonArray.foreach(v => encodeValue(v, out))
          out.writeArrayEnd()
        },
        jsonObject => {
          out.writeObjectStart()
          jsonObject.toIterable.foreach {
            case (k, v) =>
              out.writeKey(k)
              encodeValue(v, out)
          }
          out.writeObjectEnd()
        }
      )

    override def nullValue: CirceJson = CirceJson.Null
  }
its probably not ideal,I was toying with it today just to make my project work and the tests pass,so tweaking will be done later
sure @plokhotnyuk i will probably be able to do it tomorrow .My goal is for the Text case class to be with the value of the Either without types/value fields
Andriy Plokhotnyuk
@plokhotnyuk
If you will parse untrusted input then beware that Scala's maps are vulnerable under DoS attacks that exploit hash code collisions: scala/bug#11203
Timur
@Timur83176832_twitter
cool,didnt know! @plokhotnyuk but mostly its just logs
Timur
@Timur83176832_twitter
{
    "text": {
        "processing_breakdown": {
            "fetching_messages": {
                "num": 1,
                "sum": 2
            }
        }
    }
}
vs
{
    "text": {
        "Js": {
            "j": {
                "Left": {
                    "value": {
                        "processing_breakdown": {
                            "fetching_messages": {
                                "num": 1.0,
                                "sum": 2
                            }
                        }
                    }
                }
            }
        }
    }
}
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter I've released v2.7.3 with the issue fix. It is available on the Maven Central already: https://repo1.maven.org/maven2/com/github/plokhotnyuk/jsoniter-scala/jsoniter-scala-macros_2.13/2.7.3/
Timur
@Timur83176832_twitter
@plokhotnyuk great,thanks alot! Will use it
timothy
@timothyklim
Hello! Thanks for the best library for json serialization!
I have a question about ability to disable emit of DiscriminatorFieldName for ADT's. Is that possible? Because many serializers do not like unknown fields and will breaks easily. For my case it's easy to design class structure as ADT and then clients have schema to parse fields. The schema hides implementation design and do not emit extra fields.
Without this ability I have a workaround with custom JsonCodec for each ADT class to avoid extra field.
Andriy Plokhotnyuk
@plokhotnyuk
@TimothyKlim Hi, Timothy! Thanks for your feedback! Currently the no ADT tags option is not supported for derivation of codecs. The main reason is non-safe implementation of parsing for general case. plokhotnyuk/jsoniter-scala#435 is an issue where possible solutions were discussed.
Glen Marchesani
@fizzy33

I am converting from an AST based json library (well a few actually) I am looking for a recommendation. I have messaging middleware where effectively there are two layers one layer has the metadata and another layer handles the message bodies like so

case class Message(
  context: MessageContext,
  sender: NodeId,
  destination: NodeId,
  methodName: MethodName,
  event: StreamEvent,
  headers: Map[String,String],
  body: JsonStr,
)

So JsonStr is really any valid json. In the AST json world I would use the AST type. In jsoniter world that doesn't exist. So I have this

    object JsonStr {
      implicit val codec: JsonValueCodec[JsonStr] =
        new JsonValueCodec[JsonStr] {

          override def decodeValue(in: JsonReader, default: JsonStr): JsonStr =
            JsonStr(Chunk.array(in.readRawValAsBytes))

          override def encodeValue(x: JsonStr, out: JsonWriter): Unit =
            out.writeRawVal(x.rawBytes.toArray) // ??? optimize me we have at least one if not more extra array copies

          override def nullValue: JsonStr = ???
        }

      val Null = JsonStr(Chunk.array("null".getBytes))
      val EmptyObj = JsonStr(Chunk.array("{}".getBytes))
      val EmptyArr = JsonStr(Chunk.array("[]".getBytes))

    }
    case class JsonStr(rawBytes: Chunk[Byte])
So this use case with AST we parse once to AST and then in another layer process Message.body into the underlying type expected.
The use case with jsoniter it looks like I will have to parse the json twice. Once to get JsonStr and another to parse JsonStr into the type expected.
Wondering if anyone has thoughts on how best to do this with JsonIter
second question is any interest in PR with some tweaks to JsonReader and JsonWriter to support JsonStr working via a zero copy ?
Glen Marchesani
@fizzy33
third question is a ponder related to the first, is this double parsing going to ruin the performance gains of using jsonite?
any thoughts are appreciated
Andriy Plokhotnyuk
@plokhotnyuk
@fizzy33 Hi, Glen! Thanks for reaching out here! Usually reading and writing of raw bytes used for speeding up of JSON message transformation when those raw chunks go from the input to the output untouched, or for deferred parsing when raw chunks are re-parsed to case classes on demand in case of some rare conditions. I need to understand your challenge in general to propose the most efficient solution. As an example, if the body is schema-less and should be parsed always fully to some AST then you can do that without intermediate raw chunks using some custom codec. Dijon and ply-json-jsoniter libraries have such kind of codecs for their ASTs.
Andriy Plokhotnyuk
@plokhotnyuk
BTW, if it is the Chunk type is from the fs2 library then it will be quite inefficient to use it with Byte type due lack of specialization for primitives.
Glen Marchesani
@fizzy33
well something Chunk-like... wouldn't want to add fs2 as a dep and roger on the innefciencies without specialization
Well my use case is there is a general messaging layer to pass messages between processes which the Message class is the handler. Then at the lower level each class gets the Body and then turns it into whatever it needs to process. So for example most of the messages are some form of api call so one such call is just a case class e.g.
   case class QueryRequest(query: String, batching: Int)

  // the response
  case class QueryResponse(rows: fs2.Stream[F, Chunk[Row]], metadata: QueryMetadata)
Message and QueryRequest | QueryResponse are distinct layers so one cannot marshal them into instances at the same time.
Glen Marchesani
@fizzy33
fwiw I did some prelim testing on the JsonStr works well enough for now
I get better performance all around and probly can tweak it a bit later