Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 04 16:23

    plokhotnyuk on gh-pages

    Update benchmark results from J… (compare)

  • Dec 02 16:57

    plokhotnyuk on master

    More efficient parsing of nanos… (compare)

  • Dec 02 16:27

    plokhotnyuk on more-efficient-parsing-of-nanos

    (compare)

  • Dec 02 16:27

    plokhotnyuk on master

    More efficient parsing of nanos… (compare)

  • Dec 02 16:27
    plokhotnyuk closed #968
  • Dec 02 16:06
    plokhotnyuk opened #968
  • Dec 02 16:05

    plokhotnyuk on more-efficient-parsing-of-nanos

    More efficient parsing of nanos… (compare)

  • Dec 02 14:22

    plokhotnyuk on master

    Code clean up (compare)

  • Dec 02 11:56

    plokhotnyuk on master

    Code clean up (compare)

  • Dec 01 10:47

    plokhotnyuk on master

    Code clean up (compare)

  • Dec 01 09:45

    plokhotnyuk on master

    Code clean up (compare)

  • Nov 30 19:45

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 18:07

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 16:06

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 11:13

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 11:00

    plokhotnyuk on master

    Code clean up More efficient derivation of co… (compare)

  • Nov 30 10:35

    plokhotnyuk on master

    Code clean up (compare)

  • Nov 30 09:32

    plokhotnyuk on master

    Code clean up More efficient serialization of… (compare)

  • Nov 30 09:00

    plokhotnyuk on master

    Code clean up (compare)

  • Nov 30 07:56

    plokhotnyuk on master

    Code clean up (compare)

yankun1992
@yankun1992
hello everyone!I am looking for help,if I have a json like ["hello", "world", 1, 0, false] . How to parse it ?
Andriy Plokhotnyuk
@plokhotnyuk
@yankun1992 Hi! Do you have a stable structure of JSON and a data type to which it should be parsed? If no than probably you should try dijon, that uses jsoniter-scala-core under the hood.
yankun1992
@yankun1992
I
I want to parse it to Array[String]
or Array[Any]
Andriy Plokhotnyuk
@plokhotnyuk
The most efficient way parsing to Array[String] or Array[Any] is a custom codec. Have you tried dijon for that yet? How are you going to use parsed values?
yankun1992
@yankun1992
no, my json is byte code in bytebuffer, I want parse it to scala case class directly.
```json
{
    "request_id": "",
    "code": 0,
    "msg": "",
    "data": {
        "fields": [
            "exchange",
            "cal_date",
            "is_open",
            "pretrade_date"
        ],
        "items": [
            [
                "SSE",
                "19901219",
                1,
                null
            ],
            [
                "SSE",
                "19941222",
                1,
                "19941221"
            ],
            [
                "SSE",
                "19941223",
                1,
                "19941222"
            ]
        ],
        "has_more": false
    }
}
and i want to parse it to
case class TushareResponse(
                            request_id: String,
                            code: Int,
                            msg: String,
                            data: Data
                          )
case class Data(
                   fields: Array[String],
                   items: Array[Array[Option[String]]],
                   has_more: Boolean
                 )
Andriy Plokhotnyuk
@plokhotnyuk
Below is an implementation of a custom codec for Option[String] of items` elements:
        implicit val optStringCodec: JsonValueCodec[Option[String]] = new JsonValueCodec[Option[String]] {
          override def decodeValue(in: JsonReader, default: Option[String]): Option[String] = {
            val b = in.nextToken()
            if (b == 'n') {
              in.readNullOrError(None, "expected `null` value")
              None
            } else if (b == '"') {
              in.rollbackToken()
              Some(in.readString(null))
            } else if (b == 't' || b == 'f') {
              in.rollbackToken()
              Some(in.readBoolean().toString)
            } else if ((b >= '0' && b <= '9') || b == '-') {
              in.rollbackToken()
              Some(in.readBigDecimal(null).toString)
            } else in.decodeError("unexpected value")
          }

          override def encodeValue(x: Option[String], out: JsonWriter): Unit =
            x.fold(out.writeNull())(out.writeVal)

          override def nullValue: Option[String] = None
        }
Full code please see here
The expected output:
{
    "request_id": "",
    "code": 0,
    "msg": "",
    "data": {
        "fields": [
            "exchange",
            "cal_date",
            "is_open",
            "pretrade_date"
        ],
        "items": [
            [
                "SSE",
                "19901219",
                "1",
                null
            ],
            [
                "SSE",
                "19941222",
                "1",
                "19941221"
            ],
            [
                "SSE",
                "19941223",
                "1",
                "19941222"
            ]
        ],
        "has_more": false
    }
}
yankun1992
@yankun1992
Aha, I got it. Thank you ever much.
By the way, this is a really cool library! Thank you again!
Chetan Mehrotra
@chetanmeh

By the way, this is a really cool library! Thank you again!

+1 To that!

Timur
@Timur83176832_twitter
Hello. I am creating some codecs with JsonCodecMaker.makeWithoutDiscriminator and i can see in my output that my values look like
computerName":{"value":"tier2-logsingest-775446dbc8-m64bg"}
each field contains a value with that contains the value itself.Is there any way to remove the value field all together? @plokhotnyuk
Thank you!
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter Hello, Timur! Could you please share the related part of your data structures? The easiest way to unwrap single field values is to extend AnyVal like here. Another option is writing of some custom codec.
Timur
@Timur83176832_twitter
@plokhotnyuk I added extends AnyVal to everywhere,not it fails with a different error.
The case class is :
case class ReputationJson(
  ip: IPAddress,
  reliability: Option[Reliability],
  activity: Option[Array[Activity]],
  risk: Option[Risk],
  country: Option[Country],
  countryCode: Option[CountryCode],
  city: Option[City],
  coordinates: Option[Coordinates],
  activityRank: Option[Array[ActivityRank]],
  score: Score,
  scoreName: ScoreName
)
overloaded method copyOf with alternatives:
  (x$1: Array[Boolean],x$2: Int)Array[Boolean] <and>
  (x$1: Array[Double],x$2: Int)Array[Double] <and>
  (x$1: Array[Float],x$2: Int)Array[Float] <and>
  (x$1: Array[Char],x$2: Int)Array[Char] <and>
  (x$1: Array[Long],x$2: Int)Array[Long] <and>
  (x$1: Array[Int],x$2: Int)Array[Int] <and>
  (x$1: Array[Short],x$2: Int)Array[Short] <and>
  (x$1: Array[Byte],x$2: Int)Array[Byte] <and>
  [T, U](x$1: Array[U with Object], x$2: Int, x$3: Class[_ <: Array[T with Object]]): Array[T with Object] <and>
  [T](x$1: Array[T with Object], x$2: Int): Array[T with Object]
 cannot be applied to (Array[Activity], Int)
  implicit val codec: JsonValueCodec[ReputationJson] = JsonCodecMaker.make(CodecMakerConfig.withDiscriminatorFieldName(None))
Timur
@Timur83176832_twitter
@plokhotnyuk looks like its fine once i changed Array to List
Andriy Plokhotnyuk
@plokhotnyuk
Do Activity and ActivityRank types extend AnyVal? It looks like an error in the jsoniter-scala-macros module. java.util.Arrays.copyOf is used in the implementation of codecs that are generated by the make macro. I will try to reproduce and fix it. .
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter I've reproduced it in a simple test and created an issue on behalf of you: plokhotnyuk/jsoniter-scala#700
Timur
@Timur83176832_twitter
Yeah they do.cool :)
Also,I have a class with a field of either[JsonObject, List[Json]] of circe, It looks like i managed to create a custom codec from the examples in your github, but when they are deserialized it does show something like "text":{"Js":{"j":{"type":"Left","value":{" , can i paste my hierarchy for assistance? Thanks alot!
Same thing with Option[Either[Long, String]]
the reason for using circe is some manipulation we are doing on fields,and refactoring the project would be a big effort
Andriy Plokhotnyuk
@plokhotnyuk
For circe classes a custom codec like this could be a suitable solution but don't forget to test it carefully. If you need unwraping for Either values then there are examples of custom codecs for Either here and here.
Timur
@Timur83176832_twitter
Ok i will try to use the examples.Thanks alot!! @plokhotnyuk
Andriy Plokhotnyuk
@plokhotnyuk
BTW, if you need to work with some schema-less JSON then you can use the dijon library instead of circe. Or if those parts of JSON don't need to be parsed and just serialized as is then the RawVal types and a corresponding custom codec like here can be used for that.
Timur
@Timur83176832_twitter
Ok ill look into it as well.Thanks.The either example are ok but my eithers areEither[JsonObject,List[Json]](of circe)) ,i am guessing its not that straightforward implementation
Andriy Plokhotnyuk
@plokhotnyuk
Could you please create some small isolated project with tests for some expected inputs and outputs for Either[JsonObject,List[Json]]. I will fork and replace circe by jsoniter-scala in it.
Timur
@Timur83176832_twitter
case class Text(value: JsonOrString) extends AnyVal
sealed abstract class JsonOrString
  case class Str(s: String) extends JsonOrString
  case class Js(j: Either[JsonObject, List[CirceJson]]) extends JsonOrString
This is the hierarchy.Ideally i'd like the text to be serialized/deserialized as : "text":"{..." without types or value keys.
I have created custom decoders encoders for JsonObject and for CirceJson
this is the JsonObject codec:
implicit val jsonObjectCodec: JsonValueCodec[JsonObject] = new JsonValueCodec[JsonObject] {
    import io.circe.generic.encoding.ReprAsObjectEncoder.deriveReprAsObjectEncoder

    override def decodeValue(in: JsonReader, default: JsonObject): JsonObject = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        JsonObject.empty
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromBoolean(false).asObject.getOrElse(JsonObject.empty)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        JsonObject.empty
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong()).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromDoubleOrString(in.readDouble()).asObject.getOrElse(JsonObject.empty)
      } else if (b == '{') {
        val obj = scala.collection.mutable.Map[String, CirceJson]()
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), circeJsonCodec.decodeValue(in, default.asJson)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        JsonObject.fromIterable(obj)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += circeJsonCodec.decodeValue(in, CirceJson.fromFields(default.toMap)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }

        CirceJson.fromValues(arr).asObject.getOrElse(JsonObject.empty)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: JsonObject, out: JsonWriter): Unit =
      mapCodec.encodeValue(x.toMap, out)

    override def nullValue: JsonObject = JsonObject.empty
  }
this is circeJson:
implicit val circeJsonCodec: JsonValueCodec[CirceJson] = new JsonValueCodec[CirceJson] {
    import scala.jdk.CollectionConverters._
    override def decodeValue(in: JsonReader, default: CirceJson): CirceJson = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        CirceJson.fromString(in.readString(null))
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true)
        else CirceJson.fromBoolean(false)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        CirceJson.Null
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong())
        else CirceJson.fromDoubleOrString(in.readDouble())
      } else if (b == '{') {
        val obj = new java.util.LinkedHashMap[String, CirceJson]
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), decodeValue(in, default)) while (in.isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        CirceJson.fromFields(obj.asScala)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += decodeValue(in, default) while (in.isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }
        CirceJson.fromValues(arr.toList)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: CirceJson, out: JsonWriter): Unit =
      x.fold(
        out.writeNull(),
        jsonBoolean => out.writeVal(jsonBoolean),
        jsonNumber => {
          val longNumber = jsonNumber.toLong
          out.writeVal(if (longNumber.isDefined) longNumber.get else jsonNumber.toDouble)
        },
        jsonString => out.writeVal(jsonString),
        jsonArray => {
          out.writeArrayStart()
          jsonArray.foreach(v => encodeValue(v, out))
          out.writeArrayEnd()
        },
        jsonObject => {
          out.writeObjectStart()
          jsonObject.toIterable.foreach {
            case (k, v) =>
              out.writeKey(k)
              encodeValue(v, out)
          }
          out.writeObjectEnd()
        }
      )

    override def nullValue: CirceJson = CirceJson.Null
  }
its probably not ideal,I was toying with it today just to make my project work and the tests pass,so tweaking will be done later
sure @plokhotnyuk i will probably be able to do it tomorrow .My goal is for the Text case class to be with the value of the Either without types/value fields
Andriy Plokhotnyuk
@plokhotnyuk
If you will parse untrusted input then beware that Scala's maps are vulnerable under DoS attacks that exploit hash code collisions: scala/bug#11203
Timur
@Timur83176832_twitter
cool,didnt know! @plokhotnyuk but mostly its just logs
Timur
@Timur83176832_twitter
{
    "text": {
        "processing_breakdown": {
            "fetching_messages": {
                "num": 1,
                "sum": 2
            }
        }
    }
}
vs
{
    "text": {
        "Js": {
            "j": {
                "Left": {
                    "value": {
                        "processing_breakdown": {
                            "fetching_messages": {
                                "num": 1.0,
                                "sum": 2
                            }
                        }
                    }
                }
            }
        }
    }
}
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter I've released v2.7.3 with the issue fix. It is available on the Maven Central already: https://repo1.maven.org/maven2/com/github/plokhotnyuk/jsoniter-scala/jsoniter-scala-macros_2.13/2.7.3/
Timur
@Timur83176832_twitter
@plokhotnyuk great,thanks alot! Will use it
timothy
@timothyklim
Hello! Thanks for the best library for json serialization!
I have a question about ability to disable emit of DiscriminatorFieldName for ADT's. Is that possible? Because many serializers do not like unknown fields and will breaks easily. For my case it's easy to design class structure as ADT and then clients have schema to parse fields. The schema hides implementation design and do not emit extra fields.
Without this ability I have a workaround with custom JsonCodec for each ADT class to avoid extra field.
Andriy Plokhotnyuk
@plokhotnyuk
@TimothyKlim Hi, Timothy! Thanks for your feedback! Currently the no ADT tags option is not supported for derivation of codecs. The main reason is non-safe implementation of parsing for general case. plokhotnyuk/jsoniter-scala#435 is an issue where possible solutions were discussed.
Glen Marchesani
@fizzy33

I am converting from an AST based json library (well a few actually) I am looking for a recommendation. I have messaging middleware where effectively there are two layers one layer has the metadata and another layer handles the message bodies like so

case class Message(
  context: MessageContext,
  sender: NodeId,
  destination: NodeId,
  methodName: MethodName,
  event: StreamEvent,
  headers: Map[String,String],
  body: JsonStr,
)

So JsonStr is really any valid json. In the AST json world I would use the AST type. In jsoniter world that doesn't exist. So I have this

    object JsonStr {
      implicit val codec: JsonValueCodec[JsonStr] =
        new JsonValueCodec[JsonStr] {

          override def decodeValue(in: JsonReader, default: JsonStr): JsonStr =
            JsonStr(Chunk.array(in.readRawValAsBytes))

          override def encodeValue(x: JsonStr, out: JsonWriter): Unit =
            out.writeRawVal(x.rawBytes.toArray) // ??? optimize me we have at least one if not more extra array copies

          override def nullValue: JsonStr = ???
        }

      val Null = JsonStr(Chunk.array("null".getBytes))
      val EmptyObj = JsonStr(Chunk.array("{}".getBytes))
      val EmptyArr = JsonStr(Chunk.array("[]".getBytes))

    }
    case class JsonStr(rawBytes: Chunk[Byte])
So this use case with AST we parse once to AST and then in another layer process Message.body into the underlying type expected.
The use case with jsoniter it looks like I will have to parse the json twice. Once to get JsonStr and another to parse JsonStr into the type expected.
Wondering if anyone has thoughts on how best to do this with JsonIter
second question is any interest in PR with some tweaks to JsonReader and JsonWriter to support JsonStr working via a zero copy ?
Glen Marchesani
@fizzy33
third question is a ponder related to the first, is this double parsing going to ruin the performance gains of using jsonite?