Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 02 16:57

    plokhotnyuk on master

    More efficient parsing of nanos… (compare)

  • Dec 02 16:27

    plokhotnyuk on more-efficient-parsing-of-nanos

    (compare)

  • Dec 02 16:27

    plokhotnyuk on master

    More efficient parsing of nanos… (compare)

  • Dec 02 16:27
    plokhotnyuk closed #968
  • Dec 02 16:06
    plokhotnyuk opened #968
  • Dec 02 16:05

    plokhotnyuk on more-efficient-parsing-of-nanos

    More efficient parsing of nanos… (compare)

  • Dec 02 14:22

    plokhotnyuk on master

    Code clean up (compare)

  • Dec 02 11:56

    plokhotnyuk on master

    Code clean up (compare)

  • Dec 01 10:47

    plokhotnyuk on master

    Code clean up (compare)

  • Dec 01 09:45

    plokhotnyuk on master

    Code clean up (compare)

  • Nov 30 19:45

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 18:07

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 16:06

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 11:13

    plokhotnyuk on master

    More efficient derivation of co… (compare)

  • Nov 30 11:00

    plokhotnyuk on master

    Code clean up More efficient derivation of co… (compare)

  • Nov 30 10:35

    plokhotnyuk on master

    Code clean up (compare)

  • Nov 30 09:32

    plokhotnyuk on master

    Code clean up More efficient serialization of… (compare)

  • Nov 30 09:00

    plokhotnyuk on master

    Code clean up (compare)

  • Nov 30 07:56

    plokhotnyuk on master

    Code clean up (compare)

  • Nov 30 07:36

    plokhotnyuk on master

    Code clean up (compare)

corentin
@corenti13711539_twitter
@plokhotnyuk Fabulous! The warning disappears with the new version :muscle:
Repository starred :star: - thanks! :thumbsup:
corentin
@corenti13711539_twitter
Btw, any thoughts on performance implications of using .withRequireDiscriminatorFirst(false)? I'd imagine this would be affected at least to some extent by the size of the JSON as well as discriminator field position. I'm decoding JSON that's produced by another party and unfortunately won't be able to affect field order.
Andriy Plokhotnyuk
@plokhotnyuk
@corenti13711539_twitter .withRequireDiscriminatorFirst(false) doesn't affect performance, but if the hint was set in the end the expected average performance drop is about 1.5x times for each level of nested coproducts. As an example the GeoJSON sample in the benchmark sub-project has 3 nested sum types: FeatureCollection, Feature, and Polygon, so when all hints appear in the end the performance drop will be about 1.5 * 1.5 * 1.5 = 3.375 times.
Andriy Plokhotnyuk
@plokhotnyuk
From 12 JSON serializers for Scala that are tested by benchmarks only circe and spray-json serialize hints in the end.
corentin
@corenti13711539_twitter
@plokhotnyuk thanks! :thumbsup:
Andriy Plokhotnyuk
@plokhotnyuk
@corenti13711539_twitter Thank you too for the feedback and support!
yankun1992
@yankun1992
hello everyone!I am looking for help,if I have a json like ["hello", "world", 1, 0, false] . How to parse it ?
Andriy Plokhotnyuk
@plokhotnyuk
@yankun1992 Hi! Do you have a stable structure of JSON and a data type to which it should be parsed? If no than probably you should try dijon, that uses jsoniter-scala-core under the hood.
yankun1992
@yankun1992
I
I want to parse it to Array[String]
or Array[Any]
Andriy Plokhotnyuk
@plokhotnyuk
The most efficient way parsing to Array[String] or Array[Any] is a custom codec. Have you tried dijon for that yet? How are you going to use parsed values?
yankun1992
@yankun1992
no, my json is byte code in bytebuffer, I want parse it to scala case class directly.
```json
{
    "request_id": "",
    "code": 0,
    "msg": "",
    "data": {
        "fields": [
            "exchange",
            "cal_date",
            "is_open",
            "pretrade_date"
        ],
        "items": [
            [
                "SSE",
                "19901219",
                1,
                null
            ],
            [
                "SSE",
                "19941222",
                1,
                "19941221"
            ],
            [
                "SSE",
                "19941223",
                1,
                "19941222"
            ]
        ],
        "has_more": false
    }
}
and i want to parse it to
case class TushareResponse(
                            request_id: String,
                            code: Int,
                            msg: String,
                            data: Data
                          )
case class Data(
                   fields: Array[String],
                   items: Array[Array[Option[String]]],
                   has_more: Boolean
                 )
Andriy Plokhotnyuk
@plokhotnyuk
Below is an implementation of a custom codec for Option[String] of items` elements:
        implicit val optStringCodec: JsonValueCodec[Option[String]] = new JsonValueCodec[Option[String]] {
          override def decodeValue(in: JsonReader, default: Option[String]): Option[String] = {
            val b = in.nextToken()
            if (b == 'n') {
              in.readNullOrError(None, "expected `null` value")
              None
            } else if (b == '"') {
              in.rollbackToken()
              Some(in.readString(null))
            } else if (b == 't' || b == 'f') {
              in.rollbackToken()
              Some(in.readBoolean().toString)
            } else if ((b >= '0' && b <= '9') || b == '-') {
              in.rollbackToken()
              Some(in.readBigDecimal(null).toString)
            } else in.decodeError("unexpected value")
          }

          override def encodeValue(x: Option[String], out: JsonWriter): Unit =
            x.fold(out.writeNull())(out.writeVal)

          override def nullValue: Option[String] = None
        }
Full code please see here
The expected output:
{
    "request_id": "",
    "code": 0,
    "msg": "",
    "data": {
        "fields": [
            "exchange",
            "cal_date",
            "is_open",
            "pretrade_date"
        ],
        "items": [
            [
                "SSE",
                "19901219",
                "1",
                null
            ],
            [
                "SSE",
                "19941222",
                "1",
                "19941221"
            ],
            [
                "SSE",
                "19941223",
                "1",
                "19941222"
            ]
        ],
        "has_more": false
    }
}
yankun1992
@yankun1992
Aha, I got it. Thank you ever much.
By the way, this is a really cool library! Thank you again!
Chetan Mehrotra
@chetanmeh

By the way, this is a really cool library! Thank you again!

+1 To that!

Timur
@Timur83176832_twitter
Hello. I am creating some codecs with JsonCodecMaker.makeWithoutDiscriminator and i can see in my output that my values look like
computerName":{"value":"tier2-logsingest-775446dbc8-m64bg"}
each field contains a value with that contains the value itself.Is there any way to remove the value field all together? @plokhotnyuk
Thank you!
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter Hello, Timur! Could you please share the related part of your data structures? The easiest way to unwrap single field values is to extend AnyVal like here. Another option is writing of some custom codec.
Timur
@Timur83176832_twitter
@plokhotnyuk I added extends AnyVal to everywhere,not it fails with a different error.
The case class is :
case class ReputationJson(
  ip: IPAddress,
  reliability: Option[Reliability],
  activity: Option[Array[Activity]],
  risk: Option[Risk],
  country: Option[Country],
  countryCode: Option[CountryCode],
  city: Option[City],
  coordinates: Option[Coordinates],
  activityRank: Option[Array[ActivityRank]],
  score: Score,
  scoreName: ScoreName
)
overloaded method copyOf with alternatives:
  (x$1: Array[Boolean],x$2: Int)Array[Boolean] <and>
  (x$1: Array[Double],x$2: Int)Array[Double] <and>
  (x$1: Array[Float],x$2: Int)Array[Float] <and>
  (x$1: Array[Char],x$2: Int)Array[Char] <and>
  (x$1: Array[Long],x$2: Int)Array[Long] <and>
  (x$1: Array[Int],x$2: Int)Array[Int] <and>
  (x$1: Array[Short],x$2: Int)Array[Short] <and>
  (x$1: Array[Byte],x$2: Int)Array[Byte] <and>
  [T, U](x$1: Array[U with Object], x$2: Int, x$3: Class[_ <: Array[T with Object]]): Array[T with Object] <and>
  [T](x$1: Array[T with Object], x$2: Int): Array[T with Object]
 cannot be applied to (Array[Activity], Int)
  implicit val codec: JsonValueCodec[ReputationJson] = JsonCodecMaker.make(CodecMakerConfig.withDiscriminatorFieldName(None))
Timur
@Timur83176832_twitter
@plokhotnyuk looks like its fine once i changed Array to List
Andriy Plokhotnyuk
@plokhotnyuk
Do Activity and ActivityRank types extend AnyVal? It looks like an error in the jsoniter-scala-macros module. java.util.Arrays.copyOf is used in the implementation of codecs that are generated by the make macro. I will try to reproduce and fix it. .
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter I've reproduced it in a simple test and created an issue on behalf of you: plokhotnyuk/jsoniter-scala#700
Timur
@Timur83176832_twitter
Yeah they do.cool :)
Also,I have a class with a field of either[JsonObject, List[Json]] of circe, It looks like i managed to create a custom codec from the examples in your github, but when they are deserialized it does show something like "text":{"Js":{"j":{"type":"Left","value":{" , can i paste my hierarchy for assistance? Thanks alot!
Same thing with Option[Either[Long, String]]
the reason for using circe is some manipulation we are doing on fields,and refactoring the project would be a big effort
Andriy Plokhotnyuk
@plokhotnyuk
For circe classes a custom codec like this could be a suitable solution but don't forget to test it carefully. If you need unwraping for Either values then there are examples of custom codecs for Either here and here.
Timur
@Timur83176832_twitter
Ok i will try to use the examples.Thanks alot!! @plokhotnyuk
Andriy Plokhotnyuk
@plokhotnyuk
BTW, if you need to work with some schema-less JSON then you can use the dijon library instead of circe. Or if those parts of JSON don't need to be parsed and just serialized as is then the RawVal types and a corresponding custom codec like here can be used for that.
Timur
@Timur83176832_twitter
Ok ill look into it as well.Thanks.The either example are ok but my eithers areEither[JsonObject,List[Json]](of circe)) ,i am guessing its not that straightforward implementation
Andriy Plokhotnyuk
@plokhotnyuk
Could you please create some small isolated project with tests for some expected inputs and outputs for Either[JsonObject,List[Json]]. I will fork and replace circe by jsoniter-scala in it.
Timur
@Timur83176832_twitter
case class Text(value: JsonOrString) extends AnyVal
sealed abstract class JsonOrString
  case class Str(s: String) extends JsonOrString
  case class Js(j: Either[JsonObject, List[CirceJson]]) extends JsonOrString
This is the hierarchy.Ideally i'd like the text to be serialized/deserialized as : "text":"{..." without types or value keys.
I have created custom decoders encoders for JsonObject and for CirceJson
this is the JsonObject codec:
implicit val jsonObjectCodec: JsonValueCodec[JsonObject] = new JsonValueCodec[JsonObject] {
    import io.circe.generic.encoding.ReprAsObjectEncoder.deriveReprAsObjectEncoder

    override def decodeValue(in: JsonReader, default: JsonObject): JsonObject = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        JsonObject.empty
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromBoolean(false).asObject.getOrElse(JsonObject.empty)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        JsonObject.empty
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong()).asObject.getOrElse(JsonObject.empty)
        else CirceJson.fromDoubleOrString(in.readDouble()).asObject.getOrElse(JsonObject.empty)
      } else if (b == '{') {
        val obj = scala.collection.mutable.Map[String, CirceJson]()
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), circeJsonCodec.decodeValue(in, default.asJson)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        JsonObject.fromIterable(obj)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += circeJsonCodec.decodeValue(in, CirceJson.fromFields(default.toMap)) while (in
            .isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }

        CirceJson.fromValues(arr).asObject.getOrElse(JsonObject.empty)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: JsonObject, out: JsonWriter): Unit =
      mapCodec.encodeValue(x.toMap, out)

    override def nullValue: JsonObject = JsonObject.empty
  }
this is circeJson:
implicit val circeJsonCodec: JsonValueCodec[CirceJson] = new JsonValueCodec[CirceJson] {
    import scala.jdk.CollectionConverters._
    override def decodeValue(in: JsonReader, default: CirceJson): CirceJson = {
      var b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        CirceJson.fromString(in.readString(null))
      } else if (b == 'f' || b == 't') {
        in.rollbackToken()
        if (in.readBoolean()) CirceJson.fromBoolean(true)
        else CirceJson.fromBoolean(false)
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        CirceJson.Null
      } else if ((b >= '0' && b <= '9') || b == '-') {
        in.rollbackToken()
        in.setMark()
        val bs = in.readRawValAsBytes()
        val l = bs.length
        var i = 0
        while (i < l && {
                 b = bs(i)
                 b >= '0' && b <= '9'
               }) i += 1
        in.rollbackToMark()
        if (i == l) CirceJson.fromLong(in.readLong())
        else CirceJson.fromDoubleOrString(in.readDouble())
      } else if (b == '{') {
        val obj = new java.util.LinkedHashMap[String, CirceJson]
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          do obj.put(in.readKeyAsString(), decodeValue(in, default)) while (in.isNextToken(','))
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        CirceJson.fromFields(obj.asScala)
      } else if (b == '[') {
        val arr = new mutable.ArrayBuffer[CirceJson]
        if (!in.isNextToken(']')) {
          in.rollbackToken()
          do arr += decodeValue(in, default) while (in.isNextToken(','))
          if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
        }
        CirceJson.fromValues(arr.toList)
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: CirceJson, out: JsonWriter): Unit =
      x.fold(
        out.writeNull(),
        jsonBoolean => out.writeVal(jsonBoolean),
        jsonNumber => {
          val longNumber = jsonNumber.toLong
          out.writeVal(if (longNumber.isDefined) longNumber.get else jsonNumber.toDouble)
        },
        jsonString => out.writeVal(jsonString),
        jsonArray => {
          out.writeArrayStart()
          jsonArray.foreach(v => encodeValue(v, out))
          out.writeArrayEnd()
        },
        jsonObject => {
          out.writeObjectStart()
          jsonObject.toIterable.foreach {
            case (k, v) =>
              out.writeKey(k)
              encodeValue(v, out)
          }
          out.writeObjectEnd()
        }
      )

    override def nullValue: CirceJson = CirceJson.Null
  }
its probably not ideal,I was toying with it today just to make my project work and the tests pass,so tweaking will be done later
sure @plokhotnyuk i will probably be able to do it tomorrow .My goal is for the Text case class to be with the value of the Either without types/value fields
Andriy Plokhotnyuk
@plokhotnyuk
If you will parse untrusted input then beware that Scala's maps are vulnerable under DoS attacks that exploit hash code collisions: scala/bug#11203
Timur
@Timur83176832_twitter
cool,didnt know! @plokhotnyuk but mostly its just logs
Timur
@Timur83176832_twitter
{
    "text": {
        "processing_breakdown": {
            "fetching_messages": {
                "num": 1,
                "sum": 2
            }
        }
    }
}
vs
{
    "text": {
        "Js": {
            "j": {
                "Left": {
                    "value": {
                        "processing_breakdown": {
                            "fetching_messages": {
                                "num": 1.0,
                                "sum": 2
                            }
                        }
                    }
                }
            }
        }
    }
}
Andriy Plokhotnyuk
@plokhotnyuk
@Timur83176832_twitter I've released v2.7.3 with the issue fix. It is available on the Maven Central already: https://repo1.maven.org/maven2/com/github/plokhotnyuk/jsoniter-scala/jsoniter-scala-macros_2.13/2.7.3/
Timur
@Timur83176832_twitter
@plokhotnyuk great,thanks alot! Will use it
timothy
@timothyklim
Hello! Thanks for the best library for json serialization!
I have a question about ability to disable emit of DiscriminatorFieldName for ADT's. Is that possible? Because many serializers do not like unknown fields and will breaks easily. For my case it's easy to design class structure as ADT and then clients have schema to parse fields. The schema hides implementation design and do not emit extra fields.
Without this ability I have a workaround with custom JsonCodec for each ADT class to avoid extra field.