Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 11:00

    plokhotnyuk on master

    Add missing unit tests for benc… (compare)

  • 10:48

    plokhotnyuk on master

    Add missing unit tests for benc… (compare)

  • 10:17

    plokhotnyuk on master

    Add missing unit tests for benc… (compare)

  • Oct 22 13:53
    plokhotnyuk edited #237
  • Oct 22 12:59
    plokhotnyuk closed #154
  • Oct 22 12:59

    plokhotnyuk on master

    Close #154 by adding Scala docs… (compare)

  • Oct 22 12:27
    plokhotnyuk commented #154
  • Oct 22 12:22
    plokhotnyuk commented #154
  • Oct 22 12:17
    plokhotnyuk commented #237
  • Oct 22 12:16
    plokhotnyuk commented #237
  • Oct 22 12:16
    plokhotnyuk commented #237
  • Oct 22 12:14
    plokhotnyuk unlabeled #389
  • Oct 22 12:14
    plokhotnyuk labeled #389
  • Oct 22 12:13
    plokhotnyuk labeled #392
  • Oct 22 12:13
    plokhotnyuk unlabeled #392
  • Oct 22 07:05

    plokhotnyuk on master

    Add contents (compare)

  • Oct 21 14:46

    plokhotnyuk on master

    Fix DSL-JSON benchmark for Prim… (compare)

  • Oct 21 12:42
    plokhotnyuk edited #87
  • Oct 21 12:39
    plokhotnyuk closed #59
  • Oct 21 12:39
    plokhotnyuk commented #59
Gavin Bisesi
@Daenyth
Hmm ok, thank you
The f: Int => Unit callback, Int would there vary based on the array contents? The elements inside, are themselves json objects
(which I am OK to eagerly parse, just not all of them at the same time)
Andriy Plokhotnyuk
@plokhotnyuk
Type of repeated values can be generalized like here:
import java.io.ByteArrayInputStream

import com.github.plokhotnyuk.jsoniter_scala.core._
import com.github.plokhotnyuk.jsoniter_scala.macros._

object Example01 {
  def codec[A](aCodec: JsonValueCodec[A])(f: A => Unit): JsonValueCodec[Unit] = new JsonValueCodec[Unit] {
    override def decodeValue(in: JsonReader, default: Unit): Unit = {
      if (in.isNextToken('{')) {
        var p = 1
        if (!in.isNextToken('}')) {
          in.rollbackToken()
          var l = -1
          while (l < 0 || in.isNextToken(',')) {
            l = in.readKeyAsCharBuf()
            if (in.isCharBufEqualsTo(l, "data")) {
              if ((p & 0x1) != 0) p ^= 0x1
              else in.duplicatedKeyError(l)
              if (in.isNextToken('[')) {
                if (!in.isNextToken(']')) {
                  in.rollbackToken()
                  while ( {
                    f(aCodec.decodeValue(in, aCodec.nullValue))
                    in.isNextToken(',')
                  }) ()
                  if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
                }
              } else in.readNullOrTokenError(default, '[')
            } else in.skip()
          }
          if (!in.isCurrentToken('}')) in.objectEndOrCommaError()
        }
        if ((p & 0x1) != 0) in.requiredFieldError("data")
      }
    }

    override def encodeValue(x: Unit, out: _root_.com.github.plokhotnyuk.jsoniter_scala.core.JsonWriter): Unit = ???

    override def nullValue: Unit = ()
  }

  def main(args: Array[String]): Unit = {
    val aCodec = JsonCodecMaker.make[Int]
    val inputStream = new ByteArrayInputStream("""{"data": [1, 2, 3, 4] }""".getBytes("UTF-8"))
    readFromStream(inputStream)(codec(aCodec) {
      println
    })
  }
}
Andriy Plokhotnyuk
@plokhotnyuk
If types of values are unknown upfront then some AST representation and a codec for it can be used. Here is an example of the codec for the AST from the Dijon library.
Gavin Bisesi
@Daenyth
:+1: thank you
Chetan Mehrotra
@chetanmeh

In one of the scenario I am using groupedWeightedWithin operator of Akka Stream to group objects in json format upto certain size to implement batching. Say I have series of Item objects which I need to batch upto 1 MB size and send to a remote HTTP service.

Now in this case I can use jsoniter to serialize individual Item to json bytes and then use the size of byte array for weight computation. But post group I need to stitch these array of json bytes object to create a new json with some structure like

{"messages" : [
    //Item1 json bytes,
    //Item2 json bytes,
]}

Is there any easy way to model with with jsoniter where I do not derserialize the types to object ... make the required case class and then serialize it back. Basically stitch together a seq of objects in json bytes to create a new json object

Andriy Plokhotnyuk
@plokhotnyuk
I suppose that length of any item in the JSON representation is less than 1Mb. If yes, then you could use some predefined ~2Mb array with the {"messages" : [ header and then in a loop serialize all items using the writeToSubArray routine until it returns a position which exceeds 1Mb. So if limit exceeded then you should return the previous last position and set ]} in the footprint. BTW, to minimize allocations you can use some thread local pool of preallocated 2Mb arrays. Could it be a suitable solution for you?
Chetan Mehrotra
@chetanmeh
Makes sense. Given the size of grouped array instance would be known we can determine the exact size of final array with header and footer and in between , etc and allocate and fill that without further reallocations. Would give that a try
Otherwise I came up with custom codec using writeRawVal
  case class Item(name: String, age: Int)
  case class Message(messages: Seq[Array[Byte]])
  case class Message2(messages: Seq[Item])

  object Item {
    implicit val codec: JsonValueCodec[Item] = JsonCodecMaker.make[Item]
  }

  object Message {

    implicit val codec: JsonValueCodec[Message] = new JsonValueCodec[Message] {
      override def decodeValue(in: JsonReader, default: Message): Message = ???
      override def encodeValue(x: Message, out: JsonWriter): Unit = {
        out.writeObjectStart()
        out.writeKey("messages")
        out.writeArrayStart()
        x.messages.foreach(out.writeRawVal)
        out.writeArrayEnd()
        out.writeObjectEnd()
      }
      override def nullValue: Message = null.asInstanceOf[Message] // scalastyle:ignore
    }
  }
Which also works but working at lower level would allow to reduce new array allocations
Thanks Andriy for the pointer
Lev Sivashov
@lpld

Hello guys! I’m looking for a library to replace play-json in our production app, as we are starting to face delays when parsing big amounts of JSON. jsoniter-scala looks great, but we have a requirement: it is critical for us to distinguish between nulls and missing values in Json objects. I’ve came up with following type:

sealed trait JsNullable[+T] extends Product with Serializable
object JsNullable {
    case class JsValue[+T](t: T) extends JsNullable[T]
    case object JsEmpty extends JsNullable[Nothing]
    case object JsNull extends JsNullable[Nothing]
}

and here is a codec:

  def nullableCodec[T](read: JsonReader => T): JsonValueCodec[JsNullable[T]] = new JsonValueCodec[JsNullable[T]] {
    override def decodeValue(in: JsonReader, default: JsNullable[T]): JsNullable[T] = {
      if (in.isNextToken('n'))
        in.readNullOrError(JsNull, “null expected")
      else JsValue(read(in))
    }

    override def encodeValue(x: JsNullable[T], out: JsonWriter): Unit = ???
    override def nullValue: JsNullable[T] = JsEmpty
  }

But apparently it doesn’t work, because jsoniter doesn’t even call my codec if I the field is missing and I get "missing required field” exception.
Is it possible to achieve this somehow? Thanks in advance for your help!

Lev Sivashov
@lpld

I also tried wrapping my fields in Option like this:

case class TestObject(
   stringField: Option[JsNullable[String]]
)

Unfortunately it doesn’t help either, because in this case jsoniter doesn’t call my codec for both “null” and “missing” fields, and I just get None in both cases.

Andriy Plokhotnyuk
@plokhotnyuk
How about case class TestObject(stringField: Option[Either[Null, String]])? Then you can provide a custom codec for Either[Null, String] like here.
Lev Sivashov
@lpld
Thank you, I’ll try that!
Lev Sivashov
@lpld
Doesn’t work :( Same behavior as with Option[JsNullable]. I get None for both “null” and “missing”. Here’s my code:
object JsoniterTest extends App {

  case class TestObject(
    stringValue: Option[Either[Null, String]]
  )
  implicit val nullOrStringCodec = new JsonValueCodec[Either[Null, String]] {
    override def decodeValue(in: JsonReader,
                             default: Either[Null, String]): Either[Null, String] = {
      val t = in.nextToken()
      in.rollbackToken()

      if (t == '"') Right(in.readString(null))
      else in.readNullOrError(Left(null), "null expected")
    }
    override def encodeValue(x: Either[Null, String], out: JsonWriter): Unit = ???
    override def nullValue: Either[Null, String] = Left(null)
  }
  implicit val codec: JsonValueCodec[TestObject] = JsonCodecMaker.make[TestObject]

  println(readFromString[TestObject](
    """
      |{
      |"stringValue": null
      |}
      |""".stripMargin))
}
As far as I understand the codec for TestObject decides to not even call nullOrStringCodec when facing null value. It treats it as None.
Andriy Plokhotnyuk
@plokhotnyuk
Yep, I forgot that handing of null is in the codec that is derived for Option[_].
Lev Sivashov
@lpld
So, what are my options? Implement a custom codec for Option?
Andriy Plokhotnyuk
@plokhotnyuk
I bet it doesn't help because missing values are handled in codecs derived for case classes.
Lev Sivashov
@lpld
So, it’s impossible now?
Andriy Plokhotnyuk
@plokhotnyuk
Yep, no easy way for now. As an option you can use play-json-tools and stick with play-json binding while parsing to AST will be done by jsoniter-scala. Also, writing some custom macro for derivation could be an option too but will require much more work.
Lev Sivashov
@lpld
Thank you, I’ll take a look.
Marcin Szałomski
@baldram

@plokhotnyuk Hi Andriy. Does Jsoniter allow to pass explicitly the custom codec supporting the one created with JsonCodecMaker?
I would like to prevent IntelliJ from removing "unused import". I can't find a way to suppress the warning by adding some code for this. There is a solution to "Mark import as always used in this project", but then it's stored somewhere by IntelliJ locally. There is a risk of removing the import while auto-organizing.
The problem is like this:

import com.foo.bar.common.Codecs.codecOfSomething

val responseCodec = JsonCodecMaker.make[SthResponse](CodecMakerConfig)
val resultAsJson = writeToArray(someResponse)(responseCodec)

Unfortunately, the first line withcodecOfSomething is grayed by IntelliJ like it is not used. This is implicit and of course, it is in use by responseCodec, from the code emitted by macro. If I can pass it explicitly to the responseCodec, then IntelliJ would see it.
Is there a way for passing this custom codec explicitly?

Andriy Plokhotnyuk
@plokhotnyuk
Hi, Marcin! No, an injection of custom codecs during derivation can be done only using implicit vals. How about an option to define com.foo.bar.common.Codecs codec vals without implicit and then import them using the wild card import com.foo.bar.common.Codecs._ and define implicit vals locally where they are needed?
Marcin Szałomski
@baldram
This is a good idea.
I already keep the generated implicit codecs inside object Codecs to keep it away from Wartremover via build.sbt. The wild card import tricked IntelliJ good enough :smile:
I was thinking about moving the custom codec to somewhere else, but in this case, I will keep all codecs in one place.
The other solution is editing Project.xml of IntelliJ via alwaysUsedImports like here. I prefer a straightforward solution by adjusting the implementation.
Thank you!
Marcin Szałomski
@baldram

@plokhotnyuk Hi Andriy, I have one more question. I couldn't find any example for something similar in Jsoniter tests. I try to wrap a simple Map into kind of custom type to meet the legacy interface of the application after the refactoring.

There is a Root case class which has a bit more stuff (removed for simplification).
The most important are the rates in there. It looks more or less like this.

  object Wrapper {

    type RateGroups = Map[RateGroup, Seq[Rate]]

    implicit class RateGroupsExt(val rates: RateGroups) {
      def foo: Seq[Rate] = rates.flatMap(rateGroup => doSomethingExtra(rateGroup._2)).toSeq
      private def doSomethingExtra(rates: Seq[Rate]): Seq[Rate] = ???
    }
  }

  // doesn't matter here, just some stuff to get it compiled
  final case class RateGroup(foo: Int); val R1 = RateGroup(1); val R2 = RateGroup(2)

  import Wrapper.RateGroups
  final case class Root(rates: RateGroups)

Jsoniter perfectly handles a map.
This is wraped in this way to make possible a call forrates.foo or other similar methods.
This RateGroups works as well but I think it could be done much simpler.

Do you have an idea or some example for similar thing to get it simplified?

Marcin Szałomski
@baldram
If I do the simpler way final case class RateGroups(rates: Map[RateGroup, Seq[Rate]]) then it gets mixed up during the deserialization since rates are like root.rates.rates instead of just root.rates. That's why I wrapped it as above, but this implementation seems to be overcomplicated.
What do you think?
Andriy Plokhotnyuk
@plokhotnyuk
@baldram Hi, Marcin! Have you tried final case class RateGroups(rates: Map[RateGroup, Seq[Rate]]) extends AnyVal yet?
Marcin Szałomski
@baldram
Ok, I see. I tried withoutAnyVal. I need to try the Value Class approach. Let's try. Thanks for the hint.
Marcin Szałomski
@baldram

All right. I implemented a minimal example if someone else is looking at this.
Now everything is clearly visible. The AnyVal is a small but important detail here. It brings autoboxing to own types.
The below example works fine. If I remove the extend AnyVal it won't.

class RateGroupsTest extends AnyFunSuite with Matchers {

  test("should handle wrapped value") {
    // Given
    implicit val codec: JsonValueCodec[Root] = JsonCodecMaker.make

    val json =
      s"""{
         | "groups": {
         |   "R1": [{"code": "FOO"}, {"code": "BAR"}]
         |  }
         |}
         |""".stripMargin.getBytes("UTF-8")

    // When
    val root: Root = readFromArray(json)

    // Then
    root.groups.rates.head._2 should contain allElementsOf List(Rt("FOO"), Rt("BAR"))
  }
}

final case class Root(groups: RtGroups)
final case class Rt(code: String)
final case class RtGroups(rates: Map[String, Seq[Rt]]) extends AnyVal

Thank you Andriy for noticing the missing detail :smile:

Bhargava Ganti
@bhargavaganti
Guys quick help would be appreciated i have a spark dataframe which is flatten can i make it back to unflateen in scala
Andriy Plokhotnyuk
@plokhotnyuk
@bhargavaganti Probably, you will get a faster response here https://gitter.im/spark-scala/Lobby
Franco Albornoz
@dannashirn

Hey, I've got something like the following:

sealed trait Operator

object Operator {
  final case class And(conditions: List[Operator]) extends Operator
  final case class Contains(jsonPath: JsonPath, subString: String) extends Operator
}

Now this works fine if I use the make macro with recursion enabled, but now if I try to change the conditions in the and class to a :: I get the following type error which I'm struggling to get to work:

[error] .../src/main/scala/dsl/Operator.scala:14:24: type mismatch;
[error]  found   : List[dsl.Operator]
[error]  required: scala.collection.immutable.::[dsl.Operator]
[error]     JsonCodecMaker.make(CodecMakerConfig.withAllowRecursiveTypes(true).withDiscriminatorFieldName(None))
[error]                        ^

Any idea how to do something like this?

Andriy Plokhotnyuk
@plokhotnyuk
Hi, @dannashirn ! Thanks for reporting a bug. Please pick the latest version which adds support for ::: https://github.com/plokhotnyuk/jsoniter-scala/releases/tag/v2.6.1
Franco Albornoz
@dannashirn
Amazing! Will try it out.
Franco Albornoz
@dannashirn
So I was just testing out the new changes, and wouldn't it be better if instead of returning a null when an empty list is provided to a :: if it threw an in.decodeError? Given that it's a violation of the types
Andriy Plokhotnyuk
@plokhotnyuk
@dannashirn yep, you are right. This is yet another bug. I'm going to fix it this night.
Franco Albornoz
@dannashirn
Awesome! Looking forward to it
Andriy Plokhotnyuk
@plokhotnyuk
Franco Albornoz
@dannashirn
Great! Now it's working as expected.
Ujali
@ujali

Hi,
I am trying to serialize case class that contains scalaz NonEmptyList type. I keep getting NullPointerException.

case class Request(id: Long, names: NonEmptyList[String])

val StringListCodec = JsonCodecMaker.make[::[String]]

implicit val codecOfLevel: JsonValueCodec[NonEmptyList[String]] =
new JsonValueCodec[NonEmptyList[String]] {

override def decodeValue(in: JsonReader, default: NonEmptyList[String]): NonEmptyList[String] =
  StringListCodec.decodeValue(in, ::(default.head, default.tail.toList)) match {
    case head :: tail => nel(head, IList.fromList(tail))
    case _ => in.decodeError("The list has to be non empty")
  }

override def encodeValue(x: NonEmptyList[String], out: JsonWriter): Unit = StringListCodec.encodeValue(::(x.head, x.tail.toList), out)

override def nullValue: NonEmptyList[String] = null.asInstanceOf[NonEmptyList[String]]

}

implicit val codec: JsonValueCodec[Request] = JsonCodecMaker.make[Request]

val r = writeToString(Request(1L, NonEmptyList("test")))

readFromArray(ByteString(r).toArray)(codec)

Output:
val r: String = {"id":1,"names":["test"]}

java.lang.NullPointerException
at $anon$1.decodeValue(<console>:4)
at $anon$1.decodeValue(<console>:2)
at $anon$1.d0(<console>:1)
at $anon$1.decodeValue(<console>:1)
at $anon$1.decodeValue(<console>:1)
at com.github.plokhotnyuk.jsoniter_scala.core.JsonReader.read(JsonReader.scala:553)
at com.github.plokhotnyuk.jsoniter_scala.core.package$.readFromArray(package.scala:100)
... 43 elided

can someone please tell what is wrong with the codec I have added for NonEmptyList?

Andriy Plokhotnyuk
@plokhotnyuk

Hi, Ujali! Thanks for reaching out me here! Here is the code which decodes non-empty lists properly:

case class Request(id: Long, names: NonEmptyList[String])

implicit val nonEmptyListCodec: JsonValueCodec[NonEmptyList[String]] =
  new JsonValueCodec[NonEmptyList[String]] {
    private val codec = JsonCodecMaker.make[::[String]]

    override def decodeValue(in: JsonReader, default: NonEmptyList[String]): NonEmptyList[String] = {
      val ::(head, tail) = codec.decodeValue(in, if (default eq null) null else ::(default.head, default.tail.toList))
      nel(head, IList.fromList(tail))
    }

    override def encodeValue(x: NonEmptyList[String], out: JsonWriter): Unit = codec.encodeValue(::(x.head, x.tail.toList), out)

    override val nullValue: NonEmptyList[String] = null
  }
implicit val requestCodec: JsonValueCodec[Request] = JsonCodecMaker.make[Request]
val r = writeToString(Request(L, NonEmptyList("test")))
println(r)
println(readFromArray(r.getBytes("UTF-8")))

Expected output:

{"id":1,"names":["test"]}
Request(1,NonEmpty[test])
Andriy Plokhotnyuk
@plokhotnyuk
Also, both println(readFromArray("{\"id\":1,\"names\":[]}".getBytes("UTF-8"))(requestCodec)) or
println(readFromArray("{\"id\":1,\"names\":null}".getBytes("UTF-8"))(requestCodec)) will throw JsonReaderException with the "expected non-empty JSON array" message
Andriy Plokhotnyuk
@plokhotnyuk
If you will define a default value for the field with NEL (like here: case class Request(id: Long, names: NonEmptyList[String] = nel("1", IList.empty))) then those 2 lines will parse successfully and print the following output: Request(1,NonEmpty[1])
Ujali
@ujali
Thank you @plokhotnyuk for the quick response :)
Nitika Agarwal
@nitikagarw
Hi @plokhotnyuk , Is there any example that I can refer to define a custom codec instead of setting allowRecursiveTypes to true?
Andriy Plokhotnyuk
@plokhotnyuk
@nitikagarw Hi Nitika, basically by setting the allowRecursiveTypes flag to true you just allow to generate codecs for nested data structures that are defined by recursive types like case class NestedStructs(n: Option[NestedStructs] = None) or more complex with indirect recursion, and it is safe if you work with trusted counterparties. This flag is set to false by default to allow systems to accept untrusted input safely without opening Denial of Service or Denial of Wallet vulnerabilities when during parsing of malicious input of deeply nested JSON objects or arrays a lot of CPU can be eaten by stack overflow errors. You can create a custom codec that keeps the level of nesting reasonable. Here is an example of a custom codec which tracks the depth of nesting. If the depth of allowed nesting is shallow a better option would be designing of non-recursive data structures like here.
Nitika Agarwal
@nitikagarw
Thanks @plokhotnyuk !