Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 13:50

    plokhotnyuk on v2.13.23

    (compare)

  • 13:50

    plokhotnyuk on master

    Setting version to 2.13.23 Setting version to 2.13.24-SNAP… (compare)

  • 12:46

    plokhotnyuk on master

    More efficient serialization of… (compare)

  • 10:56

    plokhotnyuk on master

    More efficient serialization of… (compare)

  • 10:50

    plokhotnyuk on master

    More efficient serialization of… (compare)

  • May 17 11:12

    plokhotnyuk on master

    More efficient parsing of doubl… (compare)

  • May 16 14:31

    plokhotnyuk on master

    Code clean up (compare)

  • May 16 14:00

    plokhotnyuk on master

    More efficient parsing of doubl… (compare)

  • May 16 11:12

    plokhotnyuk on master

    Clean up of code (compare)

  • May 16 10:16

    plokhotnyuk on master

    Clean up of code (compare)

  • May 16 10:03

    plokhotnyuk on master

    Clean up of code (compare)

  • May 16 09:50

    plokhotnyuk on master

    Clean up of code (compare)

  • May 16 09:17

    plokhotnyuk on master

    Clean up of code (compare)

  • May 15 08:49

    plokhotnyuk on master

    Update jackson-module-scala to … (compare)

  • May 15 08:49
    plokhotnyuk closed #899
  • May 15 07:32
    scala-steward opened #899
  • May 15 03:01

    plokhotnyuk on master

    Update jackson-module-afterburn… (compare)

  • May 15 03:01
    plokhotnyuk closed #898
  • May 15 03:01

    plokhotnyuk on master

    Update jackson-datatype-jdk8 to… (compare)

  • May 15 03:01
    plokhotnyuk closed #897
Marcin Szałomski
@baldram
Now I will disable errors one by one. Eg. @SuppressWarnings(Array("org.wartremover.warts.Equals")). Then next will come, and next, and next...
So just after I get:
[warn] /mnt/foobar/model/FooBar.scala:12:63: [wartremover:Null] null is disabled
Marcin Szałomski
@baldram

So now I do:
@SuppressWarnings(Array("org.wartremover.warts.Equals", "org.wartremover.warts.Null"))
... and I get:

[warn] /mnt/foobar/model/FooBar.scala:12:63: [wartremover:OptionPartial] Option#get is disabled - use Option#fold instead

It looks like never ending story, so I end up in adding @SuppressWarnings(Array("org.wartremover.warts.All")).
That's why it would be great to already generate this SuppressWarnings, as this is kind of not nice thing we need to add in a code and excuse ;) for it in a comment.

If not generating Supress Warnings, I was thinking also how about implementing some custom annotation alias for Jsoniter like @SupressJsoniterWarts which would have this suppress for All errors under the hood.
Maybe also a solution. So adding this short version of Supress Warnings (@SupressJsoniterWarts) would be at least a bit nicer.
The best way would be to have it already generated, or have some configuration in build.sbt or whatever while importing a library that this will generate those @SuppressWarnings(Array("org.wartremover.warts.All")). Not sure this is possible.

Marcin Szałomski
@baldram
Is my description of a problem clear enough?
Andriy Plokhotnyuk
@plokhotnyuk
Thank you, Marcin! It was clear and helpful! Here is a commit with the proposed solution: plokhotnyuk/jsoniter-scala@a550237
Could you please clone the repo with git clone --depth 1000 --branch master git@github.com:plokhotnyuk/jsoniter-scala.git, build and publish locally a snapshot version with sbt clean publishLocal and then test the 2.8.2-SNAPSHOT version if it works for you?
Marcin Szałomski
@baldram
All right. I will do that in the evening, is it ok?
Andriy Plokhotnyuk
@plokhotnyuk
Ok, I've force pushed a better solution that passes internal tests with polluted namespaces: plokhotnyuk/jsoniter-scala@fa8b53f
Marcin Szałomski
@baldram
I've compiled a project with 2.8.2-SNAPSHOT with all SuppressWarnings removed and no linter errors :tada:
I will try in the other project to double check.
Marcin Szałomski
@baldram
@plokhotnyuk I've tried this in another project. What is important, after I bumped a version, all works without changes. Next, when removed linter ignores, I see no warnings.
All tests are passing as well, btw.
I also checked what happens if I make an issue next to codec definition and it is handled correctly by Wartremover.
Thank you Andriy!
PS: I haven't played yet with Enumeratum, but I think I will try to modify some code in a project for testing purposes tomorrow.
Andriy Plokhotnyuk
@plokhotnyuk
@baldram Thank you a lot for reporting and testing the issue with Wartremover! I've published the fix in v2.8.2 release.
Marcin Szałomski
@baldram
Thank you Andriy!
Marcin Szałomski
@baldram
@plokhotnyuk I have one more question Andriy, regarding commits about Scala 3, like "Update Scala 3.x to 3.0.0".
Is there a plan to make jsoniter-scala Scala 3 compatible? I thought it's not doable because of macros, and there is no hope for that.
However, seeing commits treating Scala 3... That would be so fabulous.
Andriy Plokhotnyuk
@plokhotnyuk
I've tried to publish jsoniter-scala-core with 3.0.0-RC3 and 3.0.0 versions but failed both attempts due some classpath problems with the sonatypeBundleRelease command: xerial/sbt-sonatype#230
Marcin Szałomski
@baldram
Sounds exciting! Does it mean that even though all those Scala 3 macro related difficulties, those moves are about trying to use Jsoniter with Scala 3, and it's close anyway?
Chris Kipp
@ckipp:matrix.org
[m]
I'm not fully sure how to actually phrase this questions, but I'm working on migrating the Build Server Protocol off of circe to jsoniter and we have some case classes that represent requests in the spec that we made codecs for, but some of these requests allow for the value that is returned to be null. However in that scenario (at least I think that's what happening) jsoniter will freak out because I believe it wants valid json, but is instead receiving null. And then you're met with this
[info]   org.eclipse.lsp4j.jsonrpc.ResponseErrorException: com.github.plokhotnyuk.jsoniter_scala.core.JsonReaderException: expected '{', offset: 0x00000000, buf:
[info] +----------+-------------------------------------------------+------------------+
[info] |          |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f | 0123456789abcdef |
[info] +----------+-------------------------------------------------+------------------+
[info] | 00000000 | 6e 75 6c 6c                                     | null             |
[info] +----------+-------------------------------------------------+------------------+

the way we are doing these requests are like

case class Reload()

object Reload {
  implicit val codec: JsonValueCodec[Reload] =
    JsonCodecMaker.make
}

But is there another way to do this that would allow for null?

Andriy Plokhotnyuk
@plokhotnyuk
@ckipp:matrix.org Hi Chris! Great news! A good option would be using of Option[Reload] instead, so null JSON values will be parsed to the None object reference. But if it is not an option for your than some delegating custom codec can be created to wrap derived one by the standard make:
  def wrapByNullAllowingCodec[A](codec: JsonValueCodec[A]): JsonValueCodec[A] = new JsonValueCodec[A] {
    override def decodeValue(in: JsonReader, default: A): A =
      if (in.isNextToken('n')) {
        in.readNullOrError[String]("x", "expected value or null")
        nullValue
      } else {
        in.rollbackToken()
        codec.decodeValue(in, default)
      }

    override def encodeValue(x: A, out: JsonWriter): Unit =
      if (x == null) out.writeNull()
      else codec.encodeValue(x, out)

    override val nullValue: A = null.asInstanceOf[A]
  }
Chris Kipp
@ckipp:matrix.org
[m]
thanks for the response! Ah, good to see your wrapByNullAllowingCodec I was actually just going down that path so it's good to see that I was headed in the right direction
Chris Kipp
@ckipp:matrix.org
[m]
cool, this worked perfectly. Thank you!
Andriy Plokhotnyuk
@plokhotnyuk
You are welcome, Chris! I'll switch to BSP definitely, so waiting for the completion of your migration with impatience!
Steven Lai
@steven-lai

I was wondering if I had:

case class(id: String, fields: Map[String, String], num: Int)

// How can I read in the following JSON:
{
  "id" : "FOO", 
  "num": 456,
  "fields" : {
    "time" : 123,
    "bar" : "baz"
  }
}

I guess a custom codec? The Stringified annotation or config doesn't seem to apply here

Steven Lai
@steven-lai
I ended up doing:
final case class StringMapValue(underlying: String) extends AnyVal
final case class(id: String, fields: Map[String, StringMapValue], num: Int)

implicit val mapValueCodec: JsonValueCodec[StringMapValue] = new JsonValueCodec[StringMapValue]() {
  override def decodeValue(in: JsonReader, default: StringMapValue): StringMapValue = {
    val b = in.nextToken()
    if (b == '"') {
      in.rollbackToken()
      StringMapValue(in.readString(null))
    } else if ((b >= '0' || b <= '9') || b == '-') {
      in.rollbackToken()
      val d = in.readDouble()
      val i = d.toInt
      if (i.toDouble == d) {
        StringMapValue(i.toString)
      } else {
        StringMapValue(d.toString)
      }
    } else if (b == 'n') {
      in.rollbackToken()
      in.readNullOrError(null, "expected `null` value")
      nullValue
    } else if (b == 't' || b == 'f') {
      in.rollbackToken()
      StringMapValue(in.readBoolean().toString)
    } else {
      in.decodeError("expected JSON value")
    }
  }

  override def encodeValue(x: StringMapValue, out: JsonWriter): Unit = {
    out.writeVal(x.underlying)
  }

  override val nullValue: StringMapValue = StringMapValue(null)
}
Andriy Plokhotnyuk
@plokhotnyuk
@steven-lai Hi, Steven! Yes, a custom codec could be a solution here. Could you please explain how values of the fields field will be used? Do you need encoding the parsed JSON back to the same representation?
Steven Lai
@steven-lai
I don't believe I do, but if I did how would I achieve that? The fields map is just a generic map for some non-fixed fields
Andriy Plokhotnyuk
@plokhotnyuk
@steven-lai Currently your codec encode all fields values (except null) as strings and then will not able to parse them. For parsed nulls it will just throw NPEs during encoding. Also it have a couple of bug in parsing of numbers and null values. To avoid such problems you can define an ADT (or sum type) for fields values and use the following codec:
import com.github.plokhotnyuk.jsoniter_scala.macros._
import com.github.plokhotnyuk.jsoniter_scala.core._

object Example01 {
  sealed trait MapValue
  case class StringMapValue(value: String) extends MapValue
  case class BooleanMapValue(value: Boolean) extends MapValue
  case class DoubleMapValue(value: Double) extends MapValue
  case class IntMapValue(value: Int) extends MapValue
  case object NullMapValue extends MapValue

  case class Data(id: String, fields: Map[String, MapValue], num: Int)

  implicit val mapValueCodec: JsonValueCodec[MapValue] = new JsonValueCodec[MapValue] {
    override def decodeValue(in: JsonReader, default: MapValue): MapValue = {
      val b = in.nextToken()
      if (b == '"') {
        in.rollbackToken()
        StringMapValue(in.readString(null))
      } else if (b >= '0' && b <= '9' || b == '-') {
        in.rollbackToken()
        val d = in.readDouble()
        val i = d.toInt
        if (i.toDouble == d) IntMapValue(i)
        else DoubleMapValue(d)
      } else if (b == 't' || b == 'f') {
        in.rollbackToken()
        BooleanMapValue(in.readBoolean())
      } else if (b == 'n') {
        in.readNullOrError(default, "expected `null` value")
        NullMapValue
      } else in.decodeError("expected JSON value")
    }

    override def encodeValue(x: MapValue, out: JsonWriter): Unit = x match {
      case NullMapValue => out.writeNull()
      case StringMapValue(x) => out.writeVal(x)
      case BooleanMapValue(x) => out.writeVal(x)
      case IntMapValue(x) => out.writeVal(x)
      case DoubleMapValue(x) => out.writeVal(x)
    }

    override def nullValue: MapValue = NullMapValue
  }

  implicit val codec: JsonValueCodec[Data] = JsonCodecMaker.make

  def main(args: Array[String]): Unit = {
    val data = readFromArray[Data]("""{
                               |  "id" : "FOO",
                               |  "num": 456,
                               |  "fields" : {
                               |    "time" : 123,
                               |    "bar" : "baz",
                               |    "bool" : true,
                               |    "double" : 123.456890123,
                               |    "null" : null
                               |  }
                               |}""".stripMargin.getBytes("UTF-8"))
    val json = writeToArray(Data(id = "BAR", num = 123,
      fields = Map[String, MapValue]("time" -> IntMapValue(123), "bar" -> StringMapValue("baz"))))

    println(data)
    println(new String(json, "UTF-8"))
  }
}
Andriy Plokhotnyuk
@plokhotnyuk
If your fields values can be arbitrary JSON values and they could be mutable then I would recommend to use Dijon's SomeJson for them that uses jsoniter-scala's codec to parse and encode it.
Another option if fields values not need to be handled and just need to be passed back to encoded JSON then for the fields field use RawVal type like here.
Steven Lai
@steven-lai
Thanks!
Dermot Haughey
@hderms
@plokhotnyuk do you have any tips for data structures that jsoniter will work better with? I'm thinking potentially Array will be marginally faster to deserialize than List, for example
Andriy Plokhotnyuk
@plokhotnyuk
@hderms Hi Dermot! Currently jsoniter-scala-macros API supports Array and all other Scala collections from the standard library, but any collection can be parsed with a custom codec that uses jsoniter-scala-core API. Throughput of collection parsing depends a lot on how fast an instance of the collection can be created. On the page with benchmark results you check results for ArrayBufferOfBooleansReading, ArrayOfBooleansReading, ListOfBooleansReading and VectorOfBooleansReading to have a rough estimation. For more concrete results you can modify and run corresponding benchmarks on your environment.
Zhenhao Li
@Zhen-hao
hi, has anyone used this library with the Alpakka Elasticsearch connector? the doc, https://doc.akka.io/docs/alpakka/current/elasticsearch.html#with-typed-source, says "The data is converted to and from JSON by Spray JSON." it seems to me a tight coupling with Spray JSON
Objektwerks
@objektwerks
@plokhotnyuk Hey, Andriy, I'm thinking of switching from Akka-Http-Upickle to Akka-Http-Jsoniter for 2 reasons: 1) uPickle requites a @nowarn annotation to work with Scala 2.13.6 due to the new and improved exhaustive pattern matching; and 2) Intellij can't understand the Akka-Http marshallers and complete directive for uPickle support. I'm hoping, beyond a major speed bump, that Akka-Http-Jsoniter support will work error-free in Intellij. I know, an impossible task, perhaps.:) Thoughts? Thanks in advance!
Andriy Plokhotnyuk
@plokhotnyuk
Hello, @objektwerks ! Thanks for trying akka-http-jsoniter-scala. I think it is quite possible to switch on it from akka-http-upickle easily. Please let me know If you will require porting of some compile-time/runtime feature from uPickle, writing of efficient custom codecs, fixing/suppressing of some compile-time warnings, etc.
Objektwerks
@objektwerks
Thanks, @plokhotnyuk Andriy. It all seems to work with Akka-Http. Yet I have 3 unique case classes with the identical structure, and Jsoniter is parsing them all as the 1st case class, which produces akka-http router -> database integrity errors. I was looking for a way to have Jsoniter provided a case class discriminator key in the json. Then I was hoping to examine the Jsonitor-generated json and not finding a convenient method to do so ( beyond a series of numbers ). So I haven't quite reached the promise land. :)
Objektwerks
@objektwerks
@plokhotnyuk Okay, I figured out the byte array to json technique: json.map(_.toChar).mkString and no case class discriminator is provided. And I'm not sure how to enable that feature. I noticed a macro make method that disables, it though.:)
Andriy Plokhotnyuk
@plokhotnyuk
Could you please provide some isolated code that reproduce your problem? You can share it as Scastie code snippet or as a pull request for an example project.
For example it can be parsing of some string constant with an assert between expected and actual parsed case classes.
Objektwerks
@objektwerks
@plokhotnyuk The code is rather complicated. Image 3 case classes A, B and C, each with this structure: id: Int, parentId: Int, installed: Int, model: String My Akka-Http router tests naturally tests A, B and C via their respective routers, and all effected database tables are good. This works with uPickle, for instance.
Yet it seems as though Jsoniter is interpreting all 3 case classes as case class A. And that naturally breaks the router-to-database code.
So is there some case class discriminator key feature in Jsoniter that might resolve this confusion?
Andriy Plokhotnyuk
@plokhotnyuk
Do you have separated codecs derived by the make call for all A, B and C types.
Objektwerks
@objektwerks
Yes, I do. See: https://github.com/objektwerks/pwa.pool/blob/master/shared/src/main/scala/pool/Codecs.scala
It all looks good to go. In the code above, the effected classes are Pump, Timer and Heater.
See: https://github.com/objektwerks/pwa.pool/blob/master/shared/src/main/scala/pool/Entity.scala
You'll note all 3 have the identical field names, just different case class names. Jsonitor thinks all 3 are a Pump.
And in my router test, I get 3 record entries in the pump table, nothing in the timer or heater tables.
Objektwerks
@objektwerks
@plokhotnyuk Andriy, I've gotta step out for a few hours. This is by no means a high priority issue for me. Any suggested resolutions would be awesome --- at your convenience. Cheers!