These are chat archives for mdedetrich/scala-json-ast

10th
Mar 2016
Ichoran
@Ichoran
Mar 10 2016 00:42
@mdedetrich - Got a few minutes to fix the key bug that was stopping parsing. Now it claims to work but I'm going to have to dig a bit more; my preliminary results are that I parse mostly numeric data at 400 MB/sec, and I don't think that's actually possible. My previous results (albeit on a slower computer with a different data file and using Grok instead of this custom parser) were around 90 MB/sec vs. Jackson's 50 MB/sec.
Matthew de Detrich
@mdedetrich
Mar 10 2016 01:32
@Ichoran well 400mb a sec is pretty freaking fast for a JSON parser
thats for sure
Ichoran
@Ichoran
Mar 10 2016 15:09
Hmm, well, it does seem to be about 3x faster than Play JSON for string parsing of string-heavy JSON, and 5x faster for number-heavy JSON, and it produces way less garbage also.
I can't find anything wrong, and it was supposed to be fast, but I'm really suspicious that something is more wrong than I think. The code is just not that clever (save for the number parsing bit).
Ichoran
@Ichoran
Mar 10 2016 15:25
2.5x faster than Spray on number-heavy, 4.5x faster on string-object heavy (with no lookup into the object)
Ichoran
@Ichoran
Mar 10 2016 15:35
2x faster than Jawn on number-heavy, 1.7x faster on string-object heavy.
@mdedetrich - All right, well, I guess this is true? Seems to be really fast, which is exactly what I wanted!
Still very rough.

Here's an example of loading and parsing and extracting stuff from a file:

scala> kse.jsonal.Json.parse(new String("/data/kerrr/test/webapp-example.json".file.gulp.yes)).right.get("web-app")("servlet")(0)("init-param")("templatePath").toString
res1: String = "templates"

where webapp-example.json is the example whose first key is "web-app" on http://json.org/example.html

Ichoran
@Ichoran
Mar 10 2016 15:49
Maybe I should spread that out a little:
kse.jsonal.Json.parse( /* parse! */
  new String("/data/kerrr/test/webapp-example.json".file.gulp.yes)
  /* this just loads the file from my drive */
).right.get. /* parses are packed in Either */
apply("web-app")("servlet")(0)("init-param")("templatePath").
toString
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:43
@Ichoran nice I will look into it, from here I will mostly be looking at how I can integrate certain parts into scala-json-ast
@eed3si9n I just looked at the status update for SBT 1.x on reddit, I assume that it would be ideal scala-json-ast is finalized before you release SBT 1.x?
Is it safe to say that Scala 1.x still has a fair bit of work (I am expecting to finish off scala-json-ast in the next couple of weeks
eugene yokota
@eed3si9n
Mar 10 2016 23:45
@mdedetrich yea.
Ichoran
@Ichoran
Mar 10 2016 23:45
@mdedetrich - I'm working on docs and finishing up...um...well, a lot of stuff.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:45
Do you have a rought minimum for SBT 1.x, is it still months off or weeks off?
Ichoran
@Ichoran
Mar 10 2016 23:46
But, anyway, I think I am probably going to be using Jsonal for JSON parsing from now on.
I would be delighted to add a conversion-to-scala-json-ast mode, though.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:46
@Ichoran sure, if you can let me know whens the best time to look at your AST (is that now?) that would be great
eugene yokota
@eed3si9n
Mar 10 2016 23:46
@mdedetrich I specifically wrote in there that i don't have any idea on when it's going to come out :)
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:47
Okay cool, so I will assume that its going to be minimum a month then which gives me plenty of time
eugene yokota
@eed3si9n
Mar 10 2016 23:47
definitely in the order of months, not weeks
Ichoran
@Ichoran
Mar 10 2016 23:47
The AST isn't going to change, but I'm adding documentation right now, so if you like English descriptions more than type signatures, maybe look in a couple hours?
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:47
:thumbsup:
@Ichoran I will look at it on Tuesday next week, so its definitely not in that much of a rush
Ichoran
@Ichoran
Mar 10 2016 23:48
Okay. I might even have the byte parsing stuff in by then.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:49
@Ichoran Yeah its going to take me a while to grok this. Actually apart from JsNumber, the AST is not that different
its mainly your parsing techniques which have all of the magic
Like I am open to adding an optional double field to JNumber for unsafe.JValue if people want to represent their JValue’s as double's
Ichoran
@Ichoran
Mar 10 2016 23:51
Not entirely. Json.Obj has a triple backing scheme that lets you parse it super-fast, construct it super-fast from a map, and look up keys super-fast (after a one-time map creation).
I didn't have that before.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:51
Ah yeah, getting to that now
Ichoran
@Ichoran
Mar 10 2016 23:51
It probably isn't the most awesome for memory usage for tiny objects.
But the "error type is part of the AST hierarchy" thing is still there.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:53
Yeah I can’t do much about that, unless I want to unseal the JValue trait
Ichoran
@Ichoran
Mar 10 2016 23:53
I have two root values: Json which is a JSON value, and Jast which is JSON value or a JastError.
It's all sealed.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:54
Well I will look into it, I am not under a huge rush as you can see
Ichoran
@Ichoran
Mar 10 2016 23:54
Which makes for a crazily huge file :(
If extension methods were truly zero-cost, I'd move more/all of the functionality out to other files.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:55
Yeah, personally I wish something like sealed package would exist
Ichoran
@Ichoran
Mar 10 2016 23:55
Agreed.
Matthew de Detrich
@mdedetrich
Mar 10 2016 23:55
Anyways, expect me to bug you every now and then :wink2:
Ichoran
@Ichoran
Mar 10 2016 23:55
Sure.