Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    armundle
    @armundle

    @rossabaker Thanks for the tips.
    I agree that Catchable might need some work in the case of failure, but I don’t think fail(…) should come into play in the case of a normal end of stream, right?
    Same with the wrap - for a successful Task it should be fine.

    The goal with Process[StreamAction, ???] is to conform to an already existing interface.

    What I am confused and unsure about is the need to catch something in the case of a successful stream Process. Why wouldn’t I run into the same issues as above if I had a stream of Process(1, 2, 3).toSource: : Process[Task, Int] ?

    Ross A. Baker
    @rossabaker
    @armundle: what happens if you Process(1, 2, 3).toSource.translate(taskToAction)?
    armundle
    @armundle
    It completes without any errors.
    Ross A. Baker
    @rossabaker
    Hmm. Maybe the await1 or fby end up handling those control exceptions differently than a pure source. The fact that scalaz-stream uses exceptions for control flow is a fact I’m usually blissfully unaware of. :)
    armundle
    @armundle
    My coworker helped me with this version of Catchable that does the trick:
      implicit val catchable: Catchable[StreamAction] = new Catchable[StreamAction] {
        override def fail[A](err: Throwable): StreamAction[A] =
          EitherT.fromDisjunction(err.getMessage.left)
    
        override def attempt[A](f: StreamAction[A]): StreamAction[\/[Throwable, A]] =
          EitherT {
            f.run.attempt.map( _ match {
              case -\/(t) => t.left[A].right[StreamError]
              case \/-(-\/(e)) => e.left[\/[Throwable,A]]
              case \/-(\/-(a)) => a.right[Throwable].right[StreamError]
            })
          }
      }
    The question still remains - why would it get an Unhandled Exception when the stream ends normally...
    lantz moore
    @mizerlou
    sometimes, endpoints will return data that looks like
    { results:[...]}
    it would be nice to have a built in streaming mode that would stream the array bound to results.
    Guillaume Massé
    @MasseGuillaume
    hello is it possible to add the gihub topic json ?
    Devon Stewart
    @blast-hardcheese
    hah, just came to ask about @mizerlou's most recent comment
    I was thinking of field-parameterized, nested parsers
    so a consumer would DeepAsyncParser("results", UnwrapArray), and data/position information would be passed between the parsers
    reaching the EOF token would immediately defer to the parent parser and all secondary output would be either ignored or returned separately in the terminating message
    Either[ParseException, Either[J, Seq[J]]] or Either[ParseException, (Option[J], Seq[J])]
    Devon Stewart
    @blast-hardcheese
    this would have the advantage of choosing to not fail over to the sync parser, which just eats memory instead of performing what its name suggests it should do
    Matt Hicks
    @darkfrog26
    any current efforts being done on supporting Scala.js? :)
    Li Haoyi
    @lihaoyi
    Is anyone here interested in streaming AST-free direct-to/from-case-class JSON parsing?
    Jawn's parser only needs a few tweaks here and there to make it work, and from there you can plug in any backend you want: create an AST, directly create case classes, re-output JSON with different formatting (streaming, with no intermediate AST)
    non/jawn#102 is the first step in getting that working, with a streaming AST-free version of uPickle working https://github.com/lihaoyi/upickle/blob/10904a40b38ec20d5ebb3139856a48c78f57c831/upickle/shared/src/main/scala/upickle/Readers.scala#L26
    Travis Brown
    @travisbrown
    @lihaoyi ooc have you looked at flatjson? I have an old experiment doing something similar with flatjson + circe-algebra.
    Li Haoyi
    @lihaoyi
    @travisbrown I haven't. flatjson scala doesn't pull up anything on google...
    I'm basically taking the jawn.Facade and augmenting it with a tiny amount of additional data to make it work, no major changes to jawn required
    Travis Brown
    @travisbrown
    @lihaoyi sorry, here's the link: https://github.com/zalando-incubator/flatjson
    Li Haoyi
    @lihaoyi
    one of my interesting discoveries is you can use jawn.Facade, take it's dual and use it as a output/serialization interface, and not just for parsing https://github.com/lihaoyi/upickle/blob/10904a40b38ec20d5ebb3139856a48c78f57c831/upickle/shared/src/main/scala/upickle/Writers.scala#L19
    in a way jawn.Facade is just a generic JSON visitor, and not something specific to parsing
    and you can compose visitors in the same way you compose ASM bytecode ClassVisitors
    @travisbrown yeah it looks a lot like that!
    Li Haoyi
    @lihaoyi
    Seems like the only restriction for AST-free one-pass parsing is that you have to encode sealed hierarchies as ["pkg.Class", {"x": 1}] rather than {"$type": "pkg.Class", "x": 1}
    because JSON key-ordering is undefined and you need to know the type up-front so you know how to one-pass parse the rest of the json dict into the object you want
    but if a user wants {"$type": "pkg.Class, ...} the library can fall back to parsing an AST just for that section of the JSON, which would be transparent except for the performance cost
    Li Haoyi
    @lihaoyi

    @travisbrown provisional benchmarks of AST-free serialization puts my WIP branch of uPickle about ~40% faster than circe on my current set of ad-hoc benchmarks, if the encoders/decoders are cached

    playJson Read 2761067
    playJson Write 3412630
    circe Read 6005895
    circe Write 5205007
    upickleDefault Read 4543628
    upickleDefault Write 3814459
    upickleLegacy Read 8393416
    upickleLegacy Write 7431523

    Circe is still significantly faster in the case where encoders/decoders are not cached, but I assume I just need to spend a bit of time micro-optimizing the encoder/decoder instantiation code and it's not a fundamental limitation (and more time optimizing should help the cached-encoder benchmark as well)

    playJson Read 1975992
    playJson Write 2811139
    circe Read 4701980
    circe Write 4252224
    upickleDefault Read 2724334
    upickleDefault Write 2443416
    upickleLegacy Read 3142672
    upickleLegacy Write 2878934

    Note that in that branch, String -> Case Class and Case Class -> String are both AST-free; my upickle Readers simply implement jawn.Facade, and the upickle Writers effectively extend jawn.Facade => Unit, and so actual definition of reader/writer instances for various types looks pretty similar to what you would see if you pattern matched over the AST (Reader example, Writer example) but it can be driven directly by the parser without any intermediate AST being constructed

    Li Haoyi
    @lihaoyi
    the patched version of jawn.Facade also gives you workflows like Case Class => Case Class, String => String (e.g. re-formatting your JSON), AST => Case Class, Case Class => AST, String => AST, AST => String all basically for free, also without any intermediate JSON AST
    guntherclass
    @guntherclass
    Hello! I have a very big json which format is { "field1": "...", "field2": "...", "field3": "...", "entries": [......] }
    This json is presented as fs2.Stream[F, Byte], and i need extract values of first 3 fields and need to get some stream of values from entries array, but i don't know how to use https://github.com/circe/circe-fs2 module here. Please tell me how can i do this or how can i get stream of json tokens for further parsing manually?
    Alex Cruise
    @acruise
    Hey folks, does Jawn have the ability to derive codecs, like argonaut does?
    Alex Cruise
    @acruise
    hm, I see that it has other Facade implementations, maybe I can get it to parse to Argonaut's AST then use the Argonaut codec
    Alex Cruise
    @acruise
    That works, but the performance is only about 10% better than with Argonaut's built-in parser
    Travis Brown
    @travisbrown
    there are several fairly trivial / noncontroversial PRs for jawn that have been open for weeks or months now—is there anything we can do to help these get merged faster?
    (at the moment I really just want a release for 2.13.0-M5)
    Travis Brown
    @travisbrown
    ^ @non
    Travis Brown
    @travisbrown
    Any review appreciated: typelevel/jawn#144 typelevel/jawn#145
    Tharindu Galappaththi
    @TharinduDG

    Hi,
    I'm stuck with an error when parsing a json response from elastic search. This is my code.

      def dropLeadingChars: Pipe[F, Byte, Byte] = {
        def go(origStr: Stream[F, Byte], beginning: Boolean) : Pull[F, Byte, Unit] = {
          logger.info(s"origStr inside dropLeadingChars -> $origStr")
          origStr
          .pull
          .unconsN(if (beginning) 1 else CHUNK_SIZE, true)
          .flatMap{
            case Some((Chunk.Bytes(chunk, _, _), str)) if beginning && !chunk.headOption.contains('[') =>
              Pull.output(Chunk.empty) >> go(str, true)
            case Some((seg, str)) =>
              logger.info(s"inside some seg is ${seg.toString} and str is $str")
              Pull.output(seg) >> go(str, false)
            case None => Pull.done
          }
        }
        in => go(in, true).stream
      }
    
    val res2: Stream[F, String] = httpClient.getQueryStream(headers, subject.indexName, query, { resp =>
      resp
        .body
        .through(dropLeadingChars)
        .dropLastIf(_ == '}')
        .dropLastIf(_ == '}')
        .chunks
        .unwrapJsonArray
        .map{ json =>
          json.as[Hit].fold(
            err => s"Invalid json: $err for payload $json\n",
            hit => {
              logger.info(s"hit -> $hit")
              val instance = hit._source.instance
              logger.info(s"instnace -> $instance")
              JsonNormalizer.normalizeOld(instance) ++ "\n"
            }
          )
        }
    })

    The error:

    ,"stack_trace":"org.typelevel.jawn.ParseException: expected true got 'text/p...' (line 1, column 2)\n\tat org.typelevel.jawn.Parser.die(Parser.scala:132)\
    Tharindu Galappaththi
    @TharinduDG
    Here is the example json that I'm trying to pass: https://jsoneditoronline.org/?id=9dca6e705acc47ca9945bafd145755b9
    Andriy Plokhotnyuk
    @plokhotnyuk
    It looks that your input is wrong (like parsing of an HTTP header instead of the body)...
    Tharindu Galappaththi
    @TharinduDG
    @plokhotnyuk I'm using resp.body to get the body from the http4s.Response
    Andriy Plokhotnyuk
    @plokhotnyuk
    Could you, please, print and check the input before processing?
    Tharindu Galappaththi
    @TharinduDG
    @plokhotnyuk I checked actually. I also created a standalone app with this piece of code and passed the json from a file. Then it parsed fine. But in production env. it gives that error when it receives the json response.
    can you think of a reason for this?
    and the response headers doesn't have any text/p part. Its application/json
    Andriy Plokhotnyuk
    @plokhotnyuk
    Just print to the log a whole body on prod... it can be something unexpected like some buffer reusage error in blaze/http4s code which serve an HTTP connection... or some runtime issue, when you processing body with 4xx/5xx status error...
    Tharindu Galappaththi
    @TharinduDG
    @plokhotnyuk Ok, I'll try that. Thanks for your support.
    Travis Brown
    @travisbrown
    Anyone want to speak up for CharBuilder? typelevel/jawn#194