Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 20:53
    scala-steward opened #159
  • 18:39

    satabin on fix-site-building

    (compare)

  • 18:39

    satabin on main

    Fix makeSite and sbt deprecatio… Merge pull request #157 from sa… (compare)

  • 18:39
    satabin closed #157
  • 16:00
    satabin commented #157
  • 15:59

    satabin on main

    Update sbt-mdoc to 2.2.20 Merge pull request #158 from sc… (compare)

  • 15:59
    satabin closed #158
  • 12:22
    scala-steward opened #158
  • Apr 17 09:10
    ybasket review_requested #157
  • Apr 17 09:10
    ybasket opened #157
  • Apr 17 09:10

    ybasket on fix-site-building

    Fix makeSite and sbt deprecatio… (compare)

  • Apr 17 09:05

    ybasket on fix-site-building

    Fix makeSite and sbt deprecatio… (compare)

  • Apr 17 08:37
    ybasket edited #150
  • Apr 17 08:36

    ybasket on replace-contextual-with-literally

    (compare)

  • Apr 17 08:36

    ybasket on main

    Replace contextual with literal… Fix macro hygiene in json selec… Fix wrong package in json selec… and 1 more (compare)

  • Apr 17 08:36
    ybasket closed #156
  • Apr 17 08:29
    ybasket synchronize #156
  • Apr 17 08:29

    ybasket on replace-contextual-with-literally

    Fix wrong package in json selec… (compare)

  • Apr 16 21:11
    ybasket synchronize #156
  • Apr 16 21:11

    ybasket on replace-contextual-with-literally

    Fix macro hygiene in json selec… (compare)

Sait Sami Kocataş
@Deliganli
sure, thanks a lot!
Lucas Satabin
@gnieh:matrix.org
[m]
ok I'm back
@Deliganli: have you found a solution for the regex step?
Sait Sami Kocataş
@Deliganli
I think it might not related to the regex per se, I tried to remove this tagname: part by hand and replay the dataset, however I end up with same exception, when the chunk ends for pure json I again get the fs2.data.json.JsonException: unexpected end of input
Lucas Satabin
@gnieh:matrix.org
[m]
I mean in the case before, the regex drops whatever comes before tagname: and this is what is missing in your data before starting to parse
namely here: {"some-key1":"some-value"}]}
Sait Sami Kocataş
@Deliganli
yes, there may be a discrepancy between the example I provided and the way I use in real scenario. I am connecting to a server and consuming text/event-stream. When first chunk arrives in the very first item, I get the exception. I am looking into it more
Lucas Satabin
@gnieh:matrix.org
[m]
ok
let me know if I can help
so far, from what I saw the input is indeed malformed when it hits the parser, but maybe there is another problem which is a bug in fs2-data :D
Sait Sami Kocataş
@Deliganli
thanks a lot for everything, it was a nice session I got 2 tricks which are debug and text.lines
Lucas Satabin
@gnieh:matrix.org
[m]
:)
then I'm happy I could help somehow :D
Sait Sami Kocataş
@Deliganli

hi everyone, after yesterday even though I still couldn't solve the problem I found out this is a standard, described below;
https://doc.akka.io/docs/akka-http/10.0/sse-support.html

I am not sure wheater it is in scope of this library here

Lucas Satabin
@gnieh:matrix.org
[m]
Hi thanks for sharing
I think it's a bit outside of scope, and I would expect this to happen upstream, before using fs2-data
however if you make your data go through text.lines after decoding, it should be easy to handle with a regex
I will look a bit more into the standard but it really looks like a line oriented format, with the keyword at the beginning of the line
so something like "^tagname:(.*)" should then capture properly after you ensured your data arrives in lines in the pipe
yes, according to the grammar given in section 6, the stream contains events separated by lines
Lucas Satabin
@gnieh:matrix.org
[m]
you also need to drop lines starting with a colon, they are comments
Lucas Satabin
@gnieh:matrix.org
[m]
then you can pipe this through fs2-data
Sait Sami Kocataş
@Deliganli
thanks a lot @gnieh:matrix.org , I also stole a bit of code from https://github.com/Spinoco/fs2-http, would have used it directly but no scala 2.13 binaries published
Damien O'Reilly
@DamienOReilly
Hi, I would like to specify a custom date-time formatter to parse particular dates in some csv files that do not conform to ISO-8601. I see that fs2.data.csv.CellDecoder#javaTimeDecoder is private. Apart from implementing something similar to javaTimeDecoder myself, is there another way to achieve this with this libary?
Lucas Satabin
@gnieh:matrix.org
[m]
hi
I have to check this part of the code again, maybe ybasket knows this on the top of his head?
ybasket
@ybasket:matrix.org
[m]
fs2.data.csv.CellDecoder#javaTimeDecoder is private as it's not safe given its (current) signature. But I totally see your use case and will create an issue to expose something appropriate. Until then, you'll have to do it yourself, sorry!
CellDecoder[String].emap(s => Either.catchOnly[DateTimeParseException](doYourParsing(s)).leftMap(new DecoderError("Couldn't parse my format", _)))
Damien O'Reilly
@DamienOReilly
No worries, thanks for your response.
Lucas Satabin
@gnieh:matrix.org
[m]
@DamienOReilly: generally speaking the CSV API will get a complete overhaul for release 1.0. It is historically the first one and has grown into something too complicated. Any feedback is also welcome on it to improve it. So if you have a pain point with it, don't hesitate to tell us
Damien O'Reilly
@DamienOReilly
Thanks @Lucas , will do

Is there anything in the lib to work with strings regards parsing to rows? For example, I am currently crunching csv files from s3, using the fs2-aws-s3 lib, and have something like:

        s3Client.readFileMultipart(BucketName(bucket), FileKey(fileKey), partSizeMB)
          .through(fs2.text.utf8Decode)
          .through(fs2.text.lines)
          .through(rows[IO]())
          .through(headers[IO, String])

However rows expects an Stream[F, Char], not Stream[F, String].

Damien O'Reilly
@DamienOReilly
I can pipe it through .through(_.flatMap(str => fs2.Stream.emits(str.toCharArray))) before the rows pipe maybe.
Lucas Satabin
@gnieh:matrix.org
[m]
yes this is the common pattern with version 0.x
you don't need the lines pipe
rows handles it, including corner cases (when new line occurs within a row)
the flatMap(Stream.emits(_)) won't be required anymore in version 1.x onward
so just replace .through(fs2.text.lines) by .flatMap(fs2.Stream.emits(_)) and it should work out of the box
Damien O'Reilly
@DamienOReilly
Nice one, thanks again
Lucas Satabin
@gnieh:matrix.org
[m]
👍️
(I really am not used to the way completion works in Element...)
Lucas Satabin
@gnieh:matrix.org
[m]
Hi there! I release fs2-data 0.10.0 with support for CBOR and configurable java.time codecs. https://twitter.com/lucassatabin/status/1373402923154161665 From now on, development will focus on the 1.0.0 release, which will bring new and better CSV pipes, as well as easier support for textual inputs (and much more in CBOR, XML, ...). Main branch will become the one targetting 1.0.0 soon(-ish) and 0.x series will be renamed maintenance and will only accept bug fixes. I wish you all a nice Sunday.
Lucas Satabin
@gnieh:matrix.org
[m]
I just promoted main branch as the default one, which is the former 1.x branch. master was renamed to 0.x and should only be targeted for bugfix PR from now on. I will update CI configuration and documentation accordingly now. This is the official start of 1.x development!
ybasket
@ybasket:matrix.org
[m]
Hey, I just released version 1.0.0-RC1 of fs2-data, featuring CE3 support, more supported input types and new CSV pipes and data types. The changes are described in detail in the release notes, feedback is welcome as usual!
Big thanks to everyone involved!
Dylan Halperin
@dylemma
Just popping in to say hi. Earlier last year I mentioned doing some overhauls in xml-spac to support cats-effect and allow for different "event" providers. It's finally reached a point where I'm just about to start adding fs2-data as a provider! dylemma/xml-spac#28 - this'll allow for easy creation of Pipe[F, XmlEvent, A] for xml streams and Pipe[F, Token, A] for json streams
ybasket
@ybasket:matrix.org
[m]
Sounds cool! Feel free to ask questions here any time 🙂
Lucas Satabin: WDYT of having some "related projects" section in the fs2-data docs?
Lucas Satabin
@gnieh:matrix.org
[m]
Hi @dylemma this sounds awesome! Thanks for letting us know! I will add a section to the website this weekend.