Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jul 29 2021 00:15
    masonedmison starred Spinoco/fs2-kafka
  • Mar 24 2021 04:23
    kevinalh starred Spinoco/fs2-kafka
  • Jan 28 2021 23:41
    mebubo starred Spinoco/fs2-kafka
  • Jan 17 2021 14:53
    PengLoveHONG starred Spinoco/fs2-kafka
  • Nov 07 2020 05:55
    nachocodexx starred Spinoco/fs2-kafka
  • May 08 2020 19:25
    ostronom closed #27
  • May 08 2020 19:18
    ostronom opened #27
  • Mar 14 2020 14:26

    AdamChlupacek on v0.4.0-M4

    (compare)

  • Mar 14 2020 14:26

    AdamChlupacek on 0.4-spinoco-compat

    Setting version to 0.4.0-M4 Setting version to 0.4.0-SNAPSH… (compare)

  • Mar 14 2020 13:34

    mraulim on 0.4-spinoco-compat

    Add timeout for publishing in k… Added trusty as a dist for runn… Fixed timeout on leader query. and 2 more (compare)

  • Mar 14 2020 13:34
    mraulim closed #26
  • Mar 14 2020 09:31
    AdamChlupacek synchronize #26
  • Mar 14 2020 09:31

    AdamChlupacek on publish-connection-timeout

    Add timeout for leaders stream. (compare)

  • Mar 13 2020 18:47
    AdamChlupacek synchronize #26
  • Mar 13 2020 18:47

    AdamChlupacek on publish-connection-timeout

    Fixed timeout on leader query. (compare)

  • Mar 13 2020 17:16
    AdamChlupacek synchronize #26
  • Mar 13 2020 17:16

    AdamChlupacek on publish-connection-timeout

    Added trusty as a dist for runn… (compare)

  • Mar 13 2020 17:13
    AdamChlupacek review_requested #26
  • Mar 13 2020 17:13
    AdamChlupacek opened #26
  • Mar 13 2020 17:07

    AdamChlupacek on publish-connection-timeout

    Add timeout for publishing in k… (compare)

Kåre Blakstad
@kareblak
which would make the whole model pretty nice
And also, N is never enough
becaause N is dependent on N-1 etc, so a snapshot of the actual accumulator, or at least the result from mapAccumulate is needed
Adam Chlupacek
@AdamChlupacek
yeah I guess. Anyhow you right that the compacted topics could be used for this.
Kåre Blakstad
@kareblak
I'm not using fs2-kafka atm, btw, so this would be a dirty impl on top of kafka-streams...
How do you guys usually store offsets with fs2-kafka? file system? db?
Adam Chlupacek
@AdamChlupacek
We do not. You have to manage them yourself. ie you could use the compacted topics for the offsets, that is actually how the native kafka client does it
Kåre Blakstad
@kareblak
Yeah, I know, but how do you usually do it?
I like the idea of handing this responsibility over to the user...
Adam Chlupacek
@AdamChlupacek
For some we use cassandra for some we use kafka. Depending on the needs
We should probably give an example how to use the persistent topics for it tho. So that people have easier time migrating.
Kåre Blakstad
@kareblak
Yeah, that would be nice, I guess
I love your Head/Tail offset apporach. It's just completely lacking in the kafka-clients
Adam Chlupacek
@AdamChlupacek
It all comes from allowing users to manage their own offsets. Plus few years of experience with using kafka in a large system :D
Kåre Blakstad
@kareblak
:thumbsup:
Andi Miller
@andimiller
have you considered adding the timestamp field in the producer?
KSQL gets a bit upset if I read data written by fs2-kafka
Adam Chlupacek
@AdamChlupacek
We are adhering to the kafka protocol, if you need some timestamps in the messages written to kafka, you can just add it there yourself. Or am I missing something? I dont want to add some additional data to everything just because of KSQL.
Andi Miller
@andimiller
there is part of the kafka protocol which has timestamps I believe
but it's optional
Adam Chlupacek
@AdamChlupacek
Oooh, I see what you mean, we are just not populating the value, but it is in protocols. all that is required is to expose it via API as of now we jsut set it to None all the me, you can PR it if you want.
Andi Miller
@andimiller
I may do if I can make it work :)
Adam Chlupacek
@AdamChlupacek
Hmm actually this may required more thought to it, now that i am reading the link :/
Andi Miller
@andimiller
I think it's a case of adding a new codec in protocols
because it bumps the magic byte
Pavel Chlupacek
@pchlupacek
@andimiller any idea what version of kafka introduced this ?
Adam Chlupacek
@AdamChlupacek
In that case yeah, a whole new codec is required
Pavel Chlupacek
@pchlupacek
if you can please, could you open issue? We’ll look onto that
Andi Miller
@andimiller
in theory 0.10.2.0
Pavel Chlupacek
@pchlupacek
:-)
Andi Miller
@andimiller
will do, I'll try and gather some info
Pavel Chlupacek
@pchlupacek
excellent thanks a lot
Kåre Blakstad
@kareblak
Can you guys provide an example of fs2-kafka writing offsets back to kafka?
pacman899
@pacman899
HI, is there a way consuming one optional msg(latest) from a topic? maybe with timeout? it suppose to return Stream of one element of Option[Msg]
Pavel Chlupacek
@pchlupacek
@/all 0.4.0-M2 release agains fs2 1.0.0-M2 was published
Vinayak Pathak
@vinayakpathak

Hello. I'm trying to use version 0.2.0 and it looks like even getting kafka.client working requires a non-trivial amount of work. I seem to need the following implicits just to make it compile:

  implicit val _concurrent: Concurrent[IO] = IO.ioConcurrentEffect
  implicit val AG: AsynchronousChannelGroup = AsynchronousChannelGroup.withThreadPool(Executors.newFixedThreadPool(8))
  implicit val ec = ExecutionContext.fromExecutor(new ForkJoinPool(8))
  implicit val scheduler = Scheduler(corePoolSize = 8).compile.toList.unsafeRunSync().head
  implicit val logger = Logger.JDKLogger[IO](java.util.logging.Logger.getAnonymousLogger).unsafeRunSync()

I'm not really familiar with various concurrency and execution related functionalities in java and so I have no idea what I'm doing here. At the same time it seems to me that if these implicits were actually required it wouldve been mentioned on the readme? Am I doing something wrong?

Pavel Chlupacek
@pchlupacek
@vinayakpathak some of this is getting to be simplified with 0.4.0 release (also please do not use 0.2.x, but rather 0.4.x branch). However we prefer the configuration of concurrency to be explicit in most cases.
Vinayak Pathak
@vinayakpathak
I see. 0.4.x uses fs2 1.0.0, no? Is that considered stable yet?
Pavel Chlupacek
@pchlupacek
the last version of 1.0 has fixes that are not available in 0.10 and liekly won't be backported. The 1.0 still may have some API changes. We are using 1.0 in production atm.
Vinayak Pathak
@vinayakpathak
Okay I'll give it a shot then
Pavel Chlupacek
@pchlupacek
generally the 0.10 version of kafka was imho never used a lot. In our production code we were at 0.9 release for a long time, and now we went straight to 1.0
Vinayak Pathak
@vinayakpathak
you mean 0.10 version of fs2 right?
Pavel Chlupacek
@pchlupacek
yes
the 0.4.0 of fs2-kafka will get likely synced with latest fs2 next week,
so no you are i think on M3
1.0.0-M3
Vinayak Pathak
@vinayakpathak
fs2-kafka's readme says M4
looks like it's going to be fs2-kafka v0.4.0-M3, which will depend on fs2 1.0.0-M4
Pavel Chlupacek
@pchlupacek
This seems to be messed up. @AdamChlupacek can you please take a look?