by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 08 19:25
    ostronom closed #27
  • May 08 19:18
    ostronom opened #27
  • Mar 14 14:26

    AdamChlupacek on v0.4.0-M4

    (compare)

  • Mar 14 14:26

    AdamChlupacek on 0.4-spinoco-compat

    Setting version to 0.4.0-M4 Setting version to 0.4.0-SNAPSH… (compare)

  • Mar 14 13:34

    mraulim on 0.4-spinoco-compat

    Add timeout for publishing in k… Added trusty as a dist for runn… Fixed timeout on leader query. and 2 more (compare)

  • Mar 14 13:34
    mraulim closed #26
  • Mar 14 09:31
    AdamChlupacek synchronize #26
  • Mar 14 09:31

    AdamChlupacek on publish-connection-timeout

    Add timeout for leaders stream. (compare)

  • Mar 13 18:47
    AdamChlupacek synchronize #26
  • Mar 13 18:47

    AdamChlupacek on publish-connection-timeout

    Fixed timeout on leader query. (compare)

  • Mar 13 17:16
    AdamChlupacek synchronize #26
  • Mar 13 17:16

    AdamChlupacek on publish-connection-timeout

    Added trusty as a dist for runn… (compare)

  • Mar 13 17:13
    AdamChlupacek review_requested #26
  • Mar 13 17:13
    AdamChlupacek opened #26
  • Mar 13 17:07

    AdamChlupacek on publish-connection-timeout

    Add timeout for publishing in k… (compare)

  • Feb 04 12:55

    AdamChlupacek on v0.4.0-M3

    (compare)

  • Feb 04 12:55

    AdamChlupacek on 0.4-spinoco-compat

    Setting version to 0.4.0-M3 Setting version to 0.4.0-SNAPSH… (compare)

  • Feb 04 12:41

    AdamChlupacek on 0.4-spinoco-compat

    Increased coursier version. (compare)

  • Feb 04 11:32

    mraulim on 0.4-spinoco-compat

    Added read timeouts for broker … Merge pull request #25 from Spi… (compare)

  • Feb 04 11:32
    mraulim closed #25
Calvin Brown
@calvinbrown085
:thumbsup:
Ernesto Periñán
@eperinan
hi all!! I'm trying to publish to a topic that it's not created yet, expecting the producer be able to create on the fly. Does anyone addressing this problem before? I've been checking the method and it's returning and F[Long] but I'm consider IMHO it should return a F[Either[E,Long]]. In my case, I'm getting the error LeaderNotAvailable which it should be encapsulated within E. Am I missing something?? I know that README talks about "leaves it up to the user to choose any the retry mechanism". any suggestions?
Pavel Chlupacek
@pchlupacek
@eperinan we do not implement kafka tooling in the fs2-kafka. So you have to create this manually. There is some tooling in tests that can help you do so, or you may use standard tooling available in kafka.
re the method, which one are you referring to ?
Ernesto Periñán
@eperinan
Thanks for your quick reply
I mean that we are not handling the exceptions in this method. IMHO, it is really intresting. What do you think?
Pavel Chlupacek
@pchlupacek
note that KafkaClient to be created requires an F : Concurrent, concurrent actually extends MonadError/ApplicativeError where E is defined as Throwable. So in fact this Either[Throwable, Long] is already there :-)
So if you want to handle error just use attempt and you get that either you asking for :-)
Ernesto Periñán
@eperinan
That sounds great for me!
Really thanks @pchlupacek I didn't take theconcurrent into account
I am going to review it. Thanks, you are awesome :muscle:
Pavel Chlupacek
@pchlupacek
np glad I can help
Daniel Karch
@danielkarch
Hi, I saw that the series/0.4 branch has been updated to fs2 1.0.0. Do you know when there might be a release ready?
Pavel Chlupacek
@pchlupacek
@danielkarch the issue why we did not release it yet, is that we couldn’t have stable travis build. We are still investigating the cause of this.
once that will be resolved we will issue a release.
Sorry for that
Daniel Karch
@danielkarch
Ah, I see. Thanks for letting me know.
Daniel Karch
@danielkarch
Hi, any news on this? We are now planning to publish to our own repository and depend on that, but I wanted to ask if you know when an official release might be happening.
Pavel Chlupacek
@pchlupacek
Daniel, we did not focus on this recently. Thanks for pinging this out. When you need this to be stable? We would push it forward.
Daniel Karch
@danielkarch
We want to start using it in production in January, but for now we can live with a version that we publish ourselves to our repo.
Pavel Chlupacek
@pchlupacek
ok, understand we will prioritise this and make sure this will get stable in week or so.
Daniel Karch
@danielkarch
Great, thank you!
countfloyd
@countfloyd
Hey everybody, just looking to use fs2-kafka in a project and wondering about the ovotech/fs2-kafka project. It seems to wrap the kafka client drivers. Why would I use one over the other? https://github.com/ovotech/fs2-kafka
Daniel Karch
@danielkarch
I asked this question in the fs2 channel and Fabio explained it. You might find it with the search.
countfloyd
@countfloyd
Great thanks @danielkarch
countfloyd
@countfloyd
Is it possible to subscribe to a topic and get a finite stream of just the already published messages? The example in the README doesn’t work because of course all of the messages will have offset <= next
This seems to work:
Stream.eval(client.offsetRangeFor(topic("offsets"), partition(0))).flatMap { case (first, next) =>
      val num = next - first
      println(s"F: $first, L: $next, N: $num")
      client.subscribe(topic("offsets"), partition(0), first).take(num)
    }
vlmiroshnikov
@vlmiroshnikov
Hi, I saw that the series/0.4 branch has been updated to fs2 1.0.0. Do you know when there might be a release ready?
Adam Chlupacek
@AdamChlupacek
@vlmiroshnikov @danielkarch I have just updated dependencies on the series/0.4, will wait for travis to see if the tests that were failing there before are now green, if so, I will get this merged today and released by the end of the day.
Adam Chlupacek
@AdamChlupacek
@/all fs2-kafka 4.0.0 released
Alex Cruise
@acruise
you mean 0.4.0?
btw would you consider this library reasonably production-ready? :)
Pavel Chlupacek
@pchlupacek
@acruise we are using this library in production
The reason why we are not up to date wit recent fs2/kafka versions is actually we don’t use them in production, and as such we do not have push to make this happen.
Alex Cruise
@acruise
thanks!
Pavel Chlupacek
@pchlupacek
@acruise welcome
Paul Roseau
@paulroseau
Hi guys, I would like to give a go at fs2-kafka instead of the official Kafka Consumer. However since there is no notion of consumer group in fs2-kafka, I am not sure about the most efficient way to replicate that behavior manually. In the production example you guys mention do you have multiple clients reading from several partitions for one topic ? If so where do you store the information regarding consumers/partitions pair in order to have consumers gracefully joining/leaving the topic ? If not, would you say the official Kafka consumer client is better suited for this use case ?
Pavel Chlupacek
@pchlupacek
We are actually using this with persistent compacted topic. There are mulitple strategies how to do that, depending on your aplication case. It is like o week or so of development to get this working, and we are happy with perfromance maintainability etc. But it is always hand work
So unless you ready for this, simple wrapper around the kafak driver may be better option. fs2-kafka really gives you clean and minimalistic approach so you know whats going on and there is no woodoo magic :-)
Paul Roseau
@paulroseau
thx @pchlupacek . I am not that familiar with all kafka features, what do you mean exactly by "persistent compacted topic" ?
Pavel Chlupacek
@pchlupacek
thats the topic, that will allow you to always persisit lat value published for any key in topic.
Essentially, this is what consumer groups do behind the scenes in official kafka driver
Paul Roseau
@paulroseau
ok so you do not rely on any other external datastore to keep track of which consumer are reading which partition ? That feature of Kafka is enough ?
Pavel Chlupacek
@pchlupacek
Yes. We use kafka solely for this and committing the messages processed. This is fastest and most reliable solution we have found so far.
Paul Roseau
@paulroseau
ok thx for your answer
Seth Lasky
@TheMover
Hey @pchlupacek , I'm just getting started with fs2-kafka and wondering if you could help. I'm running into an error when publishing to a topic that doesn't exist yet LeaderNotAvailable was thrown during property evaluation. Message: Topic: , partition: 0 has no leader available Occurred when passed generated values ( arg0 = Stream(..), arg1 = "" // 6 shrinks ) - is there any way to continue publishing if that error shows up? The topic does get created but (I think) after it checks for the leader and throws an error. Or better yet is there a way to create a topic from your client?
Pavel Chlupacek
@pchlupacek
@paulroseau this could be achieved easily via standard fs2 stream error handling. If you couldn assure that all topics exists before you start publishing just listen to error, and when this error occur create the topic. Unfortunatelly, fs2-kafka does not have implementation of the admin part of the kafka prototcol, as such you will have to resolve to standard kafka admin tools for it.
The example how this could be done can be found in tests, where we do exactly that, creating topics before every single test
Seth Lasky
@TheMover
Thanks!