Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 16 15:22
    mensfeld commented #676
  • Apr 16 15:22

    mensfeld on master

    Remove rake dependency (#676) (compare)

  • Apr 16 15:22
    mensfeld closed #676
  • Apr 16 15:13
    bilby91 opened #676
  • Apr 15 16:09
    mensfeld synchronize #668
  • Apr 15 16:09

    mensfeld on 2.0-wip

    jobs queue specs (compare)

  • Apr 14 10:03
    mensfeld closed #675
  • Apr 14 10:03
    mensfeld labeled #675
  • Apr 14 10:03
    mensfeld assigned #675
  • Apr 14 10:03
    mensfeld commented #675
  • Apr 14 10:00
    dexhacker commented #675
  • Apr 13 15:35
    mensfeld commented #675
  • Apr 13 05:10
    dexhacker opened #675
  • Apr 11 15:50
    mensfeld edited #431
  • Apr 11 15:49
    mensfeld edited #431
  • Apr 11 15:49
    mensfeld edited #431
  • Apr 11 15:49
    mensfeld edited #431
  • Apr 11 15:47
    mensfeld synchronize #668
  • Apr 11 15:47

    mensfeld on 2.0-wip

    specs (compare)

  • Apr 10 19:00
    mensfeld commented #431
Maciej Mensfeld
@mensfeld
interesting one
@nvtanh looks like an error - can you create a repro and open a GH issue? I will fix it asap
Tr.Anh
@nvtanh
@mensfeld thank you so much
I created an issue here karafka/karafka#665 and noted GH repo in this issue.
I hope that it'll fix asap
Maciej Mensfeld
@mensfeld
@nvtanh thank you. I will look into that otmorrow
Maciej Mensfeld
@mensfeld
@nvtanh can you test against this branch? karafka/karafka#666 - it should be fixed now
sorry for the trouble :pray:
Tr.Anh
@nvtanh
@mensfeld I'll try it. Thank for your support :pray:
Tr.Anh
@nvtanh
@mensfeld It's working correctly for me
Tr.Anh
@nvtanh
and when you can merge it to master?
Maciej Mensfeld
@mensfeld
@nvtanh I will craft a release today
in 1-2h tops
Maciej Mensfeld
@mensfeld
@nvtanh released. Sorry for the trouble :pray:
Tr.Anh
@nvtanh
@mensfeld thanks for your quick support :pray:
rana ali
@ranaali:matrix.org
[m]
Is there any way in karafka to add more than one consumer in the same group(Routing), also can i build an app that consumer X will not start until consumer Y and consumer Z finishes consuming message with the same key?
Sandip Subedi
@sandipsubedi
Hi everyone! new to Karafka here. Does anyone know why producers are called Responders instead of Producers? I have used Racecar before, and they used to call it Producers. So I am just curious.
Lee Hester
@LeeHester
@sandipsubedi I believe karakfa itself is the “consumer” library in the karafka ecosystem. Waterdrop is the producer library. In general, you use waterdrop to publish messages to kafka and karafka to consume those messages. Karafka has the concept of a responder, which is a consumer that also publishes messages. This is useful if you’re building a pipeline, and one consumer might read one topic, manipulate the messages it receives and publish them to ankther topic. Responder is just a special case of consumer.
1 reply
rana ali
@ranaali:matrix.org
[m]
Hello,
is there any way, in Karafka, to produce a message once several related messages have been consumed ?
Maciej Mensfeld
@mensfeld

Is there any way in karafka to add more than one consumer in the same group(Routing), also can i build an app that consumer X will not start until consumer Y and consumer Z finishes consuming message with the same key?

Well you can have as many consumers in a group as you want :) but not sure if I get the question

3 replies

can i build an app that consumer X will not start until consumer Y and consumer Z finishes consuming message with the same key?

That is kinda against the idea of Kafka message log data distribution but it can be done if you introduce intermediate trigger topics and make it event driven

@LeeHester

Hi everyone! new to Karafka here. Does anyone know why producers are called Responders instead of Producers? I have used Racecar before, and they used to call it Producers. So I am just curious.

Because I was stupid :) Responders won't be available in 2.0. They were designed to create a more complex "responses" based on events, where you would propagate data to several topics and they ensured consistency of delivery with error and correctness tracking.

@ranaali:matrix.org

is there any way, in Karafka, to produce a message once several related messages have been consumed ?

If you mean "product after batch is done" then yes. If you mean producing in regards to consumption of different topics/partitions data, then this might be tricky

rana ali
@ranaali:matrix.org
[m]
then how to do this Well you can have as many consumers in a group as you want?
Maciej Mensfeld
@mensfeld
Because it's true. You can have as many consumers s you want - it's just a matter of understanding, that always one of them will receive a given message.
Kafka is not like Redis pub-sub
rana ali
@ranaali:matrix.org
[m]
@mensfeld: can we in anyway that the message is consumed by a consumer ?
Maciej Mensfeld
@mensfeld
@ranaali:matrix.org no. Kafka is not a broadcasting engine. The only way to do it, would be to do it with different consumer groups.
1 reply
This message was deleted
Nontawat Numor
@mrnonz

Hi, I faced problem with karafka. "Karafka does not restart dead PG connections". Solution at here https://github.com/karafka/karafka/wiki/Problems-and-Troubleshooting#why-karafka-does-not-restart-dead-pg-connections

Anyone have idea for handle this error? I want to retry message which is faced with PG Connection issue. Let's say if consumed message and process is fail with PG connection. Program must reconnect to PG and then retry old message.

Maciej Mensfeld
@mensfeld
@mrnonz just handle the reconnections yourself
Nontawat Numor
@mrnonz
@mensfeld thank you so much. let's me try.
Maciej Mensfeld
@mensfeld
@mrnonz you are welcome. Karafka cannot handle broken connections unless you opt out on reentrancy and switch to manual offset management and handle many other things
you could do it but I do not recommend it
You will be much better retrying that yourself
Sandip Subedi
@sandipsubedi
Hello :wave:
Question out of curiosity: Is it possible to run Karafka within Rails without running it on a different terminal? If that's possible, is there any disadvantage of doing it that way?
Maciej Mensfeld
@mensfeld
@sandipsubedi WDYM terminal? Process?
In what type of process?
You can easily run it from within Sidekiq
Sandip Subedi
@sandipsubedi
I mean within Rails process. Without even using SideKiq or by running bundle exec karafka
Maciej Mensfeld
@mensfeld
You still did not define what is the "Rails process" for you
technically speaking @sandipsubedi there is no such thing
there is Puma, Sidekiq, Karafka, rails console
really hard for me to comment on that without you being more precise
Sandip Subedi
@sandipsubedi
@mensfeld I meant Puma.
Sorry for not being precise.
Maciej Mensfeld
@mensfeld
@sandipsubedi please open an issue in Karafka with ths. It is doable but I need to give you code snippets
althought it is not a recommended way of doing things ;)
Sandip Subedi
@sandipsubedi
I was mostly curiois. But if it's not recommended, we probably don't want to do that. If we change our mind, I will create an issue in Karafka.
Thank you for answering my question @mensfeld
Maciej Mensfeld
@mensfeld
@sandipsubedi it's not recommended similar to how Sidekiq works, etc. I will be however exposing an official API to run it within other Ruby processes with 2.0