Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 25 08:59
    mensfeld commented #630
  • Nov 24 09:47
    Azdaroth synchronize #630
  • Nov 17 17:13
    Azdaroth review_requested #630
  • Nov 17 17:10
    Azdaroth synchronize #630
  • Nov 14 20:30
    Azdaroth opened #630
  • Nov 10 15:52

    mensfeld on coditsu-remarks

    (compare)

  • Nov 10 15:52

    mensfeld on master

    quality remarks (#629) * quali… (compare)

  • Nov 10 15:52
    mensfeld closed #629
  • Nov 10 15:41
    mensfeld assigned #629
  • Nov 10 15:41
    mensfeld review_requested #629
  • Nov 10 15:41
    mensfeld ready_for_review #629
  • Nov 10 15:39
    mensfeld synchronize #629
  • Nov 10 15:39

    mensfeld on coditsu-remarks

    more remarks (compare)

  • Nov 10 15:32
    mensfeld synchronize #629
  • Nov 10 15:32

    mensfeld on coditsu-remarks

    more remarks (compare)

  • Nov 10 15:13
    mensfeld opened #629
  • Nov 10 15:13

    mensfeld on coditsu-remarks

    quality remarks (compare)

  • Nov 10 14:59

    mensfeld on return-non-zero

    (compare)

  • Nov 10 14:59

    mensfeld on master

    return non zero on missing cli … (compare)

  • Nov 10 14:59
    mensfeld closed #628
Maciej Mensfeld
@mensfeld
Hey @ruitiagomendes_gitlab I do not quite understand your question

i have karakfa and ruby sinatra api. just a advice if possible. can i have a responser that will deliver

What do you mean by dto?

would you mind describing your usecase a bit better? thanks!
Tiago Mendes
@ruitiagomendes_gitlab
Data Transfer Object. So I take that to mean an object that can be used for transferring data around.
my example is this
an user has properties to sale so they also have images/photos
but i would like to have one or multiple karafka instances running in order to process those images after upload
so the process begins on the api where the user uploads properties/apartments pictures and they are sent to the karafka for creating the user apartmanent, till here so far so good
:)
the problem is for me after apartment created i have to process the images and return the full Model of apartment with images processed by one or multiple instances of karafka
this will be received on the api endpoint that made the request
Tiago Mendes
@ruitiagomendes_gitlab
did i make it understandable? sorry for the english
Tiago Mendes
@ruitiagomendes_gitlab
made it. batch image processor takes for 5 images 4secs in response
Maciej Mensfeld
@mensfeld
Hmm
I wonder if Karafka (and Kafka) is what you want for that
Karafka 1.x is NOT a message queue engine. 2.0 will be much better in that regard but it's still not a simple thing. I even wonder why would you want to do it that way/
Gennady Grishkovtsov
@grishkovelli
Hello everyone, I'm new in Kafka/Karafka, so I have an issue. In a few days my consumer starts to process old messages,
but at that time Kafka's topic has new message. I don't know why it happenes.
Also, if I try to empty the topic Karafka continues to process old or nonexistent messages.
Any ideas? Where I'm wrong?
class KarafkaApp < Karafka::App
  setup do |config|
    config.kafka.seed_brokers = ENV['KAFKA_BROKERS'].split(' ')
    config.kafka.start_from_beginning = false
    config.kafka.offset_commit_threshold = 100
    config.client_id = 'foobar'
    config.batch_fetching = true
    config.batch_consuming = true
    config.deserializer = CustomDeserializer
  end

  consumer_groups.draw do
    topic :status do
      consumer StatusConsumer
    end
  end
end
Lee Hester
@LeeHester
Gennady, that's a tricky one. What do you mean when you say you try to "empty the kafka topic". There really isn't a way to empty a topic. You could delete it and re-create it, or create an entirely new topic and use that, but I don't know of a way to empty a topic. You may want to use kafka's command line tools or something like Yahoo's CMAK (Cluster Manager for Apache Kafka) to look at the state of your consumers in Kafka. Specifically, you probably care about the lag (how many messages behind the latest message your consumer is). You can find that in CMAK's web UI or with the kafka command line tools like this:
Lee Hester
@LeeHester
$ kafka-consumer-groups --bootstrap-server your_broker_hostname:9092 --describe --group foobar
Luong Vo
@luong-komorebi

HI @grishkovelli , happy to help. It would be nice if you can include the log of your karafka backend. From the first look, all I can guess is that

my consumer starts to process old messages, but at that time Kafka's topic has new message
Your consumer may fail to commit the offset. Which means that the next time it starts consuming, it would start from the old offset.
if I try to empty the topic Karafka continues to process old or nonexistent message
this is confusing and demands more clarification. How did you empty the topic?

Maciej Mensfeld
@mensfeld
Yeah looks like something went off before the offset has been committed
Decklyn Dubs
@DecklynDubs_twitter
What's the recommended approach for dependency injection into consumers?
I see the constructor has one parameter. I'm a bit new to ruby so curious what the golden path is here.
Maciej Mensfeld
@mensfeld
@DecklynDubs_twitter there is no golden path here. It has one parameter though I've seen people redesign it and overwrite the initialization code.
Luong Vo
@luong-komorebi

@DecklynDubs_twitter Nice question. Of course you can do it. This is my result after researching in Karafka (1.x stable). If you are gonna do DI, I suggest you dont do it the constructor way. If you want to do it the constructor way, you can do (in karafka.rb) but it is very ugly and you are breaking the layer that Karfaka opts for controlling, which is consumer classes, not instances:

consumer_groups.draw do
   topic 'SomeTopic' do
     consumer SomeConsumer.new(args1, args2)
  end
end

and in the consumer class

class SomeConsumer < ApplicationConsumer
def initialize(args1, args2)
end

# This is the ugly part, you have to add this method so that karafka/routing/topic.rb can assign a topic to your consumer
def topic=(topic)
self.class.topic = topic
end
end
Maciej Mensfeld
@mensfeld
Actually if we would make a topic a kwarg, this would be much easier
as you could easily intercept the topic kwarg and super it and go on with your life
Luong Vo
@luong-komorebi
yeah true
Although I dont do much DI in the consumer, I do recognize that this is a need and hope to see it in later versions
Maciej Mensfeld
@mensfeld
I'll make a note for 2.0 to change it if the performance is not an issue
I don;'t think it should be though, initialization of new consumer instances is fairly rare
though with virtual partitioning it may be more often so I will have to do benchmarks on it
Luong Vo
@luong-komorebi
awesome. I agree with not initializing consumer instances, too, which is why I recommend doing DI via other ways like setter methods for the class itself
Maciej Mensfeld
@mensfeld
The reason why I think building consumers with composition instead of inheritance would be hard is the underlying logic
but I will try to think about it. The current API design seems battle tested and robust, thus I would opt n not redesgining too many things in this place
not that I did not redesign at least 50% of code already for 2.0 lol
Luong Vo
@luong-komorebi
LOL
one thing I would like to suggest a change is the wiki
I do believe putting documentation in git makes more sense, since a lot of time the code grows and the wiki on Github just stays for an older version
not sure what do you think about this ?
Maciej Mensfeld
@mensfeld
@luong-komorebi You mean this? https://github.com/karafka/wiki
:)
it's a separate repository that has a sync with the main repo (karafka/karafka) wiki
there's currently 1.4 (master) and 1.3 branch (still supported)
Luong Vo
@luong-komorebi
yeah, oh , I didnt know github wiki is from another repository. Please ignore me :D That s good though. I m more familiar with the kinds of doc that are inside of codes
Maciej Mensfeld
@mensfeld
It is inside :)
It's just synced from a different repo
and the reason for it is, that wikis on github are actually a sepatet git repos
and it's just easier for me to manage that that way
Luong Vo
@luong-komorebi
:cheers: