@edenhill thanks for replying! appreciated - I have this offset bug that I can't figure out - maybe you can spot something?
My consumer here that i've built - using STORED offset. The bug behaviour is:
- i can only consume messages on the partition when the consumer is listening at the same time.
- if I produce msgs onto the partition while there is no consumer running, then I run the consumer, I just get put into this EOF section: https://gist.github.com/dragoonis/b7dfeef0b223d4879a48b53a35b0f6c4#file-kafkaconsumer-php-L61
Did this make sense ?
(dunno how to make that gist block smaller on this chat, srry)
seems like you are never committing the offsets you store (since auto commit is disabled), so whenever you restart it wont find a committed offset and thus resort to
auto.offset.reset which defaults to latest
I am committing the offset when I get EOF (no more data)
it is also kind of weird to only store (and I guess you also want commit) when you reach EOF. If your consumer is slower than the producer, for whatever reason, you will never reach EOF and thus never commit.
@edenhill should I be calling ->storeOffset() upon every entry into RD_KAFKA_RESP_ERR_NO_ERROR? If not .. where?
no, you are storing the offset for future commit, but you are never committing
I see!
offsetStore = store offset for future commit
commit = commit stored offset
Totally understand 100% :) thanks
:)
the nomenclature is kind of weird
STORED, store, commit
so your confusion is warranted