These are chat archives for allegro/hermes

16th
Mar 2017
r0ka
@r0ka
Mar 16 2017 12:04
hello guys
could you pleas advise how can I override consumer configuration ?
I want to change this property consumer.inflight.size
Adam Dubiel
@adamdubiel
Mar 16 2017 12:10
hi, we documented this here: http://hermes-pubsub.readthedocs.io/en/latest/deployment/deployment/#external-configuration - have you found it and tired it?
r0ka
@r0ka
Mar 16 2017 12:13
yup, I saw it
it describe frontend property as export HERMES_FRONTEND_OPTS="-Dfrontend.port=8090 -Dfrontend.idle.timeout=30"
what will be consumer variable? I guess HERMES_CONSUMER_OPTS , is it correct ?
Adam Dubiel
@adamdubiel
Mar 16 2017 12:28
actually its HERMES_CONSUMERS_OPTS, but its also in these docs :)
r0ka
@r0ka
Mar 16 2017 12:29
damn, how I missed that (
Adam Dubiel
@adamdubiel
Mar 16 2017 12:30
probably its not exposed enough, we repeat FRONTEND, never CONSUMERS
anyway - let me know if it will work for you
r0ka
@r0ka
Mar 16 2017 12:30
it looks like works
but I have few more questions
Adam Dubiel
@adamdubiel
Mar 16 2017 12:30
go ahead
r0ka
@r0ka
Mar 16 2017 12:31
I am testing this tool right now and see that messages arrives unordered, is it normal ?
Adam Dubiel
@adamdubiel
Mar 16 2017 12:32
ordering in Kafka can be preserved only in single partition, however when changing pull to push, we have to worry about retires, rate-limiting and so on, so even if we read messages in order from single partition, once single message has to be retried, the order is lost, thus we have no guarantees as to the ordering of messages
otherwise single retrying message could stall the consumption
r0ka
@r0ka
Mar 16 2017 12:33
make sense, clear
another problem that I see that if I send 1000 messages then my consumer receive more than 1000 messages, last test shows that consumer received 1029 messages and 102 messages are duplicated and 984 are unique
my producer sent 1000 unique messages
can't understand how it might be
Adam Dubiel
@adamdubiel
Mar 16 2017 12:36
hmm okay, couple questions from me: was kafka cluster stable during this test?
was the receiver having any problems (like timeouts or sth like this)?
just like Kafka, we have at least once guarantees. so duplicates could come from either Kafka (Kafka sender retrying messages that were acknowledged, but ack was lost somewhere) or from Hermes Consumer (retrying messages that were not ack by receiver, or that were ackd too late)
r0ka
@r0ka
Mar 16 2017 12:38

I am using docker compose configuration
from here
https://github.com/allegro/hermes/blob/master/docker/docker-compose.yml
running it locally, I don't think that there were any problems with kafka
actually this situation is reproducing every time

receiver - is simple spring boot application, nothing special, running locally

Adam Dubiel
@adamdubiel
Mar 16 2017 12:39
oh, okay, i see. i dont know about stability of Kafka in docker, but could you set ACK to ALL on the topic you are using for tests and retry it?
r0ka
@r0ka
Mar 16 2017 12:40
sure, I will update you about results
Adam Dubiel
@adamdubiel
Mar 16 2017 12:41
because while duplicates could happen, you should always get 1000 unique messages - we know it works with ACK set to all brokers. it usually works with ack-leader as well, but this depends on state of Kafka cluster and so for sensitive data we always go with ack set to "all brokers"
r0ka
@r0ka
Mar 16 2017 12:41
I see, ok, will check with "all brokers"
r0ka
@r0ka
Mar 16 2017 12:56
with ACK "all" property set all messages are delivered, sometimes with some delay and duplicates
Adam Dubiel
@adamdubiel
Mar 16 2017 12:57
so looks like docker is not the most stable setup :)
r0ka
@r0ka
Mar 16 2017 12:57
yeah, might be )
thank you Adam for helping !
Adam Dubiel
@adamdubiel
Mar 16 2017 13:08
no problem :)