These are chat archives for allegro/hermes

11th
Mar 2016
Martin Dobmeier
@martido
Mar 11 2016 12:39
Hi guys,
our understanding from looking at the code and log output of hermes-consumers is that multiple hermes-consumers instances each create a Consumer for each existing subscription. That is, there is no coordination between multiple instances. Is that understanding correct? If so, could that affect performance with an increasing number of subscriptions?
Rafał Żukowski
@rzukow
Mar 11 2016 14:29

Hi, your concern is correct. This is how it works by default and can cause problems when number of subscriptions is increasing to hundreds.

I'm not up to date with the latest changes, but I know there is another algorithm of consumer workload management to address this issue. I'm not sure if it is tested in production though, neither I see any documentation.

Quick look at the code https://github.com/allegro/hermes/blob/master/hermes-common/src/main/java/pl/allegro/tech/hermes/common/config/Configs.java shows there is a property consumer.workload.algorithm that must be set to one of https://github.com/allegro/hermes/blob/master/hermes-consumers/src/main/java/pl/allegro/tech/hermes/consumers/supervisor/workload/ConsumerWorkloadAlgorithm.java

But it looks like undocumented feature at the moment. You will get some more info as soon as some active developers will join the chat ;)
In general the solution is trying to limit number of consumers handling each subscription.
Martin Dobmeier
@martido
Mar 11 2016 15:07
thanks @rzukow for pointing us at the consumer.workload.algorithm configuration. Gave both "mirror" and "selective" a quick try, but both seem to create a Consumer on each instance. Would be nice to know the differences between those three algorithms
miguelpuyol
@miguelpuyol
Mar 11 2016 15:14
Hi all! We are trying to configure a subscription with a subscriber with Basic Auth enabled. We saw in your pl.allegro.tech.hermes.api.EndpointAddress that you are masking the password when an endpoint is set like: http://user:password@myurl.com
but then when the Hermes Consumer tries to send data to the subscriber, we get 401s
Is there a way to use these credentials to push to the subscribers?
Martin Dobmeier
@martido
Mar 11 2016 17:29
@rzukow Forget about what I said above. My test was crap. When using workload algorithm "selective" and starting more than two hermes-consumer instances you can see that consumers are only created on two of the instances. If one them is shut down, the consumer is created on the third instance after some time
very nice
Adam Dubiel
@adamdubiel
Mar 11 2016 19:27
hi guys, im having a short vacation with very limited internet access so couldnt reply earlier :)
so, @martido , we were concerned about the same thing and so we developed the selective algorithm. it has some prameters like max number of subscriptions handled by one node, so in some test cases it might seem to work like default one, cos the number is quite high (250 by default afair). we have been running it on test for some time now and fixed some non trivial bugs init. we should be moving to prod soon and thats when we add docs, so stay tuned
Adam Dubiel
@adamdubiel
Mar 11 2016 19:33
@miguelpuyol we do mask the password, but there is no basic auth support. we used it in jms consumer. since we have almost everything in place, adding basic auth should not be a big issue, if you could create one, we can implement it for http sender as well :)
*i mean: if you could creat github issue
Martin Dobmeier
@martido
Mar 11 2016 23:22
@adamdubiel Thanks, sounds good. Looking forward to the documentation. And don't worry ... enjoy your vacation ;)