These are chat archives for allegro/hermes

2nd
Jun 2016
Stevo Slavić
@sslavic
Jun 02 2016 13:53
if you don't mind, I'll create a issue on github, that basic auth credentials updating doesn't work
Adam Dubiel
@adamdubiel
Jun 02 2016 13:53
yes, go ahead
Stevo Slavić
@sslavic
Jun 02 2016 13:54
on a different note - is it exposed somehow (e.g. some metric) what is the state/mode of each subscription
Adam Dubiel
@adamdubiel
Jun 02 2016 13:55
you mean somehow different than via the api?
ah
no, this is not exposed, it can be deducted from output rate metric only
there is also a .../{susbcriptionname}/health endpoint starting from 0.8.7
where you can find some information about general health problems (i.e. too many 500 or a lot of timeouts)
Stevo Slavić
@sslavic
Jun 02 2016 13:57
endpoint is nice, would like also metric about subscription/consumer state/mode
Adam Dubiel
@adamdubiel
Jun 02 2016 13:58
this is bit trickier, since each consumer calculates it locally
so you can't get single answer, you can get list of statuses per consumer at best
Stevo Slavić
@sslavic
Jun 02 2016 14:00
yes, each consumer should report separate metric
Adam Dubiel
@adamdubiel
Jun 02 2016 14:00
this is doable then, mind to create a ticket for it as well?
Stevo Slavić
@sslavic
Jun 02 2016 14:00
nodex.subsriptiony.consumer_state
and sample/report it regularly as metric
can be number/coded or maybe string constant of state
Adam Dubiel
@adamdubiel
Jun 02 2016 14:01
yeah, we probably would have to have some mapping like 0 == normal, 1 == slow 2 == heartbeat
this would make creating checks easy (i.e. > 1 is an alert)
Stevo Slavić
@sslavic
Jun 02 2016 14:02
yes. endpoint is nice for human, while metric for automating
we're seeing some strange behavior, nothing is sent to subscriber, there were no non 2xx response from subscriber in past, lag is non-zero, and metrics are not changing - so hard to debug what's happening with subscription, would have to take heap and thread dump or something which is bad
restarting consumers app dozens of messages get delivered, but then it gets stuck again
Adam Dubiel
@adamdubiel
Jun 02 2016 14:06
is it general thing or you see it on only one subscription? how many subscriptions do you have there? and which version of Hermes are you running
Stevo Slavić
@sslavic
Jun 02 2016 14:07
there are only few subscriptions in this particular deployment and this is happening with one of them, it's hermes 0.8.6 based deployment
on a different note - it doesn't seem possible to have timeouts configurable per subscription, correct?
Adam Dubiel
@adamdubiel
Jun 02 2016 14:09
it is in 0.8.8 :)
Stevo Slavić
@sslavic
Jun 02 2016 14:09
ah, cool
Adam Dubiel
@adamdubiel
Jun 02 2016 14:10
there is "requestTimeout" in subscription policy, limited to max of 60 seconds
Stevo Slavić
@sslavic
Jun 02 2016 14:10
that one released already?
Adam Dubiel
@adamdubiel
Jun 02 2016 14:10
yes, it was released yesterday
Stevo Slavić
@sslavic
Jun 02 2016 14:10
nice
Adam Dubiel
@adamdubiel
Jun 02 2016 14:10
and as to the debug problem - by metrics are not chaning you mean i.e. latency metrics is a flat line? delivery rate is 0 or null?
Stevo Slavić
@sslavic
Jun 02 2016 14:14
yes
Adam Dubiel
@adamdubiel
Jun 02 2016 14:17
hmm i think a stacktrace would be very helpful here, unfortunately. we did have some problems with spinning up new consumers, but with ~800 subscriptions. and they would not start, not "stop working" after delivering some msgs
are there any logs from this sub?
Stevo Slavić
@sslavic
Jun 02 2016 14:18
no. generally I can see only few logging used in consumers
which is ok, metrics are better
Adam Dubiel
@adamdubiel
Jun 02 2016 14:20
yes, we try to reduce number of logs not to get overwhelmed by them. do you think you could get us a stacktrace to analyze? all threads are named, so it will be easy to find the ones that take care of your subscription
Stevo Slavić
@sslavic
Jun 02 2016 14:21
will see what more options we have, but yes, analyzing heap and thread dump is one of them
will figure it out and get back to you guys if some bug is reproduced, but it could be anything else in subscriber or environment
Adam Dubiel
@adamdubiel
Jun 02 2016 14:23
okay, let us know what you find
Stevo Slavić
@sslavic
Jun 02 2016 14:23
will do
thanks for support and understanding
Adam Dubiel
@adamdubiel
Jun 02 2016 14:24
:)