These are chat archives for allegro/hermes

25th
Feb 2016
Stevo Slavić
@sslavic
Feb 25 2016 12:36
please correct me if wrong - zookeeper.root seems to be used for additional hermes specific metadata, while kafka own stuff sees root as configured in zookeeper.connect.string, and zookeeper.root is basically always relative to root kafka uses, correct?
Adam Dubiel
@adamdubiel
Feb 25 2016 12:37
yes, its for hermes metadata, but it's not relative to Kafka root (or doesn't have to) if you use different zks for Kafka nad Hermes, or different connection strings
Stevo Slavić
@sslavic
Feb 25 2016 12:38
ah, kafka.zookeeper.connect.string
Adam Dubiel
@adamdubiel
Feb 25 2016 12:38
yes :)
Stevo Slavić
@sslavic
Feb 25 2016 12:38
cool, thanks
is there some docs, or is code only source of info on exact hermes custom metadata?
layout in zookeeper relative to kafka.zookeeper.connect.string and what is stored in the nodes
Adam Dubiel
@adamdubiel
Feb 25 2016 12:39
there are no docs currently, but i can write it all down if you need this kind of reference
Stevo Slavić
@sslavic
Feb 25 2016 12:40
would be nice, thanks
Adam Dubiel
@adamdubiel
Feb 25 2016 12:40
however it's almost 1:1 with our rest api layout. i will get down to it tomorrow then
Stevo Slavić
@sslavic
Feb 25 2016 12:41
so:
/groups
/topics/.../subscriptions
Adam Dubiel
@adamdubiel
Feb 25 2016 12:42
groups/{group-name}/topics/{topic-name}/subscriptions/{sub name}
there are some branches on each level like status, metrics etc
Stevo Slavić
@sslavic
Feb 25 2016 12:43
clear
Stevo Slavić
@sslavic
Feb 25 2016 14:06
it doesn't seem possible to include custom metadata with message, when publishing through Hermes frontend
would hermes consumers handle well if one was to publish messages directly to Kafka topics with extra custom metadata, with all rest same (message id and timestamp, and payload)?
Adam Dubiel
@adamdubiel
Feb 25 2016 14:15
what kind of metadata? we do have external metadata handling, i.e. add content from the http headers etc
which is later propagated to kafka and consumers
and yes, stuff in meadata: is treated as optional and should be no problem with adding more stuff there, it can even be absent
we have one cluster which only consumes messages from Kafka, which are published directly to Kafka and it works quite well
Stevo Slavić
@sslavic
Feb 25 2016 14:16
ok, good to know
http headers would do
couldn't find it in docs
last one mentions just id and timestamp
Adam Dubiel
@adamdubiel
Feb 25 2016 14:19
yes, docs contain most common cases now, there are some undocumented features :) you can use HermesFrontend.Builder#withHeadersPropagator to add any additional metadata from headers (or any arbitrary if you need)
they will end up in metadata.externalMetadata map
Stevo Slavić
@sslavic
Feb 25 2016 14:21
is this handling always on, or is there a feature flag, or does one have to fork and tune hermes frontend?
Adam Dubiel
@adamdubiel
Feb 25 2016 14:21
currently it's in the API only, but i think it should be moved to configuration, so you could simply add a list of header names to try and read from each request
all in all we will be moving some "fork only" options to normal configuration to make deployment easier and leave the fork option for some really advanced usecases
Stevo Slavić
@sslavic
Feb 25 2016 14:26
would have to fork, create my own main class, use in it pl.allegro.tech.hermes.frontend.HermesFrontend.frontend() factory method, configure builder appropriately and start
but it's good to know, that consumers will gracefully handle extra metadata
Adam Dubiel
@adamdubiel
Feb 25 2016 14:27
yes, or just wait for us to move it to main config :) i can make it work like this in 0.8.5. also Consumers will send those headers to subscribers
Stevo Slavić
@sslavic
Feb 25 2016 14:28
clear