Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 16:19

    spooz on consumer-groups-description

    SKYEDEN-223 | metadata (compare)

  • Jan 31 2019 16:18

    spooz on SKYEDEN-223-consumer-groups-desc

    (compare)

  • Jan 31 2019 16:07

    spooz on SKYEDEN-223-consumer-groups-desc

    SKYEDEN-223 | metadata (compare)

  • Jan 31 2019 13:27

    spooz on consumer-groups-description

    consumer groups desc (compare)

  • Jan 31 2019 13:26

    spooz on SKYEDEN-223-consumer-groups-desc

    (compare)

  • Jan 31 2019 13:25

    spooz on consumer-groups-description

    (compare)

  • Jan 31 2019 13:20

    spooz on SKYEDEN-223-consumer-groups-desc

    SKYEDEN-223 | consumer groups d… (compare)

  • Jan 31 2019 12:54
    dankraw commented #981
  • Jan 31 2019 12:38
    dankraw commented #981
  • Jan 31 2019 12:17
    druminski labeled #981
  • Jan 31 2019 12:17
    druminski labeled #981
  • Jan 31 2019 12:15
    dankraw opened #981
  • Jan 29 2019 14:28
    faderskd starred allegro/hermes
  • Jan 29 2019 14:28
    faderskd starred allegro/hermes
  • Jan 25 2019 17:45

    adamdubiel on hermes-0.15.7

    (compare)

  • Jan 25 2019 17:45

    adamdubiel on master

    Release version: 0.15.7 (compare)

  • Jan 25 2019 17:45

    druminski on changelogUpdate

    (compare)

  • Jan 25 2019 17:45

    druminski on master

    Updating changelog (#980) (compare)

  • Jan 25 2019 17:45
    druminski closed #980
  • Jan 25 2019 16:35
    druminski opened #980
Łukasz Drumiński
@druminski
Hi @tnagy81, thanks for the information, if all tests will work without spring-webmvc then I agree with you that we can remove it. I keep my fingers crossed for your test and pull request :)
Tibor Nagy
@tnagy81
@druminski :D
Tibor Nagy
@tnagy81
@druminski Hi, I just want to let you know that there is a high chance that I won't have time to work the spring-webmvc stuff just only because of vacations. I'd really like to contribute though. As far as I understand if we handle this as a small issue I don't have to create a ticket. I also want to note that if I remove spring-webmvc then Spring Actuator endpoints (like /health, /info) start to work in hermes-management which is nice to have.
1 reply
Łukasz Ciołecki
@lciolecki
Hi!
I have question for allegro/hermes#682. Does hermes support standard confluent method serialization / deserialization message?
Łukasz Drumiński
@druminski
@tnagy81 ok, imo this issue can wait as you are the only user who reported it :) However, if you need help with it then please let us know as we can work on it as well. In the meantime happy vacations!
I talked with @lciolecki on priv about subject which he raised.
Shimna
@shimnarkrishnan_twitter
What was the reason hermes stopped docker support and moved to vagrant? We are planning to use hermes with docker . What challenges should we foresee?
Shimna
@shimnarkrishnan_twitter
Also, we already have a kafka cluster running in our production . We plan to use that along with hermes. What challenges can come in our way here?
Łukasz Drumiński
@druminski

@shimnarkrishnan_twitter few years ago we had occasionally problems with docker on different laptops. On one laptops it worked, on others it didn’t, especially after docker updates. Therefore we decided to use Vagrant in our quickstart guide. However, I see that docker is much more stable technology now and our open source users are asking about it, so we are planning to bring it back in Hermes. In the near future we are going to publish new images of Hermes on dockerhub :)

As for Kafka and Hermes. Hermes for Kafka is a client. If you already have a running Kafka cluster on production and you are happy with it then run of Hermes on it should be easy and smooth :)

RaulMarina
@RaulMarina
Hi, I have my springboot application running on a docker container using 8080, I just have set up the Hermes environment using https://hermes-pubsub.readthedocs.io/en/latest/quickstart/#docker. Both application and hermes are working fine but as I needed to change the hermes-frontend port (8083) I am getting "Error: socket hang up" whenever I try to publish. What am I doing wrong?
RaulMarina
@RaulMarina
nevermind
Benson Miller
@bensonmiller
Hello Allegro team. I'm interested in moving my workload to Confluent Cloud, but Confluent Cloud doesn't expose Zk, which seems to preclude Hermes operation. Is support for Confluent Cloud (or KIP-500 mode) part of the Hermes roadmap? I'd appreciate any insight you can share. Thank you.
Łukasz Drumiński
@druminski

Hi @bensonmiller, you are the first user who would like to use Hermes with Kafka from Confluent Cloud. So far, we didn’t have that need, but definitely Hermes should be compatible with Confluent Cloud as well. Although, keep in mind that Hermes will still need a Zookeeper cluster to run.

As you noticed, Hermes uses Kafka zookeeper client to manage topics configuration. I will create an issue to replace this old client to something new. When we resolve it, then will see whether something more has to be done to be compatible with Confluent Cloud. Please let me know if this approach is ok for you :)

Benson Miller
@bensonmiller
Hello @druminski - Thank you for the response. This sounds like exactly the right approach to me. Thank you.
cisrudlow
@cisrudlow
Hi (maybe stupid question, but I haven't found an answer in the documentation). I am curious how partitioning in Hermes is resolved. I expect you are using the default partitioner, but how do you indicate (in Hermes) which partition the message should go to (partition number or key).
We have a very special case. Hermes will post messages on topic, but one (old) service will have to read from Kafka directly (akka cluster + sharding).
thx
Łukasz Drumiński
@druminski
Hi @cisrudlow, when Hermes publishes message to Kafka, it doesn't set a partition number and a key. It means that the partition will be assigned in a round-robin fashion.
Aditya Todkar
@aditodkar
Hello everyone, I am Aditya from India. I would like to contribute to allegro/hermes as a part of hacktoberfest. I am familiar with Kafka and have worked on JAVA extensively. I think I will get good grasp on Kafka by contributing to this project. Can someone please help me get started and any resources which you can share for Kafka. Currently I only know basics of Kafka. Thanks :)
Aditya Todkar
@aditodkar
Hello everyone. Is this right channel for hackoctoberfest ?
Łukasz Drumiński
@druminski
@aditodkar Welcome, very nice that you would like to contribute to Hermes project due to Hacktoberfest event. Please take a look on issues with "hacktoberfest" label. Few of them are still available, for example #1269 might be interesting for you. If you find some issue interesting then please let us know directly in the issue.
Tibor Nagy
@tnagy81
Hi @druminski,
Last time you heard form me I was stating that spring-webmvc is not a necessary dependency for hermes-manager. I went a bit deeper in this topic and unfortunately (or fortunately) I was wrong. It is needed for the static content of the console. I admit that I don't have any experience with jersey and I thought that jersey starter is enough.
Anyway, I have a few other questions:
  1. Is it allowed in case of https to define a subscription URL with non standard port?
  2. I'd like to develop a service that sends an automated notification (e.g.: email) in case a subscription is unhealthy. Can you give some basic ideas how to do that, what would be the best practice to regularly check subscriptions health?
  3. Retries and offset: As far as I understand from the code the kafka offset is committed in a separate thread. Also message processing does not stop when a previous message is in the retry loop. I'd like to understand what happens if there are too many retries and what happens if one hermes-consumer instance fails/shuts down/is killed not gracefully? What are the chances of messages being resent multiple times, losing messages, resending huge amount of already sent messages.
    • What happens to the offset commit when there is one or more message that is being retried?
    • I assume that retries stay in the message sending queue. Does that mean that if the queue is filled with retries than reading of the topic will stop until retries succeed or expire?
    • Is there any automated suspend?
Łukasz Drumiński
@druminski

Hello @tnagy81 ,

Thanks for making sure that spring-mvc is required in Hermes-management. As for your questions:

Ad 1. I think it’s possible. Anyway, no one has ever reported such a problem to us, despite the fact that we use https in our Hermes clusters.

Ad 2. Hermes-management exposes an endpoint to check a subscription health. As far as I know we don’t have any documentation for it yet, but take a look on the changelog and UnhealthyEndpoint class

I encourage you to consider use a monitoring system which periodically sends requests to hermes-management to check unhealthy subscriptions.

Very helpful is measuring a lag on Kafka consumers group. To do that you can use a dedicated tool, for example Burrow tool. Moreover Hermes-management has a SubscriptionLagSource interface. You can provide an implementation for it which verifies lag on Kafka consumers group. This interface is used by UnhealthyEndpoint class.

Ad 3. Hermes-consumers uses memory queue for sending messages. When this queue is full due to high number of retried messages then Hermes waits with reading messages from Kafka until there is a space in the memory queue. You can set the size of this memory queue by consumer.inflight.size parameter. By the way when subscriber returns only errors then most likely Hermes will lower speed of sending/retrying messages. See rate-limiting for more information.

Hermes-consumers commits offset only when message was delivered successfully or its TTL expired (retries didn’t helped). In the meantime when an instance of Hermes-consumers is killed then other instance takes its work due to Kafka-consumers autorebalance mechanism. The working instance starts sending messages from the last committed offset, so there is a chance that some messages will be delivered more than one time. This is not the only one place in Hermes when message duplication can occur. In summary Hermes has „at-least-once” message delivery guarantee.

Ad a) message offset is not committed until the message is delivered successfully or its TTL expires and is dropped

Ad b) Yes, exactly

Ad c) There is a rate-limit mechanism which slows down message sending to a few messages per minute when subscriber doesn’t receive messages successfully. However, Hermes doesn’t fully stop sending messages. If you want to stop message sending to zero then you have to suspend the subscription manually. More info about rate-limiter you can find here

I hope that my answers are helpful to you :)

cywattebled
@cywattebled
Hello guys,
We are taking a look at Hermes as an alternative to Direct Kafka connectivity.
I have a couple of questions if I may ask:
1) There doesn't seem to be a way to specify a partitioning key which means that as a consumer, you have no way to be guaranteed ordering when reading from a specific partition. Is there any plans for providing such a feature?
2) How are consumer groups managed within Hermes?
3) Hermes manages offset for consumer but seems to be expecting acknowledgement synchronously on the subscription push. We have a use-case where the work is distributed within an internal cluster and processing of the actual event may not occur on the node receiving the push http call. There would be a need for submitting the offset commit when processing is actually done. I wasn't able to find anything on the documentation. Is there such a feature under the hood?
thanks for your help
Łukasz Drumiński
@druminski

Hello @cywattebled

Nice to hear that you are considering using Hermes. Answers to your questions:

Ad 1. You are right, Hermes currently uses round-robin partitioning. It works like this because Hermes uses HTTP as communication layer for producers and subscribers. In HTTP some messages can be retried asynchronously due to errors such as 5xx. Besides, Hermes works in many instances, for example Hermes-consumers can have many instances. I don’t see how this module can easily coordinate order of messages. However, if you have an idea how this should work, then we are open on propositions.

Ad 2.

Hermes-consumer module is responsible for handling subscriptions. Each Hermes subscription has a dedicated Kafka consumer using Kafka consumer group name combined from Hermes subscription name. You can run many instances of Hermes-consumers. You can also configure how many Hermes-consumers instances should handle single subscription. By default two Hermes-consumers handles the subscription. This can be configured by consumer.workload.consumers.per.subscription parameter. If you have two Hermes-consumers handling the same subscription, then they are using the same Kafka consumer group for this subscription meaning that Kafka consumers rebalance protocol is in use. Take a look on this article as it might be helpful to you as well.

Ad 3. Hermes doesn’t have that feature. I see two solutions for this:

  1. You can increase subscription timeout and wait for the response from a subscriber
  2. Subscriber can acknowledge immediately after it receives message from Hermes and send a message on a different topic when finish processing.

I hope my answers were helpful,
Cheers
Łukasz

cywattebled
@cywattebled
@druminski
Thanks for your response.
1) Well actually, The issue with the partitioning key is that actually on the producer side. We have topics that would require data to be ordered per partition, and considering a hybrid solution where services that needs specifics Kafka features would consume directly from Kafka (and expect message to be ordered within a single partition) but with producers writing through Hermes. I understand that the consuming logic would be complicated due to the push mechanism but providing an option to provide a partitioning key shouldn't have an impact no? Without this feature, we'd be stuck to using Kafka direct.
2) Thanks, it confirms my understanding of how Hermes works. Nice article by the way :)
3)Yeah that's definitely not ideal. If we had a fix for 1), we could envision consuming direct from Kafka for this specific consumer group and managing the offset manually as needed.
Łukasz Drumiński
@druminski
@cywattebled I think we can implement a support for a „partition-key” HTTP header in Hermes. Messages with the same „partition-key” HTTP header will be sent to the same Kafka topic partition. However, this feature doesn’t guarantee an order of messages. It’s because HTTP clients and Hermes-frontend (HTTP server) work in concurrent environment - many instances, threads, etc. Therefore message A can be published on Kafka topic partition X later than message B even when message A was send faster than message B to Hermes. Please let me know whether this behavior is ok to you.
cywattebled
@cywattebled

@druminski
Thanks for the response.

Yes I agree, what I mean by ordered is ordered received by the Kafka broker but our main focus is that all event relating to the same entity land on the same partition.

I guess the partition-key header would indeed fix my issue. I was envisioning something like:
curl -X POST -H "Content-Type: application/json" http://hermes-frontend/topics/group.topic -d '{"key": "terranova-123456", "message": "Hello world!"}' where key is optional but the header would work too :).

Other question if I may, is there a way to set the partition number on the topic creation call? I don't see that option in the documentation. I guess Topics should really be created before hand and not necessarily by producers but just out of curiosity.
Thanks.

Łukasz Drumiński
@druminski

@cywattebled
The partition number can be set globally in Hermes-management by topics.partitions parameter. Hermes-management for every topic creation request creates Kafka topic using mentioned parameter which default value is 10 partitions. Hermes-management is built on spring-boot, so you can provide its configuration in many ways.

Nice to hear that the partition-key header will fix your issue. It’s is more generic solution than a key field in the body message as you don’t need to change the message schema or wrap message with key field, so already existing producers don’t have to change anything. I will consult this feature with my team and if everything is fine we will implement it soon. Let’s stay in touch.

Łukasz Drumiński
@druminski
@cywattebled we will introduce "partition-key" http header very soon. You can follow #1298
cywattebled
@cywattebled
@druminski Thanks for the feedback. Sorry for the late feedback. This would work. We are starting a PoC with Hermes in the next couple of weeks. Any rough idea on when this feature would be available?
Łukasz Drumiński
@druminski
@cywattebled nice to hear that you are starting PoC with Hermes in few weeks. As for the feature "partition-key" http header, we already implemented it in the newest version of Hermes 1.7.0 :)
Tibor Nagy
@tnagy81
Hi @druminski,
Thanks for the details last time! Just few more questions:
  1. Let's assume that there is only one message in the retry queue and many other messages are successfully delivered during the TTL of the "faulty" message. If that instance of hermes fails for some reason and another one takes over then new instance will redeliver all successful messages that came after the failing offset for that partition (!), failing message TTL will restart, failing message retry counter will be reset, right?
  2. Can I restart a suspended consumer from the current offset and not from where it was stopped? I assume not, because broker is aware of the consumer group's committed offsets but maybe you have some tricks here.
  3. Can I redeliver messages that are older than the consumer? E.g. I just created a consumer and redeliver from a timestamp which was yesterday.
  4. If I remove a subscription and later create another with the same name then the new subscription will start to consume from the latest committed offset of the deleted one because it will get the same group ID, right?
mareck
@mareckmareck
@tnagy81
  1. Right, this is one of the reasons why Hermes has “At least once delivery". Let’s say we have two partitions and two instances belonging to the consumer group. In partition one there is a message X that is somehow faulty and consumer 1 will fail to deliver it and retry later. In partition two there are other messages and consumer 2 will freely deliver them, while consumer 1 retries the message X. If consumer 1 dies and somehow gets replaced as member of consumer group and receives the same partition to handle, it will try to send the message X again. The retry timer/counter will be reset. The other messages will be unaffected - offset of the second partition was successfully committed, no need for resending.
  2. The current offset for that subscription will be locked upon suspending the subscription. Upon reactivating it, Hermes will continue consuming the messages from that point. You can manually change the value of the current offset - for example to the value of the end offset, effectively skipping the messages published in meanwhile - via the Kafka API. Another way is to activate the subscription and perform a retransmission on it via the Hermes UI. Hermes retransmission under the hood modifies current offsets positions.
  3. Yes, you can use the retransmission mechanism for that and retransmit the messages from the point in the past, effectively moving the current offset back. Of course, as long as the messages have not been deleted due of retention period. Here too you can perform the retransmission via the Hermes UI.
  4. Yes, since the consumer group id is created based on the subscription name, the id will be the same and consequently the offsets will remain. The “new” subscription will continue consuming at the point the deleted one finished.
Tibor Nagy
@tnagy81
Thank you @mareckmareck !
Tibor Nagy
@tnagy81
Hi everyone! First of all I wish you a Happy New Year!
I had a weird experience today with the consumer module. I am running two instances in docker containers. I registered 4 subscriptions using the management module's API (we created our own UI with limited accessibility) and two of them did not want to change state from PENDING to ACTIVE. I checked the logs and it seemed that the two instances were able to distribute the 4 subscriptions equally and all of the subscriptions, event the PENDING (!) were consuming kafka events. Maybe it is a coincidence but the two that were in PENDING state were assigned to the same instance. Still both were working and consuming events. I restarted that instance of hermes-consumer and in less than a minute one of the PENDING subscription changed to ACTIVE state and after a few more minutes (significantly more than the first) the one remaining subscription was also in ACTIVE state. Do you have any idea what could cause this? It is pretty hard do debug this and I don't really know where to start. I am still on version 1.4.4 but I am planning to upgrade today. Consumers / subscription is set to 1.
2 replies
anumadan
@anumadan
Hi everyone! I am starting to explore Hermes and need to assess the resource consumption in terms of minimum requirements and the bench-marking statistics. Is there any information available anywhere or though any of your personal experiences, kindly help with some inputs.
2 replies
anthonynathan
@anthonynathan
Hi. I'd like to know if there is a way to configure Hermes to connect to Kafka using SSL. I need to configure the Kafka producers and consumers to use a truststore and a keystore file and password to connect. Thank you.
Michał
@eQuitess
Hi all!
  1. It will be extremely helpful to know the possibility of using SSL in Kafka connection mentioned already by @anthonynathan .
  2. Is it possible to use Hermes with already created Kafka topics? What additional config should we provide to zk in such case? We have strict policy of topics management and the only one allowed to create anything is terraform :(
    Looking forward for some answers and thanks for your time.
Łukasz Drumiński
@druminski

Hello @anthonynathan and @eQuitess,

So far we have tested very well basic authorization between kafka-client and kafka broker. In this authorization traffic is not encrypted but in order to process messages from Kafka you need to introduce a username and password, so you avoid a scenario that an unauthorized user publishes or reads message from Kafka directly. For more information take a look at this description in changelog.

As for SSL between Kafka-client and Kafka we didn’t test it. Probably some changes also need to be introduced.

„Is it possible to use Hermes with already created Kafka topics?” - it depends. If name of those topics are in Hermes naming convention (<groupName>.<topicName>) then this might work (actually we fave a similar scenario in one of our environments in which we only use Hermes-consumers module, Kafka topics are there managed via Kafka scripts). Also you need to be aware of Hermes message internal format.

5 replies
Michał
@eQuitess

Thanks @druminski ! One more question.

Considering that we will handle proper naming of topics, will Hermes create proper metadata in Zookeeper and omit creating topics in Kafka? Or should we take care of it ourselves? If I understand correctly, Hermes store info about groups and topics in zookeeper, and I am afraid of missing some configuration steps in case of using topics not created by Hermes itself.

Łukasz Drumiński
@druminski
@eQuitess if a topic exists in Kafka but doesn't exist in Hermes, then you still need to create it in Hermes via UI or direct REST HTTP request to hermes-management module. Hermes-management omits topic creation in Kafka if it exists and creates only topic metadata in Hermes-zookeeper. In the end in logs you should see this kind of entry: "Skipping creation of topic {} on brokers, topic already exists"
Michał
@eQuitess
Thank you @druminski ! We will give it a try.
Soumya Ranjan Pattanaik
@spattanaik75
Hi Team, Could you please verify what's the status of dropping zookeeper completely out of hermes ?
1 reply