@shimnarkrishnan_twitter
Ad 1) Yes, it does. By default message size is a soft limit, warn log is emitted when message larger than declared topic message size is received.frontend.force.topic.max.message.size flag can be switched to make it a hard limit (Frontend will return
http 413 Payload Too Large status).
Ad 2) Schemas are kept in schema-registry. Zookeeper in Hermes is used as metastore to keep information about topics and subscriptions such as topic name, description, owner, max message size, attached subscriptions, subscription name, endpoint, retry policy, etc.
Ad 3) Didn’t use it but I see it supports HTTP & JSON so at first glance it should be possible to use it on Hermes
@PostMapping
public String testConsume() {
System.out.println("hi");
return "hello consume";
}
@shimnarkrishnan_twitter
Could you describe how did you create the subscription for this endpoint?
Did you create it according to the documentation (https://hermes-pubsub.readthedocs.io/en/latest/user/subscribing/#creating-subscription) ?
You can also check out example in my development environment:
- How subscribing service should work: https://github.com/jewertow/hermes-dev-environment/blob/master/modules/subscriber/src/main/java/com/github/jewertow/App.kt#L28 - it is simple Ktor app, but you can replace it with spring boot
- How to create subscription in Hermes: https://github.com/jewertow/hermes-dev-environment/blob/master/setup-data.sh
There is a README file, you can follow the instructions and easy setup working environment.
@tnagy81 I'm glad I could help
@shimnarkrishnan_twitter
- set message ttl on your subscription to a low value such as 10 seconds
- publish a message on the topic
- after 10 second refresh the subscription in a browser and verify what you see in “Last undelivered message” panel. You should see the reason why message wasn’t delivered.
if you want to track every message by hermes message tracking mechanism you have to install trace storage: https://hermes-pubsub.readthedocs.io/en/latest/configuration/message-tracking/
hermes-management module and ran into an issue because some spring auto configuration. This sounds spring related but actually the cause of it is that spring-mvc is present on the classpath and two frameworks are listening to requests. I have found spring-boot-starter-web in the dependencies of hermes-management. If you are using spring-boot-starter-jersey it is unnecessary to add starter-web because it will hook up spring MVC which you don't need for JAX-RS endpoints. My workaround was adding spring-webmwc to exclusions (I am using maven) of hermes-management dependency. My service and of course the hermes components start without any issue and endpoints are working fine. Though I did not check all the endpoints :DI wanted to make a hermes build without the spring-boot-starter-web dependency to check whether it will be successful with all the test but I did not have time to do it yet.
spring-webmvc stuff just only because of vacations. I'd really like to contribute though. As far as I understand if we handle this as a small issue I don't have to create a ticket. I also want to note that if I remove spring-webmvc then Spring Actuator endpoints (like /health, /info) start to work in hermes-management which is nice to have.
1 reply
@shimnarkrishnan_twitter few years ago we had occasionally problems with docker on different laptops. On one laptops it worked, on others it didn’t, especially after docker updates. Therefore we decided to use Vagrant in our quickstart guide. However, I see that docker is much more stable technology now and our open source users are asking about it, so we are planning to bring it back in Hermes. In the near future we are going to publish new images of Hermes on dockerhub :)
As for Kafka and Hermes. Hermes for Kafka is a client. If you already have a running Kafka cluster on production and you are happy with it then run of Hermes on it should be easy and smooth :)
Hi @bensonmiller, you are the first user who would like to use Hermes with Kafka from Confluent Cloud. So far, we didn’t have that need, but definitely Hermes should be compatible with Confluent Cloud as well. Although, keep in mind that Hermes will still need a Zookeeper cluster to run.
As you noticed, Hermes uses Kafka zookeeper client to manage topics configuration. I will create an issue to replace this old client to something new. When we resolve it, then will see whether something more has to be done to be compatible with Confluent Cloud. Please let me know if this approach is ok for you :)
We have a very special case. Hermes will post messages on topic, but one (old) service will have to read from Kafka directly (akka cluster + sharding).
thx
Last time you heard form me I was stating that spring-webmvc is not a necessary dependency for hermes-manager. I went a bit deeper in this topic and unfortunately (or fortunately) I was wrong. It is needed for the static content of the console. I admit that I don't have any experience with jersey and I thought that jersey starter is enough.
Anyway, I have a few other questions:
- Is it allowed in case of
httpsto define a subscription URL with non standard port? - I'd like to develop a service that sends an automated notification (e.g.: email) in case a subscription is unhealthy. Can you give some basic ideas how to do that, what would be the best practice to regularly check subscriptions health?
- Retries and offset: As far as I understand from the code the kafka offset is committed in a separate thread. Also message processing does not stop when a previous message is in the retry loop. I'd like to understand what happens if there are too many retries and what happens if one hermes-consumer instance fails/shuts down/is killed not gracefully? What are the chances of messages being resent multiple times, losing messages, resending huge amount of already sent messages.
- What happens to the offset commit when there is one or more message that is being retried?
- I assume that retries stay in the message sending queue. Does that mean that if the queue is filled with retries than reading of the topic will stop until retries succeed or expire?
- Is there any automated suspend?
Hello @tnagy81 ,
Thanks for making sure that spring-mvc is required in Hermes-management. As for your questions:
Ad 1. I think it’s possible. Anyway, no one has ever reported such a problem to us, despite the fact that we use https in our Hermes clusters.
Ad 2. Hermes-management exposes an endpoint to check a subscription health. As far as I know we don’t have any documentation for it yet, but take a look on the changelog and UnhealthyEndpoint class
I encourage you to consider use a monitoring system which periodically sends requests to hermes-management to check unhealthy subscriptions.
Very helpful is measuring a lag on Kafka consumers group. To do that you can use a dedicated tool, for example Burrow tool. Moreover Hermes-management has a SubscriptionLagSource interface. You can provide an implementation for it which verifies lag on Kafka consumers group. This interface is used by UnhealthyEndpoint class.
Ad 3. Hermes-consumers uses memory queue for sending messages. When this queue is full due to high number of retried messages then Hermes waits with reading messages from Kafka until there is a space in the memory queue. You can set the size of this memory queue by consumer.inflight.size parameter. By the way when subscriber returns only errors then most likely Hermes will lower speed of sending/retrying messages. See rate-limiting for more information.
Hermes-consumers commits offset only when message was delivered successfully or its TTL expired (retries didn’t helped). In the meantime when an instance of Hermes-consumers is killed then other instance takes its work due to Kafka-consumers autorebalance mechanism. The working instance starts sending messages from the last committed offset, so there is a chance that some messages will be delivered more than one time. This is not the only one place in Hermes when message duplication can occur. In summary Hermes has „at-least-once” message delivery guarantee.
Ad a) message offset is not committed until the message is delivered successfully or its TTL expires and is dropped
Ad b) Yes, exactly
Ad c) There is a rate-limit mechanism which slows down message sending to a few messages per minute when subscriber doesn’t receive messages successfully. However, Hermes doesn’t fully stop sending messages. If you want to stop message sending to zero then you have to suspend the subscription manually. More info about rate-limiter you can find here
I hope that my answers are helpful to you :)
We are taking a look at Hermes as an alternative to Direct Kafka connectivity.
I have a couple of questions if I may ask:
1) There doesn't seem to be a way to specify a partitioning key which means that as a consumer, you have no way to be guaranteed ordering when reading from a specific partition. Is there any plans for providing such a feature?
2) How are consumer groups managed within Hermes?
3) Hermes manages offset for consumer but seems to be expecting acknowledgement synchronously on the subscription push. We have a use-case where the work is distributed within an internal cluster and processing of the actual event may not occur on the node receiving the push http call. There would be a need for submitting the offset commit when processing is actually done. I wasn't able to find anything on the documentation. Is there such a feature under the hood?
thanks for your help
Hello @cywattebled
Nice to hear that you are considering using Hermes. Answers to your questions:
Ad 1. You are right, Hermes currently uses round-robin partitioning. It works like this because Hermes uses HTTP as communication layer for producers and subscribers. In HTTP some messages can be retried asynchronously due to errors such as 5xx. Besides, Hermes works in many instances, for example Hermes-consumers can have many instances. I don’t see how this module can easily coordinate order of messages. However, if you have an idea how this should work, then we are open on propositions.
Ad 2.
Hermes-consumer module is responsible for handling subscriptions. Each Hermes subscription has a dedicated Kafka consumer using Kafka consumer group name combined from Hermes subscription name. You can run many instances of Hermes-consumers. You can also configure how many Hermes-consumers instances should handle single subscription. By default two Hermes-consumers handles the subscription. This can be configured by consumer.workload.consumers.per.subscription parameter. If you have two Hermes-consumers handling the same subscription, then they are using the same Kafka consumer group for this subscription meaning that Kafka consumers rebalance protocol is in use. Take a look on this article as it might be helpful to you as well.
Ad 3. Hermes doesn’t have that feature. I see two solutions for this:
- You can increase subscription timeout and wait for the response from a subscriber
- Subscriber can acknowledge immediately after it receives message from Hermes and send a message on a different topic when finish processing.
I hope my answers were helpful,
Cheers
Łukasz
Thanks for your response.
1) Well actually, The issue with the partitioning key is that actually on the producer side. We have topics that would require data to be ordered per partition, and considering a hybrid solution where services that needs specifics Kafka features would consume directly from Kafka (and expect message to be ordered within a single partition) but with producers writing through Hermes. I understand that the consuming logic would be complicated due to the push mechanism but providing an option to provide a partitioning key shouldn't have an impact no? Without this feature, we'd be stuck to using Kafka direct.
2) Thanks, it confirms my understanding of how Hermes works. Nice article by the way :)
3)Yeah that's definitely not ideal. If we had a fix for 1), we could envision consuming direct from Kafka for this specific consumer group and managing the offset manually as needed.
@druminski
Thanks for the response.
Yes I agree, what I mean by ordered is ordered received by the Kafka broker but our main focus is that all event relating to the same entity land on the same partition.
I guess the partition-key header would indeed fix my issue. I was envisioning something like:
curl -X POST -H "Content-Type: application/json" http://hermes-frontend/topics/group.topic -d '{"key": "terranova-123456", "message": "Hello world!"}' where key is optional but the header would work too :).
Other question if I may, is there a way to set the partition number on the topic creation call? I don't see that option in the documentation. I guess Topics should really be created before hand and not necessarily by producers but just out of curiosity.
Thanks.
@cywattebled
The partition number can be set globally in Hermes-management by topics.partitions parameter. Hermes-management for every topic creation request creates Kafka topic using mentioned parameter which default value is 10 partitions. Hermes-management is built on spring-boot, so you can provide its configuration in many ways.
Nice to hear that the partition-key header will fix your issue. It’s is more generic solution than a key field in the body message as you don’t need to change the message schema or wrap message with key field, so already existing producers don’t have to change anything. I will consult this feature with my team and if everything is fine we will implement it soon. Let’s stay in touch.
Thanks for the details last time! Just few more questions:
- Let's assume that there is only one message in the retry queue and many other messages are successfully delivered during the TTL of the "faulty" message. If that instance of hermes fails for some reason and another one takes over then new instance will redeliver all successful messages that came after the failing offset for that partition (!), failing message TTL will restart, failing message retry counter will be reset, right?
- Can I restart a suspended consumer from the current offset and not from where it was stopped? I assume not, because broker is aware of the consumer group's committed offsets but maybe you have some tricks here.
- Can I redeliver messages that are older than the consumer? E.g. I just created a consumer and redeliver from a timestamp which was yesterday.
- If I remove a subscription and later create another with the same name then the new subscription will start to consume from the latest committed offset of the deleted one because it will get the same group ID, right?
- Right, this is one of the reasons why Hermes has “At least once delivery". Let’s say we have two partitions and two instances belonging to the consumer group. In partition one there is a message X that is somehow faulty and consumer 1 will fail to deliver it and retry later. In partition two there are other messages and consumer 2 will freely deliver them, while consumer 1 retries the message X. If consumer 1 dies and somehow gets replaced as member of consumer group and receives the same partition to handle, it will try to send the message X again. The retry timer/counter will be reset. The other messages will be unaffected - offset of the second partition was successfully committed, no need for resending.
- The current offset for that subscription will be locked upon suspending the subscription. Upon reactivating it, Hermes will continue consuming the messages from that point. You can manually change the value of the current offset - for example to the value of the end offset, effectively skipping the messages published in meanwhile - via the Kafka API. Another way is to activate the subscription and perform a retransmission on it via the Hermes UI. Hermes retransmission under the hood modifies current offsets positions.
- Yes, you can use the retransmission mechanism for that and retransmit the messages from the point in the past, effectively moving the current offset back. Of course, as long as the messages have not been deleted due of retention period. Here too you can perform the retransmission via the Hermes UI.
- Yes, since the consumer group id is created based on the subscription name, the id will be the same and consequently the offsets will remain. The “new” subscription will continue consuming at the point the deleted one finished.
I had a weird experience today with the consumer module. I am running two instances in docker containers. I registered 4 subscriptions using the management module's API (we created our own UI with limited accessibility) and two of them did not want to change state from PENDING to ACTIVE. I checked the logs and it seemed that the two instances were able to distribute the 4 subscriptions equally and all of the subscriptions, event the PENDING (!) were consuming kafka events. Maybe it is a coincidence but the two that were in PENDING state were assigned to the same instance. Still both were working and consuming events. I restarted that instance of hermes-consumer and in less than a minute one of the PENDING subscription changed to ACTIVE state and after a few more minutes (significantly more than the first) the one remaining subscription was also in ACTIVE state. Do you have any idea what could cause this? It is pretty hard do debug this and I don't really know where to start. I am still on version 1.4.4 but I am planning to upgrade today. Consumers / subscription is set to 1.
2 replies
2 replies
- It will be extremely helpful to know the possibility of using SSL in Kafka connection mentioned already by @anthonynathan .
- Is it possible to use Hermes with already created Kafka topics? What additional config should we provide to zk in such case? We have strict policy of topics management and the only one allowed to create anything is terraform :(
Looking forward for some answers and thanks for your time.