spooz on consumer-groups-description
SKYEDEN-223 | metadata (compare)
spooz on SKYEDEN-223-consumer-groups-desc
spooz on SKYEDEN-223-consumer-groups-desc
SKYEDEN-223 | metadata (compare)
spooz on consumer-groups-description
consumer groups desc (compare)
spooz on SKYEDEN-223-consumer-groups-desc
spooz on consumer-groups-description
spooz on SKYEDEN-223-consumer-groups-desc
SKYEDEN-223 | consumer groups d… (compare)
adamdubiel on hermes-0.15.7
adamdubiel on master
Release version: 0.15.7 (compare)
druminski on changelogUpdate
druminski on master
Updating changelog (#980) (compare)
I brought up Hermes using “vagrant up” and it successfully came up . Got the below links to access the components too.
UI & Management: http://10.10.10.10:8090
Frontend: http://10.10.10.10:8080
Graphite: http://10.10.10.10:8082
But when I actually paste the links in a browser , it says page isn’t working . Has something changed from the documentation ?
@tnagy81 Hi,
When we started implementing Hermes six years ago, spring-boot wasn’t so popular then. Hermes-management at the beginning was written in Jersey framework which in internals uses hk2 as a light-weight dependency injection framework. So we started with hk2 in hermes-management and decided to use it in hermes-{frontend, consumers} as well.
When spring-boot gained popularity we’ve rewritten hermes-management to use it. However we didn’t use spring-boot in hermes-{frontend, consumers} due to performance reasons.
Hermes-frontend has to be very fast. Therefore we’ve written it on a raw HTTP handlers in Undertow server. spring-boot-starter-web adds additional layers such as servlets implementation and custom filters. They add additional latency for request processing and we don’t want that.
So, as you see there are two reasons, historical and performance, why we don’t use sprint-boot in hermes-{frontend, consumers}. Today, replace of hk2 to spring di framework in hermes-{frontend, consumers} is not a trivial task as it requires rewrite of a lot of code. However, it would be nice to have the same dependency injection framework and configuration method in all Hermes modules. We don’t do this task in our team as we have other priorities right now but in the future it would be nice to do it.
@shimnarkrishnan_twitter
"Also what about this URL of repository? Which URL is expected?"
Both schema repo and schema registry provides http API to manage schemas so you have to provide the server URL where your repository is running.
This URL must have the following form: http://<host>:<port>. By default it is http://localhost:8888/.
"I couldn't figure out where to set the below configurations schema.repository.type: schema_registry"
You have to add these properties to your configuration files. You can look at my development hermes configuration (based on docker containers) where I have configured schema-registry: https://github.com/jewertow/hermes-dev-environment.
Schema-registry configurations:
Schema-registry container configuration: https://github.com/jewertow/hermes-dev-environment/blob/master/docker-compose.yml#L31
@shimnarkrishnan_twitter
Ad 1) Yes, it does. By default message size is a soft limit, warn
log is emitted when message larger than declared topic message size is received.frontend.force.topic.max.message.size
flag can be switched to make it a hard limit (Frontend will return
http 413 Payload Too Large
status).
Ad 2) Schemas are kept in schema-registry. Zookeeper in Hermes is used as metastore to keep information about topics and subscriptions such as topic name, description, owner, max message size, attached subscriptions, subscription name, endpoint, retry policy, etc.
Ad 3) Didn’t use it but I see it supports HTTP & JSON so at first glance it should be possible to use it on Hermes
@shimnarkrishnan_twitter
Could you describe how did you create the subscription for this endpoint?
Did you create it according to the documentation (https://hermes-pubsub.readthedocs.io/en/latest/user/subscribing/#creating-subscription) ?
You can also check out example in my development environment:
@tnagy81 I'm glad I could help
@shimnarkrishnan_twitter
if you want to track every message by hermes message tracking mechanism you have to install trace storage: https://hermes-pubsub.readthedocs.io/en/latest/configuration/message-tracking/
hermes-management
module and ran into an issue because some spring auto configuration. This sounds spring related but actually the cause of it is that spring-mvc
is present on the classpath and two frameworks are listening to requests. I have found spring-boot-starter-web
in the dependencies of hermes-management
. If you are using spring-boot-starter-jersey
it is unnecessary to add starter-web because it will hook up spring MVC which you don't need for JAX-RS endpoints. My workaround was adding spring-webmwc
to exclusions (I am using maven) of hermes-management
dependency. My service and of course the hermes components start without any issue and endpoints are working fine. Though I did not check all the endpoints :Dspring-webmvc
stuff just only because of vacations. I'd really like to contribute though. As far as I understand if we handle this as a small issue I don't have to create a ticket. I also want to note that if I remove spring-webmvc
then Spring Actuator endpoints (like /health
, /info
) start to work in hermes-management
which is nice to have.
@shimnarkrishnan_twitter few years ago we had occasionally problems with docker on different laptops. On one laptops it worked, on others it didn’t, especially after docker updates. Therefore we decided to use Vagrant in our quickstart guide. However, I see that docker is much more stable technology now and our open source users are asking about it, so we are planning to bring it back in Hermes. In the near future we are going to publish new images of Hermes on dockerhub :)
As for Kafka and Hermes. Hermes for Kafka is a client. If you already have a running Kafka cluster on production and you are happy with it then run of Hermes on it should be easy and smooth :)
Hi @bensonmiller, you are the first user who would like to use Hermes with Kafka from Confluent Cloud. So far, we didn’t have that need, but definitely Hermes should be compatible with Confluent Cloud as well. Although, keep in mind that Hermes will still need a Zookeeper cluster to run.
As you noticed, Hermes uses Kafka zookeeper client to manage topics configuration. I will create an issue to replace this old client to something new. When we resolve it, then will see whether something more has to be done to be compatible with Confluent Cloud. Please let me know if this approach is ok for you :)
https
to define a subscription URL with non standard port?Hello @tnagy81 ,
Thanks for making sure that spring-mvc is required in Hermes-management. As for your questions:
Ad 1. I think it’s possible. Anyway, no one has ever reported such a problem to us, despite the fact that we use https in our Hermes clusters.
Ad 2. Hermes-management exposes an endpoint to check a subscription health. As far as I know we don’t have any documentation for it yet, but take a look on the changelog and UnhealthyEndpoint class
I encourage you to consider use a monitoring system which periodically sends requests to hermes-management to check unhealthy subscriptions.
Very helpful is measuring a lag on Kafka consumers group. To do that you can use a dedicated tool, for example Burrow tool. Moreover Hermes-management has a SubscriptionLagSource interface. You can provide an implementation for it which verifies lag on Kafka consumers group. This interface is used by UnhealthyEndpoint class.
Ad 3. Hermes-consumers uses memory queue for sending messages. When this queue is full due to high number of retried messages then Hermes waits with reading messages from Kafka until there is a space in the memory queue. You can set the size of this memory queue by consumer.inflight.size parameter. By the way when subscriber returns only errors then most likely Hermes will lower speed of sending/retrying messages. See rate-limiting for more information.
Hermes-consumers commits offset only when message was delivered successfully or its TTL expired (retries didn’t helped). In the meantime when an instance of Hermes-consumers is killed then other instance takes its work due to Kafka-consumers autorebalance mechanism. The working instance starts sending messages from the last committed offset, so there is a chance that some messages will be delivered more than one time. This is not the only one place in Hermes when message duplication can occur. In summary Hermes has „at-least-once” message delivery guarantee.
Ad a) message offset is not committed until the message is delivered successfully or its TTL expires and is dropped
Ad b) Yes, exactly
Ad c) There is a rate-limit mechanism which slows down message sending to a few messages per minute when subscriber doesn’t receive messages successfully. However, Hermes doesn’t fully stop sending messages. If you want to stop message sending to zero then you have to suspend the subscription manually. More info about rate-limiter you can find here
I hope that my answers are helpful to you :)
Hello @cywattebled
Nice to hear that you are considering using Hermes. Answers to your questions:
Ad 1. You are right, Hermes currently uses round-robin partitioning. It works like this because Hermes uses HTTP as communication layer for producers and subscribers. In HTTP some messages can be retried asynchronously due to errors such as 5xx. Besides, Hermes works in many instances, for example Hermes-consumers can have many instances. I don’t see how this module can easily coordinate order of messages. However, if you have an idea how this should work, then we are open on propositions.
Ad 2.
Hermes-consumer module is responsible for handling subscriptions. Each Hermes subscription has a dedicated Kafka consumer using Kafka consumer group name combined from Hermes subscription name. You can run many instances of Hermes-consumers. You can also configure how many Hermes-consumers instances should handle single subscription. By default two Hermes-consumers handles the subscription. This can be configured by consumer.workload.consumers.per.subscription
parameter. If you have two Hermes-consumers handling the same subscription, then they are using the same Kafka consumer group for this subscription meaning that Kafka consumers rebalance protocol is in use. Take a look on this article as it might be helpful to you as well.
Ad 3. Hermes doesn’t have that feature. I see two solutions for this:
I hope my answers were helpful,
Cheers
Łukasz