Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 16:19

    spooz on consumer-groups-description

    SKYEDEN-223 | metadata (compare)

  • Jan 31 2019 16:18

    spooz on SKYEDEN-223-consumer-groups-desc

    (compare)

  • Jan 31 2019 16:07

    spooz on SKYEDEN-223-consumer-groups-desc

    SKYEDEN-223 | metadata (compare)

  • Jan 31 2019 13:27

    spooz on consumer-groups-description

    consumer groups desc (compare)

  • Jan 31 2019 13:26

    spooz on SKYEDEN-223-consumer-groups-desc

    (compare)

  • Jan 31 2019 13:25

    spooz on consumer-groups-description

    (compare)

  • Jan 31 2019 13:20

    spooz on SKYEDEN-223-consumer-groups-desc

    SKYEDEN-223 | consumer groups d… (compare)

  • Jan 31 2019 12:54
    dankraw commented #981
  • Jan 31 2019 12:38
    dankraw commented #981
  • Jan 31 2019 12:17
    druminski labeled #981
  • Jan 31 2019 12:17
    druminski labeled #981
  • Jan 31 2019 12:15
    dankraw opened #981
  • Jan 29 2019 14:28
    faderskd starred allegro/hermes
  • Jan 29 2019 14:28
    faderskd starred allegro/hermes
  • Jan 25 2019 17:45

    adamdubiel on hermes-0.15.7

    (compare)

  • Jan 25 2019 17:45

    adamdubiel on master

    Release version: 0.15.7 (compare)

  • Jan 25 2019 17:45

    druminski on changelogUpdate

    (compare)

  • Jan 25 2019 17:45

    druminski on master

    Updating changelog (#980) (compare)

  • Jan 25 2019 17:45
    druminski closed #980
  • Jan 25 2019 16:35
    druminski opened #980
Tibor Nagy
@tnagy81
Hi, Out of curiosity: have you ever considered to migrate all Hermes modules to Spring Boot? Why does management module use Spring and the others not?
Shimna
@shimnarkrishnan_twitter

I brought up Hermes using “vagrant up” and it successfully came up . Got the below links to access the components too.

UI & Management: http://10.10.10.10:8090
Frontend: http://10.10.10.10:8080
Graphite: http://10.10.10.10:8082

But when I actually paste the links in a browser , it says page isn’t working . Has something changed from the documentation ?

Łukasz Drumiński
@druminski

@tnagy81 Hi,

When we started implementing Hermes six years ago, spring-boot wasn’t so popular then. Hermes-management at the beginning was written in Jersey framework which in internals uses hk2 as a light-weight dependency injection framework. So we started with hk2 in hermes-management and decided to use it in hermes-{frontend, consumers} as well.
When spring-boot gained popularity we’ve rewritten hermes-management to use it. However we didn’t use spring-boot in hermes-{frontend, consumers} due to performance reasons.

Hermes-frontend has to be very fast. Therefore we’ve written it on a raw HTTP handlers in Undertow server. spring-boot-starter-web adds additional layers such as servlets implementation and custom filters. They add additional latency for request processing and we don’t want that.

So, as you see there are two reasons, historical and performance, why we don’t use sprint-boot in hermes-{frontend, consumers}. Today, replace of hk2 to spring di framework in hermes-{frontend, consumers} is not a trivial task as it requires rewrite of a lot of code. However, it would be nice to have the same dependency injection framework and configuration method in all Hermes modules. We don’t do this task in our team as we have other priorities right now but in the future it would be nice to do it.

Piotr Rżysko
@piotrrzysko
Hello @shimnarkrishnan_twitter, I’ve already merged fix for Vagrant provisioning to master. Please let me know if this change solves your problem :)
Shimna
@shimnarkrishnan_twitter
@piotrrzysko I pulled the latest code and got the file vagrant_provisioning/install_kafka.sh , but unfortunately the links dont come up yet. I think Graphite used to come up earlier too and it still comes up , but the Management and Frontend are not coming up .
Łukasz Drumiński
@druminski
@shimnarkrishnan_twitter probably you are still using previous vagrant image. Destroy it in the first place by “vagrant destroy” command, then “vagrant up” should work :)
Shimna
@shimnarkrishnan_twitter
@druminski I started using docker and the components are up now.
Łukasz Drumiński
@druminski
@shimnarkrishnan_twitter we stopped supporting docker images 3 years ago in favor of vagrant images. I recommend you to try Hermes in vagrant in the first place as we support it officially and it contains the latest version of Hermes
Shimna
@shimnarkrishnan_twitter
@druminski Thank you. Brought it up using vagrant again and it works ! There is another question. I am trying to set up schema registry and i couldn't figure out where to set the below configurations schema.repository.type: schema_registry or schema_repo
schema.repository.url: URL of repository Also what about this URL of repository? Which URL is expected?
Jacek Ewertowski
@jewertow

@shimnarkrishnan_twitter

  1. "Also what about this URL of repository? Which URL is expected?"
    Both schema repo and schema registry provides http API to manage schemas so you have to provide the server URL where your repository is running.
    This URL must have the following form: http://<host>:<port>. By default it is http://localhost:8888/.

  2. "I couldn't figure out where to set the below configurations schema.repository.type: schema_registry"
    You have to add these properties to your configuration files. You can look at my development hermes configuration (based on docker containers) where I have configured schema-registry: https://github.com/jewertow/hermes-dev-environment.

Schema-registry configurations:

Schema-registry container configuration: https://github.com/jewertow/hermes-dev-environment/blob/master/docker-compose.yml#L31

Shimna
@shimnarkrishnan_twitter
@jewertow Thank you !
Shimna
@shimnarkrishnan_twitter
A few more questions on message schema : 1) Does hermes support message size validation? 2) Read that zookeeper is the metadata store. Does this mean message schemas are stored in zookeeper? Or are they stored in kafka? 3) If we want to incorporate cloudevent spec , how should we go about it?
Łukasz Drumiński
@druminski

@shimnarkrishnan_twitter
Ad 1) Yes, it does. By default message size is a soft limit, warn log is emitted when message larger than declared topic message size is received.
frontend.force.topic.max.message.size flag can be switched to make it a hard limit (Frontend will return
http 413 Payload Too Large status).

Ad 2) Schemas are kept in schema-registry. Zookeeper in Hermes is used as metastore to keep information about topics and subscriptions such as topic name, description, owner, max message size, attached subscriptions, subscription name, endpoint, retry policy, etc.

Ad 3) Didn’t use it but I see it supports HTTP & JSON so at first glance it should be possible to use it on Hermes

Shimna
@shimnarkrishnan_twitter
@druminski thank you.
I am playing around with the subscription part. When i used a webhook site to receive messages, i am getting them. But when i use a simple spring boot rest endpoint (POST), my endpoint is not called by the consumer. I am not sure if i am missing anything in my POST endpoint. My POST api looks like the below :
@PostMapping
public String testConsume() {
System.out.println("hi");
return "hello consume";
}
Jacek Ewertowski
@jewertow

@shimnarkrishnan_twitter
Could you describe how did you create the subscription for this endpoint?
Did you create it according to the documentation (https://hermes-pubsub.readthedocs.io/en/latest/user/subscribing/#creating-subscription) ?

You can also check out example in my development environment:

  1. How subscribing service should work: https://github.com/jewertow/hermes-dev-environment/blob/master/modules/subscriber/src/main/java/com/github/jewertow/App.kt#L28 - it is simple Ktor app, but you can replace it with spring boot
  2. How to create subscription in Hermes: https://github.com/jewertow/hermes-dev-environment/blob/master/setup-data.sh
    There is a README file, you can follow the instructions and easy setup working environment.
Tibor Nagy
@tnagy81
Hi @druminski, just wanted to thank you for the quick summary last week about using different frameworks in the modules. Makes a lot of sense.
Shimna
@shimnarkrishnan_twitter
@jewertow yes i created the subscription according to the documentation. The webhooks subscription url is working . But the other subscription which i created using springboot is not working. I tried to see where the message is lost by enabling tracking but the endpoint "topics/{topicName}/subscriptions/{subscriptionName}/events/{Hermes-Message-Id}/trace" is giving empty response.
Łukasz Drumiński
@druminski

@tnagy81 I'm glad I could help

@shimnarkrishnan_twitter

  1. set message ttl on your subscription to a low value such as 10 seconds
  2. publish a message on the topic
  3. after 10 second refresh the subscription in a browser and verify what you see in “Last undelivered message” panel. You should see the reason why message wasn’t delivered.

if you want to track every message by hermes message tracking mechanism you have to install trace storage: https://hermes-pubsub.readthedocs.io/en/latest/configuration/message-tracking/

Shimna
@shimnarkrishnan_twitter
@druminski The message says Delivered in the subscription console. But hermes has not called the POST endpoint that i provided as the subscription endpoint.
Łukasz Drumiński
@druminski
@shimnarkrishnan_twitter if Hermes says "delivered" then receives 2xx status code from a subscriber. Maybe you should use tcpdump to listen tcp traffic and verify what Hermes sends to the subscriber and what the subscriber returns.
Shimna
@shimnarkrishnan_twitter
@druminski We have fixed that issue . It was not an issue with hermes . It was a minor problem at our end which we now fixed. Thanks for your support !
Tibor Nagy
@tnagy81
Hi, I was trying to use spring security with my custom hermes-management module and ran into an issue because some spring auto configuration. This sounds spring related but actually the cause of it is that spring-mvc is present on the classpath and two frameworks are listening to requests. I have found spring-boot-starter-web in the dependencies of hermes-management. If you are using spring-boot-starter-jersey it is unnecessary to add starter-web because it will hook up spring MVC which you don't need for JAX-RS endpoints. My workaround was adding spring-webmwc to exclusions (I am using maven) of hermes-management dependency. My service and of course the hermes components start without any issue and endpoints are working fine. Though I did not check all the endpoints :D
I wanted to make a hermes build without the spring-boot-starter-web dependency to check whether it will be successful with all the test but I did not have time to do it yet.
Łukasz Drumiński
@druminski
Hi @tnagy81, thanks for the information, if all tests will work without spring-webmvc then I agree with you that we can remove it. I keep my fingers crossed for your test and pull request :)
Tibor Nagy
@tnagy81
@druminski :D
Tibor Nagy
@tnagy81
@druminski Hi, I just want to let you know that there is a high chance that I won't have time to work the spring-webmvc stuff just only because of vacations. I'd really like to contribute though. As far as I understand if we handle this as a small issue I don't have to create a ticket. I also want to note that if I remove spring-webmvc then Spring Actuator endpoints (like /health, /info) start to work in hermes-management which is nice to have.
1 reply
Łukasz Ciołecki
@lciolecki
Hi!
I have question for allegro/hermes#682. Does hermes support standard confluent method serialization / deserialization message?
Łukasz Drumiński
@druminski
@tnagy81 ok, imo this issue can wait as you are the only user who reported it :) However, if you need help with it then please let us know as we can work on it as well. In the meantime happy vacations!
I talked with @lciolecki on priv about subject which he raised.
Shimna
@shimnarkrishnan_twitter
What was the reason hermes stopped docker support and moved to vagrant? We are planning to use hermes with docker . What challenges should we foresee?
Shimna
@shimnarkrishnan_twitter
Also, we already have a kafka cluster running in our production . We plan to use that along with hermes. What challenges can come in our way here?
Łukasz Drumiński
@druminski

@shimnarkrishnan_twitter few years ago we had occasionally problems with docker on different laptops. On one laptops it worked, on others it didn’t, especially after docker updates. Therefore we decided to use Vagrant in our quickstart guide. However, I see that docker is much more stable technology now and our open source users are asking about it, so we are planning to bring it back in Hermes. In the near future we are going to publish new images of Hermes on dockerhub :)

As for Kafka and Hermes. Hermes for Kafka is a client. If you already have a running Kafka cluster on production and you are happy with it then run of Hermes on it should be easy and smooth :)

RaulMarina
@RaulMarina
Hi, I have my springboot application running on a docker container using 8080, I just have set up the Hermes environment using https://hermes-pubsub.readthedocs.io/en/latest/quickstart/#docker. Both application and hermes are working fine but as I needed to change the hermes-frontend port (8083) I am getting "Error: socket hang up" whenever I try to publish. What am I doing wrong?
RaulMarina
@RaulMarina
nevermind
Benson Miller
@bensonmiller
Hello Allegro team. I'm interested in moving my workload to Confluent Cloud, but Confluent Cloud doesn't expose Zk, which seems to preclude Hermes operation. Is support for Confluent Cloud (or KIP-500 mode) part of the Hermes roadmap? I'd appreciate any insight you can share. Thank you.
Łukasz Drumiński
@druminski

Hi @bensonmiller, you are the first user who would like to use Hermes with Kafka from Confluent Cloud. So far, we didn’t have that need, but definitely Hermes should be compatible with Confluent Cloud as well. Although, keep in mind that Hermes will still need a Zookeeper cluster to run.

As you noticed, Hermes uses Kafka zookeeper client to manage topics configuration. I will create an issue to replace this old client to something new. When we resolve it, then will see whether something more has to be done to be compatible with Confluent Cloud. Please let me know if this approach is ok for you :)

Benson Miller
@bensonmiller
Hello @druminski - Thank you for the response. This sounds like exactly the right approach to me. Thank you.
cisrudlow
@cisrudlow
Hi (maybe stupid question, but I haven't found an answer in the documentation). I am curious how partitioning in Hermes is resolved. I expect you are using the default partitioner, but how do you indicate (in Hermes) which partition the message should go to (partition number or key).
We have a very special case. Hermes will post messages on topic, but one (old) service will have to read from Kafka directly (akka cluster + sharding).
thx
Łukasz Drumiński
@druminski
Hi @cisrudlow, when Hermes publishes message to Kafka, it doesn't set a partition number and a key. It means that the partition will be assigned in a round-robin fashion.
Aditya Todkar
@aditodkar
Hello everyone, I am Aditya from India. I would like to contribute to allegro/hermes as a part of hacktoberfest. I am familiar with Kafka and have worked on JAVA extensively. I think I will get good grasp on Kafka by contributing to this project. Can someone please help me get started and any resources which you can share for Kafka. Currently I only know basics of Kafka. Thanks :)
Aditya Todkar
@aditodkar
Hello everyone. Is this right channel for hackoctoberfest ?
Łukasz Drumiński
@druminski
@aditodkar Welcome, very nice that you would like to contribute to Hermes project due to Hacktoberfest event. Please take a look on issues with "hacktoberfest" label. Few of them are still available, for example #1269 might be interesting for you. If you find some issue interesting then please let us know directly in the issue.
Tibor Nagy
@tnagy81
Hi @druminski,
Last time you heard form me I was stating that spring-webmvc is not a necessary dependency for hermes-manager. I went a bit deeper in this topic and unfortunately (or fortunately) I was wrong. It is needed for the static content of the console. I admit that I don't have any experience with jersey and I thought that jersey starter is enough.
Anyway, I have a few other questions:
  1. Is it allowed in case of https to define a subscription URL with non standard port?
  2. I'd like to develop a service that sends an automated notification (e.g.: email) in case a subscription is unhealthy. Can you give some basic ideas how to do that, what would be the best practice to regularly check subscriptions health?
  3. Retries and offset: As far as I understand from the code the kafka offset is committed in a separate thread. Also message processing does not stop when a previous message is in the retry loop. I'd like to understand what happens if there are too many retries and what happens if one hermes-consumer instance fails/shuts down/is killed not gracefully? What are the chances of messages being resent multiple times, losing messages, resending huge amount of already sent messages.
    • What happens to the offset commit when there is one or more message that is being retried?
    • I assume that retries stay in the message sending queue. Does that mean that if the queue is filled with retries than reading of the topic will stop until retries succeed or expire?
    • Is there any automated suspend?
Łukasz Drumiński
@druminski

Hello @tnagy81 ,

Thanks for making sure that spring-mvc is required in Hermes-management. As for your questions:

Ad 1. I think it’s possible. Anyway, no one has ever reported such a problem to us, despite the fact that we use https in our Hermes clusters.

Ad 2. Hermes-management exposes an endpoint to check a subscription health. As far as I know we don’t have any documentation for it yet, but take a look on the changelog and UnhealthyEndpoint class

I encourage you to consider use a monitoring system which periodically sends requests to hermes-management to check unhealthy subscriptions.

Very helpful is measuring a lag on Kafka consumers group. To do that you can use a dedicated tool, for example Burrow tool. Moreover Hermes-management has a SubscriptionLagSource interface. You can provide an implementation for it which verifies lag on Kafka consumers group. This interface is used by UnhealthyEndpoint class.

Ad 3. Hermes-consumers uses memory queue for sending messages. When this queue is full due to high number of retried messages then Hermes waits with reading messages from Kafka until there is a space in the memory queue. You can set the size of this memory queue by consumer.inflight.size parameter. By the way when subscriber returns only errors then most likely Hermes will lower speed of sending/retrying messages. See rate-limiting for more information.

Hermes-consumers commits offset only when message was delivered successfully or its TTL expired (retries didn’t helped). In the meantime when an instance of Hermes-consumers is killed then other instance takes its work due to Kafka-consumers autorebalance mechanism. The working instance starts sending messages from the last committed offset, so there is a chance that some messages will be delivered more than one time. This is not the only one place in Hermes when message duplication can occur. In summary Hermes has „at-least-once” message delivery guarantee.

Ad a) message offset is not committed until the message is delivered successfully or its TTL expires and is dropped

Ad b) Yes, exactly

Ad c) There is a rate-limit mechanism which slows down message sending to a few messages per minute when subscriber doesn’t receive messages successfully. However, Hermes doesn’t fully stop sending messages. If you want to stop message sending to zero then you have to suspend the subscription manually. More info about rate-limiter you can find here

I hope that my answers are helpful to you :)

cywattebled
@cywattebled
Hello guys,
We are taking a look at Hermes as an alternative to Direct Kafka connectivity.
I have a couple of questions if I may ask:
1) There doesn't seem to be a way to specify a partitioning key which means that as a consumer, you have no way to be guaranteed ordering when reading from a specific partition. Is there any plans for providing such a feature?
2) How are consumer groups managed within Hermes?
3) Hermes manages offset for consumer but seems to be expecting acknowledgement synchronously on the subscription push. We have a use-case where the work is distributed within an internal cluster and processing of the actual event may not occur on the node receiving the push http call. There would be a need for submitting the offset commit when processing is actually done. I wasn't able to find anything on the documentation. Is there such a feature under the hood?
thanks for your help
Łukasz Drumiński
@druminski

Hello @cywattebled

Nice to hear that you are considering using Hermes. Answers to your questions:

Ad 1. You are right, Hermes currently uses round-robin partitioning. It works like this because Hermes uses HTTP as communication layer for producers and subscribers. In HTTP some messages can be retried asynchronously due to errors such as 5xx. Besides, Hermes works in many instances, for example Hermes-consumers can have many instances. I don’t see how this module can easily coordinate order of messages. However, if you have an idea how this should work, then we are open on propositions.

Ad 2.

Hermes-consumer module is responsible for handling subscriptions. Each Hermes subscription has a dedicated Kafka consumer using Kafka consumer group name combined from Hermes subscription name. You can run many instances of Hermes-consumers. You can also configure how many Hermes-consumers instances should handle single subscription. By default two Hermes-consumers handles the subscription. This can be configured by consumer.workload.consumers.per.subscription parameter. If you have two Hermes-consumers handling the same subscription, then they are using the same Kafka consumer group for this subscription meaning that Kafka consumers rebalance protocol is in use. Take a look on this article as it might be helpful to you as well.

Ad 3. Hermes doesn’t have that feature. I see two solutions for this:

  1. You can increase subscription timeout and wait for the response from a subscriber
  2. Subscriber can acknowledge immediately after it receives message from Hermes and send a message on a different topic when finish processing.

I hope my answers were helpful,
Cheers
Łukasz

cywattebled
@cywattebled
@druminski
Thanks for your response.
1) Well actually, The issue with the partitioning key is that actually on the producer side. We have topics that would require data to be ordered per partition, and considering a hybrid solution where services that needs specifics Kafka features would consume directly from Kafka (and expect message to be ordered within a single partition) but with producers writing through Hermes. I understand that the consuming logic would be complicated due to the push mechanism but providing an option to provide a partitioning key shouldn't have an impact no? Without this feature, we'd be stuck to using Kafka direct.
2) Thanks, it confirms my understanding of how Hermes works. Nice article by the way :)
3)Yeah that's definitely not ideal. If we had a fix for 1), we could envision consuming direct from Kafka for this specific consumer group and managing the offset manually as needed.
Łukasz Drumiński
@druminski
@cywattebled I think we can implement a support for a „partition-key” HTTP header in Hermes. Messages with the same „partition-key” HTTP header will be sent to the same Kafka topic partition. However, this feature doesn’t guarantee an order of messages. It’s because HTTP clients and Hermes-frontend (HTTP server) work in concurrent environment - many instances, threads, etc. Therefore message A can be published on Kafka topic partition X later than message B even when message A was send faster than message B to Hermes. Please let me know whether this behavior is ok to you.
cywattebled
@cywattebled

@druminski
Thanks for the response.

Yes I agree, what I mean by ordered is ordered received by the Kafka broker but our main focus is that all event relating to the same entity land on the same partition.

I guess the partition-key header would indeed fix my issue. I was envisioning something like:
curl -X POST -H "Content-Type: application/json" http://hermes-frontend/topics/group.topic -d '{"key": "terranova-123456", "message": "Hello world!"}' where key is optional but the header would work too :).

Other question if I may, is there a way to set the partition number on the topic creation call? I don't see that option in the documentation. I guess Topics should really be created before hand and not necessarily by producers but just out of curiosity.
Thanks.