Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 16:19

    spooz on consumer-groups-description

    SKYEDEN-223 | metadata (compare)

  • Jan 31 2019 16:18

    spooz on SKYEDEN-223-consumer-groups-desc

    (compare)

  • Jan 31 2019 16:07

    spooz on SKYEDEN-223-consumer-groups-desc

    SKYEDEN-223 | metadata (compare)

  • Jan 31 2019 13:27

    spooz on consumer-groups-description

    consumer groups desc (compare)

  • Jan 31 2019 13:26

    spooz on SKYEDEN-223-consumer-groups-desc

    (compare)

  • Jan 31 2019 13:25

    spooz on consumer-groups-description

    (compare)

  • Jan 31 2019 13:20

    spooz on SKYEDEN-223-consumer-groups-desc

    SKYEDEN-223 | consumer groups d… (compare)

  • Jan 31 2019 12:54
    dankraw commented #981
  • Jan 31 2019 12:38
    dankraw commented #981
  • Jan 31 2019 12:17
    druminski labeled #981
  • Jan 31 2019 12:17
    druminski labeled #981
  • Jan 31 2019 12:15
    dankraw opened #981
  • Jan 29 2019 14:28
    faderskd starred allegro/hermes
  • Jan 29 2019 14:28
    faderskd starred allegro/hermes
  • Jan 25 2019 17:45

    adamdubiel on hermes-0.15.7

    (compare)

  • Jan 25 2019 17:45

    adamdubiel on master

    Release version: 0.15.7 (compare)

  • Jan 25 2019 17:45

    druminski on changelogUpdate

    (compare)

  • Jan 25 2019 17:45

    druminski on master

    Updating changelog (#980) (compare)

  • Jan 25 2019 17:45
    druminski closed #980
  • Jan 25 2019 16:35
    druminski opened #980
Jacek Ewertowski
@jewertow
@tnagy81 I created pull requests that remove logback files and support JRE truststore. We are testing these changes. Stay tuned ;)
Tibor Nagy
@tnagy81
@jewertow Thx! I saw them being reviewed a few days ago. I was checking what other stuff are people working on.
Jacek Ewertowski
@jewertow
@tnagy81 Could you checkout branch hermes-supports-default-jvm-truststore, build hermes-consumers and check in your environment if this change solves your problem?
I was testing it in my docker-based development environment, but I am still not sure if it will work for you.
Tibor Nagy
@tnagy81
@jewertow I'll take a look this afternoon and let you know about the result.
Tibor Nagy
@tnagy81
@jewertow Works like a charm. Sorry for the delay of checking this.
Jacek Ewertowski
@jewertow
Great, thanks for the feedback. We will release it soon ;)
Jacek Ewertowski
@jewertow
Tibor Nagy
@tnagy81
@jewertow Cool! Thx!
Tibor Nagy
@tnagy81
Hi, Out of curiosity: have you ever considered to migrate all Hermes modules to Spring Boot? Why does management module use Spring and the others not?
Shimna
@shimnarkrishnan_twitter

I brought up Hermes using “vagrant up” and it successfully came up . Got the below links to access the components too.

UI & Management: http://10.10.10.10:8090
Frontend: http://10.10.10.10:8080
Graphite: http://10.10.10.10:8082

But when I actually paste the links in a browser , it says page isn’t working . Has something changed from the documentation ?

Łukasz Drumiński
@druminski

@tnagy81 Hi,

When we started implementing Hermes six years ago, spring-boot wasn’t so popular then. Hermes-management at the beginning was written in Jersey framework which in internals uses hk2 as a light-weight dependency injection framework. So we started with hk2 in hermes-management and decided to use it in hermes-{frontend, consumers} as well.
When spring-boot gained popularity we’ve rewritten hermes-management to use it. However we didn’t use spring-boot in hermes-{frontend, consumers} due to performance reasons.

Hermes-frontend has to be very fast. Therefore we’ve written it on a raw HTTP handlers in Undertow server. spring-boot-starter-web adds additional layers such as servlets implementation and custom filters. They add additional latency for request processing and we don’t want that.

So, as you see there are two reasons, historical and performance, why we don’t use sprint-boot in hermes-{frontend, consumers}. Today, replace of hk2 to spring di framework in hermes-{frontend, consumers} is not a trivial task as it requires rewrite of a lot of code. However, it would be nice to have the same dependency injection framework and configuration method in all Hermes modules. We don’t do this task in our team as we have other priorities right now but in the future it would be nice to do it.

Piotr Rżysko
@piotrrzysko
Hello @shimnarkrishnan_twitter, I’ve already merged fix for Vagrant provisioning to master. Please let me know if this change solves your problem :)
Shimna
@shimnarkrishnan_twitter
@piotrrzysko I pulled the latest code and got the file vagrant_provisioning/install_kafka.sh , but unfortunately the links dont come up yet. I think Graphite used to come up earlier too and it still comes up , but the Management and Frontend are not coming up .
Łukasz Drumiński
@druminski
@shimnarkrishnan_twitter probably you are still using previous vagrant image. Destroy it in the first place by “vagrant destroy” command, then “vagrant up” should work :)
Shimna
@shimnarkrishnan_twitter
@druminski I started using docker and the components are up now.
Łukasz Drumiński
@druminski
@shimnarkrishnan_twitter we stopped supporting docker images 3 years ago in favor of vagrant images. I recommend you to try Hermes in vagrant in the first place as we support it officially and it contains the latest version of Hermes
Shimna
@shimnarkrishnan_twitter
@druminski Thank you. Brought it up using vagrant again and it works ! There is another question. I am trying to set up schema registry and i couldn't figure out where to set the below configurations schema.repository.type: schema_registry or schema_repo
schema.repository.url: URL of repository Also what about this URL of repository? Which URL is expected?
Jacek Ewertowski
@jewertow

@shimnarkrishnan_twitter

  1. "Also what about this URL of repository? Which URL is expected?"
    Both schema repo and schema registry provides http API to manage schemas so you have to provide the server URL where your repository is running.
    This URL must have the following form: http://<host>:<port>. By default it is http://localhost:8888/.

  2. "I couldn't figure out where to set the below configurations schema.repository.type: schema_registry"
    You have to add these properties to your configuration files. You can look at my development hermes configuration (based on docker containers) where I have configured schema-registry: https://github.com/jewertow/hermes-dev-environment.

Schema-registry configurations:

Schema-registry container configuration: https://github.com/jewertow/hermes-dev-environment/blob/master/docker-compose.yml#L31

Shimna
@shimnarkrishnan_twitter
@jewertow Thank you !
Shimna
@shimnarkrishnan_twitter
A few more questions on message schema : 1) Does hermes support message size validation? 2) Read that zookeeper is the metadata store. Does this mean message schemas are stored in zookeeper? Or are they stored in kafka? 3) If we want to incorporate cloudevent spec , how should we go about it?
Łukasz Drumiński
@druminski

@shimnarkrishnan_twitter
Ad 1) Yes, it does. By default message size is a soft limit, warn log is emitted when message larger than declared topic message size is received.
frontend.force.topic.max.message.size flag can be switched to make it a hard limit (Frontend will return
http 413 Payload Too Large status).

Ad 2) Schemas are kept in schema-registry. Zookeeper in Hermes is used as metastore to keep information about topics and subscriptions such as topic name, description, owner, max message size, attached subscriptions, subscription name, endpoint, retry policy, etc.

Ad 3) Didn’t use it but I see it supports HTTP & JSON so at first glance it should be possible to use it on Hermes

Shimna
@shimnarkrishnan_twitter
@druminski thank you.
I am playing around with the subscription part. When i used a webhook site to receive messages, i am getting them. But when i use a simple spring boot rest endpoint (POST), my endpoint is not called by the consumer. I am not sure if i am missing anything in my POST endpoint. My POST api looks like the below :
@PostMapping
public String testConsume() {
System.out.println("hi");
return "hello consume";
}
Jacek Ewertowski
@jewertow

@shimnarkrishnan_twitter
Could you describe how did you create the subscription for this endpoint?
Did you create it according to the documentation (https://hermes-pubsub.readthedocs.io/en/latest/user/subscribing/#creating-subscription) ?

You can also check out example in my development environment:

  1. How subscribing service should work: https://github.com/jewertow/hermes-dev-environment/blob/master/modules/subscriber/src/main/java/com/github/jewertow/App.kt#L28 - it is simple Ktor app, but you can replace it with spring boot
  2. How to create subscription in Hermes: https://github.com/jewertow/hermes-dev-environment/blob/master/setup-data.sh
    There is a README file, you can follow the instructions and easy setup working environment.
Tibor Nagy
@tnagy81
Hi @druminski, just wanted to thank you for the quick summary last week about using different frameworks in the modules. Makes a lot of sense.
Shimna
@shimnarkrishnan_twitter
@jewertow yes i created the subscription according to the documentation. The webhooks subscription url is working . But the other subscription which i created using springboot is not working. I tried to see where the message is lost by enabling tracking but the endpoint "topics/{topicName}/subscriptions/{subscriptionName}/events/{Hermes-Message-Id}/trace" is giving empty response.
Łukasz Drumiński
@druminski

@tnagy81 I'm glad I could help

@shimnarkrishnan_twitter

  1. set message ttl on your subscription to a low value such as 10 seconds
  2. publish a message on the topic
  3. after 10 second refresh the subscription in a browser and verify what you see in “Last undelivered message” panel. You should see the reason why message wasn’t delivered.

if you want to track every message by hermes message tracking mechanism you have to install trace storage: https://hermes-pubsub.readthedocs.io/en/latest/configuration/message-tracking/

Shimna
@shimnarkrishnan_twitter
@druminski The message says Delivered in the subscription console. But hermes has not called the POST endpoint that i provided as the subscription endpoint.
Łukasz Drumiński
@druminski
@shimnarkrishnan_twitter if Hermes says "delivered" then receives 2xx status code from a subscriber. Maybe you should use tcpdump to listen tcp traffic and verify what Hermes sends to the subscriber and what the subscriber returns.
Shimna
@shimnarkrishnan_twitter
@druminski We have fixed that issue . It was not an issue with hermes . It was a minor problem at our end which we now fixed. Thanks for your support !
Tibor Nagy
@tnagy81
Hi, I was trying to use spring security with my custom hermes-management module and ran into an issue because some spring auto configuration. This sounds spring related but actually the cause of it is that spring-mvc is present on the classpath and two frameworks are listening to requests. I have found spring-boot-starter-web in the dependencies of hermes-management. If you are using spring-boot-starter-jersey it is unnecessary to add starter-web because it will hook up spring MVC which you don't need for JAX-RS endpoints. My workaround was adding spring-webmwc to exclusions (I am using maven) of hermes-management dependency. My service and of course the hermes components start without any issue and endpoints are working fine. Though I did not check all the endpoints :D
I wanted to make a hermes build without the spring-boot-starter-web dependency to check whether it will be successful with all the test but I did not have time to do it yet.
Łukasz Drumiński
@druminski
Hi @tnagy81, thanks for the information, if all tests will work without spring-webmvc then I agree with you that we can remove it. I keep my fingers crossed for your test and pull request :)
Tibor Nagy
@tnagy81
@druminski :D
Tibor Nagy
@tnagy81
@druminski Hi, I just want to let you know that there is a high chance that I won't have time to work the spring-webmvc stuff just only because of vacations. I'd really like to contribute though. As far as I understand if we handle this as a small issue I don't have to create a ticket. I also want to note that if I remove spring-webmvc then Spring Actuator endpoints (like /health, /info) start to work in hermes-management which is nice to have.
1 reply
Łukasz Ciołecki
@lciolecki
Hi!
I have question for allegro/hermes#682. Does hermes support standard confluent method serialization / deserialization message?
Łukasz Drumiński
@druminski
@tnagy81 ok, imo this issue can wait as you are the only user who reported it :) However, if you need help with it then please let us know as we can work on it as well. In the meantime happy vacations!
I talked with @lciolecki on priv about subject which he raised.
Shimna
@shimnarkrishnan_twitter
What was the reason hermes stopped docker support and moved to vagrant? We are planning to use hermes with docker . What challenges should we foresee?
Shimna
@shimnarkrishnan_twitter
Also, we already have a kafka cluster running in our production . We plan to use that along with hermes. What challenges can come in our way here?
Łukasz Drumiński
@druminski

@shimnarkrishnan_twitter few years ago we had occasionally problems with docker on different laptops. On one laptops it worked, on others it didn’t, especially after docker updates. Therefore we decided to use Vagrant in our quickstart guide. However, I see that docker is much more stable technology now and our open source users are asking about it, so we are planning to bring it back in Hermes. In the near future we are going to publish new images of Hermes on dockerhub :)

As for Kafka and Hermes. Hermes for Kafka is a client. If you already have a running Kafka cluster on production and you are happy with it then run of Hermes on it should be easy and smooth :)

RaulMarina
@RaulMarina
Hi, I have my springboot application running on a docker container using 8080, I just have set up the Hermes environment using https://hermes-pubsub.readthedocs.io/en/latest/quickstart/#docker. Both application and hermes are working fine but as I needed to change the hermes-frontend port (8083) I am getting "Error: socket hang up" whenever I try to publish. What am I doing wrong?
RaulMarina
@RaulMarina
nevermind
Benson Miller
@bensonmiller
Hello Allegro team. I'm interested in moving my workload to Confluent Cloud, but Confluent Cloud doesn't expose Zk, which seems to preclude Hermes operation. Is support for Confluent Cloud (or KIP-500 mode) part of the Hermes roadmap? I'd appreciate any insight you can share. Thank you.
Łukasz Drumiński
@druminski

Hi @bensonmiller, you are the first user who would like to use Hermes with Kafka from Confluent Cloud. So far, we didn’t have that need, but definitely Hermes should be compatible with Confluent Cloud as well. Although, keep in mind that Hermes will still need a Zookeeper cluster to run.

As you noticed, Hermes uses Kafka zookeeper client to manage topics configuration. I will create an issue to replace this old client to something new. When we resolve it, then will see whether something more has to be done to be compatible with Confluent Cloud. Please let me know if this approach is ok for you :)

Benson Miller
@bensonmiller
Hello @druminski - Thank you for the response. This sounds like exactly the right approach to me. Thank you.
cisrudlow
@cisrudlow
Hi (maybe stupid question, but I haven't found an answer in the documentation). I am curious how partitioning in Hermes is resolved. I expect you are using the default partitioner, but how do you indicate (in Hermes) which partition the message should go to (partition number or key).
We have a very special case. Hermes will post messages on topic, but one (old) service will have to read from Kafka directly (akka cluster + sharding).
thx
Łukasz Drumiński
@druminski
Hi @cisrudlow, when Hermes publishes message to Kafka, it doesn't set a partition number and a key. It means that the partition will be assigned in a round-robin fashion.
Aditya Todkar
@aditodkar
Hello everyone, I am Aditya from India. I would like to contribute to allegro/hermes as a part of hacktoberfest. I am familiar with Kafka and have worked on JAVA extensively. I think I will get good grasp on Kafka by contributing to this project. Can someone please help me get started and any resources which you can share for Kafka. Currently I only know basics of Kafka. Thanks :)