Burrow/Lobby

LinkedIn Burrow - Consumer lag checking for Apache Kafka

Peter Bukowinski

@pmbuko

[client-profile.MYCLIENT]
kafka-version="2.x.x"
client-id="burrow-myclient"
sasl="PLAIN"

[sasl.PLAIN]
username="Adminuser"
password="Adminpassword"
handshake-first=false

[consumer.CLUSTER]
class-name="kafka"
cluster="CLUSTER"
client-profile="MYCLIENT"

That should be all you need.

saritago

@saritago

@pmbuko Thanks! Below config worked for us. Please make a note that both sasl and tls details had to be mentioned for burrow to work. Dint work with either of them missing.

[client-profile.test]
client-id="burrow-test"
kafka-version="0.11.0.2"
sasl="PLAINSS"
tls="mytlsprofile"

[sasl.PLAINSS]
handshake-first=true
username="superuser"
password="superuserpassword"

[tls.mytlsprofile]
certfile="path/to/certfile/file/wildcard.jks"
keyfile="path/to/keyfile/file/wildcard.jks"
noverify=true

saritago

@saritago

Hi, facing issue with prometheus metrics. We are just getting burrow_kafka_topic_partition_offset and no other metrics, where as ideally we should have got all of these "burrow_kafka_consumer_partition_lag", "burrow_kafka_consumer_current_offset", "burrow_kafka_consumer_status".

Any configurations that needs to be updated to get all the metrics.
We are using kafka version 0.11.0.3. and its a secured kafka cluster.

1 reply

Lawrence Kim

@pr-lawrence

Hello, guys. I'm waiting for the next release to using Kafka 2.6.0. I wonder if you have any plans.

chetannarasimhaiah

@chetannarasimhaiah

Hi All, Any recommendations on the above issue which @saritago has raised.

Peter Bukowinski

@pmbuko

Is there a rule of thumb as to how many clusters/consumers/partitions a single instance of burrow can comfortably handle? What tends to be the limiting factor? I’ve been getting some erroneous consumer lag metrics recently and restarting burrow resolves them.

Capt

@CaptainK5

I'm running solsson/burrow 1.3.2 with Kafka 2.5.1 (cp-kafka). The endpoint "/v3/kafka/test-cluster/consumer" is returning "consumers:[]" (empty list of consumers). Our consumers are set to use auto-commits. I cannot hit the burrow endpoint to get lag information if I cannot get a list of consumers. Please let me know if anyone has any suggestions. Thank you.

Peter Bukowinski

@pmbuko

@CaptainK5 does the /v3/kafka endpoint list test-cluster?

Capt

@CaptainK5

@pmbuko '/v3/kafka' endpoint returns "clusters":["test-cluster"]

Capt

@CaptainK5

@pmbuko I did not setup client-profile, '/v3/kafka/test-cluser/' returns "kafka-version":"0.8" and a few other things null or empty, but my kafka version is 2.5.1, could this be the problem? Is client-profile required?

Capt

@CaptainK5

@pmbuko I added client-profile, but burrow does not recognize kafka-version greater than 1.0.0

Capt

@CaptainK5

@pmbuko '/v3/kafka/test-cluster' returns "kafka-version":"1.0.0" along with the other settings I've configured. "tls":null,"sasl":null. How do I know if I need to setup sasl? In client-profile, when I tried to set tls="burrow-tls" it crashes.

Capt

@CaptainK5

anyone out there can help me? i discovered that kafka.auto.commit was set to false, updated that to true. But it didn't fix the issue. I'm still getting empty list of consumers

Capt

@CaptainK5

bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list --bootstrap-server localhost:9092, list the groups

Peter Bukowinski

@pmbuko

@CaptainK5 You should make sure the version of burrow you are using is up-to-date. The latest release supports up to 2.5: https://github.com/linkedin/Burrow/blob/v1.3.4/core/internal/helpers/sarama.go#L26-L59 — the client profile section is important or it will default to using kafka version 0.8.

Capt

@CaptainK5

THANK YOU. What version of Kafka does burrow 1.3.2 support? Is there a table that maps burrow version to Kafka version?

Capt

@CaptainK5

burrow 1.3.2/Kafka 2.4.0

Ron Farkash

@ronfarkash

Hey I am not always seeing client_id and owner when I consume messages, its kind of inconsistent. Is there a way to make it consistent or debug why?

Ron Farkash

@ronfarkash

Nevermind im stupid

rajncsu

@rajncsu

Looking for a way to disable "DELETE" buttons for a consumer or disable the UI write permission for all the users

Tronyx Chpwn

@Tronyx86_twitter

When using Burrow via the Docker container, is a container restart required whenever you make a change to the configuration file? If I want to add a new notification and restart the container, it ends up sending a bunch of alerts as Burrow is sorting things out and I am trying to avoid that.

Arushi Rai

@arushi315

Hi,
I have Burrow 1.3.6 configured with Kafka 2.7, it works fine but fails when I upgrade Kafka to 2.8.1 with error - [ failed to get partition count","type":"module","coordinator":"consumer","class":"kafka","name":"local-kafka","topic":"__consumer_offsets","error":"kafka server: Request was for a topic or partition that does not exist on this broker"]
I also tested with Burrow 1.3.8 but no luck.

Peter Bukowinski

@pmbuko

@arushi315 Burrow depends on sarama, and kafka 2.8 support was added to sarama in version 1.29.0 (https://github.com/Shopify/sarama/releases/tag/v1.29.0). Burrow 1.3.7 and 1.3.8 use sarama 1.30.0 (https://github.com/linkedin/Burrow/blob/v1.3.7/go.mod#L7), so it should support kafka 2.8.

Arushi Rai

@arushi315

Thanks @pmbuko I checked that too and so tested with Burrow version 1.3.8 as well as 1.4.0 but no luck.
They all work fine with Kafka version 2.7.0 but fail with Kafka version 2.8.1.

I also created a ticket for burrow here linkedin/Burrow#737

deathyard

@deathyard

hello ?

is there any way to restart burrow

or a way such that it constantly reads config, even when changed

Where communities thrive

People

Repo info

Activity