Burrow/Lobby

LinkedIn Burrow - Consumer lag checking for Apache Kafka

Oliver Maskery

@omaskery

@LamedB_twitter

I used the latest version with docker, and when I calling the /metrics I can't see the burrow_kafka_consumer_partition_lag metics. I can see all the burrow_kafka_topic_partition_offset but I want the lag, what might be wrong?

Was there a solution to this issue? I'm having the same problem: I've successfully build and deployed Burrow v1.3.4 in a k8s cluster, it seemingly can talk to Kafka & Zookeeper successfully, but the only Prometheus metrics it exports are burrow_kafka_topic_partition_offset when really what I want is burrow_kafka_consumer_lag_total. I feel I must have done something wrong, though in a test environment with broadly the same setup I had success, so I'm confused about what I could've done wrong :sweat_smile:

1 reply

Ivan Majnarić

@iMajna

Hi guys,

Have a question whether you have an issue with group-denylist. I created a regex and config looks like this:

group-denylist="ˆ(?!crmgroup-222|issgroup-333).*$)"

Issue I'm meeting with is that I'm getting error mail if some other group which is not covered in regex like "test-group" is having an issue. I don't want to get an error email group for that one and that's the reason why I created this regex. No, anyone has similar experience or I'm doing something wrong?

Ivan Majnarić

@iMajna

EDIT:

group-denylist="ˆ(?!crmgroup-222|issgroup-333).*$"

Rodrigo Pereira

@voiprodrigo

Hello everyone.

This is probably a F.A.Q... I have a consumer group created by Burrow itself, named burrow-clustername. The lag on this group grows continuously. But it doesn't seem to be impacting lag detection for the normal groups. Is this normal?

Rushikesh Thakur

@RushikeshThakur

Hello Everyone, could you tell me .How to customize email notification template in burrow ?

Rushikesh Thakur

@RushikeshThakur

Hello Everyone, Does Burrow generated load on kafka. while getting data from Kafka Cluster ?

saritago

@saritago

Hi, We are trying to setup burrow for a secured kafka cluster. Is there any documentation for this? We have SASl_PLAIN authentication setup. Has someone done similar setup?

Peter Bukowinski

@pmbuko

@saritago Have you looked at this section of the docs? https://github.com/linkedin/Burrow/wiki/Configuration#sasl-profile. You’ll need to add this section to the burrow config and then reference it by name in the client-profile section of the config — https://github.com/linkedin/Burrow/wiki/Configuration#client-profile.

saritago

@saritago

@pmbuko did try it, but was still getting same “ "error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

So wanted to look at a sample.. have mentioned details as below

Sasl=PLAIN

[sasl.PLAIN]
username="Adminuser"
password="Adminpassword"
handshake-first=false

saritago

@saritago

any other settings in need to add? Have added above details in respective sections..

Peter Bukowinski

@pmbuko

[client-profile.MYCLIENT]
kafka-version="2.x.x"
client-id="burrow-myclient"
sasl="PLAIN"

[sasl.PLAIN]
username="Adminuser"
password="Adminpassword"
handshake-first=false

[consumer.CLUSTER]
class-name="kafka"
cluster="CLUSTER"
client-profile="MYCLIENT"

That should be all you need.

saritago

@saritago

@pmbuko Thanks! Below config worked for us. Please make a note that both sasl and tls details had to be mentioned for burrow to work. Dint work with either of them missing.

[client-profile.test]
client-id="burrow-test"
kafka-version="0.11.0.2"
sasl="PLAINSS"
tls="mytlsprofile"

[sasl.PLAINSS]
handshake-first=true
username="superuser"
password="superuserpassword"

[tls.mytlsprofile]
certfile="path/to/certfile/file/wildcard.jks"
keyfile="path/to/keyfile/file/wildcard.jks"
noverify=true

saritago

@saritago

Hi, facing issue with prometheus metrics. We are just getting burrow_kafka_topic_partition_offset and no other metrics, where as ideally we should have got all of these "burrow_kafka_consumer_partition_lag", "burrow_kafka_consumer_current_offset", "burrow_kafka_consumer_status".

Any configurations that needs to be updated to get all the metrics.
We are using kafka version 0.11.0.3. and its a secured kafka cluster.

1 reply

Lawrence Kim

@pr-lawrence

Hello, guys. I'm waiting for the next release to using Kafka 2.6.0. I wonder if you have any plans.

chetannarasimhaiah

@chetannarasimhaiah

Hi All, Any recommendations on the above issue which @saritago has raised.

Peter Bukowinski

@pmbuko

Is there a rule of thumb as to how many clusters/consumers/partitions a single instance of burrow can comfortably handle? What tends to be the limiting factor? I’ve been getting some erroneous consumer lag metrics recently and restarting burrow resolves them.

Capt

@CaptainK5

I'm running solsson/burrow 1.3.2 with Kafka 2.5.1 (cp-kafka). The endpoint "/v3/kafka/test-cluster/consumer" is returning "consumers:[]" (empty list of consumers). Our consumers are set to use auto-commits. I cannot hit the burrow endpoint to get lag information if I cannot get a list of consumers. Please let me know if anyone has any suggestions. Thank you.

Peter Bukowinski

@pmbuko

@CaptainK5 does the /v3/kafka endpoint list test-cluster?

Capt

@CaptainK5

@pmbuko '/v3/kafka' endpoint returns "clusters":["test-cluster"]

Capt

@CaptainK5

@pmbuko I did not setup client-profile, '/v3/kafka/test-cluser/' returns "kafka-version":"0.8" and a few other things null or empty, but my kafka version is 2.5.1, could this be the problem? Is client-profile required?

Capt

@CaptainK5

@pmbuko I added client-profile, but burrow does not recognize kafka-version greater than 1.0.0

Capt

@CaptainK5

@pmbuko '/v3/kafka/test-cluster' returns "kafka-version":"1.0.0" along with the other settings I've configured. "tls":null,"sasl":null. How do I know if I need to setup sasl? In client-profile, when I tried to set tls="burrow-tls" it crashes.

Capt

@CaptainK5

anyone out there can help me? i discovered that kafka.auto.commit was set to false, updated that to true. But it didn't fix the issue. I'm still getting empty list of consumers

Capt

@CaptainK5

bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list --bootstrap-server localhost:9092, list the groups

Peter Bukowinski

@pmbuko

@CaptainK5 You should make sure the version of burrow you are using is up-to-date. The latest release supports up to 2.5: https://github.com/linkedin/Burrow/blob/v1.3.4/core/internal/helpers/sarama.go#L26-L59 — the client profile section is important or it will default to using kafka version 0.8.

Capt

@CaptainK5

THANK YOU. What version of Kafka does burrow 1.3.2 support? Is there a table that maps burrow version to Kafka version?

Capt

@CaptainK5

burrow 1.3.2/Kafka 2.4.0

Ron Farkash

@ronfarkash

Hey I am not always seeing client_id and owner when I consume messages, its kind of inconsistent. Is there a way to make it consistent or debug why?

Ron Farkash

@ronfarkash

Nevermind im stupid

rajncsu

@rajncsu

Looking for a way to disable "DELETE" buttons for a consumer or disable the UI write permission for all the users

Tronyx Chpwn

@Tronyx86_twitter

When using Burrow via the Docker container, is a container restart required whenever you make a change to the configuration file? If I want to add a new notification and restart the container, it ends up sending a bunch of alerts as Burrow is sorting things out and I am trying to avoid that.

Arushi Rai

@arushi315

Hi,
I have Burrow 1.3.6 configured with Kafka 2.7, it works fine but fails when I upgrade Kafka to 2.8.1 with error - [ failed to get partition count","type":"module","coordinator":"consumer","class":"kafka","name":"local-kafka","topic":"__consumer_offsets","error":"kafka server: Request was for a topic or partition that does not exist on this broker"]
I also tested with Burrow 1.3.8 but no luck.

Peter Bukowinski

@pmbuko

@arushi315 Burrow depends on sarama, and kafka 2.8 support was added to sarama in version 1.29.0 (https://github.com/Shopify/sarama/releases/tag/v1.29.0). Burrow 1.3.7 and 1.3.8 use sarama 1.30.0 (https://github.com/linkedin/Burrow/blob/v1.3.7/go.mod#L7), so it should support kafka 2.8.

Arushi Rai

@arushi315

Thanks @pmbuko I checked that too and so tested with Burrow version 1.3.8 as well as 1.4.0 but no luck.
They all work fine with Kafka version 2.7.0 but fail with Kafka version 2.8.1.

I also created a ticket for burrow here linkedin/Burrow#737

deathyard

@deathyard

hello ?

is there any way to restart burrow

or a way such that it constantly reads config, even when changed

Where communities thrive

People

Repo info

Activity