Burrow/Lobby

LinkedIn Burrow - Consumer lag checking for Apache Kafka

Suramya Shah

@ss22ever

Hi I am trying to use burrow with amazon msk version 1.1.1 with traffic between clients and brokers as only TLS encrypted traffic is allowed.
I tried using the following properties:

[tls.tlsonly]
certfile="/opt/ssl/kafkakeystore/client.crt"
keyfile="/opt/ssl/kafkakeystore/client.key"
cafile="/opt/ssl/kafkakeystore/ca.crt"
noverify=false

But I am getting the error level":"error","ts":1581337063.4007235,"msg":"failed to start client","type":"module","coordinator":"cluster","class":"kafka","name":"pre-glp1-glp2-int-msk3","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

KyleBoyer

@KyleBoyer

If a topic is removed from a consumer group’s subscribed list, when will Burrow remove that topic from the consumer group’s results?

Ivan Majnaric

@IvanMajnari_twitter

Hello :=)
Sorry for silly question maybe, but, is there any implementation of metrics being sent to prometheus from Burrow?

KyleBoyer

@KyleBoyer

@IvanMajnari_twitter try https://github.com/shakil/burrow_exporter

Typo^

https://github.com/shamil/burrow_exporter

adosapati

@adosapati

Hey, Anyone monitoring storm consumers using burrow?

we use 1.2.0 version of burrow and storm consumers don't appear there

Ivan Majnaric

@IvanMajnari_twitter

@KyleBoyer Thanks! Will try it tomorrow! Do you maybe know how reliable that exporter really? Have you tried it? :)

KyleBoyer

@KyleBoyer

Yup, I have/currently use it! Works really well!

frankhenderson

@frankhenderson

hi everyone :) I've got Burrow 1.3.2 and generalmills/burrowui running and I'm wondering about the one consumer that seems unlike all the others ... its lag is very high and its topic is __consumer_offsets -- Is my setup missing something? Or is it normal that the total lag for this consumer keeps increasing and it has an ERR state?

frankhenderson

@frankhenderson

This lagging consumer's name seems related to a section of my config which I called [consumer.myconsumers] ... the consumer's name is burrow-myconsumers

And that section of my config looks like this:

[consumer.myconsumers]
class-name="kafka"
cluster="mycluster"
servers=["host1:6667","host2:6667","host3:6667"]

frankhenderson

@frankhenderson

All of my other consumers have a total lag of 0. As for kafka version, I have this file: kafka_2.12-2.4.0.jar

frankhenderson

@frankhenderson

I've seen my other consumers occasionally have total lag > 0 and then later they recover back to total lag == 0 ... but the lag for this one special consumer that refers to topic __consumer_offsets keeps going up. It's over 14 million now and in ERR state.

Evgeniy Zakharov

@EvgeniyZ

hi there, I'm experiencing issues with burrow on windows 10.
I cloned a repository and run commands
docker-compose build
and
docker-compose up
and
burrow dashboard failed to start, I checked logs and see

Reading configuration from /etc/burrow
Failed reading configuration: Config File "burrow" Not Found in "[/etc/burrow]"

I assume that the configuration file does not copy into container volume,
but it should be copied by default from docker-config folder

Any suggestions on what should I do?

Ivan Majnarić

@iMajna

Hi all,
Question about burrow capability regarding authentication. I have Kafka which can handle both SASL_PLAINTEXT(GSSAPI) and SSL authentication, but my Zookeeper has only sasl as a client Auth scheme.
Can I somehow connect Burrow to that cluster? Tried with SSL but as you can imagine I'm getting "zk: not authenticated". I havent even tried Kerberos authentication with keytab because I can see in docs that its not really supported.
Anyone has any ideas?

Durai

@duraimurugantest

Hi All,
Im working on to setup burrowv3 to monitor 2 kafka cluster consumers in a same configuration,
and added 2 kafka clusters end point in the same configuration by dividing it based on cluster name
But it works for only one cluster not for other cluster where i can see output of prod cluster but not of stg

Myquestion is - Can we use 2 kafka cluster endpoints in the same configuration to monitor?
If yes, Can someone help me to make necessary changes.

Ivan Majnarić

@iMajna

Question about burrow email notifer.
Is it possible to adjust in some settings (bcs seems I can't find it) that I receive only limited number of notif emails about CG error status? Example: once CG is triggered and being marked as STOP burrow is not stopping with sending emails. Can I limit that somehow? Thanks :)

Leon

@LamedB_twitter

I used the latest version with docker, and when I calling the /metrics I can't see the burrow_kafka_consumer_partition_lag metics. I can see all the burrow_kafka_topic_partition_offset but I want the lag, what might be wrong?

Peter Bukowinski

@pmbuko

Greetings! @bai I see a prometheus metrics exporter was added here linkedin/Burrow#628 but is not documentation. I’m interested in using the feature. I see there’s a /metrics endpoint, but am confused about how to use it.

Peter Bukowinski

@pmbuko

Oops. Never mind. I upgraded a burrow instance and was trying to hit the metrics endpoint, but my url was incorrect and pointing at an older instance. It’s working.

Todd Palino

@toddpalino

I've just created a PR to change whitelist/blacklist to allowlist/denylist. This is going to be a breaking change - use of the old names will cause the app to print out a sensible message and panic.

I'll leave the PR there for a little bit before merging it for any discussion or concerns

Peter Bukowinski

@pmbuko

@toddpalino :thumbsup:

Lester Fan

@lesterfan

Hi everyone, I have a question on how Burrow's "completeness" value works. From what I can tell, completeness is defined as the percentage of information Burrow has in its internal sliding window. So with an intervals=10 configuration and a group which has committed 6 offsets for a partition, the completeness for that (group, partition) pair would be 0.6. As the window only moves forwards upon seeing a committed offset, this should mean that completeness should never decrease unless expire-group seconds passes (default 7 days). Does this sound accurate?

Peter Bukowinski

@pmbuko

I’m looking for a sanity check on burrow and promethues. I still need a “middle-man” to poll the /metrics endpoint on a regular interval and dump that data into prometheus, right?

Prakash-droid

@Prakash-droid

I am getting EOF error while sending emails . Anyone encountered this error before? please let me know. level":"error","ts":1600157776.7377965,"msg":"failed to send","type":"module","coordinator":"notifier","class":"email","name":"msk","cluster":"dc-msk-ci","group":"perf-consumer-81913","id":"8c8c88b3-bd4f-4e1d-b3fb-699fe3b1a834","status":"ERR","error":"EOF"}

Config file entry for notifier:
[notifier.msk]
class-name="email"
interval=30
threshold=3
group-allowlist="^perf.*$"
group-denylist="^perf-consumer-46470$"
extras={ key1="value1", key2="value2" }
send-close=false
template-open="/root/go/src/github.com/linkedin/Burrow/config/msk-email.tmpl"
server="10.242.48.73"
port=4001
from="xxx@xx.com"
to="xxx@xx.com"

Oliver Maskery

@omaskery

@LamedB_twitter

I used the latest version with docker, and when I calling the /metrics I can't see the burrow_kafka_consumer_partition_lag metics. I can see all the burrow_kafka_topic_partition_offset but I want the lag, what might be wrong?

Was there a solution to this issue? I'm having the same problem: I've successfully build and deployed Burrow v1.3.4 in a k8s cluster, it seemingly can talk to Kafka & Zookeeper successfully, but the only Prometheus metrics it exports are burrow_kafka_topic_partition_offset when really what I want is burrow_kafka_consumer_lag_total. I feel I must have done something wrong, though in a test environment with broadly the same setup I had success, so I'm confused about what I could've done wrong :sweat_smile:

1 reply

Ivan Majnarić

@iMajna

Hi guys,

Have a question whether you have an issue with group-denylist. I created a regex and config looks like this:

group-denylist="ˆ(?!crmgroup-222|issgroup-333).*$)"

Issue I'm meeting with is that I'm getting error mail if some other group which is not covered in regex like "test-group" is having an issue. I don't want to get an error email group for that one and that's the reason why I created this regex. No, anyone has similar experience or I'm doing something wrong?

Ivan Majnarić

@iMajna

EDIT:

group-denylist="ˆ(?!crmgroup-222|issgroup-333).*$"

Rodrigo Pereira

@voiprodrigo

Hello everyone.

This is probably a F.A.Q... I have a consumer group created by Burrow itself, named burrow-clustername. The lag on this group grows continuously. But it doesn't seem to be impacting lag detection for the normal groups. Is this normal?

Rushikesh Thakur

@RushikeshThakur

Hello Everyone, could you tell me .How to customize email notification template in burrow ?

Rushikesh Thakur

@RushikeshThakur

Hello Everyone, Does Burrow generated load on kafka. while getting data from Kafka Cluster ?

saritago

@saritago

Hi, We are trying to setup burrow for a secured kafka cluster. Is there any documentation for this? We have SASl_PLAIN authentication setup. Has someone done similar setup?

Peter Bukowinski

@pmbuko

@saritago Have you looked at this section of the docs? https://github.com/linkedin/Burrow/wiki/Configuration#sasl-profile. You’ll need to add this section to the burrow config and then reference it by name in the client-profile section of the config — https://github.com/linkedin/Burrow/wiki/Configuration#client-profile.

saritago

@saritago

@pmbuko did try it, but was still getting same “ "error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

So wanted to look at a sample.. have mentioned details as below

Sasl=PLAIN

[sasl.PLAIN]
username="Adminuser"
password="Adminpassword"
handshake-first=false

saritago

@saritago

any other settings in need to add? Have added above details in respective sections..

Peter Bukowinski

@pmbuko

[client-profile.MYCLIENT]
kafka-version="2.x.x"
client-id="burrow-myclient"
sasl="PLAIN"

[sasl.PLAIN]
username="Adminuser"
password="Adminpassword"
handshake-first=false

[consumer.CLUSTER]
class-name="kafka"
cluster="CLUSTER"
client-profile="MYCLIENT"

That should be all you need.

saritago

@saritago

@pmbuko Thanks! Below config worked for us. Please make a note that both sasl and tls details had to be mentioned for burrow to work. Dint work with either of them missing.

[client-profile.test]
client-id="burrow-test"
kafka-version="0.11.0.2"
sasl="PLAINSS"
tls="mytlsprofile"

[sasl.PLAINSS]
handshake-first=true
username="superuser"
password="superuserpassword"

[tls.mytlsprofile]
certfile="path/to/certfile/file/wildcard.jks"
keyfile="path/to/keyfile/file/wildcard.jks"
noverify=true

saritago

@saritago

Hi, facing issue with prometheus metrics. We are just getting burrow_kafka_topic_partition_offset and no other metrics, where as ideally we should have got all of these "burrow_kafka_consumer_partition_lag", "burrow_kafka_consumer_current_offset", "burrow_kafka_consumer_status".

Any configurations that needs to be updated to get all the metrics.
We are using kafka version 0.11.0.3. and its a secured kafka cluster.

1 reply

Lawrence Kim

@pr-lawrence

Hello, guys. I'm waiting for the next release to using Kafka 2.6.0. I wonder if you have any plans.

Where communities thrive

People

Repo info

Activity