Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jul 27 21:25
    ashishvashisht1 opened #761
  • Jul 26 15:03
    swapnildangorechc opened #760
  • Jul 14 16:31
    nkostoulas opened #759
  • Jul 13 09:32
    DaveCLowe edited #758
  • Jul 13 09:29
    DaveCLowe opened #758
  • Jun 15 02:22
    swoogi review_requested #757
  • Jun 15 02:22
    swoogi opened #757
  • Jun 15 02:22
    swoogi closed #754
  • Jun 15 02:21
    swoogi synchronize #754
  • Jun 15 02:18
    swoogi review_requested #756
  • Jun 15 02:18
    swoogi opened #756
  • Jun 15 02:15
    swoogi closed #755
  • Jun 15 01:53
    swoogi review_requested #755
  • Jun 15 01:53
    swoogi opened #755
  • Jun 15 01:21
    swoogi review_requested #754
  • Jun 15 01:21
    swoogi opened #754
  • Jun 12 08:08
    daodennis closed #539
  • May 27 14:58
    sungpeo edited #753
  • May 27 14:55
    sungpeo opened #753
  • May 27 14:55
    sungpeo review_requested #753
Evgeniy Zakharov
@EvgeniyZ
I assume that the configuration file does not copy into container volume,
but it should be copied by default from docker-config folder
Any suggestions on what should I do?
Ivan Majnarić
@iMajna
Hi all,
Question about burrow capability regarding authentication. I have Kafka which can handle both SASL_PLAINTEXT(GSSAPI) and SSL authentication, but my Zookeeper has only sasl as a client Auth scheme.
Can I somehow connect Burrow to that cluster? Tried with SSL but as you can imagine I'm getting "zk: not authenticated". I havent even tried Kerberos authentication with keytab because I can see in docs that its not really supported.
Anyone has any ideas?
Durai
@duraimurugantest

Hi All,
Im working on to setup burrowv3 to monitor 2 kafka cluster consumers in a same configuration,
and added 2 kafka clusters end point in the same configuration by dividing it based on cluster name
But it works for only one cluster not for other cluster where i can see output of prod cluster but not of stg

Myquestion is - Can we use 2 kafka cluster endpoints in the same configuration to monitor?
If yes, Can someone help me to make necessary changes.

kafka.PNG
Ivan Majnarić
@iMajna
Question about burrow email notifer.
Is it possible to adjust in some settings (bcs seems I can't find it) that I receive only limited number of notif emails about CG error status? Example: once CG is triggered and being marked as STOP burrow is not stopping with sending emails. Can I limit that somehow? Thanks :)
Leon
@LamedB_twitter
I used the latest version with docker, and when I calling the /metrics I can't see the burrow_kafka_consumer_partition_lag metics. I can see all the burrow_kafka_topic_partition_offset but I want the lag, what might be wrong?
Peter Bukowinski
@pmbuko
Greetings! @bai I see a prometheus metrics exporter was added here linkedin/Burrow#628 but is not documentation. I’m interested in using the feature. I see there’s a /metrics endpoint, but am confused about how to use it.
Peter Bukowinski
@pmbuko
Oops. Never mind. I upgraded a burrow instance and was trying to hit the metrics endpoint, but my url was incorrect and pointing at an older instance. It’s working.
Todd Palino
@toddpalino
I've just created a PR to change whitelist/blacklist to allowlist/denylist. This is going to be a breaking change - use of the old names will cause the app to print out a sensible message and panic.
I'll leave the PR there for a little bit before merging it for any discussion or concerns
Peter Bukowinski
@pmbuko
@toddpalino :thumbsup:
Lester Fan
@lesterfan
Hi everyone, I have a question on how Burrow's "completeness" value works. From what I can tell, completeness is defined as the percentage of information Burrow has in its internal sliding window. So with an intervals=10 configuration and a group which has committed 6 offsets for a partition, the completeness for that (group, partition) pair would be 0.6. As the window only moves forwards upon seeing a committed offset, this should mean that completeness should never decrease unless expire-group seconds passes (default 7 days). Does this sound accurate?
Peter Bukowinski
@pmbuko
I’m looking for a sanity check on burrow and promethues. I still need a “middle-man” to poll the /metrics endpoint on a regular interval and dump that data into prometheus, right?
Prakash-droid
@Prakash-droid
I am getting EOF error while sending emails . Anyone encountered this error before? please let me know. level":"error","ts":1600157776.7377965,"msg":"failed to send","type":"module","coordinator":"notifier","class":"email","name":"msk","cluster":"dc-msk-ci","group":"perf-consumer-81913","id":"8c8c88b3-bd4f-4e1d-b3fb-699fe3b1a834","status":"ERR","error":"EOF"}
Config file entry for notifier:
[notifier.msk]
class-name="email"
interval=30
threshold=3
group-allowlist="^perf.*$"
group-denylist="^perf-consumer-46470$"
extras={ key1="value1", key2="value2" }
send-close=false
template-open="/root/go/src/github.com/linkedin/Burrow/config/msk-email.tmpl"
server="10.242.48.73"
port=4001
from="xxx@xx.com"
to="xxx@xx.com"
Oliver Maskery
@omaskery

@LamedB_twitter

I used the latest version with docker, and when I calling the /metrics I can't see the burrow_kafka_consumer_partition_lag metics. I can see all the burrow_kafka_topic_partition_offset but I want the lag, what might be wrong?

Was there a solution to this issue? I'm having the same problem: I've successfully build and deployed Burrow v1.3.4 in a k8s cluster, it seemingly can talk to Kafka & Zookeeper successfully, but the only Prometheus metrics it exports are burrow_kafka_topic_partition_offset when really what I want is burrow_kafka_consumer_lag_total. I feel I must have done something wrong, though in a test environment with broadly the same setup I had success, so I'm confused about what I could've done wrong :sweat_smile:

1 reply
Ivan Majnarić
@iMajna

Hi guys,

Have a question whether you have an issue with group-denylist. I created a regex and config looks like this:

group-denylist="ˆ(?!crmgroup-222|issgroup-333).*$)"

Issue I'm meeting with is that I'm getting error mail if some other group which is not covered in regex like "test-group" is having an issue. I don't want to get an error email group for that one and that's the reason why I created this regex. No, anyone has similar experience or I'm doing something wrong?

Ivan Majnarić
@iMajna
EDIT:
group-denylist="ˆ(?!crmgroup-222|issgroup-333).*$"
Rodrigo Pereira
@voiprodrigo
Hello everyone.
This is probably a F.A.Q... I have a consumer group created by Burrow itself, named burrow-clustername. The lag on this group grows continuously. But it doesn't seem to be impacting lag detection for the normal groups. Is this normal?
Rushikesh Thakur
@RushikeshThakur
Hello Everyone, could you tell me .How to customize email notification template in burrow ?
Rushikesh Thakur
@RushikeshThakur
Hello Everyone, Does Burrow generated load on kafka. while getting data from Kafka Cluster ?
saritago
@saritago
Hi, We are trying to setup burrow for a secured kafka cluster. Is there any documentation for this? We have SASl_PLAIN authentication setup. Has someone done similar setup?
Peter Bukowinski
@pmbuko
@saritago Have you looked at this section of the docs? https://github.com/linkedin/Burrow/wiki/Configuration#sasl-profile. You’ll need to add this section to the burrow config and then reference it by name in the client-profile section of the config — https://github.com/linkedin/Burrow/wiki/Configuration#client-profile.
saritago
@saritago
@pmbuko did try it, but was still getting same “ "error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
So wanted to look at a sample.. have mentioned details as below
Sasl=PLAIN
[sasl.PLAIN]
username="Adminuser"
password="Adminpassword"
handshake-first=false
saritago
@saritago
any other settings in need to add? Have added above details in respective sections..
Peter Bukowinski
@pmbuko
[client-profile.MYCLIENT]
kafka-version="2.x.x"
client-id="burrow-myclient"
sasl="PLAIN"

[sasl.PLAIN]
username="Adminuser"
password="Adminpassword"
handshake-first=false

[consumer.CLUSTER]
class-name="kafka"
cluster="CLUSTER"
client-profile="MYCLIENT"
That should be all you need.
saritago
@saritago
@pmbuko Thanks! Below config worked for us. Please make a note that both sasl and tls details had to be mentioned for burrow to work. Dint work with either of them missing.
[client-profile.test]
client-id="burrow-test"
kafka-version="0.11.0.2"
sasl="PLAINSS"
tls="mytlsprofile"

[sasl.PLAINSS]
handshake-first=true
username="superuser"
password="superuserpassword"

[tls.mytlsprofile]
certfile="path/to/certfile/file/wildcard.jks"
keyfile="path/to/keyfile/file/wildcard.jks"
noverify=true
saritago
@saritago

Hi, facing issue with prometheus metrics. We are just getting burrow_kafka_topic_partition_offset and no other metrics, where as ideally we should have got all of these "burrow_kafka_consumer_partition_lag", "burrow_kafka_consumer_current_offset", "burrow_kafka_consumer_status".

Any configurations that needs to be updated to get all the metrics.
We are using kafka version 0.11.0.3. and its a secured kafka cluster.

1 reply
Lawrence Kim
@pr-lawrence
Hello, guys. I'm waiting for the next release to using Kafka 2.6.0. I wonder if you have any plans.
chetannarasimhaiah
@chetannarasimhaiah
Hi All, Any recommendations on the above issue which @saritago has raised.
Peter Bukowinski
@pmbuko
Is there a rule of thumb as to how many clusters/consumers/partitions a single instance of burrow can comfortably handle? What tends to be the limiting factor? I’ve been getting some erroneous consumer lag metrics recently and restarting burrow resolves them.
Capt
@CaptainK5
I'm running solsson/burrow 1.3.2 with Kafka 2.5.1 (cp-kafka). The endpoint "/v3/kafka/test-cluster/consumer" is returning "consumers:[]" (empty list of consumers). Our consumers are set to use auto-commits. I cannot hit the burrow endpoint to get lag information if I cannot get a list of consumers. Please let me know if anyone has any suggestions. Thank you.
Peter Bukowinski
@pmbuko
@CaptainK5 does the /v3/kafka endpoint list test-cluster?
Capt
@CaptainK5
@pmbuko '/v3/kafka' endpoint returns "clusters":["test-cluster"]
Capt
@CaptainK5
@pmbuko I did not setup client-profile, '/v3/kafka/test-cluser/' returns "kafka-version":"0.8" and a few other things null or empty, but my kafka version is 2.5.1, could this be the problem? Is client-profile required?
Capt
@CaptainK5
@pmbuko I added client-profile, but burrow does not recognize kafka-version greater than 1.0.0
Capt
@CaptainK5
@pmbuko '/v3/kafka/test-cluster' returns "kafka-version":"1.0.0" along with the other settings I've configured. "tls":null,"sasl":null. How do I know if I need to setup sasl? In client-profile, when I tried to set tls="burrow-tls" it crashes.
Capt
@CaptainK5
anyone out there can help me? i discovered that kafka.auto.commit was set to false, updated that to true. But it didn't fix the issue. I'm still getting empty list of consumers
Capt
@CaptainK5
bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --list --bootstrap-server localhost:9092, list the groups
Peter Bukowinski
@pmbuko
@CaptainK5 You should make sure the version of burrow you are using is up-to-date. The latest release supports up to 2.5: https://github.com/linkedin/Burrow/blob/v1.3.4/core/internal/helpers/sarama.go#L26-L59 — the client profile section is important or it will default to using kafka version 0.8.
Capt
@CaptainK5
THANK YOU. What version of Kafka does burrow 1.3.2 support? Is there a table that maps burrow version to Kafka version?
Capt
@CaptainK5
burrow 1.3.2/Kafka 2.4.0
Ron Farkash
@ronfarkash
Hey I am not always seeing client_id and owner when I consume messages, its kind of inconsistent. Is there a way to make it consistent or debug why?
Ron Farkash
@ronfarkash
Nevermind im stupid
rajncsu
@rajncsu
Looking for a way to disable "DELETE" buttons for a consumer or disable the UI write permission for all the users