Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 17:38
    mhowlett opened #2484
  • 12:02

    edenhill on v1.2.0-RC1b

    (compare)

  • 12:02

    edenhill on master

    nuget packaging/s3: support mor… Travis: fix ARCH env var (compare)

  • 11:06

    edenhill on v1.2.0-RC1

    (compare)

  • 11:06

    edenhill on master

    Bump version defines for v1.2.0… (compare)

  • Aug 19 20:43

    edenhill on master

    Optimization: avoid atomic fata… Optimize varint decoding, incre… (compare)

  • Aug 19 20:43

    edenhill on prodopt

    (compare)

  • Aug 19 20:43
    edenhill closed #2483
  • Aug 19 17:36

    edenhill on master

    Tidy up rd_kafka_new() conf fre… (compare)

  • Aug 19 17:36

    edenhill on newconffree

    (compare)

  • Aug 19 17:36
    edenhill closed #2479
  • Aug 19 15:52
    mhowlett commented #2479
  • Aug 19 15:47
    edenhill synchronize #2483
  • Aug 19 15:47

    edenhill on prodopt

    Optimize varint decoding, incre… (compare)

  • Aug 19 15:47

    edenhill on master

    Minor refactoring and making fu… Consumer: Fix memory leak for a… Test 0098: skip if auth/sec is … and 2 more (compare)

  • Aug 19 15:47

    edenhill on constxnfix

    (compare)

  • Aug 19 15:47
    edenhill closed #2482
  • Aug 19 15:11
    edenhill synchronize #2483
  • Aug 19 15:10

    edenhill on prodopt

    Optimize varint decoding, incre… (compare)

  • Aug 19 15:09
    edenhill synchronize #2483
adityanahan
@adityanahan
i am also trying
rd_kafka_conf_set(conf, "queue.buffering.max.ms", "500", NULL, 0); //linger ms
and
rd_kafka_conf_set(conf, "socket.max.fails ", "10", NULL, 0);
adityanahan
@adityanahan
i see these kind of logs
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3781, rtt 866.61ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3782, rtt 527.77ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3783, rtt 533.93ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3784, rtt 533.96ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3785, rtt 555.18ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3786, rtt 571.58ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3787, rtt 645.75ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3788, rtt 646.47ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3789, rtt 646.82ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3790, rtt 659.20ms)
Jul 3 12:47:03 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 3791, rtt 659.71ms)
sent and recevied should coorelate right
**8
if we take one example
Jul 3 12:49:32 localhost rdkafka[20826]: SEND: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Sent ProduceRequest (v3, 4043 bytes @ 0, CorrId 8722)
Jul 3 12:49:33 localhost rdkafka[20826]: RECV: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Received ProduceResponse (v3, 54 bytes, CorrId 8722, rtt 1585.88ms)
this is sent and received log for the same corrId
does this mean that, after 1585.88 ms we are getting a ack for the above message
Magnus Edenhill
@edenhill
yep
adityanahan
@adityanahan
when do we start seeing connection errors
if we don't get the response of acks in time
and we exceed our allowed failures ? is that the one governed by socket.fails ?
Magnus Edenhill
@edenhill
request.timeout.ms and socket.timeout.ms
you shouldnt change socket.max.fails
adityanahan
@adityanahan
i c
how does that affect ?
rd_kafka_conf_set(conf, "socket.timeout.ms", "300000", NULL, 0);
rd_kafka_conf_set(conf, "request.timeout.ms", "300000", NULL, 0);
for now changed it to 300 secs
adityanahan
@adityanahan
hi magnus
i am getting these logs
est (v3, 2738 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 3612 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 3177 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 6666 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 7971 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 5789 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 3177 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 3611 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 3613 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 5355 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 4046 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: RETRY: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Retrying ProduceRequest (v3, 3614 bytes, retry 1/3)
Jul 3 14:08:38 localhost rdkafka[15626]: FAIL: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: 19 request(s) timed out: disconnect (average rtt 2295.578ms)
Jul 3 14:08:38 localhost rdkafka[15626]: ERROR: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: 19 request(s) timed out: disconnect (average rtt 2295.578ms)
Jul 3 14:08:38 localhost rdkafka[15626]: FEATURE: rdkafka#producer-1: [thrd:ssl://10.6.39.62:9093/bootstrap]: ssl://10.6.39.62:9093/1001: Updated enabled protocol features -ApiVersion to MsgVer1,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2
after increasing the time u suggested
some more
Magnus Edenhill
@edenhill
That's what happens if your produce requests time out, they will be retried. Increase request.timeout.ms and socket.timeout.ms . If the network link is bad there isnt much a client can do than to retry
adityanahan
@adityanahan
ok
alian
@lianfulei
2019-07-18 08:33:24,418 - [ERROR] : [kafka] consumer error 182.168.1.3:9092/3: Disconnected (after 307230ms in state UP)
2019-07-18 08:33:24,419 - [ERROR] : [kafka] consumer error 192.168.1.2:9092/2: Disconnected (after 307230ms in state UP)
2019-07-18 08:33:24,419 - [ERROR] : [kafka] comsuming from : topic,partition 0,offset 0, Error Local: Broker transport failure.
2019-07-18 08:33:24,419 - [ERROR] : [kafka] consumer error 192.168.1.1:9092/1: Disconnected (after 307229ms in state UP)
2019-07-18 08:33:24,419 - [ERROR] : [kafka] consumer error 3/3 brokers are down
2019-07-18 08:33:25,418 - [ERROR] : [kafka] consumer error 192.168.1.3:9092/3: Connect to ipv4#192.168.1.3:9092 failed: 套接字操作尝试一个无法连接的主机。.. (after 999ms in state CONNECT)
2019-07-18 08:33:25,418 - [ERROR] : [kafka] consumer error 192.168.1.1:9092/1: Connect to ipv4#192.168.1.3:9092 failed: 套接字操作尝试一个无法连接的主机。.. (after 998ms in state CONNECT)
2019-07-18 08:33:25,419 - [ERROR] : [kafka] consumer error 192.168.1.2:9092/2: Connect to ipv4#192.168.1.3:9092 failed: 套接字操作尝试一个无法连接的主机。.. (after 999ms in state CONNECT)
What about this mistake?
@edenhill hello
alian
@lianfulei
Hello, Author. Can you help me?
Idan Adar
@IdanAdar
Hi. I know this place is for rdkafka and not node-rdkafka, but I'm out of ideas... It seems that with node:8.13-alpine all is good, but with node:8.16-rdkafka stuff break. We think it has to do with OpenSSL but it's not clear where the fault it... rdkafka, node-rdkafka, Node? I'm hoping for some community assistance here... Please see this ticket: Blizzard/node-rdkafka#649 cc: @edenhill
Magnus Edenhill
@edenhill
@IdanAdar Since OpenSSL is not ABI compatible between minor versions librdkafka needs to be built using the same OpenSSL version as on the system. E.g., librdkafka built for openssl 1.0.2s will not work with openssl 1.1.0
Idan Adar
@IdanAdar
Node:8:16-alpine is based on Alpine 3.9 which does support/provide OpenSSL 1.1.0 and Node supports this as well
Idan Adar
@IdanAdar
Could it be that the SSL version shipped in alpine is not compatible?
Idan Adar
@IdanAdar
@edenhill when do you intend on upgrading OpenSSL to 1.1.1?
Magnus Edenhill
@edenhill
@IdanAdar upgrade where? librdkafka will build fine with both OpenSSL 1.0.x and 1.1.x
Idan Adar
@IdanAdar
If so then I just don’t understand why is this failing for pretty much everyone who use node:8.16-alpine. It seems non of the involved parties is willing to help investigate this. Not alpine, not node, not node-rdkafka...
Magnus Edenhill
@edenhill
@IdanAdar Well, I don't know the first thing about node so I can't help at this point, but will be happy to answer specific questions that are not related to Node, or C modules in node.
Peter Bukowinski
@pmbuko
Is there a simple way to get topic-partition(s) that a consumer that’s part of a consumer group has been assigned without parsing the event string? We’re using confluent-kafka-go.
Peter Bukowinski
@pmbuko
Here’s a comment directly from the developer I’m assisting:

"We run an application on multiple hosts, each with a single consumer instance connected to one topic and being assigned one or more partitions. What I want to do is, on the host side, if the host receives a packet that is EOF from its consumer, it calls a helper function in the consumer code (Seek()) that offsets the partition pointer back to the beginning of the partition so we can start consuming from the beginning again, giving us the illusion of infinite data that we can replay at any rate we like, no matter how fast/slow new data is coming in.

Every time the consumer receives an EOF event, we want to reset the partition that that event came from to the beginning."

Stefan Breunig
@breunigs
Does librdkafka have a "recommended" approach for batch consuming if I want the batch to be per-partition? In the rdkafka_consume_batch.cpp example it appears to retrieve messages from all partitions. From my (client) side I guess I can break whenever I receive a message that's not from the previous partition, but then I have to seek/set the offset or handle this 1 message. I tested if I can use the partition EOF signal, but in my observations librdkafka will pass messages from various partitions even before reaching a partition's EOF. I guess manually handling assigned partitions might work, but that sounds weird as well. Right now we're simply grouping within the client, which works, but maybe this is something already offered by the lib?
Magnus Edenhill
@edenhill
@pmbuko call Assignment() to get the consumer's assignment
@breunigs what you can do is extract the per-partition queue, see queue_get_partition() and forward it to NULL and poll each partition queue explicitly.
Stefan Breunig
@breunigs
thank you