Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    al-razi
    @allyraza
    just installed php-rdkafka on my macbook when I run my producer script I get a seg fault
    the producer script is from examples
    al-razi
    @allyraza
    php70, librdkafka 0.9.3, php70-rdkafka, kafka 0.10
    Craig Patrick
    @cpats007
    Hi - anyone know how I perform a simple check to see if rdkafka is connected or not?
    al-razi
    @allyraza
    channel is dead
    Aaron Gong
    @ais-one
    its holiday season
    Craig Patrick
    @cpats007
    anyone around?
    Aaron Gong
    @ais-one
    @patrick try the api to get topics as producer or as consumer
    Efimov Evgenij
    @edefimov

    Hi everyone! Does anybody know how to interrupt librdkafka internal threads using a signal and let the application process some signal? I have daemon+worker application and kafka is used inside the worker. When daemon receives SIGTERM it sends one to all workers. The problem is when the application is consuming a topic, the signal handler inside worker is not called. However using strace I can see, that process actually receives SIGTERM and hangs inside system calls like this:

    --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=30489, si_uid=1000} ---
    rt_sigreturn({mask=~[KILL TERM STOP RTMIN RT_1]}) = -1 EINTR (Interrupted system call)
    futex(0x5602a96ca6bc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 17, {1484151715, 231482000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
    futex(0x5602a96ca690, FUTEX_WAKE_PRIVATE, 1) = 0
    rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], ~[KILL TERM STOP RTMIN RT_1], 8) = 0
    rt_sigprocmask(SIG_SETMASK, ~[KILL TERM STOP RTMIN RT_1], NULL, 8) = 0
    rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], ~[KILL TERM STOP RTMIN RT_1], 8) = 0
    rt_sigprocmask(SIG_SETMASK, ~[KILL TERM STOP RTMIN RT_1], NULL, 8) = 0

    Lines after rt_sigreturn are repeated until the worker is killed after timeout.
    I'm using asynchronous signal processing from php 7.1 and when I replace kafka consuming with sleep function everything works well.

    Environment: php 7.1, php-rdkafka 3.0.0, librdkafka 0.9.2-237-gd3dcc0 (d3dcc0198517160b9c8e374da2e963f563eb2c6f)

    apocello2008
    @apocello2008
    Hi! How can i use correlation id in this lib? I need process signin over kafka, how can i do this?
    Magnus Edenhill
    @edenhill
    @apocello2008 what do you mean by process signin? client authentication?
    apocello2008
    @apocello2008
    @edenhill Yes. client send login and passwd, i process this information and make unique token and after this i need to send token to this user. But if i send this data to "consumer group" all consumers in this group get this info. Maybe i can create personal "consumer group" for every user but i think 100 000+ groups is bad idea..
    Craig Patrick
    @cpats007
    anyone have any ideas of how to improve performance? on php-fpm my app response times more than double when writing to Kafka
    Magnus Edenhill
    @edenhill
    @cpats007 measure and profile to find out what is taking time. Possibly the produce+ack. Then see queue.buffering.max.ms and socket.blocking.max.ms
    Craig Patrick
    @cpats007
    thanks @edenhill - so I'm not doing any ack at this stage - this is simply a "fire and forget" function - I just want to dump stuff into Kafka from PHP. I have altered the socket.blocking.max.ms down to 100 and still the same, plus we can't use the SIGIO stuff because it's php-fpm. I'll look at the queue.buffering.max.ms now
    it's literally logging the request information into Kafka from the app, one big JSON string, nothing complicated
    Craig Patrick
    @cpats007
    I have adjusted the queue.buffering.max.ms and it has made a big difference, thanks @edenhill ! - I'll monitor now and see how it goes, but really appreciate that - is there any way I can see if a message has failed to send - like the config values I'm changing wont alter the fact that it'll still send the message right? if that makes sense
    Magnus Edenhill
    @edenhill
    you register a delivery report callback that'll be called once for each message to tell if it succeeded delivery or failed (and why)
    if you use acks=0 (which you typically dont want to do, hello message loss) then the delivery report callback will be triggered as soon as the message has been sent to the broker.
    Craig Patrick
    @cpats007
    I tried that - but it didn't seem to work, the method was unsupported or something unless I was using the wrong methodology
    Craig Patrick
    @cpats007
    setDrMsgCb is not available in the Conf() object?
    Magnus Edenhill
    @edenhill
    Looks like it is. maybe you have an old version
    Craig Patrick
    @cpats007
    hmmmm
    Magnus Edenhill
    @edenhill
    anyhoo, Ive never used the php client so I dont know. Not erally sure what Im doing here! ;)
    Craig Patrick
    @cpats007
    ha ha ha, well you've certainly given me a huge step forward!
    could it be because I'm using the stubs? and maybe the stubs are outdated?
    Craig Patrick
    @cpats007
    Okay, so by the looks of it, I have version 0.9.2 (latest released version) but there is a version 3.0 - however it wont install
    Paul Dragoonis
    @dragoonis
    Happy new year everyone :)
    @edenhill can you spare time to look at a bug report that I'm about to file with librdkafka ? (seg fault) Happening with PHP 5.6 and PHP7
    Magnus Edenhill
    @edenhill
    sure
    Paul Dragoonis
    @dragoonis
    I'm busy gathering all info from the checklist on your "create issue" page.
    Magnus Edenhill
    @edenhill
    I will need a C backtrace or similar though.
    Paul Dragoonis
    @dragoonis
    ..................FFF........S.SS..[New Thread 0x7fffdfccf700 (LWP 20884)]
    [Thread 0x7fffdfccf700 (LWP 20884) exited]
    [New Thread 0x7fffdfccf700 (LWP 20886)]
    [Thread 0x7fffdfccf700 (LWP 20886) exited]
    .......[New Thread 0x7fffdfccf700 (LWP 20887)]
    [Thread 0x7fffdfccf700 (LWP 20887) exited]
    [New Thread 0x7fffdfccf700 (LWP 20888)]
    [Thread 0x7fffdfccf700 (LWP 20888) exited]
    ...........
    Program received signal SIGSEGV, Segmentation fault.
    0x00007fffec678a3b in rd_kafka_new () from /lib64/librdkafka.so.1
    (gdb) quit
    Magnus Edenhill
    @edenhill
    bt
    Paul Dragoonis
    @dragoonis
    Do you want me to type bt to get the gdb backtrace? (it's not me running this, it's another team member)
    Sure! Will do
    Magnus Edenhill
    @edenhill
    yah
    Paul Dragoonis
    @dragoonis
    @edenhill 0.92 was segfaulting, we're pulling down latest librdkafka version (runtime) => 0.9.2-257-g65c8bc and it's working OK
    So you've patched it. It may be this. edenhill/librdkafka#850 - I'm getting a bt so we know what issue we ran into though. To close this off :)
    Magnus Edenhill
    @edenhill
    :+1:
    Paul Dragoonis
    @dragoonis
    @edenhill can you tell me when the next stable release is planned?
    Magnus Edenhill
    @edenhill
    0.9.3 is a maint release which is due in about 1-2 weeks
    then there'll be a new feature release in a month or so
    Paul Dragoonis
    @dragoonis
    i have bt
    Magnus Edenhill
    @edenhill
    it'd be great if you want to test the 0.9.3.x branch to make sure the above crash is fixed
    Magnus Edenhill
    @edenhill
    yep