arnaud-lb/php-rdkafka

Thin PHP binding for librdkafka - Kafka client

Paul Dragoonis

@dragoonis

also does the Producer class require access to remote zookeeper?

$rk = new RdKafka\Producer($conf);

Magnus Edenhill

@edenhill

@dragoonis ZK is not used by clients.

@dragoonis you decide what protocol version to use, see this https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility

Paul Dragoonis

@dragoonis

@edenhill thanks for replying! appreciated - I have this offset bug that I can't figure out - maybe you can spot something?

My consumer here that i've built - using STORED offset. The bug behaviour is:

i can only consume messages on the partition when the consumer is listening at the same time.
if I produce msgs onto the partition while there is no consumer running, then I run the consumer, I just get put into this EOF section: https://gist.github.com/dragoonis/b7dfeef0b223d4879a48b53a35b0f6c4#file-kafkaconsumer-php-L61

Did this make sense ?

(dunno how to make that gist block smaller on this chat, srry)

Magnus Edenhill

@edenhill

seems like you are never committing the offsets you store (since auto commit is disabled), so whenever you restart it wont find a committed offset and thus resort to auto.offset.reset which defaults to latest

Paul Dragoonis

@dragoonis

I am committing the offset when I get EOF (no more data)

Magnus Edenhill

@edenhill

it is also kind of weird to only store (and I guess you also want commit) when you reach EOF. If your consumer is slower than the producer, for whatever reason, you will never reach EOF and thus never commit.

Paul Dragoonis

@dragoonis

@edenhill should I be calling ->storeOffset() upon every entry into RD_KAFKA_RESP_ERR_NO_ERROR? If not .. where?

Magnus Edenhill

@edenhill

no, you are storing the offset for future commit, but you are never committing

Paul Dragoonis

@dragoonis

I see!

Magnus Edenhill

@edenhill

offsetStore = store offset for future commit

commit = commit stored offset

Paul Dragoonis

@dragoonis

Totally understand 100% :) thanks

Magnus Edenhill

@edenhill

the nomenclature is kind of weird

STORED, store, commit

so your confusion is warranted

Srigi

@srigi

Hello everyone. I'm coming here from issues on GH asking before filling the issue.
I'm having troubles running even the simplest demo of RdKafka. I'm running the macOS 10.12.1 (Sierra). My PHP is installed via homebrew. I tried both 5.6 and 7.0.

When I install/enable the rdkafka extension (via homebrew or manually compile, tried both) and then run $rk = new RdKafka\Producer(); I get segmentation fault. I tried both Apache & cli mode for the code.
However in phpinfo() everything looks OK. I tried homebrew's v0.9.1 and manually installed 1.0.0. v2.0.0 won't compile.
Any ideas? Is there a way get more info from that segfault?

Srigi

@srigi

OK, for PHP7, you must use rdkafka 2.0.x.

Paul Dragoonis

@dragoonis

@srigi hello! I'm using PHP5.6, with Kafka 0.10. Only started few days go. Did you get your segfault fixed?

For php-rdkafka, there's a php7 branch you can use, which will be more uptodate than the 2.0.x tag. https://github.com/arnaud-lb/php-rdkafka/tree/php7

The readme says "git checkout php7" to get the php7 code. Hope that helps.

Srigi

@srigi

Well, today I tried start with the clean-sheet. I reinstalled cli-tools (compilers) from Apple, I reinstalled PHP7 with homebrew and then I compiled as @dragoonis suggested. Cloned & switched branch to php7. phpize, ./configure, make all, make install. All good, extesnsion enabled successfully (checked in phpinfo). But still segfault on $rk = new RdKafka\Producer().

I also tried running w/o php.ini with command

php -n -d extension="/usr/local/Cellar/php70/7.0.13_6/lib/php/extensions/no-debug-non-zts-20151012/rdkafka.so" cli-producer.php

For my project I will be forced to use Docker for development. In container everything works fine :)

Magnus Edenhill

@edenhill

@srigi does it write a 'core' file? if not, run ulimit -c unlimited first. Then do something like: gdb php core and then bt in gdb to get the backtrace

Paul Dragoonis

@dragoonis

@edenhill How do I get a 'core' file dumped onto disk when the segfault happens?

Trying to debug a segfault at my end, but ain't got a 'core' file.

This is from the PHP CLI btw, no apache/fpm here.

Magnus Edenhill

@edenhill

ulimit -c unlimited

Paul Dragoonis

@dragoonis

Yes, I ran that.

Magnus Edenhill

@edenhill

make sure the current directory is writable and there is no core file already

Paul Dragoonis

@dragoonis

done.

Magnus Edenhill

@edenhill

after that: YMMV depending on OS

Paul Dragoonis

@dragoonis

i done

gdb `which php`

Then it says: Reading symbols from /usr/bin/php...(no debugging symbols found)...done.

I got something with strace

Paul Dragoonis

@dragoonis

Does this make any sense? https://gist.githubusercontent.com/dragoonis/2ee1b2666de0c7282017623f51e51f50/raw/b09602c82d5f5b6431eef39d71943f0d7c3f9080/kafka.out

Magnus Edenhill

@edenhill

strace typically isnt usable for tracking down segfaults, it will only show syscalls so it is too coarse grained

what did 'bt' in gdb say?

Paul Dragoonis

@dragoonis

@edenhill i figured it out. It's nothing to do with kafka.

If you have a class constant, and you refer to it using the wrong case (like all lowercase, when it's defined as uppercase) PHP5.6 segfaults

Paul Dragoonis

@dragoonis

5.6.15 segfaults but 5.6.28 does not - must have been patched :)

Nico

@Nico_beBee_twitter

Hi, have a question, when I create a new TopicPartition, warnings and other advises like deprecateds, throw exceptions, but if I reset the php error handler, the problem persists. (PHP 5.6)

Arnaud Le Blanc

@arnaud-lb

Hi @Nico_beBee_twitter
Could you show the exact error messages ?

Nico

@Nico_beBee_twitter

There are no error messages, the error handler change and warnings, deprecated notifications and similar messages from other parts of the code causes exception when a TopicPartition object is created.

Nico

@Nico_beBee_twitter

For example:
// This error not throw exception
@trigger_error('', E_USER_DEPRECATED);
$partition = new TopicPartition($topic, $partition, $offset);
// This error now throw exception
@trigger_error('', E_USER_DEPRECATED);

Nastasia Saby

@NastasiaSaby

Hi everybody, can I ask you something? I'm trying to use the high level consumer with the example given in the documentation, but I have a problem.
The KafkaConsumer seems to commit my messages automatically when doing "consume" even if I put the "auto.commit.enable" or the "enable.auto.commit" to 0.
Is there something I do wrong? Could anyone help me?
I am working with PHP7 and rdkafka 2.0.1.

Magnus Edenhill

@edenhill

@NastasiaSaby that's enable.auto.commit (for the high level consumer), and it really should work.

what librdkafka version?

Where communities thrive

People

Repo info

Activity