Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Paul Dragoonis
    @dragoonis
    still some issues, but this is good progress - thanks @edenhill
    Magnus Edenhill
    @edenhill
    👍
    Paul Dragoonis
    @dragoonis
    I'm replacing another tech stack with a kafka solution, I need to learn kafka FAST! what's your suggestions on best places to learn from ?
    Paul Dragoonis
    @dragoonis
    @edenhill this is what's being set in server.properties
    #advertised.listeners=PLAINTEXT://your.host.name:9092
    it's commented out ..
    but these are at the end of the file
    port=9092
    create.topics=users:1:1
    advertised.host.name=xx.xx.xx.xx
    advertised.port=9092
    Magnus Edenhill
    @edenhill
    Make sure that host and port is reachable from the client
    And I believe.. Host.name was replaced by advertised.listeners
    Paul Dragoonis
    @dragoonis
    indeed, i am on kafka 0.10
    so i removed advertised.host/port, advertised.listeners=PLAINTEXT://<my ip>:9092
    however when kafka boots, it doesn't fully finish booting, just hangs there (after making the topic)
    Magnus Edenhill
    @edenhill
    Maybe it fails to reach other brokers, or zk
    Paul Dragoonis
    @dragoonis
    Only have 1 broker, this one
    it can access ZK
    Paul Dragoonis
    @dragoonis
    @edenhill which versions of kafka does php-rdkafka support? README says
    PHP-rdkafka is a thin librdkafka binding providing a working PHP 5 / PHP 7 Kafka 0.8 / 0.9 client.
    does this mean 0.10 is not supported ?
    Magnus Edenhill
    @edenhill
    It supports all >=0.8 versions
    Paul Dragoonis
    @dragoonis
    That's good to hear
    Paul Dragoonis
    @dragoonis
    @edenhill can you help me find out what version of the kafka protocol my PHP ext is using?
    also does the Producer class require access to remote zookeeper?
    $rk = new RdKafka\Producer($conf);
    Magnus Edenhill
    @edenhill
    @dragoonis ZK is not used by clients.
    @dragoonis you decide what protocol version to use, see this https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility
    Paul Dragoonis
    @dragoonis

    @edenhill thanks for replying! appreciated - I have this offset bug that I can't figure out - maybe you can spot something?

    My consumer here that i've built - using STORED offset. The bug behaviour is:

    Did this make sense ?

    (dunno how to make that gist block smaller on this chat, srry)
    Magnus Edenhill
    @edenhill
    seems like you are never committing the offsets you store (since auto commit is disabled), so whenever you restart it wont find a committed offset and thus resort to auto.offset.reset which defaults to latest
    Paul Dragoonis
    @dragoonis
    I am committing the offset when I get EOF (no more data)
    Magnus Edenhill
    @edenhill
    it is also kind of weird to only store (and I guess you also want commit) when you reach EOF. If your consumer is slower than the producer, for whatever reason, you will never reach EOF and thus never commit.
    Paul Dragoonis
    @dragoonis
    @edenhill should I be calling ->storeOffset() upon every entry into RD_KAFKA_RESP_ERR_NO_ERROR? If not .. where?
    Magnus Edenhill
    @edenhill
    no, you are storing the offset for future commit, but you are never committing
    Paul Dragoonis
    @dragoonis
    I see!
    Magnus Edenhill
    @edenhill
    offsetStore = store offset for future commit
    commit = commit stored offset
    Paul Dragoonis
    @dragoonis
    Totally understand 100% :) thanks
    Magnus Edenhill
    @edenhill
    :)
    the nomenclature is kind of weird
    STORED, store, commit
    so your confusion is warranted
    Srigi
    @srigi

    Hello everyone. I'm coming here from issues on GH asking before filling the issue.
    I'm having troubles running even the simplest demo of RdKafka. I'm running the macOS 10.12.1 (Sierra). My PHP is installed via homebrew. I tried both 5.6 and 7.0.

    When I install/enable the rdkafka extension (via homebrew or manually compile, tried both) and then run $rk = new RdKafka\Producer(); I get segmentation fault. I tried both Apache & cli mode for the code.
    However in phpinfo() everything looks OK. I tried homebrew's v0.9.1 and manually installed 1.0.0. v2.0.0 won't compile.
    Any ideas? Is there a way get more info from that segfault?

    Srigi
    @srigi
    OK, for PHP7, you must use rdkafka 2.0.x.
    Paul Dragoonis
    @dragoonis
    @srigi hello! I'm using PHP5.6, with Kafka 0.10. Only started few days go. Did you get your segfault fixed?
    For php-rdkafka, there's a php7 branch you can use, which will be more uptodate than the 2.0.x tag. https://github.com/arnaud-lb/php-rdkafka/tree/php7
    The readme says "git checkout php7" to get the php7 code. Hope that helps.
    Srigi
    @srigi
    Well, today I tried start with the clean-sheet. I reinstalled cli-tools (compilers) from Apple, I reinstalled PHP7 with homebrew and then I compiled as @dragoonis suggested. Cloned & switched branch to php7. phpize, ./configure, make all, make install. All good, extesnsion enabled successfully (checked in phpinfo). But still segfault on $rk = new RdKafka\Producer().
    I also tried running w/o php.ini with command php -n -d extension="/usr/local/Cellar/php70/7.0.13_6/lib/php/extensions/no-debug-non-zts-20151012/rdkafka.so" cli-producer.php
    For my project I will be forced to use Docker for development. In container everything works fine :)
    Magnus Edenhill
    @edenhill
    @srigi does it write a 'core' file? if not, run ulimit -c unlimited first. Then do something like: gdb php core and then bt in gdb to get the backtrace
    Paul Dragoonis
    @dragoonis
    @edenhill How do I get a 'core' file dumped onto disk when the segfault happens?
    Trying to debug a segfault at my end, but ain't got a 'core' file.
    This is from the PHP CLI btw, no apache/fpm here.
    Magnus Edenhill
    @edenhill
    ulimit -c unlimited