by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Frédéric G. MARAND
    @fgm
    Issue is at arnaud-lb/php-rdkafka#88
    Paul Dragoonis
    @dragoonis
    Hello everyone! :D
    Hey @edenhill great to see you here :)
    Paul Dragoonis
    @dragoonis
    For the life of me I can't get the broker address to change from 127.0.0.1 :(
        private function initialize(array $brokers)
        {
            $conf = new ConsumerConf();
            $conf->set('metadata.broker.list', implode(',', $brokers));
            $consumer = new Consumer($conf);
    
            $consumer->addBrokers(implode(',', $brokers));
            $this->setConsumer($consumer);
        }
    didn't work with just ->addBrokers(), tried the metadata.broker.list too, still tries to go to 127.0.0.1 :( :(
    Paul Dragoonis
    @dragoonis
    Nobody seems to be around - I've raised this bug - arnaud-lb/php-rdkafka#92
    Magnus Edenhill
    @edenhill
    You shouldn't use addBrokers, config is enough
    Paul Dragoonis
    @dragoonis
    let me remove addBrokers()
    Magnus Edenhill
    @edenhill
    Can you print the string you set?
    Paul Dragoonis
    @dragoonis
    No change - yes here's the string.
            $conf = new ConsumerConf();
            $conf->set('metadata.broker.list', 'xx.xx.xx.xx:9092');
            $consumer = new Consumer($conf);
    %3|1479475193.723|ERROR|rdkafka#consumer-1| 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused
    Magnus Edenhill
    @edenhill
    Ah, 127.0.0.1 is coming from the broker. Check its advertised.listener config
    Paul Dragoonis
    @dragoonis
    interesting, do you think ti's talking to remote broker
    but broker isn't listening on external IP ?
    suggestion on how I can run sniffer on remote server, on specific port ? to check incoming traffic
    I'm running kafka using this docker IMG - https://hub.docker.com/r/wurstmeister/kafka/
    aha
    docker-compose.yml was set to this .. tweaking
          KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
    Paul Dragoonis
    @dragoonis
    I'm able to connect to the broker now, that ENV var becomes advertised.host.name
    still some issues, but this is good progress - thanks @edenhill
    Magnus Edenhill
    @edenhill
    👍
    Paul Dragoonis
    @dragoonis
    I'm replacing another tech stack with a kafka solution, I need to learn kafka FAST! what's your suggestions on best places to learn from ?
    Paul Dragoonis
    @dragoonis
    @edenhill this is what's being set in server.properties
    #advertised.listeners=PLAINTEXT://your.host.name:9092
    it's commented out ..
    but these are at the end of the file
    port=9092
    create.topics=users:1:1
    advertised.host.name=xx.xx.xx.xx
    advertised.port=9092
    Magnus Edenhill
    @edenhill
    Make sure that host and port is reachable from the client
    And I believe.. Host.name was replaced by advertised.listeners
    Paul Dragoonis
    @dragoonis
    indeed, i am on kafka 0.10
    so i removed advertised.host/port, advertised.listeners=PLAINTEXT://<my ip>:9092
    however when kafka boots, it doesn't fully finish booting, just hangs there (after making the topic)
    Magnus Edenhill
    @edenhill
    Maybe it fails to reach other brokers, or zk
    Paul Dragoonis
    @dragoonis
    Only have 1 broker, this one
    it can access ZK
    Paul Dragoonis
    @dragoonis
    @edenhill which versions of kafka does php-rdkafka support? README says
    PHP-rdkafka is a thin librdkafka binding providing a working PHP 5 / PHP 7 Kafka 0.8 / 0.9 client.
    does this mean 0.10 is not supported ?
    Magnus Edenhill
    @edenhill
    It supports all >=0.8 versions
    Paul Dragoonis
    @dragoonis
    That's good to hear
    Paul Dragoonis
    @dragoonis
    @edenhill can you help me find out what version of the kafka protocol my PHP ext is using?
    also does the Producer class require access to remote zookeeper?
    $rk = new RdKafka\Producer($conf);
    Magnus Edenhill
    @edenhill
    @dragoonis ZK is not used by clients.
    @dragoonis you decide what protocol version to use, see this https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility
    Paul Dragoonis
    @dragoonis

    @edenhill thanks for replying! appreciated - I have this offset bug that I can't figure out - maybe you can spot something?

    My consumer here that i've built - using STORED offset. The bug behaviour is:

    Did this make sense ?

    (dunno how to make that gist block smaller on this chat, srry)
    Magnus Edenhill
    @edenhill
    seems like you are never committing the offsets you store (since auto commit is disabled), so whenever you restart it wont find a committed offset and thus resort to auto.offset.reset which defaults to latest
    Paul Dragoonis
    @dragoonis
    I am committing the offset when I get EOF (no more data)
    Magnus Edenhill
    @edenhill
    it is also kind of weird to only store (and I guess you also want commit) when you reach EOF. If your consumer is slower than the producer, for whatever reason, you will never reach EOF and thus never commit.
    Paul Dragoonis
    @dragoonis
    @edenhill should I be calling ->storeOffset() upon every entry into RD_KAFKA_RESP_ERR_NO_ERROR? If not .. where?
    Magnus Edenhill
    @edenhill
    no, you are storing the offset for future commit, but you are never committing