rust-rdkafka/Lobby

Home of rust-rdkafka and kafka-view

Timothy Shoaf

@timshoaf

4+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++;[++++++++++++++++++++++++++++++++++++++4\65+.6555

;;;;;;;;;7

-*-.

Timothy Shoaf

@timshoaf

ऑ .

Timothy Shoaf

@timshoaf

oh wow, sorry guys, just got back into town, clearly my cat has been asking questions on my behalf

Ben Sully

@sd2k

hey everyone! is there any effort going into migrating rdkafka from futures 0.1 to std::future? happy to contribute (although i'm pretty clueless about async rust right now)

Thijs Cadier

@thijsc

@sd2k There hasn't been a lot of activity lately. My company depends on this crate, so we're willing to spend some more time on it going forward. Don't think anyone has worked on std future support yet.

Ben Sully

@sd2k

@thijsc I made a PR (fede1024/rust-rdkafka#143) and tests pass on nightly, but it obviously won't compile on stable until Rust 1.39. I'm also a bit skeptical that I did everything right, particularly with the executor::block_on calls

Sheriff

@sheriffio

Hi I'm having troubling getting my rust app to build with this library. I'm running on a mac and I get the following errors: error: linking with cc failed: exit code: 1

ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Please help!!

Thijs Cadier

@thijsc

@sheriffio I can't tell without the full error, but it's likely that it can't link openssl. If you have it installed via homebrew you might need to set some env vars.

Sheriff

@sheriffio

@thijsc unfortunately it wasn't openssl link. But what did find was something was up with some libraries in my apps target folder, maybe file link collisions or something. As soon as I deleted my target folder and rebuilt everything worked. Thank you for getting back to me, it helped me look at the problem differently.

Nikhil Benesch

@benesch

Hi folks, I’ll be helping out with rdkafka maintanance! I recently landed a bunch of build system fixes that should smooth out the linking errors that a lot of folks have been running into.

Philippe GASSMANN

@zenria

Hello, since a while (I think since the release of macos catalina), I've not able to build librdkafka-sys anymore on my system, it ends up in a strange error

Building and linking librdkafka statically
Running command: "./configure --disable-sasl --disable-ssl --disable-lz4" in dir: librdkafka
thread 'main' panicked at 'Command failed with error: No such file or directory (os error 2)', /Users/philou/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-1.0.0/build.rs:25:21

The error seems to come from the relative execution of configure: if I replace build.rs run_command_or_fail function with the one of the master it solves the issue
Do you know when a new release of rust-rdkafka will be out?

Ben Sully

@sd2k

@benesch hey! I'm wondering what you'd like to do about the two async PRs (#143 and #166) - the first is mine, and the second is based off mine. @dbcfd has made a PR to merge #166 into my branch, but it makes a lot of changes that I wouldn't know how to review. I could either keep my branch as is and have the PRs handled separately, or merge #166 in and just have a single large PR - let me know which you'd prefer.

Nate Mara

@nate-onesignal

@benesch what's the status of the migration to futures-0.3/tokio-0.2?

Nikhil Benesch

@benesch

The master branch is fully async/await/futures v0.3/tokio v0.2 enabled

Would love to hear from a few more folks before pulling the trigger on releasing 0.23

There is also the open question of fede1024/rust-rdkafka#192, though it's not blocking for 0.23

Mohammed Abubakar

@moh-abk

make the push

Oron Sharabi

@oronsh

Hey, how can I use regex in order to consume from * topics please?

["users.*.messages"]

Thanks :)

Nate Mara

@nate-onesignal

@benesch congrats on the release of 0.23! I noticed that the docs are not building https://docs.rs/crate/rdkafka/0.23.0, could you look into that?

Nikhil Benesch

@benesch

Thanks, @nate-onesignal! We've finally got things sorted with the new docs.rs environment for 0.23.1: https://docs.rs/rdkafka/0.23.1/rdkafka/

@oronsh you'll need to implement such a feature yourself. You can get a list of all topics with https://docs.rs/rdkafka/0.23.1/rdkafka/client/struct.Client.html#method.fetch_metadata, and then filter for topics that match your regex, and then subscribe to those topics.

Calvin Brown

@calvinbrown085

Hey all! I have a quick question, when using the MessageStream is there a way to get a batch of records per poll, instead of iteration over them as they come in 1 by one?

Nikhil Benesch

@benesch

Afraid not @calvinbrown085. You’d have to implement the batching in your own code.

vijay sali

@vijaysali_twitter

Hi folks,

I am building application with rdkafka-rust, version pinned to 0.22. I am compiling this to windows binary from linux box.
I have installed all the toolchain for windows.

Following this example

I am getting following errors during build.

error: linking with `/usr/bin/x86_64-w64-mingw32-gcc` failed: exit code: 1

= note: /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x2bf): undefined reference to `rd_kafka_opaque'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x3d8): undefined reference to `rd_kafka_queue_get_consumer'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x43b): undefined reference to `rd_kafka_conf_set_opaque'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x454): undefined reference to `rd_kafka_conf_set_log_cb'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x46d): undefined reference to `rd_kafka_conf_set_stats_cb'

Found nothing on web, any help is appreciated :)

Nikhil Benesch

@benesch

You’re mostly on your own if you’re trying to cross-compile, I’m afraid; there’s just so much voodoo involved

At the very least sure you’re using the cmake build; cross compilation with the default build system is extra unsupported

And here’s the list of environment variables that I needed to set to xcompile from macOS to Linux: https://github.com/MaterializeInc/materialize/blob/7696c8cdaba6b5d0d9225e82322d5c33fa8a09fb/bin/xcompile#L45-L58

Note that those are the variables required for a large project (Materialize) that depends on rdkafka in addition to a few other *-sys crates; not all of them are necessarily applicable to rdkafka

@vijaysali_twitter

vijay sali

@vijaysali_twitter

Thanks @benesch , I shall try out these.

Daniel Samson

@daniel-samson

Hey guys, I am working on a migration tool with rdkafka-rust. Is there a way to mix rdkafka-rust with rayon instead of tokio?

i can't seem to to get the await to work? any ideas?

  metrics.into_par_iter().for_each(|metric: Metric| {
            let json = serde_json::to_string(&metric);
            if let Err(e) = &json {
                log::error!("{}", e)
                // TODO: IMPORTANT: FIX the deserialization issue where everything comes back as null
            }

            if let Ok(j) = &json {
                log::info!("{}", j);
                    producer.send(FutureRecord::to("METRICS").payload(j).key(""), 0);
                    pb.inc(1);
            }
        });

Nikhil Benesch

@benesch

Afraid not. Rayon is for parallelizing CPU intensive tasks; tokio is for parallelizing IO intesive tasks. (Give or take.)

If you need to mix the two worlds, you’ll probably want to use a channel. Do your CPU intensive stuff in parallel with Rayon, and then send the records to produce to a Tokio runtime over a channel.

Michal 'vorner' Vaner

@vorner

I think someone experimented with futures-enabled rayon thing. Like, submitting something to run on rayon giving you a future + being able to await inside rayon and switching tasks there. But I think this didn't get over the finish line.

John Halbert

@j-halbert

Hi all! I'm running into a linking issue with lz4. Multple functions are being redefined - once from rocksdb and once from rdkafka. Is there a way to disable inclusion of lz4 with rdkafka?

Gerard Klijs

@gklijs

Hi, since the default of rdkafka is now async, I guess that's how it's typically used? Asking because I'm considering making async the default for schema_registry_converter.

Adam Chalmers

@adamchalmers

Hi all, I'm a real Kafka noob. My FuturesProducer has connected to the broker, but when I try to send a message I get this:

KafkaError (Message production error: MessageTimedOut (Local: Message timed out)), OwnedMessage {
  payload: Some([214, 29, 23]), key: Some([]), 
  topic: 'auditlog.json', 
  timestamp: NotAvailable, 
  partition: -1, 
  offset: -1001, 
  headers: Some(OwnedHeaders { ptr: 0x7f84c4000b90 }) 
}"

Two questions:

Is there anything in the error message that tells me what I've done wrong?
What exactly is a "key" of the FutureRecord? The docs say it's optional, but doesn't have any more detail.

anthonyjchriste

@anthonyjchriste

aloha folks. Quick question. The API for send: https://docs.rs/rdkafka/0.24.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send shows a return type of Result<(i32, i64), (KafkaError, OwnedMessage)>. I'm trying to decode what the two values are that are returned on success. I think the i32 is the partition? Can someone else me with what the i64 is?

Siddartha Guthikonda

@sidhhu

Has any tried StreamConsumer with kafka-0.8.2.1

it dies with an error
INFO [librdkafka] librdkafka: MAXPOLL [thrd:main]: GroupCoordinator/83: Broker does not support KIP-62 (requires Apache Kafka >= v0.10.1.0): consumer configuration max.poll.interval.ms (300000) is effectively limited by session.timeout.ms (6000) with this broker version

Where communities thrive

People

Repo info

Activity