Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
  • Oct 28 15:10
    benesch commented #299
  • Oct 23 14:48
    benesch commented #295
  • Oct 23 09:01
    durch closed #295
  • Oct 23 08:59
    durch commented #295
  • Oct 22 19:19
    benesch commented #295
  • Oct 22 19:17
    benesch commented #295
  • Oct 22 19:14
    benesch commented #295
  • Oct 22 18:09
    durch commented #295
  • Oct 22 18:09
    durch commented #295
  • Oct 22 18:07
    durch commented #295
  • Oct 22 18:06
    durch commented #295
  • Oct 22 18:02
    durch commented #295
  • Oct 22 10:14
    mfelsche opened #299
  • Oct 20 04:35
    benesch closed #298
  • Oct 20 04:35
    benesch commented #298
  • Oct 20 04:35

    benesch on master

    deps: update to tokio v0.3.0 Merge pull request #297 from Qu… (compare)

  • Oct 20 04:35
    benesch closed #297
  • Oct 20 04:35
    benesch commented #297
  • Oct 19 18:27
    QuentinPerez edited #297
  • Oct 19 17:27
    krojew opened #298
Timothy Shoaf
Timothy Shoaf
ऑ .
Timothy Shoaf
oh wow, sorry guys, just got back into town, clearly my cat has been asking questions on my behalf
Ben Sully
hey everyone! is there any effort going into migrating rdkafka from futures 0.1 to std::future? happy to contribute (although i'm pretty clueless about async rust right now)
Thijs Cadier
@sd2k There hasn't been a lot of activity lately. My company depends on this crate, so we're willing to spend some more time on it going forward. Don't think anyone has worked on std future support yet.
Ben Sully
@thijsc I made a PR (fede1024/rust-rdkafka#143) and tests pass on nightly, but it obviously won't compile on stable until Rust 1.39. I'm also a bit skeptical that I did everything right, particularly with the executor::block_on calls

Hi I'm having troubling getting my rust app to build with this library. I'm running on a mac and I get the following errors: error: linking with cc failed: exit code: 1

ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Please help!!

Thijs Cadier
@sheriffio I can't tell without the full error, but it's likely that it can't link openssl. If you have it installed via homebrew you might need to set some env vars.
@thijsc unfortunately it wasn't openssl link. But what did find was something was up with some libraries in my apps target folder, maybe file link collisions or something. As soon as I deleted my target folder and rebuilt everything worked. Thank you for getting back to me, it helped me look at the problem differently.
Nikhil Benesch
Hi folks, I’ll be helping out with rdkafka maintanance! I recently landed a bunch of build system fixes that should smooth out the linking errors that a lot of folks have been running into.
Hello, since a while (I think since the release of macos catalina), I've not able to build librdkafka-sys anymore on my system, it ends up in a strange error
Building and linking librdkafka statically
Running command: "./configure --disable-sasl --disable-ssl --disable-lz4" in dir: librdkafka
thread 'main' panicked at 'Command failed with error: No such file or directory (os error 2)', /Users/philou/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-1.0.0/build.rs:25:21
The error seems to come from the relative execution of configure: if I replace build.rs run_command_or_fail function with the one of the master it solves the issue
Do you know when a new release of rust-rdkafka will be out?
Ben Sully
@benesch hey! I'm wondering what you'd like to do about the two async PRs (#143 and #166) - the first is mine, and the second is based off mine. @dbcfd has made a PR to merge #166 into my branch, but it makes a lot of changes that I wouldn't know how to review. I could either keep my branch as is and have the PRs handled separately, or merge #166 in and just have a single large PR - let me know which you'd prefer.
Nate Mara
@benesch what's the status of the migration to futures-0.3/tokio-0.2?
Nikhil Benesch
The master branch is fully async/await/futures v0.3/tokio v0.2 enabled
Would love to hear from a few more folks before pulling the trigger on releasing 0.23
There is also the open question of fede1024/rust-rdkafka#192, though it's not blocking for 0.23
Mohammed Abubakar
make the push
Oron Sharabi
Hey, how can I use regex in order to consume from * topics please?
Thanks :)
Nate Mara
@benesch congrats on the release of 0.23! I noticed that the docs are not building https://docs.rs/crate/rdkafka/0.23.0, could you look into that?
Nikhil Benesch
Thanks, @nate-onesignal! We've finally got things sorted with the new docs.rs environment for 0.23.1: https://docs.rs/rdkafka/0.23.1/rdkafka/
@oronsh you'll need to implement such a feature yourself. You can get a list of all topics with https://docs.rs/rdkafka/0.23.1/rdkafka/client/struct.Client.html#method.fetch_metadata, and then filter for topics that match your regex, and then subscribe to those topics.
Calvin Brown
Hey all! I have a quick question, when using the MessageStream is there a way to get a batch of records per poll, instead of iteration over them as they come in 1 by one?
Nikhil Benesch
Afraid not @calvinbrown085. You’d have to implement the batching in your own code.
vijay sali

Hi folks,

I am building application with rdkafka-rust, version pinned to 0.22. I am compiling this to windows binary from linux box.
I have installed all the toolchain for windows.

Following this example

I am getting following errors during build.

error: linking with `/usr/bin/x86_64-w64-mingw32-gcc` failed: exit code: 1

= note: /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x2bf): undefined reference to `rd_kafka_opaque'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x3d8): undefined reference to `rd_kafka_queue_get_consumer'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x43b): undefined reference to `rd_kafka_conf_set_opaque'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x454): undefined reference to `rd_kafka_conf_set_log_cb'
          /home/vagrant/code/demo-example/target/x86_64-pc-windows-gnu/release/deps/demo_example-feb82719e9d890fc.demo_example.h10zmb5r-cgu.12.rcgu.o:demo_example.h:(.text+0x46d): undefined reference to `rd_kafka_conf_set_stats_cb'

Found nothing on web, any help is appreciated :)

Nikhil Benesch
You’re mostly on your own if you’re trying to cross-compile, I’m afraid; there’s just so much voodoo involved
At the very least sure you’re using the cmake build; cross compilation with the default build system is extra unsupported
And here’s the list of environment variables that I needed to set to xcompile from macOS to Linux: https://github.com/MaterializeInc/materialize/blob/7696c8cdaba6b5d0d9225e82322d5c33fa8a09fb/bin/xcompile#L45-L58
Note that those are the variables required for a large project (Materialize) that depends on rdkafka in addition to a few other *-sys crates; not all of them are necessarily applicable to rdkafka
vijay sali
Thanks @benesch , I shall try out these.
Daniel Samson
Hey guys, I am working on a migration tool with rdkafka-rust. Is there a way to mix rdkafka-rust with rayon instead of tokio?
i can't seem to to get the await to work? any ideas?
  metrics.into_par_iter().for_each(|metric: Metric| {
            let json = serde_json::to_string(&metric);
            if let Err(e) = &json {
                log::error!("{}", e)
                // TODO: IMPORTANT: FIX the deserialization issue where everything comes back as null

            if let Ok(j) = &json {
                log::info!("{}", j);
                    producer.send(FutureRecord::to("METRICS").payload(j).key(""), 0);
Nikhil Benesch
Afraid not. Rayon is for parallelizing CPU intensive tasks; tokio is for parallelizing IO intesive tasks. (Give or take.)
If you need to mix the two worlds, you’ll probably want to use a channel. Do your CPU intensive stuff in parallel with Rayon, and then send the records to produce to a Tokio runtime over a channel.
Michal 'vorner' Vaner
I think someone experimented with futures-enabled rayon thing. Like, submitting something to run on rayon giving you a future + being able to await inside rayon and switching tasks there. But I think this didn't get over the finish line.
John Halbert
Hi all! I'm running into a linking issue with lz4. Multple functions are being redefined - once from rocksdb and once from rdkafka. Is there a way to disable inclusion of lz4 with rdkafka?
Gerard Klijs
Hi, since the default of rdkafka is now async, I guess that's how it's typically used? Asking because I'm considering making async the default for schema_registry_converter.
Adam Chalmers
Hi all, I'm a real Kafka noob. My FuturesProducer has connected to the broker, but when I try to send a message I get this:
KafkaError (Message production error: MessageTimedOut (Local: Message timed out)), OwnedMessage {
  payload: Some([214, 29, 23]), key: Some([]), 
  topic: 'auditlog.json', 
  timestamp: NotAvailable, 
  partition: -1, 
  offset: -1001, 
  headers: Some(OwnedHeaders { ptr: 0x7f84c4000b90 }) 
Two questions:
  1. Is there anything in the error message that tells me what I've done wrong?
  2. What exactly is a "key" of the FutureRecord? The docs say it's optional, but doesn't have any more detail.
aloha folks. Quick question. The API for send: https://docs.rs/rdkafka/0.24.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send shows a return type of Result<(i32, i64), (KafkaError, OwnedMessage)>. I'm trying to decode what the two values are that are returned on success. I think the i32 is the partition? Can someone else me with what the i64 is?
Siddartha Guthikonda
Has any tried StreamConsumer with kafka-
it dies with an error
INFO [librdkafka] librdkafka: MAXPOLL [thrd:main]: GroupCoordinator/83: Broker does not support KIP-62 (requires Apache Kafka >= v0.10.1.0): consumer configuration max.poll.interval.ms (300000) is effectively limited by session.timeout.ms (6000) with this broker version