Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 16 18:28
    benesch commented #211
  • Jan 16 18:28
    benesch commented #211
  • Jan 16 17:56
    tkasu commented #211
  • Jan 16 17:56
    tkasu commented #211
  • Jan 16 08:40
    FSMaxB-dooshop commented #211
  • Jan 16 08:39
    FSMaxB-dooshop commented #211
  • Jan 15 19:55
    benesch commented #211
  • Jan 15 19:28
    tkasu opened #211
  • Jan 15 04:00
    benesch closed #185
  • Jan 15 04:00
    benesch commented #185
  • Jan 15 03:58
    benesch commented #209
  • Jan 15 03:58

    benesch on master

    derive Debug for ClientConfig Merge pull request #209 from pr… (compare)

  • Jan 15 03:58
    benesch closed #209
  • Jan 14 01:32
    prettynatty commented #210
  • Jan 14 01:12
    prettynatty edited #210
  • Jan 14 00:34
    prettynatty commented #210
  • Jan 14 00:32
    prettynatty commented #210
  • Jan 14 00:31
    prettynatty edited #210
  • Jan 14 00:20
    prettynatty edited #210
  • Jan 14 00:20
    prettynatty edited #210
Thijs Cadier
@thijsc
That makes me think it's probably not very idiomatic
James Brown
@Roguelazer
Any chance for a librdkafka-sys 0.11.5 bump? I think I may have hit one of the bugs fixed in it...
Kirill Taran
@kirillt
Hi, guys
I'm looking at asynchronous example (https://github.com/fede1024/rust-rdkafka/blob/master/examples/asynchronous_processing.rs) and one thing isn't clear for me. Why do we use io_thread and block it? Can't we just spawn futures on the pool?
Kirill Taran
@kirillt
Is it just workaround for DeliveryFuture not being Send? Why it isn't Send?
Separating thread just for writing into Kafka looks suboptimal if we have sparse writing events..
Kirill Taran
@kirillt
Sure it's bad idea, but what particarly can happen if I mark the future manually as Send?
Mike Lubinets
@mersinvald
Hi! How should I use BaseConsumer to make it commit offset only after request have been handled successfully?
Rust
@LooMaclin
@mersinvald you should disable auto commit(on consumer config: enable.auto.commit) and manually handle it after processing with this function: https://docs.rs/rdkafka/0.17.0/rdkafka/consumer/trait.Consumer.html#method.commit_message
Wang Chen
@cjen07
Hi all, I plan to run the customer example in my own project, but got can't find crate rdkafka_sys error, I have added rdkafka = { version = "0.17.0", features = ["ssl", "sasl"] } in the deps. How to solve this? Thanks a lot in advance!
Alessio Coltellacci
@NotBad4U

Hi!
I get this error when I build with the feature "ssl"

error: failed to run custom build command for `openssl v0.9.24`
process didn't exit successfully: `/home/alessio/Projects/clevercloud/http-rust-agent/target/debug/build/openssl-332bd61629278021/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Unable to detect OpenSSL version', /home/alessio/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.9.24/build.rs:16:14
note: Run with `RUST_BACKTRACE=1` for a backtrace.

rdkafka_sys use the sys-openssl crate and there is an issue related to this: sfackler/rust-openssl#994
Do you have a workaround to fix this problem ?

FYI I have openssl 1.1.1
Alessio Coltellacci
@NotBad4U
Sorry for the inconvenience, I upgraded all my dependencies and now rust-openssl is up to date and work with SSL 1.1.1.
David Bresnick
@gravaton
Has anyone had success building and working with this library on a Windows platform? Building librdkafka inline is challenging at best and finding a way to get the DLL installed for dynamic linking is also not very viable. Anyone have thoughts on this?
Michal 'vorner' Vaner
@vorner
Hello. I'm wondering if kafka would be usable for a certain use case. That one needs priorities and I'm trying to think of a way to emulate them. The basic idea is to have multiple topics (eg. high, mid, low) and the client would read from high, then go for mid only if high was empty, etc. But I'm not sure how to do it properly ‒ one idea is create multiple consumer clients in the same application. But rdkafka seems to start bunch of threads in the backgroud ‒ will it create them multiple times, or share them? Another idea is to somehow subscribe and unsubscribe the lower priorities depending on situation, but that sounds a bit crazy. But the best way would be if there was an API to pull messages out of the consumer selectively based on topic somehow. Any recommendation, what works, what doesn't?
James Sewell
@jamessewell
Hi all - I'm fairly new to all of this. I want to write batches of kafka messages (say batch size of 500) to another bit of software
I want to to use tokio, and I suppose I need at least once as well
The at_least once example isn't sync is it?
Thijs Cadier
@thijsc
Hi all, how is it going?
Still using rdkafka here
Frank B Greco Jr
@frankgreco
I'm getting the following error with v1 on macOS. Any ideas?
  = note: Undefined symbols for architecture x86_64:
            "_ZSTD_createCStream", referenced from:
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_initCStream", referenced from:
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_endStream", referenced from:
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_decompress", referenced from:
                _rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_compressBound", referenced from:
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_isError", referenced from:
                _rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_compressStream", referenced from:
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_freeCStream", referenced from:
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_getFrameContentSize", referenced from:
                _rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_getErrorCode", referenced from:
                _rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
            "_ZSTD_getErrorName", referenced from:
                _rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
                _rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
Timothy Shoaf
@timshoaf
POL{.,k/9i\
Timothy Shoaf
@timshoaf
4+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++;[++++++++++++++++++++++++++++++++++++++4\65+.6555
;;;;;;;;;7
-*-.
0
Timothy Shoaf
@timshoaf
ऑ .
Timothy Shoaf
@timshoaf
oh wow, sorry guys, just got back into town, clearly my cat has been asking questions on my behalf
Ben Sully
@sd2k
hey everyone! is there any effort going into migrating rdkafka from futures 0.1 to std::future? happy to contribute (although i'm pretty clueless about async rust right now)
Thijs Cadier
@thijsc
@sd2k There hasn't been a lot of activity lately. My company depends on this crate, so we're willing to spend some more time on it going forward. Don't think anyone has worked on std future support yet.
Ben Sully
@sd2k
@thijsc I made a PR (fede1024/rust-rdkafka#143) and tests pass on nightly, but it obviously won't compile on stable until Rust 1.39. I'm also a bit skeptical that I did everything right, particularly with the executor::block_on calls
Sheriff
@sheriffio

Hi I'm having troubling getting my rust app to build with this library. I'm running on a mac and I get the following errors: error: linking with cc failed: exit code: 1

ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Please help!!

Thijs Cadier
@thijsc
@sheriffio I can't tell without the full error, but it's likely that it can't link openssl. If you have it installed via homebrew you might need to set some env vars.
Sheriff
@sheriffio
@thijsc unfortunately it wasn't openssl link. But what did find was something was up with some libraries in my apps target folder, maybe file link collisions or something. As soon as I deleted my target folder and rebuilt everything worked. Thank you for getting back to me, it helped me look at the problem differently.
Nikhil Benesch
@benesch
Hi folks, I’ll be helping out with rdkafka maintanance! I recently landed a bunch of build system fixes that should smooth out the linking errors that a lot of folks have been running into.
Philippe GASSMANN
@zenria
Hello, since a while (I think since the release of macos catalina), I've not able to build librdkafka-sys anymore on my system, it ends up in a strange error
Building and linking librdkafka statically
Running command: "./configure --disable-sasl --disable-ssl --disable-lz4" in dir: librdkafka
thread 'main' panicked at 'Command failed with error: No such file or directory (os error 2)', /Users/philou/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-1.0.0/build.rs:25:21
The error seems to come from the relative execution of configure: if I replace build.rs run_command_or_fail function with the one of the master it solves the issue
Do you know when a new release of rust-rdkafka will be out?
Ben Sully
@sd2k
@benesch hey! I'm wondering what you'd like to do about the two async PRs (#143 and #166) - the first is mine, and the second is based off mine. @dbcfd has made a PR to merge #166 into my branch, but it makes a lot of changes that I wouldn't know how to review. I could either keep my branch as is and have the PRs handled separately, or merge #166 in and just have a single large PR - let me know which you'd prefer.
Nate Mara
@nate-onesignal
@benesch what's the status of the migration to futures-0.3/tokio-0.2?
Nikhil Benesch
@benesch
The master branch is fully async/await/futures v0.3/tokio v0.2 enabled
Would love to hear from a few more folks before pulling the trigger on releasing 0.23
There is also the open question of fede1024/rust-rdkafka#192, though it's not blocking for 0.23
Mohammed Abubakar
@moh-abk
make the push
Oron Sharabi
@oronsh
Hey, how can I use regex in order to consume from * topics please?
like
["users.*.messages"]
Thanks :)
Nate Mara
@nate-onesignal
@benesch congrats on the release of 0.23! I noticed that the docs are not building https://docs.rs/crate/rdkafka/0.23.0, could you look into that?
Nikhil Benesch
@benesch
Thanks, @nate-onesignal! We've finally got things sorted with the new docs.rs environment for 0.23.1: https://docs.rs/rdkafka/0.23.1/rdkafka/
@oronsh you'll need to implement such a feature yourself. You can get a list of all topics with https://docs.rs/rdkafka/0.23.1/rdkafka/client/struct.Client.html#method.fetch_metadata, and then filter for topics that match your regex, and then subscribe to those topics.