I'm looking at asynchronous example (https://github.com/fede1024/rust-rdkafka/blob/master/examples/asynchronous_processing.rs) and one thing isn't clear for me. Why do we use io_thread and block it? Can't we just spawn futures on the pool?
Separating thread just for writing into Kafka looks suboptimal if we have sparse writing events..
@mersinvald you should disable auto commit(on consumer config: enable.auto.commit) and manually handle it after processing with this function: https://docs.rs/rdkafka/0.17.0/rdkafka/consumer/trait.Consumer.html#method.commit_message
Hi!
I get this error when I build with the feature "ssl"
error: failed to run custom build command for `openssl v0.9.24`
process didn't exit successfully: `/home/alessio/Projects/clevercloud/http-rust-agent/target/debug/build/openssl-332bd61629278021/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Unable to detect OpenSSL version', /home/alessio/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.9.24/build.rs:16:14
note: Run with `RUST_BACKTRACE=1` for a backtrace.rdkafka_sys use the sys-openssl crate and there is an issue related to this: sfackler/rust-openssl#994
Do you have a workaround to fix this problem ?
FYI I have openssl 1.1.1
Hello. I'm wondering if kafka would be usable for a certain use case. That one needs priorities and I'm trying to think of a way to emulate them. The basic idea is to have multiple topics (eg. high, mid, low) and the client would read from high, then go for mid only if high was empty, etc. But I'm not sure how to do it properly ‒ one idea is create multiple consumer clients in the same application. But rdkafka seems to start bunch of threads in the backgroud ‒ will it create them multiple times, or share them? Another idea is to somehow subscribe and unsubscribe the lower priorities depending on situation, but that sounds a bit crazy. But the best way would be if there was an API to pull messages out of the consumer selectively based on topic somehow. Any recommendation, what works, what doesn't?
I want to to use tokio, and I suppose I need at least once as well
The at_least once example isn't sync is it?
Still using rdkafka here
I'm getting the following error with v1 on macOS. Any ideas?
= note: Undefined symbols for architecture x86_64:
"_ZSTD_createCStream", referenced from:
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_initCStream", referenced from:
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_endStream", referenced from:
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_decompress", referenced from:
_rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_compressBound", referenced from:
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_isError", referenced from:
_rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_compressStream", referenced from:
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_freeCStream", referenced from:
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_getFrameContentSize", referenced from:
_rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_getErrorCode", referenced from:
_rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
"_ZSTD_getErrorName", referenced from:
_rd_kafka_zstd_decompress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
_rd_kafka_zstd_compress in librdkafka_sys-3b712273a172d6f0.rlib(rdkafka_zstd.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
4+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++;[++++++++++++++++++++++++++++++++++++++4\65+.6555
;;;;;;;;;7
-*-.
0
@thijsc I made a PR (fede1024/rust-rdkafka#143) and tests pass on nightly, but it obviously won't compile on stable until Rust 1.39. I'm also a bit skeptical that I did everything right, particularly with the
executor::block_on calls
Hi I'm having troubling getting my rust app to build with this library. I'm running on a mac and I get the following errors: error: linking with cc failed: exit code: 1
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Please help!!
@thijsc unfortunately it wasn't openssl link. But what did find was something was up with some libraries in my apps target folder, maybe file link collisions or something. As soon as I deleted my target folder and rebuilt everything worked. Thank you for getting back to me, it helped me look at the problem differently.
Building and linking librdkafka statically
Running command: "./configure --disable-sasl --disable-ssl --disable-lz4" in dir: librdkafka
thread 'main' panicked at 'Command failed with error: No such file or directory (os error 2)', /Users/philou/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-1.0.0/build.rs:25:21configure: if I replace build.rs run_command_or_fail function with the one of the master it solves the issueDo you know when a new release of rust-rdkafka will be out?
@benesch hey! I'm wondering what you'd like to do about the two async PRs (#143 and #166) - the first is mine, and the second is based off mine. @dbcfd has made a PR to merge #166 into my branch, but it makes a lot of changes that I wouldn't know how to review. I could either keep my branch as is and have the PRs handled separately, or merge #166 in and just have a single large PR - let me know which you'd prefer.
Would love to hear from a few more folks before pulling the trigger on releasing 0.23
There is also the open question of fede1024/rust-rdkafka#192, though it's not blocking for 0.23
like
["users.*.messages"]
Thanks :)
@benesch congrats on the release of 0.23! I noticed that the docs are not building https://docs.rs/crate/rdkafka/0.23.0, could you look into that?
Thanks, @nate-onesignal! We've finally got things sorted with the new docs.rs environment for 0.23.1: https://docs.rs/rdkafka/0.23.1/rdkafka/
@oronsh you'll need to implement such a feature yourself. You can get a list of all topics with https://docs.rs/rdkafka/0.23.1/rdkafka/client/struct.Client.html#method.fetch_metadata, and then filter for topics that match your regex, and then subscribe to those topics.