These are chat archives for rust-lang/rust

16th
Nov 2016
Christopher Andrews
@CryptArchy
Nov 16 2016 00:14 UTC

So it took some doing, but I figured it out. So, the documentation for the Kafka Consumer has this critical bit of information in it:

Individual messages are embedded in the retrieved messagesets and can be processed using the messages() iterator. Due to this embedding, individual messsages's lifetime is bound to the MessageSet they are part of. Typically, client code access the raw data/bytes, parses it into custom data types and passes that for further processing within the application. Altough unconvenient, this helps in reducing the number of allocations within the pipeline of processing incoming messages.

This led me to adding this to my code:
use std::sync::mpsc::{channel, Sender, Receiver, TryRecvError};
use kafka::consumer::{Consumer, FetchOffset, GroupOffsetStorage};
use kafka::client::fetch::Message as KafkaMessage;

#[derive(Debug)]
struct Message {
    key: String,
    value: String,
}

impl<'a> From<&'a KafkaMessage<'a>> for Message {
    fn from(msg: &'a KafkaMessage) -> Self {
        Message {
            key: String::from_utf8_lossy(msg.key).into_owned(),
            value: String::from_utf8_lossy(msg.value).into_owned(),
        }
    }
}

fn start_kafka(tx: Sender<Message>) {
    let mut consumer = Consumer::from_hosts(vec!["example.com:9092".to_owned()])
        .with_topic_partitions("test".to_owned(), &[0])
        .with_fallback_offset(FetchOffset::Earliest)
        .with_group("mygroup".to_owned())
        .with_offset_storage(GroupOffsetStorage::Kafka)
        .create()
        .unwrap();

    loop {
        for ms in consumer.poll().unwrap().iter() {
            for m in ms.messages().iter() {
                debug!("{:?}", m);
                tx.send(m.into());
            }
            consumer.consume_messageset(ms);
        }
        consumer.commit_consumed().unwrap();
        thread::yield_now();
    }
}
Add my own new type, then figure out the correct way to convert the raw byte data into some other form (most importantly some other form that my new data structure has ownership over) and then send that along the channel.
This, at last, compiles.
Christopher Andrews
@CryptArchy
Nov 16 2016 00:19 UTC
And if I'm now understanding the ownership model, the new Message type is owned by the inner loop, then gets moved into the channel so that the receiver should ultimately end up in possession of the data.
Matanel Levi
@matanelevi
Nov 16 2016 20:21 UTC
Someone knows about a cool rust workshop for beginners
?
Remi Rampin
@remram44
Nov 16 2016 21:19 UTC
workshop?
Ian Henry
@eeyun
Nov 16 2016 21:34 UTC
@matanelevi not certain about workshops, but my first suggestion is work your way through the rust book https://doc.rust-lang.org/stable/book/ and rust by example http://rustbyexample.com/ and then maybe check out the rust exercises on exercism.io or http://rosettacode.org/wiki/Rosetta_Code
Matanel Levi
@matanelevi
Nov 16 2016 21:37 UTC
@eeyun thanks, but it's not for me. I've got a few hours to present Rust for some people
So I thought about a workshop
Any ideas? Focus on concurrency
Matanel Levi
@matanelevi
Nov 16 2016 21:47 UTC
@remram44 you can think about it as an excersice with stages
Robert Winslow
@rw
Nov 16 2016 22:11 UTC

@matanelevi A good workshop strategy is to build a commandline tool together. I like having each person write their own, with discussion and guidance from you. So it's a lab environment. e.g. write a simple grep clone: go from something that just echos the input, to one that filters the input lines based on a fixed string, to one that does it in parallel (and demonstrate CPU usage in the parallel case using something like top).

This approach enables you to introduce concepts as they arise (ownership, etc).

The downside is that Rust's ownership system has a steep learning curve for first-time users, so that could be a problem.

There may be pedagogical coding problems that are very good at explaining ownership. You might look for something like that.

Matanel Levi
@matanelevi
Nov 16 2016 22:14 UTC
@rw thanks! I'll think about it