rayon/Lobby

Rayon: A data parallelism library for Rust -- https://github.com/rayon-rs/rayon

Miguel Raz Guzmán Macedo

@miguelraz

Thinking of stuff like
Dot product calculation,
Independently processing many strings in a big file
or anything numeric/related to linear algebra

Josh Stone

@jistone:fedora.im

[m]

rayon-demo (in the rayon repo) has some, not sure if you'd call them "good" though ;)

Miguel Raz Guzmán Macedo

@miguelraz

@jistone:fedora.im yup, those seem just like what I wanted. What's the command to run them though? cargo bench in the rayon/rayon-demo dir?

Ah, yes...
https://github.com/rayon-rs/rayon/wiki/Guide-to-Development

:sweat:

whoknow

@whoknow:matrix.org

[m]

Hi, somewhere in my code I need to check a big vector (also some times it can be small) for finding some data in it (Im using data.iter().find(...) in old method) its work good, I didn't notice any kind of performance lost in it, but I was thinking changing it so it search in parallel using rayon make sense? I mean there is any cost for rayon to spawn the threads everytime Im calling it? (as I said it would so many times...)
at the end is it worth it for use it in this kind of situations?

Josh Stone

@jistone:fedora.im

[m]

rayon keeps a threadpool, so after the first time it won't be spawning any new threads

3 replies

(unless you explicitly create your own ThreadPool each time)

whoknow

@whoknow:matrix.org

[m]

also one other thing, is it possible to use rayon inside normal loops?

2 replies

WGH

@WGH:torlan.ru

[m]

No clue about Windows, never really used it for any development work

On Linux, I'd use perf, if that helps

No, you can't

Josh Stone

@jistone:fedora.im

[m]

the language for loops are hard-coded to the regular Iterator trait

but if you can rewrite that with Iterator::for_each, then this can translate to ParallelIterator::for_each

I have a crate that hacks that transformation with a procedural macro: https://crates.io/crates/rayon-macro

Miguel Raz Guzmán Macedo

@miguelraz

Let’s say I have a dir with many nested dirs (at different levels of , all containing .txt files that can be processed independently.
What’s the Rayon idiomatic way of processing them? (for argument’s sake, a wc on each file will suffice)

Miguel Raz Guzmán Macedo

@miguelraz

Answering my own question :)
https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html#generate-jpg-thumbnails-in-parallel

WcaleNieWolny

@WcaleNieWolny

pub fn fast_split(data: &[i8], cache: Vec<usize>) -> Vec<i8>{
        let mut out = Vec::<i8>::with_capacity(data.len());
    //TODO: resize

        for i in 0..data.len(){
            out[cache[i]] = data[i]
        }

        out
    }

how would I make this run in parallel? Like using rayon or something like that

Josh Stone

@jistone:fedora.im

[m]

@WcaleNieWolny: arbitrary parallel writes are difficult in Rust, but the best way is probably to temporarily cast that out vector to &mut [AtomicI8].

you can do that on nightly with https://doc.rust-lang.org/std/sync/atomic/struct.AtomicI8.html#method.from_mut_slice

or with your own unsafe pointer cast the same way, https://doc.rust-lang.org/src/core/sync/atomic.rs.html#2064

then in rayon, (0..data.len()).into_par_iter().for_each(|i| /* atomic store */)

it may not parallelize well though -- memory bound and probably cache-unfriendly

mr.rusty

@mr.rusty:matrix.org

[m]

Hi everyone! Just wanted to know if things like the JoinHandle are possible with Rayon. Or are there alternatives to wait for a task to finish?

Josh Stone

@jistone:fedora.im

[m]

mr.rusty: most of the rayon api blocks for completion, but for spawn you can use a scope, as that waits for its spawns to complete before returning.

mr.rusty

@mr.rusty:matrix.org

[m]

Thank your, i think that'll help me :)

mr.rusty

@mr.rusty:matrix.org

[m]

:point_up: Edit: Thank you, i think that'll help me :)

Miguel Raz Guzmán Macedo

@miguelraz

Hello @jistone:fedora.im !

I'm thinking of using some of my Xmas break to implement this - rayon-rs/rayon#329

What's a good way to start?

Josh Stone

@jistone:fedora.im

[m]

I don't know how Niko thought that would work, and he never got back to that issue with a better write-up...

@miguelraz: you might have to do your own research on that one

maybe looking into what @wagnerf42 was doing

wagnerf42

@wagnerf42

yeah i do have a scan

https://github.com/wagnerf42/diam/blob/main/examples/scan.rs

each task has its own state

wagnerf42

@wagnerf42

i use it in several places but i think it's always in use with a par_windows

Miguel Raz Guzmán Macedo

@miguelraz

Oh sweet, thanks @wagnerf42!
What would you say are the big blockers before shaping that up into something that could land in Rayon?

Also - I'm a huge fan of your course. I've been scouring for HPC courses that exploited Rust and yours is the only one I've been able to find so far!

@jntrnr

I might be missing something - but if you do .par_bridge() on an iterator that's potentially infinite, is there a way to get the values back out without collecting? Can you chunk it? If not, is there another way to work with a potentially unbound iterator?

wagnerf42

@wagnerf42

@miguelraz hey thanks a lot. thanks to let me know. i have a lot of material i should upload

wagnerf42

@wagnerf42

@jntrnr https://is.gd/QFQpOF

with this trick you can process your sequential iterator by chunks

@miguelraz for the scan, i'm guess there is not really any blocker before landing it in rayon

wagnerf42

@wagnerf42

it's just that it is somehow weaker than the sequential one. as i said i use it in one or two places but it's always in conjunction with par_windows.

@jntrnr

@jntrnr https://is.gd/QFQpOF

If I'm reading this right, you're saying to break into blocks that are outside of rayon? How would you know what size to split into so that rayon is efficient? Feels like something rayon should do so that it's able to tune for efficiency

wagnerf42

@wagnerf42

@jntrnr yes it should. however it is currently not the case. hence you should fall back to do it manually

the good way to do it is to grow the block size exponentially until it takes long enough

Nicolas Ochsner

@sirno

hello all, quick question, i need to both init rng::thread_rng() and pass a Sender to a for_each loop. Unfortunately for_each_init requires Sync as a trait bound, which is not satisfied by Sender. Is there a way to get both, the functionality of for_each_init and for_each_with at the same time?

Where communities thrive

People

Repo info

Activity