:sweat:
Hi, somewhere in my code I need to check a big vector (also some times it can be small) for finding some data in it (Im using data.iter().find(...) in old method) its work good, I didn't notice any kind of performance lost in it, but I was thinking changing it so it search in parallel using rayon make sense? I mean there is any cost for rayon to spawn the threads everytime Im calling it? (as I said it would so many times...)
at the end is it worth it for use it in this kind of situations?
at the end is it worth it for use it in this kind of situations?
rayon keeps a threadpool, so after the first time it won't be spawning any new threads
3 replies
(unless you explicitly create your own
ThreadPool each time)
also one other thing, is it possible to use rayon inside normal loops?
2 replies
On Linux, I'd use
perf, if that helps
No, you can't
the language
for loops are hard-coded to the regular Iterator trait
but if you can rewrite that with
Iterator::for_each, then this can translate to ParallelIterator::for_each
I have a crate that hacks that transformation with a procedural macro: https://crates.io/crates/rayon-macro
Answering my own question :)
https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html#generate-jpg-thumbnails-in-parallel
https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html#generate-jpg-thumbnails-in-parallel
@WcaleNieWolny: arbitrary parallel writes are difficult in Rust, but the best way is probably to temporarily cast that
out vector to &mut [AtomicI8].
you can do that on nightly with https://doc.rust-lang.org/std/sync/atomic/struct.AtomicI8.html#method.from_mut_slice
or with your own unsafe pointer cast the same way, https://doc.rust-lang.org/src/core/sync/atomic.rs.html#2064
then in rayon,
(0..data.len()).into_par_iter().for_each(|i| /* atomic store */)
it may not parallelize well though -- memory bound and probably cache-unfriendly
Hi everyone! Just wanted to know if things like the JoinHandle are possible with Rayon. Or are there alternatives to wait for a task to finish?
I'm thinking of using some of my Xmas break to implement this - rayon-rs/rayon#329
What's a good way to start?
I don't know how Niko thought that would work, and he never got back to that issue with a better write-up...
@miguelraz: you might have to do your own research on that one
maybe looking into what @wagnerf42 was doing
each task has its own state
Also - I'm a huge fan of your course. I've been scouring for HPC courses that exploited Rust and yours is the only one I've been able to find so far!
<3
with this trick you can process your sequential iterator by chunks
@miguelraz for the scan, i'm guess there is not really any blocker before landing it in rayon
@jntrnr https://is.gd/QFQpOF
If I'm reading this right, you're saying to break into blocks that are outside of rayon? How would you know what size to split into so that rayon is efficient? Feels like something rayon should do so that it's able to tune for efficiency
the good way to do it is to grow the block size exponentially until it takes long enough
hello all, quick question, i need to both init
rng::thread_rng() and pass a Sender to a for_each loop. Unfortunately for_each_init requires Sync as a trait bound, which is not satisfied by Sender. Is there a way to get both, the functionality of for_each_init and for_each_with at the same time?
@sirno: can you chain the two operations? e.g.
map_init to do some computation with the rng, then for_each_with to send your result.
1 reply
I'm getting a warning 👉 unused implementer of futures::Future that must be used, futures do nothing unless you .await or poll them.
What should I await here?
What should I await here?
async move {
let mut con = redis.get_tokio_connection_manager().await.unwrap();
redis::Cmd::sadd(format!("appearances:{}", user_id), appearance)
.query_async::<_, i32>(&mut con)
.await
.unwrap_or_else(|e| panic!("sadd failed with {}", e));
};the code is inside into_part_iter().for_each()