perf, if that helps
for loops are hard-coded to the regular Iterator trait
Iterator::for_each, then this can translate to ParallelIterator::for_each
https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html#generate-jpg-thumbnails-in-parallel
out vector to &mut [AtomicI8].
(0..data.len()).into_par_iter().for_each(|i| /* atomic store */)
@jntrnr https://is.gd/QFQpOF
If I'm reading this right, you're saying to break into blocks that are outside of rayon? How would you know what size to split into so that rayon is efficient? Feels like something rayon should do so that it's able to tune for efficiency
rng::thread_rng() and pass a Sender to a for_each loop. Unfortunately for_each_init requires Sync as a trait bound, which is not satisfied by Sender. Is there a way to get both, the functionality of for_each_init and for_each_with at the same time?
map_init to do some computation with the rng, then for_each_with to send your result.
1 reply
What should I await here?
async move {
let mut con = redis.get_tokio_connection_manager().await.unwrap();
redis::Cmd::sadd(format!("appearances:{}", user_id), appearance)
.query_async::<_, i32>(&mut con)
.await
.unwrap_or_else(|e| panic!("sadd failed with {}", e));
};the code is inside into_part_iter().for_each()
Hello All, trying to figure out some interesting performance behaviors I'm running into while using Rayon - mostly looking for ideas on debugging/profiling/logging whatever tooling might help shed some light on the behavior I'm seeing. Essentially I started a project using the ripunzip crate which downloads & unzips a file with the unzipping part done in parallel by using into_par_iter on index inside zip and a fancy cache on the HTTP stream, that works great. I'm now trying to use rayon to do multiple download/unzip operations in parallel, simple enough to make a vector of URLs then do something like
let results = url.into_par_iter().map(|url| download_streaming_unzip(url, "C:\wherever\my_unzipped_files\")
.filter_map(Result::err)
.collect();
And that somewhat works. But then the performance has some quirks that I don't understand. I have a test set of 4 URLs that it downloads & unzips, 3 medium sized files and one large file. Predictably the smaller ones finish up first, but then the remainder of the long one takes longer to finish than if I had just run it by itself. Feels a lot like as the smaller ones finish up, work for the iterator on the large file isn't getting farmed out to the rest of the thread pool, or perhaps the backend buffer is just in a worst state as far as memory layout but that seems less likely to me. The CPU usage looks way down during the end of the operation which is what is leading me to believe the work isn't getting farmed out as expected.
download_streaming_unzip is also using rayon? "using into_par_iter on index inside zip" -- it may be that it initially doesn't split as much because the other threads are busy with the smaller files. You could force maximal splitting .with_max_len(1) and see if that helps.