Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Feb 02 21:19
    HopedWall closed #922
  • Feb 02 12:28
    yu-re-ka synchronize #1012
  • Feb 02 12:28
    yu-re-ka synchronize #1012
  • Feb 02 12:28
    yu-re-ka edited #1012
  • Feb 02 12:27
    yu-re-ka synchronize #1012
  • Feb 02 12:19
    yu-re-ka synchronize #1012
  • Feb 01 12:17
    romancardenas closed #1015
  • Feb 01 12:13
    romancardenas opened #1015
  • Feb 01 00:25
    DarkFenX closed #1014
  • Jan 31 22:50
    DarkFenX edited #1014
  • Jan 31 22:50
    DarkFenX edited #1014
  • Jan 31 22:46
    DarkFenX edited #1014
  • Jan 31 22:41
    DarkFenX edited #1014
  • Jan 31 22:40
    DarkFenX opened #1014
  • Jan 22 22:18

    cuviper on rayon-core-v1.10.2

    (compare)

  • Jan 22 22:17
    bors[bot] closed #1008
  • Jan 22 22:17
    bors[bot] closed #952
  • Jan 22 22:17
    bors[bot] closed #1013
  • Jan 22 22:17

    bors[bot] on master

    Release rayon-core 1.10.2 Merge #1013 1013: Release rayo… (compare)

  • Jan 22 22:10

    bors[bot] on staging.tmp

    (compare)

WGH
@WGH:torlan.ru
[m]
No clue about Windows, never really used it for any development work
On Linux, I'd use perf, if that helps
No, you can't
Josh Stone
@jistone:fedora.im
[m]
the language for loops are hard-coded to the regular Iterator trait
but if you can rewrite that with Iterator::for_each, then this can translate to ParallelIterator::for_each
I have a crate that hacks that transformation with a procedural macro: https://crates.io/crates/rayon-macro
Miguel Raz Guzmán Macedo
@miguelraz
Let’s say I have a dir with many nested dirs (at different levels of , all containing .txt files that can be processed independently.
What’s the Rayon idiomatic way of processing them? (for argument’s sake, a wc on each file will suffice)
WcaleNieWolny
@WcaleNieWolny
pub fn fast_split(data: &[i8], cache: Vec<usize>) -> Vec<i8>{
        let mut out = Vec::<i8>::with_capacity(data.len());
    //TODO: resize

        for i in 0..data.len(){
            out[cache[i]] = data[i]
        }

        out
    }
how would I make this run in parallel? Like using rayon or something like that
Josh Stone
@jistone:fedora.im
[m]
@WcaleNieWolny: arbitrary parallel writes are difficult in Rust, but the best way is probably to temporarily cast that out vector to &mut [AtomicI8].
or with your own unsafe pointer cast the same way, https://doc.rust-lang.org/src/core/sync/atomic.rs.html#2064
then in rayon, (0..data.len()).into_par_iter().for_each(|i| /* atomic store */)
it may not parallelize well though -- memory bound and probably cache-unfriendly
mr.rusty
@mr.rusty:matrix.org
[m]
Hi everyone! Just wanted to know if things like the JoinHandle are possible with Rayon. Or are there alternatives to wait for a task to finish?
Josh Stone
@jistone:fedora.im
[m]
mr.rusty: most of the rayon api blocks for completion, but for spawn you can use a scope, as that waits for its spawns to complete before returning.
mr.rusty
@mr.rusty:matrix.org
[m]
Thank your, i think that'll help me :)
mr.rusty
@mr.rusty:matrix.org
[m]
:point_up: Edit: Thank you, i think that'll help me :)
Miguel Raz Guzmán Macedo
@miguelraz
Hello @jistone:fedora.im !
I'm thinking of using some of my Xmas break to implement this - rayon-rs/rayon#329
What's a good way to start?
Josh Stone
@jistone:fedora.im
[m]
I don't know how Niko thought that would work, and he never got back to that issue with a better write-up...
@miguelraz: you might have to do your own research on that one
maybe looking into what @wagnerf42 was doing
wagnerf42
@wagnerf42
yeah i do have a scan
each task has its own state
wagnerf42
@wagnerf42
i use it in several places but i think it's always in use with a par_windows
Miguel Raz Guzmán Macedo
@miguelraz
Oh sweet, thanks @wagnerf42!
What would you say are the big blockers before shaping that up into something that could land in Rayon?
Also - I'm a huge fan of your course. I've been scouring for HPC courses that exploited Rust and yours is the only one I've been able to find so far!
<3
JT
@jntrnr
I might be missing something - but if you do .par_bridge() on an iterator that's potentially infinite, is there a way to get the values back out without collecting? Can you chunk it? If not, is there another way to work with a potentially unbound iterator?
wagnerf42
@wagnerf42
@miguelraz hey thanks a lot. thanks to let me know. i have a lot of material i should upload
wagnerf42
@wagnerf42
with this trick you can process your sequential iterator by chunks
@miguelraz for the scan, i'm guess there is not really any blocker before landing it in rayon
wagnerf42
@wagnerf42
it's just that it is somehow weaker than the sequential one. as i said i use it in one or two places but it's always in conjunction with par_windows.
JT
@jntrnr

@jntrnr https://is.gd/QFQpOF

If I'm reading this right, you're saying to break into blocks that are outside of rayon? How would you know what size to split into so that rayon is efficient? Feels like something rayon should do so that it's able to tune for efficiency

wagnerf42
@wagnerf42
@jntrnr yes it should. however it is currently not the case. hence you should fall back to do it manually
the good way to do it is to grow the block size exponentially until it takes long enough
Nicolas Ochsner
@sirno
hello all, quick question, i need to both init rng::thread_rng() and pass a Sender to a for_each loop. Unfortunately for_each_init requires Sync as a trait bound, which is not satisfied by Sender. Is there a way to get both, the functionality of for_each_init and for_each_with at the same time?
Josh Stone
@jistone:fedora.im
[m]
@sirno: can you chain the two operations? e.g. map_init to do some computation with the rng, then for_each_with to send your result.
1 reply
simanacci
@simanacci
I'm getting a warning 👉 unused implementer of futures::Future that must be used, futures do nothing unless you .await or poll them.
What should I await here?
async move {
    let mut con = redis.get_tokio_connection_manager().await.unwrap();
    redis::Cmd::sadd(format!("appearances:{}", user_id), appearance)
        .query_async::<_, i32>(&mut con)
        .await
        .unwrap_or_else(|e| panic!("sadd failed with {}", e));
};
the code is inside into_part_iter().for_each()
I'm trying to run the SADDs in parallel.
WGH
@WGH:torlan.ru
[m]
@simanacci: this will perform terribly though, I'd suggest to use pipelining instead
simanacci
@simanacci
@simanacci: this will perform terribly though, I'd suggest to use pipelining instead
:thumbsup:
David Corrigan
@davidcorrigan714

Hello All, trying to figure out some interesting performance behaviors I'm running into while using Rayon - mostly looking for ideas on debugging/profiling/logging whatever tooling might help shed some light on the behavior I'm seeing. Essentially I started a project using the ripunzip crate which downloads & unzips a file with the unzipping part done in parallel by using into_par_iter on index inside zip and a fancy cache on the HTTP stream, that works great. I'm now trying to use rayon to do multiple download/unzip operations in parallel, simple enough to make a vector of URLs then do something like
let results = url.into_par_iter().map(|url| download_streaming_unzip(url, "C:\wherever\my_unzipped_files\")
.filter_map(Result::err)
.collect();

And that somewhat works. But then the performance has some quirks that I don't understand. I have a test set of 4 URLs that it downloads & unzips, 3 medium sized files and one large file. Predictably the smaller ones finish up first, but then the remainder of the long one takes longer to finish than if I had just run it by itself. Feels a lot like as the smaller ones finish up, work for the iterator on the large file isn't getting farmed out to the rest of the thread pool, or perhaps the backend buffer is just in a worst state as far as memory layout but that seems less likely to me. The CPU usage looks way down during the end of the operation which is what is leading me to believe the work isn't getting farmed out as expected.

Josh Stone
@jistone:fedora.im
[m]
so download_streaming_unzip is also using rayon? "using into_par_iter on index inside zip" -- it may be that it initially doesn't split as much because the other threads are busy with the smaller files. You could force maximal splitting .with_max_len(1) and see if that helps.