Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Nov 26 18:19
    dcompiler closed #994
  • Nov 25 22:05
    dcompiler opened #994
  • Nov 21 10:54
    bobi6666 opened #993
  • Nov 19 00:16

    cuviper on rayon-core-v1.10.1

    (compare)

  • Nov 19 00:14
    bors[bot] closed #992
  • Nov 19 00:14

    bors[bot] on master

    Fix a sleep race with broadcast… Release 1.10.1 Merge #992 992: Fix a sleep ra… (compare)

  • Nov 19 00:05

    bors[bot] on staging.tmp

    (compare)

  • Nov 19 00:05

    bors[bot] on staging

    Fix a sleep race with broadcast… Release 1.10.1 Merge #992 992: Fix a sleep ra… (compare)

  • Nov 19 00:05

    bors[bot] on staging.tmp

    Fix a sleep race with broadcast… Release 1.10.1 [ci skip][skip ci][skip netlify… (compare)

  • Nov 19 00:05

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Nov 19 00:03
    cuviper synchronize #992
  • Nov 18 23:45
    cuviper opened #992
  • Nov 18 21:10

    cuviper on v1.6.0

    (compare)

  • Nov 18 21:10

    cuviper on rayon-core-v1.10.0

    (compare)

  • Nov 18 20:56
    bors[bot] closed #991
  • Nov 18 20:55

    bors[bot] on master

    Release rayon 1.6.0 / rayon-cor… Merge #991 991: Release rayon … (compare)

  • Nov 18 20:45

    bors[bot] on staging.tmp

    (compare)

  • Nov 18 20:45

    bors[bot] on staging

    Release rayon 1.6.0 / rayon-cor… Merge #991 991: Release rayon … (compare)

  • Nov 18 20:45

    bors[bot] on staging.tmp

    Release rayon 1.6.0 / rayon-cor… [ci skip][skip ci][skip netlify… (compare)

  • Nov 18 20:45

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

WGH
@WGH:torlan.ru
[m]
But this whole question looks silly: I can imagine a code that breaks from parallelization because it assumes particular determinism
But other way around, i.e. breaking code that never assumed anything by removing nondeterminism... sounds weird
1 reply
enjoinedmot
@enjoinedmot:matrix.org
[m]
thank you WGH
HackerFoo
@hackerfoo:sandbox.hackerfoo.com
[m]
Rayon tests fail on my machine. I'm not sure if it's the new compiler or M1 Mac specific: rayon-rs/rayon#956
Ritchie Vink
@ritchie46

Hi all.. We use rayon extensively in polars and can have multiple levels of par_iter.

Now my understanding is that if the outer par_iter has sufficient groups and the inner par_iter also has many groups we can get stackoverflow due to work stealing.

I believe this can be resolved by setting with_min_len and that is a solution I want to investigate. But before I do so I have a question of another possible path.

Is it possible to retrieve the depth of par_iter calls? . Something like this. This shows two levels of nesting, but we can actually have more.


// this would be level 1
fn inner_work(seq) {
    if par_level == 0 {
        seq.par_iter().some_work()
    } else {
       seq.iter().some_work()
    }
}

// this would be level 0
fn outer_work(seq) {
   seq.par_iter().map(inner_work).collect()
}
wagnerf42
@wagnerf42
hi
we could actually write an adaptor giving you access to that info but i don't think any of the actual adaptors will allow you to retrieve it
outside of manually splitting with rayon::iter::split
Ritchie Vink
@ritchie46
That would be great. Would that be in the scope of the project? It would make it much easier for use to predict if some parallelizing job might sefgault due to SO or not.
Josh Stone
@jistone:fedora.im
[m]
could you pass your own par_level parameter through inner_work/some_work()?
I guess it depends whether you just want par_iter nesting, or the full depth of work-stealing nesting -- but we don't really track that
Ritchie Vink
@ritchie46

could you pass your own par_level parameter through inner_work/some_work()?

In this trivial example we can. But in polars we can have aribitrary nesting as expressions may call expressions. It is very hard to track in there are so many operation that may par_iter and may be very deep.

I guess it depends whether you just want par_iter nesting, or the full depth of work-stealing nesting -- but we don't really track

I am only interested in the par_iter nesting. Then we could apply a rule of thumb that the outer level has parallelization priority and can make the inner levels 1..n sequential.

xmac94x
@xmac94x:matrix.org
[m]
Hi everyone :) I just noticed some weird behavior on our game velorem where we are waiting for some threads to finish in places where we wouldn't have thought they would. I opened a issue with the details: rayon-rs/rayon#969 . Not sure if this is a bug, but we noticed that a threadpool.spawn inside a parallel iterator will let the parallel iterator wait until the spawn completed. Which is not always what we want. Do you know any good workaround for that problem ? Maybe already others experienced it (:
Josh Stone
@jistone:fedora.im
[m]
xmac94x: I think this is the nature of work-stealing, but see my reply on the issue for gory details
(also, don't take my word as gospel!)
xmac94x
@xmac94x:matrix.org
[m]
I understand that rayon is using work-stealing, but not why its not having a check to not steal unreleated work inside a par_iter ? Or even steal work from other scopes
Josh Stone
@jistone:fedora.im
[m]
it has no concept of what is "related"
there's one global queue for each pool, and each thread within the pool has a local deque
Miguel Raz Guzmán Macedo
@miguelraz
Hello!
I'm looking for a good collection of serial vs Rayon's parallelization benchmarks snippets
Thinking of stuff like
Dot product calculation,
Independently processing many strings in a big file
or anything numeric/related to linear algebra
Josh Stone
@jistone:fedora.im
[m]
rayon-demo (in the rayon repo) has some, not sure if you'd call them "good" though ;)
Miguel Raz Guzmán Macedo
@miguelraz
@jistone:fedora.im yup, those seem just like what I wanted. What's the command to run them though? cargo bench in the rayon/rayon-demo dir?
:sweat:
whoknow
@whoknow:matrix.org
[m]
Hi, somewhere in my code I need to check a big vector (also some times it can be small) for finding some data in it (Im using data.iter().find(...) in old method) its work good, I didn't notice any kind of performance lost in it, but I was thinking changing it so it search in parallel using rayon make sense? I mean there is any cost for rayon to spawn the threads everytime Im calling it? (as I said it would so many times...)
at the end is it worth it for use it in this kind of situations?
Josh Stone
@jistone:fedora.im
[m]
rayon keeps a threadpool, so after the first time it won't be spawning any new threads
3 replies
(unless you explicitly create your own ThreadPool each time)
whoknow
@whoknow:matrix.org
[m]
also one other thing, is it possible to use rayon inside normal loops?
2 replies
WGH
@WGH:torlan.ru
[m]
No clue about Windows, never really used it for any development work
On Linux, I'd use perf, if that helps
No, you can't
Josh Stone
@jistone:fedora.im
[m]
the language for loops are hard-coded to the regular Iterator trait
but if you can rewrite that with Iterator::for_each, then this can translate to ParallelIterator::for_each
I have a crate that hacks that transformation with a procedural macro: https://crates.io/crates/rayon-macro
Miguel Raz Guzmán Macedo
@miguelraz
Let’s say I have a dir with many nested dirs (at different levels of , all containing .txt files that can be processed independently.
What’s the Rayon idiomatic way of processing them? (for argument’s sake, a wc on each file will suffice)
WcaleNieWolny
@WcaleNieWolny
pub fn fast_split(data: &[i8], cache: Vec<usize>) -> Vec<i8>{
        let mut out = Vec::<i8>::with_capacity(data.len());
    //TODO: resize

        for i in 0..data.len(){
            out[cache[i]] = data[i]
        }

        out
    }
how would I make this run in parallel? Like using rayon or something like that
Josh Stone
@jistone:fedora.im
[m]
@WcaleNieWolny: arbitrary parallel writes are difficult in Rust, but the best way is probably to temporarily cast that out vector to &mut [AtomicI8].
or with your own unsafe pointer cast the same way, https://doc.rust-lang.org/src/core/sync/atomic.rs.html#2064
then in rayon, (0..data.len()).into_par_iter().for_each(|i| /* atomic store */)
it may not parallelize well though -- memory bound and probably cache-unfriendly
mr.rusty
@mr.rusty:matrix.org
[m]
Hi everyone! Just wanted to know if things like the JoinHandle are possible with Rayon. Or are there alternatives to wait for a task to finish?
Josh Stone
@jistone:fedora.im
[m]
mr.rusty: most of the rayon api blocks for completion, but for spawn you can use a scope, as that waits for its spawns to complete before returning.
mr.rusty
@mr.rusty:matrix.org
[m]
Thank your, i think that'll help me :)
mr.rusty
@mr.rusty:matrix.org
[m]
:point_up: Edit: Thank you, i think that'll help me :)
Miguel Raz Guzmán Macedo
@miguelraz
Hello @jistone:fedora.im !
I'm thinking of using some of my Xmas break to implement this - rayon-rs/rayon#329
What's a good way to start?
Josh Stone
@jistone:fedora.im
[m]
I don't know how Niko thought that would work, and he never got back to that issue with a better write-up...