Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Nov 26 18:19
    dcompiler closed #994
  • Nov 25 22:05
    dcompiler opened #994
  • Nov 21 10:54
    bobi6666 opened #993
  • Nov 19 00:16

    cuviper on rayon-core-v1.10.1

    (compare)

  • Nov 19 00:14
    bors[bot] closed #992
  • Nov 19 00:14

    bors[bot] on master

    Fix a sleep race with broadcast… Release 1.10.1 Merge #992 992: Fix a sleep ra… (compare)

  • Nov 19 00:05

    bors[bot] on staging.tmp

    (compare)

  • Nov 19 00:05

    bors[bot] on staging

    Fix a sleep race with broadcast… Release 1.10.1 Merge #992 992: Fix a sleep ra… (compare)

  • Nov 19 00:05

    bors[bot] on staging.tmp

    Fix a sleep race with broadcast… Release 1.10.1 [ci skip][skip ci][skip netlify… (compare)

  • Nov 19 00:05

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Nov 19 00:03
    cuviper synchronize #992
  • Nov 18 23:45
    cuviper opened #992
  • Nov 18 21:10

    cuviper on v1.6.0

    (compare)

  • Nov 18 21:10

    cuviper on rayon-core-v1.10.0

    (compare)

  • Nov 18 20:56
    bors[bot] closed #991
  • Nov 18 20:55

    bors[bot] on master

    Release rayon 1.6.0 / rayon-cor… Merge #991 991: Release rayon … (compare)

  • Nov 18 20:45

    bors[bot] on staging.tmp

    (compare)

  • Nov 18 20:45

    bors[bot] on staging

    Release rayon 1.6.0 / rayon-cor… Merge #991 991: Release rayon … (compare)

  • Nov 18 20:45

    bors[bot] on staging.tmp

    Release rayon 1.6.0 / rayon-cor… [ci skip][skip ci][skip netlify… (compare)

  • Nov 18 20:45

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

parazyd
@parazyd:dark.fi
[m]
hmm ok, I'll try something out in a bit
Thanks
Aleksey Kladov
@matklad
Question: I want to process a number of items in parallel, so I do xs.into_par_iter().... The problem is, the lenght of xs is at most 4, and Items are very heavy, so I want to make sure that rayon splits each of vec's element into a separate stolable job. What's the right API for that?
Aleksey Kladov
@matklad
And perhaps more generally: where can I read how rayon decides to split slices?
Aleksey Kladov
@matklad
Hm, actually my testing seems to show that rayon just distributes the work across all cores out of the box... Is there some official confirmatio to that end somewhere?
Josh Stone
@cuviper
@matklad I don't think we document any guarantees about how the default mode works, but you can force .with_max_len(1)
the details of how are found in src/iter/plumbing/ if you want to go reading
WGH
@WGH:torlan.ru
[m]
isn't the bottom line, as things are right now, .with_min_len is useful if every work unit is small, otherwise, there's no need to touch anything?
Josh Stone
@cuviper
hopefully yes, the default "adaptive" splitting should work pretty well in most situations
Josh Stone
@jistone:fedora.im
[m]
(ooh, I haven't tried matrix here before, neat!)
Zhengyi Yang
@zhengyi-yang
Hi guys, does anyone know if it is possible to turn off work stealing in rayon's ThreadPool? Thanks a lot
Josh Stone
@jistone:fedora.im
[m]
@zhengyi-yang: not really -- I did have some ideas about adding a "critical section" primitive where you could prevent stealing while a particular job is blocked. Can you elaborate why you want this?
oliver-giersch
@oliver-giersch
hello, I have a cargo workspace with a bunch of crates. In one of them I collect benchmarks using criterion (which depends on rayon) and in one of the other crates I want to use rayon as a git dependency (master), in order to be able to use std::ops::ControlFlow, which is already integrated but not in the latest version from May 21. However, rayon-core appears to explicitly prohibit this mixing of different versions. Is there a work around for this restriction?
Josh Stone
@jistone:fedora.im
[m]
@oliver-giersch: I think maybe if you use cargo [patch...] to override git rayon, it might still use the published rayon-core, but I'm not sure. The best answer is that I should just publish a new release for you. :)
I would like to get rayon-rs/rayon#926 done for a release too
Zhengyi Yang
@zhengyi-yang
@jistone:fedora.im Thanks josh. I just want to do some experiments on how work-stealing affects the overall performance. I will try to use the naive thread for comparison.
Aleksey Kladov
@matklad
Question: is it possible to "rate limit" a particular parallel iterator? I have a concurrent application, which has background and forground processing, and both parts sometimes do things in parallell. For foreground, I want to always use all ther threads from the global threadpool, all the time. For background, I want it to occupy, say, at most half of the threads.
My first naive thought is to just add a semaphor to background "task", but I think that'll just tie up the threads
My second thought is to use min_len to make sure that .par_iter splits the backghound work into at most n_cpus / 2 chunks, which I think will work (although it might be somewat suboptimal with respect to work-stealing and load balancing).
Is there a more proper way to do what I want here?
Josh Stone
@jistone:fedora.im
[m]
beyond limiting splits, there's no way to prioritize way within a single pool
if you can use multiple pools, you can give the background pool's threads lower OS priority (no builtin rayon method for that, but you can make calls from a start_handler)
Josh Stone
@jistone:fedora.im
[m]
@oliver-giersch: I just published rayon 1.5.2
oliver-giersch
@oliver-giersch
@jistone:fedora.im Hey thanks! Much appreciated
epilys
@epilys:matrix.org
[m]
Hello all
Would a PR adding wasm target code to run jobs on the main process (without WebWorkers/threads) be accepted?
so that you can compile crates that use rayon to wasm without having to do any changes.
Josh Stone
@jistone:fedora.im
[m]
@epilys there are two similar issues already open:
rayon-rs/rayon#591
rayon-rs/rayon#861
spawn is the biggest challenge that I know of, if code expects that to make independent progress
I think this shouldn't be hard-coded for wasm targets, but it could be a feature that users may want to enable in wasm builds
epilys
@epilys:matrix.org
[m]
Yeah that makes more sense. My team needs it for our compiler so we are interested in contributing it.
thank you 😊
forbisc
@forbisc
If I have two different threads that are spawning task into different rayon::scope, and they share the global thread pool. Is there any reason to believe that rayon would fairly share the threads among the different scopes? Basically I'm wondering if it would give roughly equal time or resources to each scope, or is that behavior not determined?
Josh Stone
@jistone:fedora.im
[m]
@forbisc: there's no concept of fairness. the threads use work stealing when they are idle, taking tasks from any other thread's queue, with no visibility into their "origin" like scopes.
task stealing happens at the front of the queue, essentially FIFO, but it's random which thread is stolen from.
broadly speaking, rayon is opportunistic, trying to maximize execution throughput. I think tokio would be a better fit if you're concerned about fairness or latency.
forbisc
@forbisc
Thank you! Makes sense. Rayon does not make any guarantees about which thread will be stolen from (if multiple threads have tasks waiting).
zr40
@zr40:zr40.nl
[m]
I'm iterating over a list of files that each contain a number of items that I'd want to process in parallel. Right now I'm basically doing files.into_par_iter().filter_map(get_items_from_file).flat_map_iter(process_item).collect(). However this way get_items_from_file can only return all the items in a file at once. What would be a way to provide these items to the iterator as soon as they're available?
Josh Stone
@jistone:fedora.im
[m]
zr40: it sounds like get_items_from_file would do well in a flat_map_iter
zr40
@zr40:zr40.nl
[m]
how would that let get_items_from_file return items before it’s done?
Josh Stone
@jistone:fedora.im
[m]
I was thinking as a regular Iterator, but I can't guess more without knowing what that function is doing
zr40
@zr40:zr40.nl
[m]
the function takes one file and returns the N items that are present in the file
Josh Stone
@jistone:fedora.im
[m]
well sure, but can you do that incrementally in Iterator::next?
zr40
@zr40:zr40.nl
[m]
ah, you meant that I could implement an Iterator. Sure I can do that. get_items_from_file is built like a state machine so that shouldn't be too hard to adapt
enjoinedmot
@enjoinedmot:matrix.org
[m]
Hello people! I recently inherited a project with rayon in its dependencies, and I made a patch that changed .into_par_iter() to .into_iter() after finding the proeject wouldn't compile due to a trait bound error.
If anyone with specific knowledge has a moment, could you let me know if this should change at all the result of the logic that was changed? To me it looks like it should only be a performance hit, but I wanted to make sure.
WGH
@WGH:torlan.ru
[m]
So you basically removed parallelism here, no?
9 replies
enjoinedmot
@enjoinedmot:matrix.org
[m]
It does compile and seems to work after the change to .into_iter() Also, this might be out of scope, but actually just satisfying the trait bound and restoring the logic to what it was would be great, but I didn't get success right away. If anyone has general or specific advice I would be very grateful.
It seemed to me like that was the case, but I just jumped into these crates so I thought I might ask an expert. 😄
Dirkjan Ochtman
@djc
you should probably explain more about the trait bound issue so you can just keep it going in parallel :)