I don't really know rayon internals, but at first glance it doesn't look normal
me to collect the items encountered so far?
e.g. if you make a blocking
install/join call into your first pool, and that makes another blocking call into the second pool, that will all be synced as they return
beyond that, I don't think we have a good guide, just existing implementations as an example I guess
this README may help too: https://github.com/rayon-rs/rayon/tree/master/src/iter/plumbing
ah cool. I joined with weechat-matrix which is sometimes flaky so I wasn't sure it worked :)
Anyway, I'm learning rayon, and wondering - how could I stop iter::repeat from spawning new jobs, for example when I catch SIGINT?
I had hoped there would be some kind of take_while(), but turns out it doesn't work.
I suppose you could use fallible
try_*, and check for a flag inside your handler function
I'm trying to use this example: https://github.com/Detegr/rust-ctrlc#example-usage
And before the blocking recv, there is my ThreadPool which I spawn, and inside it there's iter::repeat
(0..100).into_par_iter()
.try_for_each(|x| {
if sigint_raised() {
return Err(());
}
// ...
});sigint_raised would probably be implemented by checking some atomic variable
and
SIGINT handler would set such variable
hmm ok, I'll try something out in a bit
Thanks
Question: I want to process a number of items in parallel, so I do
xs.into_par_iter().... The problem is, the lenght of xs is at most 4, and Items are very heavy, so I want to make sure that rayon splits each of vec's element into a separate stolable job. What's the right API for that?
the details of how are found in
src/iter/plumbing/ if you want to go reading
isn't the bottom line, as things are right now,
.with_min_len is useful if every work unit is small, otherwise, there's no need to touch anything?
@zhengyi-yang: not really -- I did have some ideas about adding a "critical section" primitive where you could prevent stealing while a particular job is blocked. Can you elaborate why you want this?
hello, I have a cargo workspace with a bunch of crates. In one of them I collect benchmarks using criterion (which depends on rayon) and in one of the other crates I want to use rayon as a git dependency (master), in order to be able to use
std::ops::ControlFlow, which is already integrated but not in the latest version from May 21. However, rayon-core appears to explicitly prohibit this mixing of different versions. Is there a work around for this restriction?
@oliver-giersch: I think maybe if you use cargo
[patch...] to override git rayon, it might still use the published rayon-core, but I'm not sure. The best answer is that I should just publish a new release for you. :)
I would like to get rayon-rs/rayon#926 done for a release too
Question: is it possible to "rate limit" a particular parallel iterator? I have a concurrent application, which has background and forground processing, and both parts sometimes do things in parallell. For foreground, I want to always use all ther threads from the global threadpool, all the time. For background, I want it to occupy, say, at most half of the threads.
My first naive thought is to just add a semaphor to background "task", but I think that'll just tie up the threads
My second thought is to use min_len to make sure that
.par_iter splits the backghound work into at most n_cpus / 2 chunks, which I think will work (although it might be somewat suboptimal with respect to work-stealing and load balancing).
Is there a more proper way to do what I want here?
beyond limiting splits, there's no way to prioritize way within a single pool
if you can use multiple pools, you can give the background pool's threads lower OS priority (no builtin rayon method for that, but you can make calls from a
start_handler)