Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jun 16 21:44
    cuviper opened #869
  • May 29 22:52
    evergreen-trading-systems closed #866
  • May 29 07:24
    bkkgbkjb opened #868
  • May 26 22:15
    stanbar closed #867
  • May 26 17:10
    stanbar opened #867
  • May 25 19:32
    evergreen-trading-systems edited #866
  • May 25 19:31
    evergreen-trading-systems opened #866
  • May 24 23:13
    rocallahan opened #865
  • May 24 07:29
    yangfengzzz opened #864
  • May 21 17:23
    bors[bot] closed #863
  • May 21 17:23
    bors[bot] closed #734
  • May 21 17:23

    bors[bot] on master

    Move `slice::Chunks*` to a new … Add `par_rchunks*` for slices Merge #863 863: Add `par_rchun… (compare)

  • May 21 17:14

    bors[bot] on staging.tmp

    (compare)

  • May 21 17:14

    bors[bot] on staging

    Move `slice::Chunks*` to a new … Add `par_rchunks*` for slices Merge #863 863: Add `par_rchun… (compare)

  • May 21 17:14

    bors[bot] on staging.tmp

    Move `slice::Chunks*` to a new … Add `par_rchunks*` for slices [ci skip][skip ci][skip netlify… (compare)

  • May 21 17:14

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • May 21 15:32
    bors[bot] closed #862
  • May 21 15:32

    bors[bot] on master

    Use in_place_scope to spawn ski… Merge #862 862: Use in_place_s… (compare)

  • May 21 15:23

    bors[bot] on staging.tmp

    (compare)

  • May 21 15:23

    bors[bot] on staging

    adjust the latest release date Use in_place_scope to spawn ski… Merge #862 862: Use in_place_s… (compare)

Eric Tu
@ec2
For some context, I'm getting a stack overflow on the global threadpool sometimes (non consistently) when I have a nested par_iter.map()
by default, it uses the standard library's own default for new threads, which I think is 2MB
ulimit -s only applies to your application's main thread, and can be "unlimited", but pthreads have to pre-allocate their stack
@ec2 ^
ah, here it confirms 2 MiB currently: https://doc.rust-lang.org/std/thread/index.html#stack-size
Eric Tu
@ec2
I see I see. Cool, @cuviper. So whats kinda the best approach to this stack overflow issue that I'm having? I've seen some Github issues where people were talking about using with_min_len(x) on the par_iter. But I'm not entirely sure exactly how Rayon already splits it, so I'm not sure what heuristic to use to set x. Or should I just double the stack size on my threadpool?
James Webber
@jamestwebber
hello! long time fan first time gitter...I'm trying to parallelize into two Counter objects (https://docs.rs/counter/0.5.2/counter/) and I implemented a newtype to implement ParExtend, but it seems like I'm running out of memory. Basically I'm wondering if I'm doing something dumb and my lovely parallel iterator is building a gigantic vector before counting things
i expect the counter to fit in memory but it's plausible the vector wouldn't... I'm running on shared compute so I think it's getting killed for memory usage but not 100% sure (just says "killed")
Josh Stone
@cuviper
@ec2 it's hard to say, often it's a judgement call for what seems like a minimum decent "chunk" of work to do serialized. You might even find that you don't need nested par_iter at all -- if the outer loop already provides enough parallelism, just let the inner run serially.
@jamestwebber can you share your ParExtend code?
James Webber
@jamestwebber
yeah, is it fine to paste in here? or I could make a gist
Josh Stone
@cuviper
if it's more than a few lines, then yes a gist or something please
James Webber
@jamestwebber
i kind cargo-cult-copied some of the existing implementations for foreign types
okay i'll make a gist
thanks!
Josh Stone
@cuviper
ok, so that's using the LinkedList<Vec<_>> strategy as rayon does for many of the collection types
but you'll note, that does collect all items in memory before ever touching the final collection
which is okay if you have mostly unique items, but probably not if you're trying to count many duplicates
@jamestwebber I think you'd probably be better off with a plain fold+reduce -- accumulating a Counter in the fold, then combining them in reduce
James Webber
@jamestwebber
ah that makes sense. i guess i was just hoping to be fancy
I wasn't sure whether the unzip would incrementally feed things into the Counter or not
James Webber
@jamestwebber
man, as a mostly-python programmer, Rust is so weird. Someone tells me what to do, I write it down and fix all the warnings the IDE tells me about, and then it just...works? wtf?
Josh Stone
@cuviper
heh, yes, it's a great feeling :)
James Webber
@jamestwebber
thanks for the help!
Dirkjan Ochtman
@djc
how can I drain a Vec in rayon?
I see that vec::Drain type exists and I found ParallelDrain* traits for HashMap/BinaryHeap, but not Vec
Dirkjan Ochtman
@djc
never mind, figured it out
Josh Stone
@cuviper
@djc what were you missing?
looking at ParallelDrainFull when you needed ParallelDrainRange?
Dirkjan Ochtman
@djc
yes
Niko Matsakis
@nikomatsakis
@cuviper r=me on rayon-rs/rayon#860
Josh Stone
@cuviper
@nikomatsakis still wait for Friday?
well you just approved with that date in place...
Niko Matsakis
@nikomatsakis
sorry :) I didn’t understand that detail
I’m fine to release whenever
Josh Stone
@cuviper
it's fine, I can even push the date change directly to master 💪
Josh Stone
@cuviper
:rocket:
Niko Matsakis
@nikomatsakis
nice
Dirkjan Ochtman
@djc
I have a workload where tasks (a) are pretty similarly sized, (b) are best executed in FIFO order, (c) need some stack-bound context, (d) need some context types that I currently pop/push from a Mutex-based pool
Currently I'm just using into_par_iter().for_each(), but of course this does work stealing and therefore can run out of order
What's the best alternative where I can stimulate FIFO ordering, and keep the context types around rather than pooling them?
Dirkjan Ochtman
@djc
Should I drop down to the crossbeam level and build something with its scope and channels, or is there something at the rayon level that could facilitate this kind of workload?
I'm a bit surprised that rayon::scope documents itself as slower than parallel iterators, since it seems like a lower-level primitive
Josh Stone
@cuviper
@djc I think Firefox does something like that for style layout using scope_fifo, with context maintained in their own tls
parallel iterators are built on join, which has the advantage of being entirely stack based, whereas scope has to box every spawn
depending on the size of work, that may be irrelevant
Dirkjan Ochtman
@djc
I did an experiment crossbeam::scope and flume channels and it was a good deal slower than my current setup
upon further reflection, I think FIFO itself causes a substantial slowdown compared to the divide-and-conquer strategy used by the parallel iterator setup