Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 05 21:49
    cuviper edited #711
  • Dec 05 21:49
    cuviper synchronize #711
  • Dec 05 21:17
    cuviper synchronize #711
  • Dec 05 19:41
    cuviper synchronize #711
  • Dec 05 17:25
    cuviper opened #711
  • Nov 21 09:18
    kentfredric opened #710
  • Nov 21 00:42

    cuviper on rayon-core-v1.6.1

    (compare)

  • Nov 21 00:42

    cuviper on v1.2.1

    (compare)

  • Nov 21 00:35
    bors[bot] closed #709
  • Nov 21 00:35

    bors[bot] on master

    Avoid mem::uninitialized in par… Avoid mem::uninitialized in the… cargo fmt and 4 more (compare)

  • Nov 20 23:28
    cuviper labeled #707
  • Nov 20 23:06

    bors[bot] on staging.tmp

    (compare)

  • Nov 20 23:06

    bors[bot] on staging

    Avoid mem::uninitialized in par… Avoid mem::uninitialized in the… cargo fmt and 4 more (compare)

  • Nov 20 23:06

    bors[bot] on staging.tmp

    Avoid mem::uninitialized in par… Avoid mem::uninitialized in the… cargo fmt and 4 more (compare)

  • Nov 20 23:06

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Nov 20 20:15
    cuviper opened #709
  • Nov 20 00:38
    cuviper closed #708
  • Nov 18 19:28
    calebwin closed #699
  • Nov 16 01:21
    bachue opened #708
  • Nov 08 19:41
    silwol synchronize #707
Josh Stone
@cuviper
(cache being their TLS caching, not CPU-level)
Steven Joruk
@steven-joruk
I want to use itertools' multi_cartesian_product with rayon's ParallelIndexedIterator so I can create chunks for rayon to work with, without collecting first as it's too big. Is implementing the relevant traits for my own new type the right approach? Is there a better way to make them compatible upstream?
Caleb Winston
@calebwin
i'm really curious about this - how does Rayon maintain a single global thread pool for all Rayon users?
Caleb Winston
@calebwin
@cuviper is it a thread-local variable?
Paul Reesman
@reesmanp

Hey all, quick question for you...
I am trying to par_iter over a vec I have and I am getting:

 the method `par_iter` exists but the following trait bounds were not satisfied:
`[island_model::island::Island] : rayon::iter::IntoParallelRefIterator`
 `std::vec::Vec<island_model::island::Island> : rayon::iter::IntoParallelRefIterator`

Would it be because the type within the vec does not implement Send?

self.islands.par_iter().for_each(|&mut island| {
   |                          ^^^^^^^^ method not found in `std::vec::Vec<island_model::island::Island>`
Paul Reesman
@reesmanp
Ok, so even when I make the vec Vec<Arc<RefCell<Island>>> it doesn't work. Can I get any help on this?
Caleb Winston
@calebwin
it seems like, yeah, there is the Send trait implementation requirement for Island
also seems like the provided implementation of IntoParallelRefIterator requires Island to be Sized...?
i could be wrong though
Paul Reesman
@reesmanp
How would I go about implementing Send for it? I can't find an example
Caleb Winston
@calebwin
you don't, it's automatically derived if all of its fields implement it
what does Island look like?
you need to make sure that everything inside an Island is all Send
i should mention that I have never used Rayon myself...
Paul Reesman
@reesmanp
Well Island doesn't have Send, thus the error so not everything in it has send.
Seems to almost be there now... just had to do a unsafe impl Send for Island {}
Caleb Winston
@calebwin
no, Island does implicitly have Send if all of Island's fields have Send
you don't need to manually derive it, I think
Denis Lisov
@tanriol
@reesmanp Are you really sure that sending Island to a different thread is safe?
Caleb Winston
@calebwin
yeah, if Island isn't Send then it probably isn't thread-safe
doing an unsafe impl is literally unsafe and exactly what you want Rust to prevent
can you show us what Island's fields are?
Caleb Winston
@calebwin
also, can someone with more Rayon knowledge than me explain how Rayon manages a static mut threadpool in a safe way?
Denis Lisov
@tanriol
By setting the global pool static exactly once and protecting it with Once?
Caleb Winston
@calebwin
so after it is set, it is never mutated?
Denis Lisov
@tanriol
The static mut itself is never mutated after that. There's probably some internal mutability in it, but that uses safe wrappers.
Caleb Winston
@calebwin
ok that's good to know..
Matthew Pocock
@drdozer
I just tried rayon for the first time - my second thing didn't compile - I have a for x in y.iter().map... { ... } and when I convert that over to par_iter it complains that the expression isn't an IntoIterator. So is the for syntax not supported for parallel computations?
Josh Stone
@cuviper
@drdozer for loops are solely tied to IntoIterator and Iterator::next(). Try converting that to a for_each call, and then that can convert to par_iter
Matthew Pocock
@drdozer
thanks @cuviper
Matthew Pocock
@drdozer
so I'm calling vec_of_expensive_objects.par_iter().map(...).reduce(|l, r| ...)
in some circumstances I'd like to return either l or r from the reduce, and in others I want to combine them to make a new value
I'm not sure how to express this - I'm getting the dreaded "value used here after move"
I definitely don't want to implement copy() on the objects, as they are multi-gigabyte data structures
should I wrap everything in Arc boxes? or is there a lighter-weight fix?
Josh Stone
@cuviper
@drdozer I can't see enough to guess what is your "value used here after move"
you're iterating by reference here, which seems like a good start
Matthew Pocock
@drdozer
"move occurs because r has type std::option::Option<tokens::tok_coords::TokenTable>, which does not implement the Copy trait"
TokenTable is my datastructure
        let tok = g.prots.par_iter()
          .map(|p| tokenize_raw_protein(p, &params, g_co, &coordinate_packing).seq)
          .reduce(|| None, |l, r| {
            match (l, r) {
              (None, _) => r,
              (_, None) => l,
              (Some(ref l), Some(ref r)) => Some(l.union(r))
            }
          });
that's the whole map-reduce - the move errors are on the two None lines
Josh Stone
@cuviper
so, match (l, r) moves those values out of the l and r bindings, then they can't be used further
but you can bind them anew in your pattern, like (None, r) => r
Matthew Pocock
@drdozer
oh thanks! That works :) I'm such a noob
Josh Stone
@cuviper
np
Matthew Pocock
@drdozer
will a rayon parallel iterator map reduce like the one I've built be freeing all the memory for the intermediate results as it goes? Or will it effectively be keeping hold of it all or large chunks of it until it returns the single summary value?
Josh Stone
@cuviper
@drdozer do you mean those l and r values? As far as rayon is concerned, we've moved those into your closure. Your code will drop locals like normal, as soon as that closure returns.
technically those will drop as soon as the match ends, since you moved them into that expression, but this happens to be the same as the closure end