What is the idiomatic way to do the following in Rayon:

• Starting at a given state
• A function that, given a state, generates additional states
• Generate all states in parallel, collecting them (e.g. into a vector).

The simplest way is to use an Arc<RwLock<Vec<State>>> and use a ThreadPoolBuilder.install(|| reach_state(first_state)), where reach_state will generate the sub-states, then par_iter on each generated sub-state, pushing it into the vector and recursively invoke reach_state on it.

This works. However, this consumes large amounts of stack stack due to the recursive calls to reach_state.

Is there an idiomatic way to setup a queue of items, and tell Rayon to process items in parallel from the queue, where this processing may add new items to the queue? This pattern is common in graph algorithms (basically, parallel variants of Dijkstra's algorithm, with either an explicit input graph or a graph that is being built dynamically as it is being searched).

The underlying libraries (e.g. crossbeam) can obviously do this, but with nothing like Rayon's ease of use.

@cuviper - ah, scope seems to do almost exactly what I need (with a few lines of code). I'll give it a try. Thanks!

@wagnerf42 - thanks for the code. This got complex pretty fast... Especially compared to "replace iter with par_iter" :-)

well. the inner workings are not easy but the use is easy.

here is an example with a tree traversal. (i fixed a bug btw)

If you look at the main it's just a call to descendant in order to get the parallel iterator. the descendant function is literally what you wanted.

use rayon::iter::plumbing::{bridge_unindexed, Folder, UnindexedConsumer, UnindexedProducer};
use rayon::prelude::*;
use std::collections::VecDeque;
use std::fmt::Debug;
use std::iter::once;
use std::marker::PhantomData;

struct TreeProducer<'b, S, B, I> {
    to_explore: VecDeque<Vec<S>>, // invariant: no one is empty here
    seen: Vec<S>,
    breed: &'b B,
    phantom: PhantomData<I>,
}

impl<'b, S, B, I> UnindexedProducer for TreeProducer<'b, S, B, I>
where
    S: Send + Debug,
    B: Fn(&S) -> I + Send + Sync,
    I: Iterator<Item = S> + Send,
{
    type Item = S;
    fn split(mut self) -> (Self, Option<Self>) {
        // explore while front is of size one.
        while self
            .to_explore
            .front()
            .map(|f| f.len() == 1)
            .unwrap_or(false)
        {
            let front_node = self.to_explore.pop_front().unwrap().pop().unwrap();
            let next_to_explore: Vec<_> = (self.breed)(&front_node).collect();
            if !next_to_explore.is_empty() {
                self.to_explore.push_back(next_to_explore);
            }
            self.seen.push(front_node);
        }
        // now take half of the front.
        let f = self.to_explore.front_mut();
        let right_children = f.map(|f| {
            let n = f.len() / 2;
            let mut c = Vec::with_capacity(n);
            for _ in 0..n {
                c.push(f.pop().unwrap());
            }
            c
        });
        let right = right_children.map(|c| TreeProducer {
            to_explore: once(c).collect(),
            seen: Vec::new(),
            breed: self.breed,
            phantom: PhantomData,
        }); //TODO: if nothing we could split seen
        eprintln!("right: {}", right.is_some());
        (self, right)
    }
    fn fold_with<F>(self, mut folder: F) -> F
    where
        F: Folder<Self::Item>,
    {
        // start by consuming everything seen
        for s in self.seen {
            folder = folder.consume(s);
            if folder.full() {
                return folder;
            }
        }
        // now do all remaining explorations
        for mut e in self.to_explore {
            while let Some(s) = e.pop() {
                e.extend((self.breed)(&s));
                folder = folder.consume(s);
                if folder.full() {
                    return folder;
                }
            }
        }
        folder
    }
}

struct TreeIter<S, B, I> {
    initial_state: S,
    breed: B,
    phantom: PhantomData<I>,
}

impl<S, B, I> ParallelIterator for TreeIter<S, B, I>
where
    S: Send + Debug,
    B: Fn(&S) -> I + Send + Sync,
    I: Iterator<Item = S> + Send,
{
    type Item = S;
    fn drive_unindexed<C>(self, consumer: C) -> C::Result
    where
        C: UnindexedConsumer<Self::Item>,
    {
        let producer = TreeProducer {
            to_explore: once(vec![self.initial_state]).collect(),
            seen: Vec::new(),
            breed: &self.breed,
            phantom: PhantomData,
        };
        bridge_unindexed(producer, consumer)
    }
}

fn descendants<S, B, I>(root: S, breed: B) -> TreeIter<S, B, I>
where
    S: Send,
    B: Fn(&S) -> I + Send + Sync,
    I: Iterator<Item = S> + Send,
{
    TreeIter {
        initial_state: root,
        breed,
        phantom: PhantomData,
    }
}

#[derive(Debug)]
struct Node {
    content: u32,
    left: Option<Box<Node>>,
    right: Option<Box<Node>>,
}

fn main() {
    let t = Node {
        content: 10,
        left: Some(Box::new(Node {
            content: 3,
            left: None,
            right: None,
        })),
        right: Some(Box::new(Node {
            content: 14,
            left: None,
            right: Some(Box::new(Node {
                content: 18,
                left: None,
                right: None,
            })),
        })),
    };
    let v: Vec<&Node> = descendants(&t, |r| {
        r.left
            .as_ref()
            .into_iter()
            .chain(r.right.as_ref())
            .map(|n| &**n)
    })
    .collect();
    eprintln!("v: {:?}", v);
}

@wagnerf42 - thanks for the code. This got complex pretty fast... Especially compared to "replace iter with par_iter" :-)

I don't suppose there's some crate somewhere that has something similar to:

fn par_deque<T>(first: T, task: Fn(T, &RwLock<VecDeque<T>>);

So I could write something along the lines of:

par_deque(first_data, |data, pending| {
    ... do stuff with the data ...
    pending.write().unwrap().push_back(... more data ...);
    ... do stuff with the data ...
    pending.write().unwrap().push_back(... even more data ...);
    ... etc. ...
});

@cuviper Is there anything left in this PR that needs to be changed? We'd really like this functionality for rust-gpu so that we can support rem_euclid on SPIR-V platforms. 🙂

hi,

here is one:

use rayon::iter::plumbing::{bridge_unindexed, Folder, UnindexedConsumer, UnindexedProducer};
use rayon::prelude::*;
use std::collections::VecDeque;
use std::fmt::Debug;
use std::iter::once;
use std::marker::PhantomData;

struct TreeProducer<'b, S, B, I> {
    to_explore: VecDeque<Vec<S>>, // invariant: no one is empty here
    seen: Vec<S>,
    breed: &'b B,
    phantom: PhantomData<I>,
}

impl<'b, S, B, I> UnindexedProducer for TreeProducer<'b, S, B, I>
where
    S: Send + Debug,
    B: Fn(&S) -> I + Send + Sync,
    I: Iterator<Item = S> + Send,
{
    type Item = S;
    fn split(mut self) -> (Self, Option<Self>) {
        // explore while front is of size one.
        while self
            .to_explore
            .front()
            .map(|f| f.len() == 1)
            .unwrap_or(false)
        {
            let front_node = self.to_explore.pop_front().unwrap().pop().unwrap();
            self.to_explore
                .push_back((self.breed)(&front_node).collect());
            self.seen.push(front_node);
        }
        // now take half of the front.
        let f = self.to_explore.front_mut();
        let right_children = f.map(|f| {
            let n = f.len() / 2;
            let mut c = Vec::with_capacity(n);
            for _ in 0..n {
                c.push(f.pop().unwrap());
            }
            c
        });
        let right = right_children.map(|c| TreeProducer {
            to_explore: once(c).collect(),
            seen: Vec::new(),
            breed: self.breed,
            phantom: PhantomData,
        }); //TODO: if nothing we could split seen
        (self, right)
    }
    fn fold_with<F>(self, mut folder: F) -> F
    where
        F: Folder<Self::Item>,
    {
        // start by consuming everything seen
        for s in self.seen {
            folder = folder.consume(s);
            if folder.full() {
                return folder;
            }
        }
        // now do all remaining explorations
        for mut e in self.to_explore {
            while let Some(s) = e.pop() {
                e.extend((self.breed)(&s));
                folder = folder.consume(s);
                if folder.full() {
                    return folder;
                }
            }
        }
        folder
    }
}

struct TreeIter<S, B, I> {
    initial_state: S,
    breed: B,
    phantom: PhantomData<I>,
}

impl<S, B, I> ParallelIterator for TreeIter<S, B, I>
where
    S: Send + Debug,
    B: Fn(&S) -> I + Send + Sync,
    I: Iterator<Item = S> + Send,
{
    type Item = S;
    fn drive_unindexed<C>(self, consumer: C) -> C::Result
    where
        C: UnindexedConsumer<Self::Item>,
    {
        let producer = TreeProducer {
            to_explore: once(vec![self.initial_state]).collect(),
            seen: Vec::new(),
            breed: &self.breed,
            phantom: PhantomData,
        };
        bridge_unindexed(producer, consumer)
    }
}

fn descendants<S, B, I>(root: S, breed: B) -> TreeIter<S, B, I>
where
    S: Send,
    B: Fn(&S) -> I + Send + Sync,
    I: Iterator<Item = S> + Send,
{
    TreeIter {
        initial_state: root,
        breed,
        phantom: PhantomData,
    }
}

fn main() {
    let v: Vec<u32> = descendants(10, |e| {
        once(e / 2).chain(once(e / 2 + 1)).filter(|&e| e > 2)
    })
    .collect();
    eprintln!("v: {:?}", v);
}

note that you don't necessarily need to collect. if you do it also could be more efficient to fold into Vec and collect into LinkedList<Vec>

What is the idiomatic way to do the following in Rayon:

• Starting at a given state
• A function that, given a state, generates additional states
• Generate all states in parallel, collecting them (e.g. into a vector).

The simplest way is to use an Arc<RwLock<Vec<State>>> and use a ThreadPoolBuilder.install(|| reach_state(first_state)), where reach_state will generate the sub-states, then par_iter on each generated sub-state, pushing it into the vector and recursively invoke reach_state on it.

This works. However, this consumes large amounts of stack stack due to the recursive calls to reach_state.

Is there an idiomatic way to setup a queue of items, and tell Rayon to process items in parallel from the queue, where this processing may add new items to the queue? This pattern is common in graph algorithms (basically, parallel variants of Dijkstra's algorithm, with either an explicit input graph or a graph that is being built dynamically as it is being searched).

The underlying libraries (e.g. crossbeam) can obviously do this, but with nothing like Rayon's ease of use.

Note that this causes compile failures in rustc_middle currently:

    Checking rustc_builtin_macros v0.0.0 (/home/joshua/rustc3/compiler/rustc_builtin_macros)
error[E0277]: `*mut u8` cannot be sent between threads safely
    --> compiler/rustc_middle/src/ty/mod.rs:2788:14
     |
2788 |             .for_each(|&body_id| f(self.hir().body_owner_def_id(body_id)));
     |              ^^^^^^^^ `*mut u8` cannot be sent between threads safely
     |
     = help: within `rustc_arena::DropType`, the trait `Send` is not implemented for `*mut u8`
     = note: required because it appears within the type `rustc_arena::DropType`
     = note: required because of the requirements on the impl of `Send` for `Unique<rustc_arena::DropType>`
     = note: required because it appears within the type `RawVec<rustc_arena::DropType>`
     = note: required because it appears within the type `Vec<rustc_arena::DropType>`
     = note: required because of the requirements on the impl of `Send` for `RefCell<Vec<rustc_arena::DropType>>`
     = note: required because it appears within the type `DropArena`
     = note: required because it appears within the type `Arena<'tcx>`
     = note: required because of the requirements on the impl of `std::marker::Sync` for `WorkerLocal<Arena<'tcx>>`
     = note: required because it appears within the type `&'tcx WorkerLocal<Arena<'tcx>>`
     = note: required because it appears within the type `GlobalCtxt<'tcx>`
     = note: required because it appears within the type `&'tcx GlobalCtxt<'tcx>`
     = note: required because it appears within the type `context::TyCtxt<'tcx>`
     = note: required because it appears within the type `&context::TyCtxt<'tcx>`
     = note: required because it appears within the type `[closure@compiler/rustc_middle/src/ty/mod.rs:2788:23: 2788:74]`

error[E0277]: `*mut u8` cannot be sent between threads safely
    --> compiler/rustc_middle/src/ty/context.rs:1738:13
     |
1738 |             sync::assert_sync::<ImplicitCtxt<'_, '_>>();
     |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `*mut u8` cannot be sent between threads safely
     | 
    ::: /home/joshua/rustc3/compiler/rustc_data_structures/src/sync.rs:420:32
     |
420  | pub fn assert_sync<T: ?Sized + Sync>() {}
     |                                ---- required by this bound in `assert_sync`
     |
     = help: within `rustc_arena::DropType`, the trait `Send` is not implemented for `*mut u8`
     = note: required because it appears within the type `rustc_arena::DropType`
     = note: required because of the requirements on the impl of `Send` for `Unique<rustc_arena::DropType>`
     = note: required because it appears within the type `RawVec<rustc_arena::DropType>`
     = note: required because it appears within the type `Vec<rustc_arena::DropType>`
     = note: required because of the requirements on the impl of `Send` for `RefCell<Vec<rustc_arena::DropType>>`
     = note: required because it appears within the type `DropArena`
     = note: required because it appears within the type `Arena<'_>`
     = note: required because of the requirements on the impl of `std::marker::Sync` for `WorkerLocal<Arena<'_>>`
     = note: required because it appears within the type `&WorkerLocal<Arena<'_>>`
     = note: required because it appears within the type `GlobalCtxt<'_>`
     = note: required because it appears within the type `&GlobalCtxt<'_>`
     = note: required because it appears within the type `context::TyCtxt<'_>`
     = note: required because it appears within the type `ImplicitCtxt<'_, '_>`

error: aborting due to 2 previous errors

No guarantees for the uniformity of the shuffle with such an ordering function in sort. There should be literature on it; IUUC it is biased. Sorting by a cached random key should be the way to go if needed.

… TempFile doesn't reside on a local drive"

This reverts commit 79657e2339b58bc01fe1b85a448bb073d57d90bb.

The change regressed on Windows 7 -- rust-lang/rust#81051

… TempFile doesn't reside on a local drive"

This reverts commit 6ec777c2f6496b4fe1d78cc6d6871a3dc931a185, which was cherry picked to 11.x from 79657e2339b58bc01fe1b85a448bb073d57d90bb.

The change regressed on Windows 7 -- rust-lang/rust#81051

We need the same revert for LLVM12, no?

I think its fine to backport #90 too, it is only relevant to unstable functionality (-Csplit-debuginfo=packed requires -Z unstable-options)

I suppose if the item already exists at any index, you want it moved to the new location?

For my particular use-case I wish to keep the item at the current index, but I need to modify it as well. However, performance is not very critical at all, so I am completely okay with handling this by doing a second lookup.

I think you are right, that the most reasonable behavior would be to move the item it.

I guess the way to completely support my use-case would be to add shift_insert methods to both VacantEntry and OccupiedEntry, but I am not sure if this is worth the added complexity.

I wrote this simple snippet to test the panic_handler

use ndarray::{Array2, Array3, ArrayView2, Axis};
use rayon::ThreadPoolBuilder;
let panic_handler = |_| {
    println!("############ Test ############");
};
ThreadPoolBuilder::new().num_threads(4).panic_handler(panic_handler).build_global().unwrap();

let data = Array3::zeros((100, 100, 100));
Zip::from(data.axis_iter(Axis(2))).par_apply(|slice: ArrayView2<f32>| {
    panic!("Hola");
});

The threads do panic. I see several errors messages that look like

thread 'thread 'thread 'thread '<unnamed><unnamed><unnamed><unnamed>' panicked at '' panicked at '' panicked at '' panicked at 'HolaHolaHolaHola', ', ', ', test.rstest.rstest.rstest.rs::::10101010::::9999
idem
...

However, I never see my panic_handler message. Why? Isn't this exactly the goal of a panic_handler?

Moreover, why do I see that many error messages? I understand that each thread must end or panic before quitting, but it looks like rayon is trying and trying again to run what I asked in Zip::from. Can I ask rayon to never retry? To leave as soon as there an error?

Oh, ok, it's clearer now :) Thank you!

I added @tspiteri's fallback behind an autocfg gate, but I haven't been able to confirm whether it actually works as designed - I can't find any traces of the fallback method in the output binary (with stable rustc) even if I delete the method in the trait implementation entirely.

I really like that sort_by idea! Here's an example for future readers:

Rust Playground

use indexmap::IndexSet;
use rand::{
    distributions::{Distribution, Standard},
    Rng,
};
use std::cmp::Ordering;

#[derive(Debug)]
struct RandomOrdering(Ordering);

impl Into<Ordering> for RandomOrdering {
    fn into(self) -> Ordering {
        self.0
    }
}

impl Distribution<RandomOrdering> for Standard {
    fn sample<R: Rng + ?Sized>(&self, rng: &mut R) -> RandomOrdering {
        RandomOrdering(match rng.gen_range(0..2) {
            0 => Ordering::Less,
            1 => Ordering::Equal,
            _ => Ordering::Greater,
        })
    }
}

fn main() {
    let mut letters: IndexSet<char> = ('a'..='z').collect();
    println!("{:?}", letters);
    letters.sort_by(|_, _| rand::random::<RandomOrdering>().into());
    println!("{:?}", letters);
}

@cuviper @janmarthedal Thank you so much. Today, I confirmed that my program got faster using your bit and set_bit. It helped me a lot.

• cgmath 0.17 -> 0.18
• glium 0.28 -> 0.29
• num 0.2 -> 0.3
• rand 0.7 -> 0.8
• rand_xorshift 0.2 -> 0.3

Build succeeded:

• Check (1.36.0)
• Demo (nightly)
• Demo (stable)
• Format
• Test (macos-latest, beta)
• Test (macos-latest, nightly)
• Test (macos-latest, stable)
• Test (ubuntu-latest, beta)
• Test (ubuntu-latest, nightly)
• Test (ubuntu-latest, stable)
• Test (ubuntu-latest, stable-i686)
• Test (windows-latest, beta)
• Test (windows-latest, nightly)
• Test (windows-latest, stable)
• WebAssembly