ParallelMap and Sow/Reap - not behaving as expected? [duplicate]

Question

This question already has an answer here:

• How to collect result continuously (interruptible calculation) when running parallel calculations? 4 answers

Is ParallelMap and Parallelize@Map expected to work with Sow and Reap construct.

This is a rather brief toy code to find all Primes just to force the use of Sow/Reap inside Map

Reap[Map[
   If[PrimeQ@#, Sow@#] &,
   Range[10]
   ]][[2]]

and the output is, as expected

{{2, 3, 5, 7}}

But when I do

Reap[Parallelize@Map[
    If[PrimeQ@#, Sow@#] &,
    Range[1000]
    ]][[2]]

I got the output

{}

And the same was the output when I ran

Reap[
  ParallelMap[
   If[PrimeQ@#, Sow@#] &,
   Range[10]
   ]
  ][[2]]

2 questions :

1) How is this explainable ?

2) If this is the case, how do I efficiently I spawn parallel jobs which accumulate something ?

Answer 1

I guess this will work! Check documentation for SetSharedFunction but be careful of the parallel overhead of following implementation.

SetSharedFunction[PSow];
PSow[x_] := Sow[x];
Reap[Parallelize@Map[If[PrimeQ@#, PSow@#] &, Range[10]]][[2]]

{{2, 5, 3, 7}}

Answer 2

I would recommend parallelizing these types of calculations using the ParallelCombine paradigm.

First, let's make a function out of your example:

fun = Function[list, Reap[Map[If[PrimeQ@#, Sow@#] &, list]][[2, 1]]]

The non-parallel version will be fun[Range[200]]. Parallelize it using

ParallelCombine[
 fun,
 Range[200],
 Join
]

I spelt out Join here because in other applications you may need a different function there. Here it can be omitted.

---

This is clearly more work than the (also valid) SetSharedFunction approach, but it is going to avoid some overhead. SetSharedFunction simply forces the function to be evaluated on the main kernel. If the equivalent of PrimeQ for your real calculation gives True for a sizeable fraction of the values, the SetSharedFunction-based parallelization won't buy much performance, or might even be slower than a serial computation.