Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Is ParallelMap and Parallelize@Map expected to work with Sow and Reap construct.

This is a rather brief toy code to find all Primes just to force the use of Sow/Reap inside Map

Reap[Map[
   If[PrimeQ@#, Sow@#] &,
   Range[10]
   ]][[2]]

and the output is, as expected

{{2, 3, 5, 7}}

But when I do

Reap[Parallelize@Map[
    If[PrimeQ@#, Sow@#] &,
    Range[1000]
    ]][[2]]

I got the output

{}

And the same was the output when I ran

Reap[
  ParallelMap[
   If[PrimeQ@#, Sow@#] &,
   Range[10]
   ]
  ][[2]]

2 questions :

1) How is this explainable ?

2) If this is the case, how do I efficiently I spawn parallel jobs which accumulate something ?

share|improve this question

marked as duplicate by Szabolcs, Leonid Shifrin, Artes, Michael E2, rm -rf Oct 4 '13 at 1:05

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

2 Answers 2

I guess this will work! Check documentation for SetSharedFunction but be careful of the parallel overhead of following implementation.

SetSharedFunction[PSow];
PSow[x_] := Sow[x];
Reap[Parallelize@Map[If[PrimeQ@#, PSow@#] &, Range[10]]][[2]]

{{2, 5, 3, 7}}

share|improve this answer

I would recommend parallelizing these types of calculations using the ParallelCombine paradigm.

First, let's make a function out of your example:

fun = Function[list, Reap[Map[If[PrimeQ@#, Sow@#] &, list]][[2, 1]]]

The non-parallel version will be fun[Range[200]]. Parallelize it using

ParallelCombine[
 fun,
 Range[200],
 Join
]

I spelt out Join here because in other applications you may need a different function there. Here it can be omitted.


This is clearly more work than the (also valid) SetSharedFunction approach, but it is going to avoid some overhead. SetSharedFunction simply forces the function to be evaluated on the main kernel. If the equivalent of PrimeQ for your real calculation gives True for a sizeable fraction of the values, the SetSharedFunction-based parallelization won't buy much performance, or might even be slower than a serial computation.

share|improve this answer
    
Thanks for sueggesting ParallelCombine. An interesting find! –  my account_ram Oct 3 '13 at 21:53
    
But how is this behavior of Reap@Map@Sow giving {} explainable? –  my account_ram Oct 3 '13 at 21:54
    
@myaccount_ram You were running Sow in a subkernel and Reap in the main kernel, so it can't work. These are separate processes that do not share any memory. The main kernel merely submits evaluations to the subkernels and collects results. –  Szabolcs Oct 3 '13 at 21:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.