Build recursion in parallel?

Question

Problem

Let's define a simple recursion.

f[1] = 1;
f[n_] := f[n] = f[n - 1]*n;

If I evaluate f in parallel

ParallelTable[f[i], {i, 10}];

and then query f

?f
Global`f
f[1]=1
f[n_]:=f[n]=f[n-1] n

Question

How can I tell the sub kernels to build up the recursion, so that

?f

Global`f

f[1]=1
....
f[10]=3628800

f[n_]:=f[n]=f[n-1] n

Thanks !

PS: of course the actual function is not that trivial.

On the other hand it does not re-export to the different kernels the known definitions. Table[f[i], {i, 5000}]; // Timing (* {0.005126,Null} ) ParallelTable[f[i], {i, 5000}]; // Timing ( {6.69067,Null} *) — chris, Commented Aug 7, 2012 at 17:22
Indeed. I can remember that I once ran into that problem but I couldn't find a solution to distribute DownValues from the subkernels. Maybe you can formulate your question more generally (with a different title?), so you get more attention! — sebhofer, Commented Aug 7, 2012 at 18:05

chris · Accepted Answer · 2012-08-07 20:25:51Z

1

@sebhofer provides a partial answer via the command

SetSharedFunction[f]

which collects to the master kernel the accumulated downvalues. On the other hand, the slave kernels seem unaware of these.

Indeed

ParallelTable[f[i], {i, 5000}]; 
Table[f[i], {i, 5000}]; // Timing

(* {0.005126,Null} *)

which suggests the master kernel does remember the previous definitions. On the other hand,

ParallelTable[f[i], {i, 5000}]; // Timing

(* {6.69067,Null} *)

still takes quite some time.

answered Aug 7, 2012 at 20:25

chris

23.1k5 gold badges62 silver badges151 bronze badges

$\begingroup$ The reason the second invocation takes a while has changed, though--f exists only in the master kernel and is fetched using a callback every time it's referenced in one of the slaves. This is still basically a serial operation, but now it involves a lot of communication overhead. Unfortunately it isn't easy to do what you want given the way the Parallel` package works. I thought you could perhaps try something like f[n_]:=g[n]=f[n-1] n with SetSharedFunction[g]; DistributeDefinitions[f], but this doesn't seem to work... sorry, no time for a more considered response! $\endgroup$
– Oleksandr R.
Commented Aug 8, 2012 at 8:20

Add a comment |

1 Answer 1