Nesting Parallel processes

I just attempted to run code that had nested ParallelMap[] functions. It generates the error message:

ParallelMap::subpar: Parallel computations cannot be nested; proceeding with sequential evaluation.

I can represent a simplified version of the code's structure with the following:

f[list_] := ParallelMap[# + 0.1 &, list]
dataLists = RandomReal[{0, 1}, {5, 100, 3}];
Dimensions[dataLists]
ParallelMap[f[#] &, dataLists];


When ParallelMap calls f[datLists] I have extra kernels available for use, but it appears that whatever manages the distribution of processing jobs can't reach them.

I can appreciate that nesting parallel computations presents some complications, but not having the ability to nest them doesn't seem very Mathematica like.

As my code has many functions at the same (2nd) level in the hierarchy as f[list_], this constrains my possible solutions quite a bit and makes optimizing performance by using parallel computation time consuming. I think I can either parallelize at the top level or parallelize all the functions at the second level, but obviously not both.

Also, this 2nd level of functions tend to greater complexity than the simple example above. They look more like:

If[#[[1]] == 0, 0, 1 - NProbability[x < #[[1]], x \[Distributed] #[[2]]]] &,
Transpose[{listOfReals, myDistributions}]


Just speculative brainstorming, but maybe (probably) I haven't thought of all the possible ways to attack a problem like this.

• Rather than mapping or parallel mapping, could I recast this problem in other ways better suited to the single available level of parallelization?
• Could one flatten the computations then cull out the answers afterwards?
• Could one turn this into a matrix operation of some kind?

As CUDA and OpenCL don't look like a ready or easy solution for me (see: CUDA and GPU hardware and compatibility questions) I'd hoped to get everything I could out of Mathematica's parallel powers.

-
Could you just Quiet the message and get sequential evaluation automatically at all levels except the outermost? – Andrew Moylan Feb 27 '12 at 22:31

This limitation exists, and as far as I can see we have to live with it for now. It does indeed make it impractical to use Parallel operations inside functions---the functions won't work in parallel kernels.
I recommend you parallelize at the top level unless you have a specific reason to do it at lower levels. The longer the a "unit calculation" that is run in a subkernel, the less parallelization overhead will matter. This overhead can be significant if you are running the subkernels on different machines from your main kernel (as you seem to be based on your Lightweight Grid question). Could you give some indication of how long it takes for your functions to finish, and how long the list dataLists is? If dataLists is long enough compared to the number of processors you have available, then probably parallelization at the highest level will be most efficient.
Alternatively you could just use Off[ParallelMap::subpar].
For a ParallelMap equivalent that works both in the master kernels and subkernels without producing messages, one might try: maybeParallelMap := If[SystemParallel\$SubKernel, Map, ParallelMap]. Multiple levels of parallelization aren't easily handled within the framework of the Parallel  package, not least since subkernels can't (officially) communicate with one another except via the master kernel. – Oleksandr R. Feb 27 '12 at 22:42
@Jagra if you have a lot of independent computations and simply want better control over the parallelization granularity then I would suggest trying something with ParallelSubmit`. This can be wrapped around any part of the overall computation at any level. – Oleksandr R. Feb 28 '12 at 1:55