# Nesting Parallel processes

I just attempted to run code that had nested ParallelMap[] functions. It generates the error message:

ParallelMap::subpar: Parallel computations cannot be nested; proceeding with sequential evaluation.

I can represent a simplified version of the code's structure with the following:

f[list_] := ParallelMap[# + 0.1 &, list]
dataLists = RandomReal[{0, 1}, {5, 100, 3}];
Dimensions[dataLists]
ParallelMap[f[#] &, dataLists];


When ParallelMap calls f[datLists] I have extra kernels available for use, but it appears that whatever manages the distribution of processing jobs can't reach them.

I can appreciate that nesting parallel computations presents some complications, but not having the ability to nest them doesn't seem very Mathematica like.

As my code has many functions at the same (2nd) level in the hierarchy as f[list_], this constrains my possible solutions quite a bit and makes optimizing performance by using parallel computation time consuming. I think I can either parallelize at the top level or parallelize all the functions at the second level, but obviously not both.

Also, this 2nd level of functions tend to greater complexity than the simple example above. They look more like:

If[#[[1]] == 0, 0, 1 - NProbability[x < #[[1]], x \[Distributed] #[[2]]]] &,
Transpose[{listOfReals, myDistributions}]


Just speculative brainstorming, but maybe (probably) I haven't thought of all the possible ways to attack a problem like this.

• Rather than mapping or parallel mapping, could I recast this problem in other ways better suited to the single available level of parallelization?
• Could one flatten the computations then cull out the answers afterwards?
• Could one turn this into a matrix operation of some kind?

As CUDA and OpenCL don't look like a ready or easy solution for me (see: CUDA and GPU hardware and compatibility questions) I'd hoped to get everything I could out of Mathematica's parallel powers.

• Could you just Quiet the message and get sequential evaluation automatically at all levels except the outermost? Commented Feb 27, 2012 at 22:31

This limitation exists, and as far as I can see we have to live with it for now. It does indeed make it impractical to use Parallel operations inside functions---the functions won't work in parallel kernels.

I recommend you parallelize at the top level unless you have a specific reason to do it at lower levels. The longer the a "unit calculation" that is run in a subkernel, the less parallelization overhead will matter. This overhead can be significant if you are running the subkernels on different machines from your main kernel (as you seem to be based on your Lightweight Grid question). Could you give some indication of how long it takes for your functions to finish, and how long the list dataLists is? If dataLists is long enough compared to the number of processors you have available, then probably parallelization at the highest level will be most efficient.

Alternatively you could just use Off[ParallelMap::subpar].

• For a ParallelMap equivalent that works both in the master kernels and subkernels without producing messages, one might try: maybeParallelMap := If[SystemParallel\$SubKernel, Map, ParallelMap]. Multiple levels of parallelization aren't easily handled within the framework of the Parallel  package, not least since subkernels can't (officially) communicate with one another except via the master kernel. Commented Feb 27, 2012 at 22:42
• @OleksandrR. You raise an interesting point. In my situation my processes run discretely at both the top and secondary levels. By this I mean the top level parallelizations have no passing of information between them. At the 2nd level, again no passing of information between any set of process running within a top level process. Certainly each of the discrete top level process will receive and aggregate information returned from sub level process, but this really just gives us the most elementary way of parallel processing. Commented Feb 28, 2012 at 1:22
• I have an extensive program. 30+ custom functions & +300 lines of functional code. 3 top level data sets of {12000, 2} dimensions. Code generates a subset of permutations of each. Lots of processing - up to 90 min. to run on 2 core iMac. 4 types of calculations take the most time: Fitting parameters for custom distributions; NExpectation[] at ea. time step (using the above distributions); NSolve[]to find the intersection of pairs of PDFs; & NProbability[]. All these map over lengthy lists, I might get the most performance boost from parallelizing these at level 2 rather then just level 1. Commented Feb 28, 2012 at 1:41
• @Jagra if you have a lot of independent computations and simply want better control over the parallelization granularity then I would suggest trying something with ParallelSubmit. This can be wrapped around any part of the overall computation at any level. Commented Feb 28, 2012 at 1:55
• On a completely different script, I am also facing the error that @Jagra posted. Interestingly, when I use the exact same code with Map instead of ParallelMap it takes ages to finish the calculations (some 12 hours splitted among 4 computers for Map vs <1hour in one of those computers using ParallelMap). @Szabolcs -- will there be any reason why using Off[ParallelMap::subpar]` would still allow the warnings to show up? It's happening here
– Sos
Commented Apr 22, 2015 at 11:48