Monitor ParallelMap (again) - how to predict the number of function calls efficiently?

Question

I am currently implementing a function to let me monitor the progress of a computation running in parallel (similar to the answers in Monitor doesn't work with ParallelTable). The goal is to have a function that can be wrapped around any parallel computation, to show the progress of it (e.g.: parallelProgress[ParallelTable[...]]). Since something like that is nigh to impossible to implement for "any parallel computation", I settled for now to implementing it for the four big ones: ParallelTable, ParallelMap, ParallelDo, ParallelSum. After figuring out the basics, the implementation for ParallelTable, ParallelDo, ParallelSum were rather straightforward (and almost identical). ParallelMap however is giving me some problems, mainly due to the levelspec parameter. The problem is to figure out the total number of function calls automatically and quick. For example in

ParallelMap[f,Evaluate[Range[10]]]

the number of times f is called is easily calculated using Length. For levelspec different from {1} you can calculate it using Dimensions, as long as you have a rectangular array. However, in

ParallelMap[f,Evaluate[Range[Range[10]]],{2}]

the answer is not so straightforward. Here is what i did so far:

SetAttributes[parallelProgress, {HoldAll}]
parallelProgress[expr_] :=
 ...
 Module[{...,max,...},
  ...
  Count[holdexpr,
    HoldPattern[
      ParallelMap[
        Alternatives[Function[par_, body_, attr_: {}], Function[body_],f_Symbol], mapExpr_, levelSpec_: {1}, 
        OptionsPattern[ParallelMap]
    ] /; (
      dim = Dimensions[mapExpr]; 
      If[MatchQ[levelSpec, n_Integer /; (n >= 0 && n <= Length[dim])], 
        max += Times @@ Flatten[Table[dim[[i]], {i, 1, levelSpec}]], 
      If[MatchQ[levelSpec, {n_Integer} /; (n >= 0 && n <= Length[dim])], 
        max += Times @@ dim[[levelSpec[[1]]]], 
      If[MatchQ[levelSpec, {m_Integer,n_Integer} /; (n >= 0 && m >= 0 && m <= n && n <= Length[dim])], 
        max += Times @@ dim[[levelSpec[[1]] ;; levelSpec[[2]]]], 
      Map[max++&, mapExpr, levelSpec]]]]; True
    )],
    \[Infinity]];
 ...
 ]

I use patternmatching so that cases like parallelProgress[{ParallelMap[...],ParallelMap[...]}] can also be handled. For all ParallelMap instances found I first try to calculate the number of calls via Dimensions. If that is not possible, I resort to Map[max++, mapExpr, levelSpec], which in my tests worked actually better than Length[Level[mapExpr, levelSpec]]. However, this strikes me as being unnecessarily verbose. Is there a way to compute the number of function calls more easily and/or faster?

Edit:

Fixed a typo in the code.

Regarding the use of Length[Level[...]]: I compared this method with my simple Map[n++&,...]:

Array[List, {100, 20, 1000}];
{2};
AbsoluteTiming[Module[{n = 0}, Map[n++ &, %%, %]; n]]
AbsoluteTiming[Length[Level[%%%, %%]]]
%[[1]]/%%[[1]]

With a variety of arrays and level specifications. While i didn't do a comprehensive study, my observations were as follows:

• With deeper levels Level is slightly faster (the lowest ratio I ever got from the code above was ~0.6)
• With shallow levels and large lists Map is much faster (the highest ratio I ever got from the code above was ~100 !!)

but most importantly:

• Level seemed to crash (run out of memory, I guess) for smaller Lists, compared to Map.

All in all this leads me to prefer Map[n++ &,...] over Length[Level[...]] even though the latter is more concise (which with Mathematica usually means faster).

Answer 1

I not certain why you seem not to like Length@Level. With this and Monitor you can construct a monitorParallelMap that appears to achieve your goal.

ClearAll[monitorParallelMap];
monitorParallelMap[foo_, expr_, levelSpec_: {1}, 
  opts : OptionsPattern[ParallelMap]] :=
 Module[{res, exprCount, progress = 0},
  LaunchKernels[];
  SetSharedVariable[progress];
  exprCount = Length@Level[expr, levelSpec];
  res =
   Monitor[
    ParallelMap[(Pause[.25]; progress++; foo[#]) &, expr, levelSpec, opts],
    Column[{ToString@progress <> " of " <> ToString@exprCount, 
      ProgressIndicator[progress, {0, exprCount}]}, 
     Alignment -> Center]
    ];
  UnsetShared [progress];
  res
  ]

monitorParallelMap gets the number of calls with Lenght@Level and uses a SetSharedVariable across the kernels to update the progress. Pause[.25]; was added for the demonstration to slow things down enough so we can see what is happening. Monitor displays the progress as the ParallelMap progresses.

Then

monitorParallelMap[f, Range[Range[10]], {2}]

displays an updating ProgressIndicator as it evaluates

and then the results of the ParallelMap.

{{f[1]}, 
 {f[1], f[2]}, 
 {f[1], f[2], f[3]}, 
 {f[1], f[2], f[3], f[4]}, 
 {f[1], f[2], f[3], f[4], f[5]}, 
 {f[1], f[2], f[3], f[4], f[5], f[6]}, 
 {f[1], f[2], f[3], f[4], f[5], f[6], f[7]}, 
 {f[1], f[2], f[3], f[4], f[5], f[6], f[7], f[8]}, 
 {f[1], f[2], f[3], f[4], f[5], f[6], f[7], f[8], f[9]}, 
 {f[1], f[2], f[3], f[4], f[5], f[6], f[7], f[8], f[9], f[10]}}

For heavier parallel processing with monitoring you may find this answer (19542) interesting.

Hope this helps.