22
$\begingroup$

I will use a very simple example for demonstrating two issues I face with ParallelTable, one with respect to the speed and one with respect to the use of memory. These issues might be related.

Let us start with the speed.

args=RandomReal[1, {10^7}];

Table[x, {x, args}]; // Timing
(* {0.312002,Null} *)

ParallelTable[x, {x, args}]; // Timing
(* {8.40845,Null} *)

This is surprising. I used Timing instead of AbsoluteTiming, so the result shows that the master kernel needs 8 seconds for handling the results of my 4 subkernels. What is it doing all the time?

This question has been asked before, see here. There is a reference to a great answer of @OleksandrR (see here) to another question, in which he showed that the bottleneck is a call to MemberQ with second argument $Aborted, which in turn forces Mathematica to do a lot of unpacking packed arrays.

We can test this in our example by adding a definition to MemberQ that never actually will be used, it only prints a message when MemberQ is called. One should never modify built in functions, but this seems to be pretty safe.

On["Packing"]; On[General::stop];
Unprotect[MemberQ];Clear[MemberQ];
MemberQ[_, $Aborted] /; (Print["MemberQ[arg, $Aborted] called."]; False) := Null;
ParallelTable[x, {x, args}]; 
Unprotect[MemberQ]; Clear[MemberQ]; 

(*
Developer`FromPackedArray::punpack1: Unpacking array with dimensions {544}. >>
Developer`FromPackedArray::punpackl1: Unpacking array with dimensions {544,2} to level 1. >>
Developer`FromPackedArray::punpack1: Unpacking array with dimensions {2}. >>
Developer`FromPackedArray::punpack1: Unpacking array with dimensions {2}. >>
General::stop: Further output of Developer`FromPackedArray::punpack1 will be suppressed during this calculation. >>
MemberQ[arg, $Aborted] called.
*)

When I run this command with Off[General::stop], I see about 650 unpacking array messages. So it might very well be that all this unpacking is responsible for the 8 seconds master kernel time.

However, what we see here is not quite in accordance with OleksandrR's statement. All the unpacking is done before MemberQ[_, $Aborted] is called. That explains why the following dirty trick: simply redefine MemberQ[_, $Aborted] as False, so that no unpacking would be needed, does not work.

On["Packing"]; On[General::stop];
Unprotect[MemberQ];Clear[MemberQ];
MemberQ[_, $Aborted] = False;
ParallelTable[x, {x, args}]; 
Unprotect[MemberQ]; Clear[MemberQ]; 

(* Developer`FromPackedArray::punpack1: Unpacking array with dimensions {544}. >>
...
*)

For the same reason, @Szabolcs solution with a locally redefined function MemberQ does not work either.

In my example, there is a lot of unpacking, but not because of a call to MemberQ. Unfortunately, Mathematica does not tell me which function is initiating the unpacking.

There is another issue with this example, maybe related. Start with a fresh kernel and obeserve the amount of used memory.

Dynamic[MemoryInUse[], UpdateInterval->1]

args=RandomReal[1, {10^7}];

On my computer, the amount of memory increases from 22 MB to 103 MB, about as expected.

Table[x, {x, args}];

No change in the amount of used memory, as expected.

ParallelTable[x, {x, args}];

During the computation, the amount of used memory goes up to at least 584 MB, and ends with 304 MB. So this computation costs permanently 200 MB of memory. Each time when we execute the ParallelTable command we loose 200 MB of memory. That looks like a bug to me; just as with Table I would have expected no change in the amount of memory in use.

Also the following command gives an increase of 200 MB of used memory:

ParallelTable[x, {x, args}, Method->"CoarsestGrained"];

And with the following command, the kernel silently crashes:

ParallelTable[x, {x, args}, Method->"FinestGrained"];

I am a little bit surprised that such an elementary function as ParallelTable shows such strange behaviour. Any comment is highly welcomed.

$\endgroup$
  • 1
    $\begingroup$ In a related bit of anguish, I too am experiencing speed issues with ParallelTable. It's not 100% clear to me that it is for the same reasons as the OP but I've done some spelunking ala @Mr.Wizard and see lots of PackedArray action. Doesn't seem a likely problem here because the slowdown happens even with the smallest problem. Here's a link to the question $\endgroup$ – flip Jan 13 '16 at 0:46
5
$\begingroup$

I do not have time to explore this deeply, but a quick Trace of the issue indicates that the functionality simply is not written to preserve packing. Observe:

args = RandomReal[1, {10000}];

On["Packing"];

tr = Trace @ ParallelTable[x, {x, args}, Method -> "ItemsPerEvaluation" -> 100];

FirstPosition[tr, Developer`FromPackedArray::punpack1]
{5, 22, 5, 5, 1, 1, 5, 3, 1, 1, 1}

That gives us the position of the first unpacking message in the trace. Stepping back a bit gives:

tr[[5, 22, 5, 5, 1, 1, 5, 2]] // Short
Take[Parallel`Combine`Private`ha[0.363938,<<9998>>,0.75942],{1,100}]

Which shows that at some point the packed List of reals has been converted to an expression with the head Parallel`Combine`Private`ha A bit of spelunking reveals:

(* handling of evaluated/unevaluated arguments:
    normal case: eval on master, do not eval on subkernel
    Unevaluated[args]: neither on master nor subkernel
*)

SetAttributes[ha, HoldAllComplete]

parallelCombine[func_, h_[exprs___], comb_, o:OptionsPattern[]] :=
With[{cmds = ha[exprs], nk=$KernelCount}, With[{items = Length[cmds]}, (* omitted *)

So ha is a holding head intended to allow Unevaluated expressions. The destructuring replacement h_[exprs___] :> ha[exprs] would unpack a packed array even if ha were List.

I think that this function will need to be rewritten to check for a packed array and bypass this operation if one is found. I'll try to implement when I have time. However I suspect that other parts of this functionality will also unpack therefore that alone may be insufficient.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.