I will use a very simple example for demonstrating two issues I face with ParallelTable, one with respect to the speed and one with respect to the use of memory. These issues might be related.
Let us start with the speed.
args=RandomReal[1, {10^7}];
Table[x, {x, args}]; // Timing
(* {0.312002,Null} *)
ParallelTable[x, {x, args}]; // Timing
(* {8.40845,Null} *)
This is surprising. I used Timing instead of AbsoluteTiming, so the result shows that the master kernel needs 8 seconds for handling the results of my 4 subkernels. What is it doing all the time?
This question has been asked before, see here. There is a reference to a great answer of @OleksandrR (see here) to another question, in which he showed that the bottleneck is a call to MemberQ with second argument $Aborted, which in turn forces Mathematica to do a lot of unpacking packed arrays.
We can test this in our example by adding a definition to MemberQ that never actually will be used, it only prints a message when MemberQ is called. One should never modify built in functions, but this seems to be pretty safe.
On["Packing"]; On[General::stop];
Unprotect[MemberQ];Clear[MemberQ];
MemberQ[_, $Aborted] /; (Print["MemberQ[arg, $Aborted] called."]; False) := Null;
ParallelTable[x, {x, args}];
Unprotect[MemberQ]; Clear[MemberQ];
(*
Developer`FromPackedArray::punpack1: Unpacking array with dimensions {544}. >>
Developer`FromPackedArray::punpackl1: Unpacking array with dimensions {544,2} to level 1. >>
Developer`FromPackedArray::punpack1: Unpacking array with dimensions {2}. >>
Developer`FromPackedArray::punpack1: Unpacking array with dimensions {2}. >>
General::stop: Further output of Developer`FromPackedArray::punpack1 will be suppressed during this calculation. >>
MemberQ[arg, $Aborted] called.
*)
When I run this command with Off[General::stop], I see about 650 unpacking array messages. So it might very well be that all this unpacking is responsible for the 8 seconds master kernel time.
However, what we see here is not quite in accordance with OleksandrR's statement. All the unpacking is done before MemberQ[_, $Aborted]
is called. That explains why the following dirty trick: simply redefine MemberQ[_, $Aborted]
as False, so that no unpacking would be needed, does not work.
On["Packing"]; On[General::stop];
Unprotect[MemberQ];Clear[MemberQ];
MemberQ[_, $Aborted] = False;
ParallelTable[x, {x, args}];
Unprotect[MemberQ]; Clear[MemberQ];
(* Developer`FromPackedArray::punpack1: Unpacking array with dimensions {544}. >>
...
*)
For the same reason, @Szabolcs solution with a locally redefined function MemberQ does not work either.
In my example, there is a lot of unpacking, but not because of a call to MemberQ. Unfortunately, Mathematica does not tell me which function is initiating the unpacking.
There is another issue with this example, maybe related. Start with a fresh kernel and obeserve the amount of used memory.
Dynamic[MemoryInUse[], UpdateInterval->1]
args=RandomReal[1, {10^7}];
On my computer, the amount of memory increases from 22 MB to 103 MB, about as expected.
Table[x, {x, args}];
No change in the amount of used memory, as expected.
ParallelTable[x, {x, args}];
During the computation, the amount of used memory goes up to at least 584 MB, and ends with 304 MB. So this computation costs permanently 200 MB of memory. Each time when we execute the ParallelTable command we loose 200 MB of memory. That looks like a bug to me; just as with Table I would have expected no change in the amount of memory in use.
Also the following command gives an increase of 200 MB of used memory:
ParallelTable[x, {x, args}, Method->"CoarsestGrained"];
And with the following command, the kernel silently crashes:
ParallelTable[x, {x, args}, Method->"FinestGrained"];
I am a little bit surprised that such an elementary function as ParallelTable shows such strange behaviour. Any comment is highly welcomed.
ParallelTable
. It's not 100% clear to me that it is for the same reasons as the OP but I've done some spelunking ala @Mr.Wizard and see lots ofPackedArray
action. Doesn't seem a likely problem here because the slowdown happens even with the smallest problem. Here's a link to the question $\endgroup$