# Parallel calculation is 5 times slower than the non-parallel one

I have used code from Tell ParallelMap[] to use just specific kernels :

chunkenize[data_, nkernels_] :=
Partition[data, Quotient[Length[data], nkernels]]

MyParallelMap[f_, data_, kernels_] :=
Module[{chunks = chunkenize[data, Length[kernels]]},
Block[{subdata},
MapIndexed[ParallelEvaluate[subdata = #1, kernels[[First[#2]]]] &,
chunks];
DistributeDefinitions[f];
ParallelEvaluate[Map[f, subdata], kernels]]]


with

m = 5 10^3;
f[y_] := (3 y^3)/Sqrt[y^2 + 1] // N[#, 10^5] &
kernels = LaunchKernels[]


to get

{"KernelObject"[1, "local"], "KernelObject"[2, "local"],
"KernelObject"[3, "local"], "KernelObject"[4, "local"],
"KernelObject"[5, "local"], "KernelObject"[6, "local"]}

Then

ClearSystemCache[];
MyParallelMap[f, Range[m], kernels]; // AbsoluteTiming


yields

{100.934, Null}

whereas the non-parallel version

ClearSystemCache[];
Range[m] // f; // AbsoluteTiming


is almost 5 times faster:

{21.9527, Null}

If I do it with N[#, 10^4] & instead of N[#, 10^5] &, then the parallel calculation is "only" about 2 times slower. If I do it for N[Log[#], 10^4] & instead of f[], the parallel calculation is about 2--3 times faster -- something to be expected, I think. What could make f[] so drastically different from Log[]?

More importantly, why would the parallel calculation be so much slower than the non-parallel one in any case? Any fix to this? Thank you.

• @MichaelE2 : I think I know this, but how is it relevant to my question? In this case each of the 6 kernels is supposed to work just on one of the 6 chunks into which the data is partitioned, which could give an up to 6-fold improvement in the execution time. In the case of Log[], I get a 2- or 3-fold improvement, but for f[] I get it 5 times worse with the parallelization. Feb 1, 2017 at 2:29
• Sorry, I think I misread some of your code. On my machine, each chunk is processed sequentially, one subkernel running at a time. If that's any help. Feb 1, 2017 at 3:15
• @MichaelE2 : Thank you for your comment. On my computer, I do seem to get a 2- or 3-fold gain by the parallelization -- but only for Log[] and not for the f[]. In any case, could you please tell me how I can see if on my computer the chunks are processed sequentially or in parallel? Feb 1, 2017 at 3:43
• I'm on a Mac and use Activity Monitor. I believe there's something similar on Windows, but I don't know Windows. There's a built-in tool: See the menu Evaluation > Parallel Kernel Status... It can show a kernel as "Busy" when it's just waiting, though. -- I just ran Log and it took ages (175 sec. with all kernels running, vs 45 sec. another time with kernels running in sequence). I don't understand the Log result at all. There are a few others on this site who understand parallel processing much better than I do. I hope they'll see your question and be able to answer it. Feb 1, 2017 at 3:53
• Thank you Michael for your advice. I have now used the Windows Task Manager Performance tab. For N[Log[#], 10^4] &, it showed all 12 threads running, with about 50% CPU usage, and about 3 sec execution time. For f[], only one of the 12 threads seemed substantially engaged, with no more than 12% CPU usage and execution time about 100 sec. Feb 1, 2017 at 5:03

First of all I would recommend to read this answer which explains the most common pitfalls when parallelizing code in Mathematica.

I think there is nothing wrong with your code and it does what you expect it to do. There are two problems which make it perform so bad:

Your problem is not really well suited for efficient parallel execution: you are doing a relatively cheap operation and produce a relatively large result. This will cause a relatively large overhead when the results are sent back to the master kernel.

You can see that this is true by only returning a summary of your results, e.g. the Total or ByteCount of your results. That should show that the parallel execution really gives you some speedup:

MyParallelMap[f_, data_, kernels_] := Module[{
chunks = chunkenize[data, Length[kernels]]
},
Block[{subdata},
MapIndexed[ParallelEvaluate[subdata = #1, kernels[[First[#2]]]] &,
chunks];
DistributeDefinitions[f];
ParallelEvaluate[Map[f, subdata] // ByteCount, kernels]
]
]


and then compare:

MyParallelMap[f, Range[m], kernels] // ByteCount // AbsoluteTiming


to :

Map[f, Range[m]] // ByteCount // AbsoluteTiming


# Mathematica specific inefficiencies

On the other hand I think your code suffers more than necessary from such communication overhead between kernels, it is not clear why it takes several seconds to copy a few megabytes from one kernel to the other. To some extend that is probably just a consequence of the high level language, but there are some cases known where Mathematica adds unnecessary inefficiencies which one could consider bugs, for references see the above link.

I think that you are suffering from one of those cases where the (unavoidable) communication overhead alone doesn't really seem to explain the differences in runtime. Unfortunately I don't know the reason or any cure for that, and as you can see from the mentioned answer it is nontrivial to identify and solve such problems.

As a final note I just wanted to mention that your code will do exactly what

 ParallelMap[f, Range[m], Method -> "CoarsestGrained"]


would do but unfortunately that has the same efficiency issues as your code.

• Thank you very much for the detailed and thoughtful answer. I did learn from it. I think now I can make a guess as to why Log seemed to behave differently: apparently because Log is harder to compute than f with the same precision, and so, the communication overhead is percentage-wise less for Log. Do you think this guess is reasonable? Feb 1, 2017 at 18:40
• @IosifPinelis: Could be an explanation, and of course the result will be somewhat smaller as well as you only compute 10^4 digits and not 10^5. But honestly my own experience with such high precision floating point numbers is limited. Such computations are completely done in software and are much slower than hardware precision and at that high precision probably not that well tested and optimized. I would even suspect that the bad parallel performance could somehow be related to these high precision numbers. Just out of curiosity: what are you doing that needs that many digits computed? Feb 1, 2017 at 20:32
• Comparisons in arxiv.org/abs/1511.03247 of a new summation formula with the Euler--Maclaurin (EM) one show that the new one will usually perform better. An additional potential advantage of the new formula is that the computation of the integrals/antiderivatives can be easily parallelized/vectorized, in contrast with that of the consecutive derivatives. To an extent, this contrast holds also for the coefficients in the new formula, vs. the Bernoulli numbers in EM. However, parallelization is useful only for large enough execution times; hence, high precision. Feb 2, 2017 at 1:20
• @IosifPinelis: that sounds interesting but is out of my scope. I think if you want to see speedup from parallelization in Mathematica you will need to do some pre-reduction of your results on the parallel kernels and only return a condensed result back to the master. I don't know whether that is possible for your application, but if e.g. you need to sum all these coefficients, each kernel could return a partial sum and the master would only need to sum the Length[kernel] results. While with other platforms it might be possible to see speedup without that, it will of course also help there... Feb 2, 2017 at 8:32
• Thank you Albert for your latest comment as well. Indeed, the main lesson that I took from your answer was to condense/reduce as much as possible the results of the computation in each subkernel, before transferring the results to the master -- to minimize the volume of transfer. Such reduction certainly seems possible in the situations of interest to me. Before your answer, I did not appreciate how much time it would take to transfer a large amount of data. Feb 2, 2017 at 14:21