Parallel calculation is 5 times slower than the non-parallel one

Question

I have used code from Tell ParallelMap[] to use just specific kernels :

chunkenize[data_, nkernels_] := 
 Partition[data, Quotient[Length[data], nkernels]]

MyParallelMap[f_, data_, kernels_] := 
 Module[{chunks = chunkenize[data, Length[kernels]]}, 
  Block[{subdata}, 
   MapIndexed[ParallelEvaluate[subdata = #1, kernels[[First[#2]]]] &, 
    chunks];
   DistributeDefinitions[f];
   ParallelEvaluate[Map[f, subdata], kernels]]]

with

m = 5 10^3;
f[y_] := (3 y^3)/Sqrt[y^2 + 1] // N[#, 10^5] &
kernels = LaunchKernels[]

to get

{"KernelObject"[1, "local"], "KernelObject"[2, "local"],
"KernelObject"[3, "local"], "KernelObject"[4, "local"],
"KernelObject"[5, "local"], "KernelObject"[6, "local"]}

Then

ClearSystemCache[];
MyParallelMap[f, Range[m], kernels]; // AbsoluteTiming

yields

{100.934, Null}

whereas the non-parallel version

ClearSystemCache[];
Range[m] // f; // AbsoluteTiming

is almost 5 times faster:

{21.9527, Null}

If I do it with N[#, 10^4] & instead of N[#, 10^5] &, then the parallel calculation is "only" about 2 times slower. If I do it for N[Log[#], 10^4] & instead of f[], the parallel calculation is about 2--3 times faster -- something to be expected, I think. What could make f[] so drastically different from Log[]?

More importantly, why would the parallel calculation be so much slower than the non-parallel one in any case? Any fix to this? Thank you.

Answer 1

First of all I would recommend to read this answer which explains the most common pitfalls when parallelizing code in Mathematica.

I think there is nothing wrong with your code and it does what you expect it to do. There are two problems which make it perform so bad:

Communication overhead in general

Your problem is not really well suited for efficient parallel execution: you are doing a relatively cheap operation and produce a relatively large result. This will cause a relatively large overhead when the results are sent back to the master kernel.

You can see that this is true by only returning a summary of your results, e.g. the Total or ByteCount of your results. That should show that the parallel execution really gives you some speedup:

MyParallelMap[f_, data_, kernels_] := Module[{
    chunks = chunkenize[data, Length[kernels]]
  },
  Block[{subdata},
    MapIndexed[ParallelEvaluate[subdata = #1, kernels[[First[#2]]]] &, 
    chunks];
    DistributeDefinitions[f];
    ParallelEvaluate[Map[f, subdata] // ByteCount, kernels]
  ]
]

and then compare:

MyParallelMap[f, Range[m], kernels] // ByteCount // AbsoluteTiming

to :

Map[f, Range[m]] // ByteCount // AbsoluteTiming

Mathematica specific inefficiencies

On the other hand I think your code suffers more than necessary from such communication overhead between kernels, it is not clear why it takes several seconds to copy a few megabytes from one kernel to the other. To some extend that is probably just a consequence of the high level language, but there are some cases known where Mathematica adds unnecessary inefficiencies which one could consider bugs, for references see the above link.

I think that you are suffering from one of those cases where the (unavoidable) communication overhead alone doesn't really seem to explain the differences in runtime. Unfortunately I don't know the reason or any cure for that, and as you can see from the mentioned answer it is nontrivial to identify and solve such problems.

As a final note I just wanted to mention that your code will do exactly what

 ParallelMap[f, Range[m], Method -> "CoarsestGrained"]

would do but unfortunately that has the same efficiency issues as your code.