Optimization: Differences between Map and ParallelMap

I just was studying Map and ParallelMap in a MacBook Air of 2 kernels. So, I tried comparing the AbosoluteTiming of these input lines and the outputs sorprised me because I thought a ParallelMap would run faster than Map. Also, the Method -> "CoarsestGrained" aborted.

On["Packing"];
Needs["Developer"];

data = RandomReal[1, 5 10^6];

In[202]:= Map[#^2 &, data]; // AbsoluteTiming
Out[202]= {0.730642, Null}

In[204]:= ParallelMap[#^2 &, data]; // AbsoluteTiming
Out[204]= {9.1048, Null}

In[205]=  ParallelMap[#^2 &, data, Method -> "CoarsestGrained"]; // AbsoluteTiming
During evaluation of In[205]:= \$Aborted[]
During evaluation of In[205]:= FromPackedArray::unpack: Unpacking array in call toHoldForm.
Out[205]= {7.14583, Null}

In[207]:= ParallelMap[#^2 &, data, Method -> "EvaluationsPerKernel" -> 2]; // AbsoluteTiming
Out[207]= {4.5805, Null}


So, I have various questions: Why is faster Map than ParrallelMap?, Why did the method "CoarsestGrained" abort?, Why is the method of "FinestGrained" so slow (I didn't show its out because I aborted it? and What is the meaning of the number 2 in the method of "EvaluationsPerKernel"?

• Not entirely a duplicate, but related: (144056). Mar 19, 2020 at 18:52

"Why is faster Map than ParrallelMap?"

Communication overhead between the kernels (see below). Also not all task are equally amenable to parallelization.

"Why is the method of "FinestGrained" so slow?"

Because communication overhead is maximal in this case (see below).

"What is the meaning of the number 2 in the method of "EvaluationsPerKernel"?"

"EvaluationsPerKernel" -> k means that tasks are partitioned in a way that each kernel is supplied with at most k chunks of jobs. You can find a very brief description of the Method option in the "Options" section of doc page of ParallelMap.

When a kernel has finished its chunk, it requests a new one (from a master kernel). It must wait until the data is exchanged. This is the communication overhead. The smaller "EvaluationsPerKernel", the fewer rounds of this process are needed. If I had to guess, I'd say that "CoarsestGrained" is equivalent to "EvaluationsPerKernel"->1 and that "FinestGrained" should have chunk size 1 ("ItemsPerEvaluation" -> 1) so that "EvaluationsPerKernel" is maximal. The latter is why it is so inefficient when all tasks are roughly of the same size. (Still, "FinestGrained" can be a good choice when the runtime of the jobs varies greatly.)

Why did the method "CoarsestGrained" abort?

That's indeed odd and I have no explanation for that. It seems to be related to the message handling, because it works fine when I rund verything with Off["Packing"];. (In general, error handling is super slow.)

My suggestion for this piece of code:

Do data^2. This employs vectorization and is an order of magnitude faster.

• thanks for the answer, I already understood the difference between "CoarsestGrained" and "FinestGrained". However, it is still no clear to me the difference between "EvaluationPerKernel" and "ItemsPerEvaluation". I tried with "EvaluationsPerKernel"->1 and "EvaluationsPerKernel"-> 2 and it gave me a runtime similar to the "CorasestGrained" but with "ItemsPerEvaluation" -> 1 it took so much time like "FinestGrained".
• I tried to explain that this is because "FinestGrained" is just "ItemsPerEvaluation" -> 1 and "CoarsestGrained" is "EvaluationsPerKernel"->1... "ItemsPerEvaluation" is the number of items per chunk. "EvaluationsPerKernel" is the number of chunks (per kernel). Mar 19, 2020 at 20:59
• Well, computations are good to parallelization if they can be made independent of each other and if not too much data has to be moved. The first condition is satisfied by your example; but the seconds is not really satisfied: You want one flop perp double precision number. And memory is much, much slower than the CPU's number crunching capabilities. In such cases, it is very important that data is streamed appropriately into the processor. And Mathematica's Parallel-commands are a bit too coarse for doing that. Mar 19, 2020 at 21:30