4
$\begingroup$

I just was studying Map and ParallelMap in a MacBook Air of 2 kernels. So, I tried comparing the AbosoluteTiming of these input lines and the outputs sorprised me because I thought a ParallelMap would run faster than Map. Also, the Method -> "CoarsestGrained" aborted. 

On["Packing"];
Needs["Developer`"];

data = RandomReal[1, 5 10^6];

In[202]:= Map[#^2 &, data]; // AbsoluteTiming
Out[202]= {0.730642, Null}

In[204]:= ParallelMap[#^2 &, data]; // AbsoluteTiming
Out[204]= {9.1048, Null}

In[205]=  ParallelMap[#^2 &, data, Method -> "CoarsestGrained"]; // AbsoluteTiming
     During evaluation of In[205]:= $Aborted[]
     During evaluation of In[205]:= FromPackedArray::unpack: Unpacking array in call toHoldForm.
Out[205]= {7.14583, Null}

In[207]:= ParallelMap[#^2 &, data, Method -> "EvaluationsPerKernel" -> 2]; // AbsoluteTiming
Out[207]= {4.5805, Null}

So, I have various questions: Why is faster Map than ParrallelMap?, Why did the method "CoarsestGrained" abort?, Why is the method of "FinestGrained" so slow (I didn't show its out because I aborted it? and What is the meaning of the number 2 in the method of "EvaluationsPerKernel"?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Not entirely a duplicate, but related: (144056). $\endgroup$
    – Henrik Schumacher
    Commented Mar 19, 2020 at 18:52

1 Answer 1

Reset to default
9
$\begingroup$

"Why is faster Map than ParrallelMap?"

Communication overhead between the kernels (see below). Also not all task are equally amenable to parallelization.

"Why is the method of "FinestGrained" so slow?"

Because communication overhead is maximal in this case (see below).

"What is the meaning of the number 2 in the method of "EvaluationsPerKernel"?"

"EvaluationsPerKernel" -> k means that tasks are partitioned in a way that each kernel is supplied with at most k chunks of jobs. You can find a very brief description of the Method option in the "Options" section of doc page of ParallelMap.

When a kernel has finished its chunk, it requests a new one (from a master kernel). It must wait until the data is exchanged. This is the communication overhead. The smaller "EvaluationsPerKernel", the fewer rounds of this process are needed. If I had to guess, I'd say that "CoarsestGrained" is equivalent to "EvaluationsPerKernel"->1 and that "FinestGrained" should have chunk size 1 ("ItemsPerEvaluation" -> 1) so that "EvaluationsPerKernel" is maximal. The latter is why it is so inefficient when all tasks are roughly of the same size. (Still, "FinestGrained" can be a good choice when the runtime of the jobs varies greatly.)

Why did the method "CoarsestGrained" abort?

That's indeed odd and I have no explanation for that. It seems to be related to the message handling, because it works fine when I rund verything with Off["Packing"];. (In general, error handling is super slow.)

My suggestion for this piece of code:

Do data^2. This employs vectorization and is an order of magnitude faster.

Improve this answer
$\endgroup$
4
  • $\begingroup$ thanks for the answer, I already understood the difference between "CoarsestGrained" and "FinestGrained". However, it is still no clear to me the difference between "EvaluationPerKernel" and "ItemsPerEvaluation". I tried with "EvaluationsPerKernel"->1 and "EvaluationsPerKernel"-> 2 and it gave me a runtime similar to the "CorasestGrained" but with "ItemsPerEvaluation" -> 1 it took so much time like "FinestGrained". $\endgroup$
    – Ask8
    Commented Mar 19, 2020 at 20:55
  • 2
    $\begingroup$ I tried to explain that this is because "FinestGrained" is just "ItemsPerEvaluation" -> 1 and "CoarsestGrained" is "EvaluationsPerKernel"->1... ``"ItemsPerEvaluation"` is the number of items per chunk. "EvaluationsPerKernel" is the number of chunks (per kernel). $\endgroup$
    – Henrik Schumacher
    Commented Mar 19, 2020 at 20:59
  • $\begingroup$ Ok, I go it!! thaks. I also want to ask: How can I know what kind of taks are amenable to parallelize? $\endgroup$
    – Ask8
    Commented Mar 19, 2020 at 21:19
  • 1
    $\begingroup$ Well, computations are good to parallelization if they can be made independent of each other and if not too much data has to be moved. The first condition is satisfied by your example; but the seconds is not really satisfied: You want one flop perp double precision number. And memory is much, much slower than the CPU's number crunching capabilities. In such cases, it is very important that data is streamed appropriately into the processor. And Mathematica's Parallel-commands are a bit too coarse for doing that. $\endgroup$
    – Henrik Schumacher
    Commented Mar 19, 2020 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.