1
$\begingroup$

I'm trying to parallelize a very simple code that performs a stochastic gradient descent optimization (on a dual core cpu w. hyperthreading, so 4 parallel kernels).

I use ParallelMap to compute the gradient over a randomly sampled subset of variables. I am puzzled by the fact that when the EXACT SAME definition is used inside or outside a function the performance sees a dramatic drop.

These are the definitions, where SGDgrad and SGDgradPar only differ for the use of Map vs ParallelMap

MyDer[strenghts_, efforts_, i_] := Module[{sum},
  sum = strenghts.efforts;
  2*strenghts[[i]]^2*efforts[[i]]/sum - 
   strenghts[[i]] ((strenghts^2).(efforts^2))/sum^2
  ]
SGDgrad[strenghts_, efforts_, rsamp_] := 
 SparseArray[
  Map[# -> MyDer[strenghts[[All, #[[2]]]], 
      efforts[[All, #[[2]]]], #[[1]]] &, rsamp], {Length@strenghts, 
   Length@strenghts[[1]]}]
SGDgradPar[strenghts_, efforts_, rsamp_] := 
  SparseArray[
   ParallelMap[# -> 
      MyDer[strenghts[[All, #[[2]]]], 
       efforts[[All, #[[2]]]], #[[1]]] &, rsamp, 
    Method -> "CoarsestGrained"], {Length@strenghts, 
    Length@strenghts[[1]]}];

As a test this can be run on (small - 50 samples) random data:

strenghts = RandomReal[1, {20, 10000}];
efforts = RandomReal[1, {20, 10000}];

rsamp = RandomSample[Flatten[Table[{i, j}, {i, Length@strenghts}, {j, 
      Length@strenghts[[1]]}], 1], 50];

I try to run the same code as a function (SGDgrad and SGDgradPar) and as a simple line of code. For the parallelized version there is a 20x performance difference!

SGDgrad[strenghts, efforts, rsamp]; // AbsoluteTiming
SGDgradPar[strenghts, efforts, rsamp]; // AbsoluteTiming

{0.001407, Null}
{0.547823, Null}

SparseArray[
   Map[# -> 
      MyDer[strenghts[[All, #[[2]]]], 
       efforts[[All, #[[2]]]], #[[1]]] &, rsamp], {Length@strenghts, 
    Length@strenghts[[1]]}]; // AbsoluteTiming
SparseArray[
   ParallelMap[# -> 
      MyDer[strenghts[[All, #[[2]]]], 
       efforts[[All, #[[2]]]], #[[1]]] &, rsamp, 
    Method -> "CoarsestGrained"], {Length@strenghts, 
    Length@strenghts[[1]]}]; // AbsoluteTiming

{0.001153, Null}
{0.026422, Null}

If I make the dataset bigger (50000 samples):

rsamp = RandomSample[Flatten[Table[{i, j}, {i, Length@strenghts}, {j, 
      Length@strenghts[[1]]}], 1], 50000];

Then timings for the same operations are:

{1.20238, Null}
{1.87731, Null}


{1.20966, Null}
{1.01134, Null}

The advantage of parallelization is very small (which I find strange) and only when batches are large. Such advantage is completely spoiled if I parallelize inside a function rather than outside...

Can someone explain these differences? Am I doing something wrong?

I have already tried distributing the definitions of the variables and MyDer, and nothing changes.

$\endgroup$
  • $\begingroup$ It is actually spelt, strengths not strenghts. Just in case others will wonder why the name they typed doesn't appear to be defined... $\endgroup$ – Szabolcs Mar 14 '17 at 14:22
  • $\begingroup$ oops... autocomplete leads to propagation of misspellings... $\endgroup$ – CupiDio Mar 14 '17 at 16:46
2
$\begingroup$

Here is a much smaller example demonstrating the difference:

In[17]:= ParallelMap[# -> 
     MyDer[strenghts[[All, #[[2]]]], 
      efforts[[All, #[[2]]]], #[[1]]] &, rsamp, 
   Method -> "CoarsestGrained"]; // AbsoluteTiming

Out[17]= {0.065301, Null}

In[13]:= With[{strenghts = strenghts, efforts = efforts},
   ParallelMap[# -> 
      MyDer[strenghts[[All, #[[2]]]], 
       efforts[[All, #[[2]]]], #[[1]]] &, rsamp, 
    Method -> "CoarsestGrained"]
   ]; // AbsoluteTiming

Out[13]= {0.488711, Null}

When you use a function (or With above), strenghts and efforts are inlined into the pure function that you are mapping. These arrays are sent to the subkernels once for each evaluation. The literal value of the arrays is part of the expression describing the pure function. The pure function becomes a huge expression that is slow to transfer.

When you use variables, they are sent (distributed, see DistributeDefinitions) once and re-used multiple times. The function that you are mapping does not contain the literal value of these arrays. It only contains references to them. The function will be sent to subkernels once for each evaluation. But this time the function is a small expression that does not take a long time to transfer.


The advantage of parallelization is very small (which I find strange) and only when batches are large. Such advantage is completely spoiled if I parallelize inside a function rather than outside...

In Mathematica, parallelization involves explicitly transferring data between the main kernel and the subkernels. This transfer is expensive (i.e. it takes a long time). Parallelization is only worth it if the computation takes considerably longer than the data transfer.

$\endgroup$
  • $\begingroup$ Thanks, having MyDer[] use a global rather than local variable solved the issue, i.e. timing is now the same as not using the function. I know that transfer time can be the biggest factor here. But then why if I define something like: MyDer[] := Block[{}, 1 + 1;] I still have that ParallelMap[MyDer[]&, rsamp] is worse than Map[MyDer[]&, rsamp]? $\endgroup$ – CupiDio Mar 14 '17 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.