I'm trying to parallelize a very simple code that performs a stochastic gradient descent optimization (on a dual core cpu w. hyperthreading, so 4 parallel kernels).
I use ParallelMap to compute the gradient over a randomly sampled subset of variables. I am puzzled by the fact that when the EXACT SAME definition is used inside or outside a function the performance sees a dramatic drop.
These are the definitions, where SGDgrad and SGDgradPar only differ for the use of Map vs ParallelMap
MyDer[strenghts_, efforts_, i_] := Module[{sum},
sum = strenghts.efforts;
2*strenghts[[i]]^2*efforts[[i]]/sum -
strenghts[[i]] ((strenghts^2).(efforts^2))/sum^2
]
SGDgrad[strenghts_, efforts_, rsamp_] :=
SparseArray[
Map[# -> MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp], {Length@strenghts,
Length@strenghts[[1]]}]
SGDgradPar[strenghts_, efforts_, rsamp_] :=
SparseArray[
ParallelMap[# ->
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp,
Method -> "CoarsestGrained"], {Length@strenghts,
Length@strenghts[[1]]}];
As a test this can be run on (small - 50 samples) random data:
strenghts = RandomReal[1, {20, 10000}];
efforts = RandomReal[1, {20, 10000}];
rsamp = RandomSample[Flatten[Table[{i, j}, {i, Length@strenghts}, {j,
Length@strenghts[[1]]}], 1], 50];
I try to run the same code as a function (SGDgrad and SGDgradPar) and as a simple line of code. For the parallelized version there is a 20x performance difference!
SGDgrad[strenghts, efforts, rsamp]; // AbsoluteTiming
SGDgradPar[strenghts, efforts, rsamp]; // AbsoluteTiming
{0.001407, Null}
{0.547823, Null}
SparseArray[
Map[# ->
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp], {Length@strenghts,
Length@strenghts[[1]]}]; // AbsoluteTiming
SparseArray[
ParallelMap[# ->
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp,
Method -> "CoarsestGrained"], {Length@strenghts,
Length@strenghts[[1]]}]; // AbsoluteTiming
{0.001153, Null}
{0.026422, Null}
If I make the dataset bigger (50000 samples):
rsamp = RandomSample[Flatten[Table[{i, j}, {i, Length@strenghts}, {j,
Length@strenghts[[1]]}], 1], 50000];
Then timings for the same operations are:
{1.20238, Null}
{1.87731, Null}
{1.20966, Null}
{1.01134, Null}
The advantage of parallelization is very small (which I find strange) and only when batches are large. Such advantage is completely spoiled if I parallelize inside a function rather than outside...
Can someone explain these differences? Am I doing something wrong?
I have already tried distributing the definitions of the variables and MyDer, and nothing changes.
strengths
notstrenghts
. Just in case others will wonder why the name they typed doesn't appear to be defined... $\endgroup$ – Szabolcs Mar 14 '17 at 14:22