I want to distribute the value of a variable to a specific kernel in a parallel computing setup. Using DistributeDefinitions
of course does the job, but it distributes the value to every active kernel, which is unnecessary and takes too much time.
In my case I do have an array a
and a listable function f
operating on this array. A simple test scenario would be:
a = Table[Random[],{i,1000}];
f[x_] = x^2;
now I want to manually split up this calculation so that that each kernel has the same amounts of function calls to f
, which would look schematically like this (I assume 1000/$KernelCount is an Integer):
handle = {};
For[i = 1, i <= $KernelCount, i++,
AppendTo[handle,
ParallelSubmit[{i}, f[a[[(i - 1)*1000/$KernelCount + 1 ;; i*1000/$KernelCount]]]]
];
];
test = WaitAll[handle];
Of course this does not work, because the kernels do not have any knowledge about a
. I could use DistributeDefinitions[a]
. But this is as mentioned before exactly what I do not want to, because each kernel only has to know about a certain part of a
, so distributing the whole array a
would be a waste of time. This is of course only a test scenario, the real scenario consists of way more data bundled in the array a
and a more complicated function f
, but the task remains the same.
I thought of using ParallelEvaluate
to distribute part i of a
to kernel i, but I did not come up with a solution. Any hints are appreciated.