# Distributing value of variable to specific kernel

I want to distribute the value of a variable to a specific kernel in a parallel computing setup. Using DistributeDefinitions of course does the job, but it distributes the value to every active kernel, which is unnecessary and takes too much time. In my case I do have an array a and a listable function f operating on this array. A simple test scenario would be:

a = Table[Random[],{i,1000}];
f[x_] = x^2;


now I want to manually split up this calculation so that that each kernel has the same amounts of function calls to f, which would look schematically like this (I assume 1000/$KernelCount is an Integer): handle = {}; For[i = 1, i <=$KernelCount, i++,
AppendTo[handle,
ParallelSubmit[{i}, f[a[[(i - 1)*1000/$KernelCount + 1 ;; i*1000/$KernelCount]]]]
];
];
test = WaitAll[handle];


Of course this does not work, because the kernels do not have any knowledge about a. I could use DistributeDefinitions[a]. But this is as mentioned before exactly what I do not want to, because each kernel only has to know about a certain part of a, so distributing the whole array a would be a waste of time. This is of course only a test scenario, the real scenario consists of way more data bundled in the array a and a more complicated function f, but the task remains the same. I thought of using ParallelEvaluate to distribute part i of a to kernel i, but I did not come up with a solution. Any hints are appreciated.

-

Round 2:

test=WaitAll[
Table[(b=a[[(i1)*1000/$KernelCount+1;;i*1000/$KernelCount]];
ParallelSubmit[{b},(Pause[1.0];$KernelID->f[b])]), {i,1,$KernelCount}]];


Inside Table chop out the bit of a you want to send to a specific Kernel and place in b. Pass b as a closure into ParallelSubmit rather than i. Apply f to b within the parallel evaluation. Note the Pause is just to force a different Kernel each time because this test case is so fast without it \$KernelID (just to show which Kernel is being used) would (in my case) always be 1.

-
Thanks, this is what I was looking for. I can't explain why I did not see that in the first place, but whatever :). –  Wizard Mar 20 at 12:07

Why not just use ParallelMap and let it sort out which bits of a to send to each kernel?

And example...

a = Range[100]


Create a local store in each kernel to demonstrate which parts of a arrive....

ParallelEvaluate[z = {}];


Now execute...

ParallelMap[f, a];


Now check each z in each kernel...

ParallelEvaluate[z]

{{33,34,35,36,65,66,67,68,97,98,99,100},
{29,30,31,32,61,62,63,64,93,94,95,96},
{25,26,27,28,57,58,59,60,89,90,91,92},
{21,22,23,24,53,54,55,56,85,86,87,88},
{16,17,18,19,20,49,50,51,52,81,82,83,84},
{11,12,13,14,15,45,46,47,48,77,78,79,80},
{6,7,8,9,10,41,42,43,44,73,74,75,76},
{1,2,3,4,5,37,38,39,40,69,70,71,72}}

-
Unfortunately not an option in my more complicated case, that would have taken way longer to explain. Essential is that I need a solution to the problem above because I cannot use ParallelMap. Short version is that in my case f is a listable compiled function and the real parallelism takes place on the compiled level. –  Wizard Mar 18 at 19:45
Is a too big to distribute a copy of it to all kernels? –  Ymareth Mar 18 at 21:41
Essentially my compiled function f does not take very long to compute, but I think long enough to benefit from parallization. If I distributing the whole data array to all kernels takes about as long as calculating the function on a single multicore machine, but if I could split up the data array and only distribute a small fraction I guess the scenario could benefit from parallel computing especially if there are many machines in the distibuted computing cluster. –  Wizard Mar 19 at 9:22