I want to distribute the value of a variable to a specific kernel in a parallel computing setup. Using DistributeDefinitions of course does the job, but it distributes the value to every active kernel, which is unnecessary and takes too much time. In my case I do have an array a and a listable function f operating on this array. A simple test scenario would be:

a = Table[Random[],{i,1000}];
f[x_] = x^2;

now I want to manually split up this calculation so that that each kernel has the same amounts of function calls to f, which would look schematically like this (I assume 1000/$KernelCount is an Integer):

handle = {};
For[i = 1, i <= $KernelCount, i++, 
       ParallelSubmit[{i}, f[a[[(i - 1)*1000/$KernelCount + 1 ;; i*1000/$KernelCount]]]]
test = WaitAll[handle];

Of course this does not work, because the kernels do not have any knowledge about a. I could use DistributeDefinitions[a]. But this is as mentioned before exactly what I do not want to, because each kernel only has to know about a certain part of a, so distributing the whole array a would be a waste of time. This is of course only a test scenario, the real scenario consists of way more data bundled in the array a and a more complicated function f, but the task remains the same. I thought of using ParallelEvaluate to distribute part i of a to kernel i, but I did not come up with a solution. Any hints are appreciated.


Round 2:


Inside Table chop out the bit of a you want to send to a specific Kernel and place in b. Pass b as a closure into ParallelSubmit rather than i. Apply f to b within the parallel evaluation. Note the Pause is just to force a different Kernel each time because this test case is so fast without it $KernelID (just to show which Kernel is being used) would (in my case) always be 1.

  • $\begingroup$ Thanks, this is what I was looking for. I can't explain why I did not see that in the first place, but whatever :). $\endgroup$ – Wizard Mar 20 '14 at 12:07

Why not just use ParallelMap and let it sort out which bits of a to send to each kernel?

And example...

a = Range[100]

Create a local store in each kernel to demonstrate which parts of a arrive....

ParallelEvaluate[z = {}];

Now execute...

ParallelMap[f, a];

Now check each z in each kernel...


  • $\begingroup$ Unfortunately not an option in my more complicated case, that would have taken way longer to explain. Essential is that I need a solution to the problem above because I cannot use ParallelMap. Short version is that in my case f is a listable compiled function and the real parallelism takes place on the compiled level. $\endgroup$ – Wizard Mar 18 '14 at 19:45
  • $\begingroup$ Is a too big to distribute a copy of it to all kernels? $\endgroup$ – Ymareth Mar 18 '14 at 21:41
  • $\begingroup$ Essentially my compiled function f does not take very long to compute, but I think long enough to benefit from parallization. If I distributing the whole data array to all kernels takes about as long as calculating the function on a single multicore machine, but if I could split up the data array and only distribute a small fraction I guess the scenario could benefit from parallel computing especially if there are many machines in the distibuted computing cluster. $\endgroup$ – Wizard Mar 19 '14 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.