1
$\begingroup$

I want to distribute the value of a variable to a specific kernel in a parallel computing setup. Using DistributeDefinitions of course does the job, but it distributes the value to every active kernel, which is unnecessary and takes too much time. In my case I do have an array a and a listable function f operating on this array. A simple test scenario would be:

a = Table[Random[],{i,1000}];
f[x_] = x^2;

now I want to manually split up this calculation so that that each kernel has the same amounts of function calls to f, which would look schematically like this (I assume 1000/$KernelCount is an Integer):

handle = {};
For[i = 1, i <= $KernelCount, i++, 
      AppendTo[handle, 
       ParallelSubmit[{i}, f[a[[(i - 1)*1000/$KernelCount + 1 ;; i*1000/$KernelCount]]]]
   ];
];
test = WaitAll[handle];

Of course this does not work, because the kernels do not have any knowledge about a. I could use DistributeDefinitions[a]. But this is as mentioned before exactly what I do not want to, because each kernel only has to know about a certain part of a, so distributing the whole array a would be a waste of time. This is of course only a test scenario, the real scenario consists of way more data bundled in the array a and a more complicated function f, but the task remains the same. I thought of using ParallelEvaluate to distribute part i of a to kernel i, but I did not come up with a solution. Any hints are appreciated.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Round 2:

test=WaitAll[
  Table[(b=a[[(i1)*1000/$KernelCount+1;;i*1000/$KernelCount]];
         ParallelSubmit[{b},(Pause[1.0];$KernelID->f[b])]),
            {i,1,$KernelCount}]];

Inside Table chop out the bit of a you want to send to a specific Kernel and place in b. Pass b as a closure into ParallelSubmit rather than i. Apply f to b within the parallel evaluation. Note the Pause is just to force a different Kernel each time because this test case is so fast without it $KernelID (just to show which Kernel is being used) would (in my case) always be 1.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, this is what I was looking for. I can't explain why I did not see that in the first place, but whatever :). $\endgroup$
    – Wizard
    Mar 20, 2014 at 12:07
0
$\begingroup$

Why not just use ParallelMap and let it sort out which bits of a to send to each kernel?

And example...

a = Range[100]

Create a local store in each kernel to demonstrate which parts of a arrive....

ParallelEvaluate[z = {}];

Now execute...

ParallelMap[f, a];

Now check each z in each kernel...

ParallelEvaluate[z]

{{33,34,35,36,65,66,67,68,97,98,99,100},
{29,30,31,32,61,62,63,64,93,94,95,96},
{25,26,27,28,57,58,59,60,89,90,91,92},
{21,22,23,24,53,54,55,56,85,86,87,88},
{16,17,18,19,20,49,50,51,52,81,82,83,84},
{11,12,13,14,15,45,46,47,48,77,78,79,80},
{6,7,8,9,10,41,42,43,44,73,74,75,76},
{1,2,3,4,5,37,38,39,40,69,70,71,72}}
Improve this answer
$\endgroup$
3
  • $\begingroup$ Unfortunately not an option in my more complicated case, that would have taken way longer to explain. Essential is that I need a solution to the problem above because I cannot use ParallelMap. Short version is that in my case f is a listable compiled function and the real parallelism takes place on the compiled level. $\endgroup$
    – Wizard
    Mar 18, 2014 at 19:45
  • $\begingroup$ Is a too big to distribute a copy of it to all kernels? $\endgroup$
    – Ymareth
    Mar 18, 2014 at 21:41
  • $\begingroup$ Essentially my compiled function f does not take very long to compute, but I think long enough to benefit from parallization. If I distributing the whole data array to all kernels takes about as long as calculating the function on a single multicore machine, but if I could split up the data array and only distribute a small fraction I guess the scenario could benefit from parallel computing especially if there are many machines in the distibuted computing cluster. $\endgroup$
    – Wizard
    Mar 19, 2014 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.