1
$\begingroup$

Related to this question and in particular its answer I am trying to wrap everything into a new parallelTab function that takes into account some cost per job, computes the optimal distribution of jobs to the cores and uses ParallelSubmit to, well, submit them.

Let's just focus on the case that the optimal distribution is already known, e.g. here

ClearAll[fun, vals, distribute, f, submit];
CloseKernels[];
fun[x_] := (Pause[.05*x]; x^2);
vals = Range[1, 12];
distribute = {{1, 3, 6, 10}, {2, 4, 12}, {5, 7, 8}, {9, 11}};
LaunchKernels[4];

fun is the function with imbalanced timing and distribute tells us how to submit jobs 1 to 12 to 4 parallel kernels so that the computation time is minimized. If the number of kernels is known I can proceed like this

f[i_] := Table[fun[x], {x, vals[[distribute[[i]]]]}];
DistributeDefinitions[f];
AbsoluteTiming[submit = {ParallelSubmit[f[1]], ParallelSubmit[f[2]], ParallelSubmit[f[3]], ParallelSubmit[f[4]]}; Print[submit]; Flatten[WaitAll[submit]][[Ordering@Flatten@distribute]]]

which works fine and yields a factor of 1.7 speedup compared to the naive ParallelTable approach and almost a factor of 4 compared to the Table. However, the goal is to let parallelTab have an argument, say numKernels, that specifies the number of cores to be used. For this purpose, I would like to automate the ParallelSubmit part in the above code, so that a list with necessary ParallelSubmits of proper f[i] is generated (this particular code is not working and is the part I am asking for help - let's assume numKernels has been fixed to 4). What I tried is this:

submit2 = Table[ParallelSubmit[f[i]], {i, 4}]
rules = Table[{Rule[i, j]}, {j, 1, 4}];
submit3 = ParallelSubmit[f[i]] /. rules

Neither of them works. To my understanding, submit2 fails because ParallelSubmit has Attributes HoldAllComplete, resulting in a list {ParallelSubmit[f[i]],ParallelSubmit[f[i]],...}. Actually on the first glance, submit3 seems to do what I want, see here

enter image description here

However, a WaitAll yields failure (although at least one kernel seems to do what it is supposed to):

enter image description here

How can I automate the procedure of ParallelSubmitting the tasks. I believe that must be possible but my knowledge about advanced usage of Hold and its relatives is just too little to solve that trick on my own.

$\endgroup$
1
$\begingroup$

The following works :

submit4 = Table[With[{i = i}, ParallelSubmit[f[i]]], {i, 4}]
WaitAll[submit4]

{{1, 9, 36, 100}, {4, 16, 144}, {25, 49, 64}, {81, 121}}

You can find a explanation about With here (chapter entitled "With", paragraph "Advanced Use")

$\endgroup$
  • $\begingroup$ Thank you very much! Also very nice reference for explanation. Using Withfor such injections inside Hold is very useful, good to know that for future :) $\endgroup$ – Lukas Feb 24 '16 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.