Related to this question and in particular its answer I am trying to wrap everything into a new parallelTab
function that takes into account some cost per job, computes the optimal distribution of jobs to the cores and uses ParallelSubmit
to, well, submit them.
Let's just focus on the case that the optimal distribution is already known, e.g. here
ClearAll[fun, vals, distribute, f, submit];
CloseKernels[];
fun[x_] := (Pause[.05*x]; x^2);
vals = Range[1, 12];
distribute = {{1, 3, 6, 10}, {2, 4, 12}, {5, 7, 8}, {9, 11}};
LaunchKernels[4];
fun
is the function with imbalanced timing and distribute
tells us how to submit jobs 1
to 12
to 4 parallel kernels so that the computation time is minimized. If the number of kernels is known I can proceed like this
f[i_] := Table[fun[x], {x, vals[[distribute[[i]]]]}];
DistributeDefinitions[f];
AbsoluteTiming[submit = {ParallelSubmit[f[1]], ParallelSubmit[f[2]], ParallelSubmit[f[3]], ParallelSubmit[f[4]]}; Print[submit]; Flatten[WaitAll[submit]][[Ordering@Flatten@distribute]]]
which works fine and yields a factor of 1.7
speedup compared to the naive ParallelTable
approach and almost a factor of 4
compared to the Table
. However, the goal is to let parallelTab
have an argument, say numKernels
, that specifies the number of cores to be used. For this purpose, I would like to automate the ParallelSubmit
part in the above code, so that a list with necessary ParallelSubmit
s of proper f[i]
is generated (this particular code is not working and is the part I am asking for help - let's assume numKernels
has been fixed to 4
). What I tried is this:
submit2 = Table[ParallelSubmit[f[i]], {i, 4}]
rules = Table[{Rule[i, j]}, {j, 1, 4}];
submit3 = ParallelSubmit[f[i]] /. rules
Neither of them works. To my understanding, submit2
fails because ParallelSubmit
has Attributes
HoldAllComplete
, resulting in a list {ParallelSubmit[f[i]],ParallelSubmit[f[i]],...}
. Actually on the first glance, submit3
seems to do what I want, see here
However, a WaitAll
yields failure (although at least one kernel seems to do what it is supposed to):
How can I automate the procedure of ParallelSubmit
ting the tasks. I believe that must be possible but my knowledge about advanced usage of Hold
and its relatives is just too little to solve that trick on my own.