Related to this question and in particular its answer I am trying to wrap everything into a new parallelTab function that takes into account some cost per job, computes the optimal distribution of jobs to the cores and uses ParallelSubmit to, well, submit them.
Let's just focus on the case that the optimal distribution is already known, e.g. here
ClearAll[fun, vals, distribute, f, submit];
CloseKernels[];
fun[x_] := (Pause[.05*x]; x^2);
vals = Range[1, 12];
distribute = {{1, 3, 6, 10}, {2, 4, 12}, {5, 7, 8}, {9, 11}};
LaunchKernels[4];
fun is the function with imbalanced timing and distribute tells us how to submit jobs 1 to 12 to 4 parallel kernels so that the computation time is minimized. If the number of kernels is known I can proceed like this
f[i_] := Table[fun[x], {x, vals[[distribute[[i]]]]}];
DistributeDefinitions[f];
AbsoluteTiming[submit = {ParallelSubmit[f[1]], ParallelSubmit[f[2]], ParallelSubmit[f[3]], ParallelSubmit[f[4]]}; Print[submit]; Flatten[WaitAll[submit]][[Ordering@Flatten@distribute]]]
which works fine and yields a factor of 1.7 speedup compared to the naive ParallelTable approach and almost a factor of 4 compared to the Table. However, the goal is to let parallelTab have an argument, say numKernels, that specifies the number of cores to be used. For this purpose, I would like to automate the ParallelSubmit part in the above code, so that a list with necessary ParallelSubmits of proper f[i] is generated (this particular code is not working and is the part I am asking for help - let's assume numKernels has been fixed to 4). What I tried is this:
submit2 = Table[ParallelSubmit[f[i]], {i, 4}]
rules = Table[{Rule[i, j]}, {j, 1, 4}];
submit3 = ParallelSubmit[f[i]] /. rules
Neither of them works. To my understanding, submit2 fails because ParallelSubmit has Attributes HoldAllComplete, resulting in a list {ParallelSubmit[f[i]],ParallelSubmit[f[i]],...}. Actually on the first glance, submit3 seems to do what I want, see here
However, a WaitAll yields failure (although at least one kernel seems to do what it is supposed to):
How can I automate the procedure of ParallelSubmitting the tasks. I believe that must be possible but my knowledge about advanced usage of Hold and its relatives is just too little to solve that trick on my own.
