# Finer granularity with ParallelDo

ParallelDo only parallelizes by the innermost variable. Is it possible to distribute the calculation over all the points (all the variable) separately? let me demonstrate:

CloseKernels[]; LaunchKernels[6];
ParallelDo[Print@Row[{i, j, k}, ", "], {i, 2}, {j, 10}, {k, 10}]


This only runs on two kernels, because i only has two values.

Method "FinestGrained" is of no help either.

As a workaround I can do

WorkUnits = Flatten[Table[{i, j, k}, {i, 2}, {j, 5}, {k, 5}], 2];
ParallelDo[With[{i = w[[1]], j = w[[2]], k = w[[3]]},
Print@Row[{i, j, k}, ", "]
]
, {w, WorkUnits}]


But this seems very clumsy.

Is there a more elegant way to parallelize the calculation over all variables, not just the first in a ParallelDo?

-
I think there was a similar question about ParallelTable, which has the same behaviour, but I can't find it ... I tend to use a workaround similar to yours: make a flat list of "work items" and then ParallelMap a function over them. –  Szabolcs Jan 20 '14 at 18:00
@Szabolcs Aha, Thanks. Probably a seperate Q, but, do you have a more general way of {i = w[[1]], j = w[[2]], k = w[[3]]}. I remember seeing that somewhere, but can't find it. –  Ajasja Jan 20 '14 at 18:21
When I did parallel calculations like this, I had a function defined for the task at hand. Something like Clear[f]; f[{i_,j_,k_}] := ...; ParallelMap[f, workItems]. It's admittedly not the most concise way if your function is actually quite small, but at that time the important things for me was keeping the code relatively organized and getting the bloody thing done finally (the parallel stuff ran for several hours). So I didn't have much time/energy left to think about the most elegant solution. –  Szabolcs Jan 20 '14 at 19:21
Instead of Flatten@Table I tend to use Tuples[{Range[2],Range[5],Range[5]}] but again this is just a personal preference and it's not actually better than a flattened table. Well, actually Table has the nasty property that it does not generate packed arrays if you have more than a single iterator, while Tuple does generate a packed array. But I think in most practical applications of this type that won't matter much. –  Szabolcs Jan 20 '14 at 19:23
In fact I think that the question I remembered was about why Table doesn't keep arrays packed when there are multiple iterators, and not about parallelization ... the answer to that was that when there are multiple iterators then the result may be ragged, as in Table[{i, j}, {i, 5}, {j, i}]. But I'm digressing and making the comment thread too long. –  Szabolcs Jan 20 '14 at 19:28

I'm not sure if it would be considered more elegant, but here's an idea:

Since your goal is to have each evaluation go to a separate kernel, one-by-one, we can submit evaluations manually:

WaitAll@Table[
ParallelSubmit[
Print@Row[{i, j, k}, ", "]
],
{i, 2}, {j, 5}, {k, 5}
]


One inconvenience with ParallelSubmit is that the easiest way to use it is to submit each evaluation separately (i.e. use the finest possible granularity). If the evaluations are small, then this is not efficient: Mathematica will spend most of its time submitting evaluations and retrieving results, not actually performing the computations. In this case you were asking specifically for the finest granularity, so it might be a good solution for your problem. It's good to be aware of this overhead problem though.

Your workaround doesn't suffer from this problem. Here's a slightly different way to do the same:

workItems = Tuples[{Range[2], Range[5], Range[5]}];

Clear[f] (* make sure there are no leftover definitions from prior uses of f *)
f[{i_, j_, k_}] := Print@Row[{i, j, k}, ", "]

ParallelMap[f, workItems, Method -> ...]


You are still free to quickly set the granularity to a coarser one if needed and reduce the communication overhead. With ParallelSubmit this would require enough extra work to outweigh the other conveniences.

-
Defining a function is not a bad idea, and I can get around the {i = w[[1]], j = w[[2]], k = w[[3]]} boring to type and error prone construct:) –  Ajasja Jan 20 '14 at 19:50