ParallelDo with two loop counters

Question

why isn't Mathematica using all the cores in the following code.

ParallelDo[
 Print["i=" <> ToString[i] <> "; " <> "j=" <> ToString[j]], 
{i, 1, 2}, {j, 1, 2}]

The output is

(kernel 4) i=1; j=1
(kernel 3) i=2; j=1
(kernel 4) i=1; j=2
(kernel 3) i=2; j=2

the machine has 4 physical cores but only two are being used. I'm using Mathematica 11 on Windows 10. Thanks in advance.

P.S. in comparison

ParallelDo[
Print["i=" <> ToString[i]], {i, 1, 4}
]

has the output

(kernel 4) i=1
(kernel 3) i=2
(kernel 2) i=3
(kernel 1) i=4

Answer 1

Performance-wise, it wouldn't make sense because the overhead is much larger when evaluating it on 4 cores. Please look at the settings of Method in Parallelize and apply it to your ParallelDo

ParallelDo[
 Print["i=" <> ToString[i] <> "; " <> "j=" <> ToString[j]], {i, 1, 
  10}, {j, 1, 10}, Method -> "FinestGrained"]

This uses all 8 cores I have available, but remember that finding the right chunk-size is important to optimize speed. Distributing a computation on all cores is not always the best choice.

If you want a deeper insight, you can look at the output of the following

ParallelTable[
  $KernelID, {i, 1, 10}, {j, 1, 10},
  Method -> "FinestGrained"] // Column

(*
{8,8,8,8,8,8,8,8,8,8}
{7,7,7,7,7,7,7,7,7,7}
{6,6,6,6,6,6,6,6,6,6}
{5,5,5,5,5,5,5,5,5,5}
{4,4,4,4,4,4,4,4,4,4}
{3,3,3,3,3,3,3,3,3,3}
{2,2,2,2,2,2,2,2,2,2}
{1,1,1,1,1,1,1,1,1,1}
{8,8,8,8,8,8,8,8,8,8}
{7,7,7,7,7,7,7,7,7,7}
*)

That seems to indicate that your outer iteration needs to be at least 32 to use all cores.

Edit

If you know what you are doing (because finding the sweet-spot in parallelization is not always trivial), you can parallelize to whatever level you like. One solution is to use only a 1-dim loop and recover the indices inside the loop. Here is an example that is equivalent of {i,1,4} and {j,1,4} but doing every element in parallel:

ParallelDo[Print[{QuotientRemainder[x, 4] + 1, $KernelID}], {x, 0, 7}]

It uses all 8 kernels I have. If you have a more complex iterator pattern, you can create the iterators upfront

iter = Flatten[Table[{i, j, k}, {i, 3}, {j, 4}, {k, 5}], 2];
ParallelDo[Print[{it, $KernelID}], {it, iter}]

Again, each element is processed in parallel.

Answer 2

Parallel computing is based on splitting the computation into independent subtasks, that can be executed by (independent) subkernels.

Consider a computation with two counters, say {i,1,2},{j,1,8}.

The outer counter allows only a splitting into two independent subtasks: i=1; {j,1,8}, i=2;{j,1,8}. So these subtasks will be executed on two subkernels. Each of the subkernels has to execute a command with a counter {j,1,8}. Since a subkernel cannot control parallel computations, no other subkernels can be used for the execution of this command. Thus, in this example it is impossible to use more than two subkkernels for the computation.

More general, when using multiple counters, the maximum number of subtasks is the number of elements determined by first counter and therefore it is impossible to use more subkernels than this number. All examples in the other answers demonstrate this.

If you really want to use more subkernels (be aware of the overhead), you have to rewrite your task, e.g. as done by Halirutan.

Edit

Let us consider the situation where the range for the inner counter is a little bit larger.

On my quadcore computer, I can launch 8 subkernels:

LaunchKernels[8];

The following command uses only two subkernels:

ParallelDo[{i,j, i+j}, {i,1,2},{j,1,10^6}] // AbsoluteTiming
(* {0.49775,Null} *)

The outer loop is done in parallel and on each subkernel the inner loop is executed as a normal Do. Since the inner loop is pretty much work, we want to do that in parallel. Then the outer loop must be executed with Do:

Do[ParallelDo[{i,j, i+j}, {j,1,10^6}], {i,1,2}] // AbsoluteTiming
(* {0.409451,Null} *)

This is not much better, due to the overhead of parallel computing. We can reduce this overhead:

Do[ParallelDo[{i,j, i+j}, {j,1,10^6}, Method->"CoarsestGrained"], {i,1,2}] // AbsoluteTiming
(* {0.265565,Null} *)

Halirutan's solution works nice for small examples. It has the disadvantage that all points are created by the main kernel, have to be stored in the memory and have to be sent to the subkernels.

(iter=Flatten[Table[{i,j},{i,1,2},{j,1,10^6}],1];
   ParallelDo[{i,j,i+j},{it,iter}, Method->"CoarsestGrained"]) // AbsoluteTiming
(* {13.4777,Null} *)

So this should not be the default method.

Answer 3

Try this

LaunchKernels@8; 
ParallelDo[Print["i=" <> ToString[i] <> "; " <> "j=" <> ToString[j]], {i, 1,  10}, {j, 1, 10}]
CloseKernels;