3
$\begingroup$

why isn't Mathematica using all the cores in the following code.

ParallelDo[
 Print["i=" <> ToString[i] <> "; " <> "j=" <> ToString[j]], 
{i, 1, 2}, {j, 1, 2}]

The output is

(kernel 4) i=1; j=1
(kernel 3) i=2; j=1
(kernel 4) i=1; j=2
(kernel 3) i=2; j=2

the machine has 4 physical cores but only two are being used. I'm using Mathematica 11 on Windows 10. Thanks in advance.

P.S. in comparison

ParallelDo[
Print["i=" <> ToString[i]], {i, 1, 4}
]

has the output

(kernel 4) i=1
(kernel 3) i=2
(kernel 2) i=3
(kernel 1) i=4
Improve this question
$\endgroup$
  • $\begingroup$ My observation is that Mathematica in default uses half of the number of cores. So if you want use all of them, you need to change the default value. $\endgroup$ – Αλέξανδρος Ζεγγ May 27 '18 at 12:53
  • 1
    $\begingroup$ @Αλέξανδρος Ζεγγ, ParallelDo[ Print["i=" <> ToString[i]], {i, 1, 4} ], uses all 4 cores $\endgroup$ – user1740587 May 27 '18 at 12:54
  • $\begingroup$ First observe that a physical core is not the same as a subkernel for parallel computations. On my computer, your command initiates the default of four subkernels, but indeed it uses only two subkernels. That seems to be determined by the limit of the first counter. When you use {i,1,2},{j,1,8} only two subkernels are used, but when you use {j,1,8},{i,1,2} all 4 subkernels are used. $\endgroup$ – Fred Simons May 27 '18 at 14:00
  • $\begingroup$ @Fred Simons , thanks for your input, I agree with you, but it doesn't answer how I can use all kernels while using multiple counters. $\endgroup$ – user1740587 May 27 '18 at 14:03
4
$\begingroup$

Performance-wise, it wouldn't make sense because the overhead is much larger when evaluating it on 4 cores. Please look at the settings of Method in Parallelize and apply it to your ParallelDo

ParallelDo[
 Print["i=" <> ToString[i] <> "; " <> "j=" <> ToString[j]], {i, 1, 
  10}, {j, 1, 10}, Method -> "FinestGrained"]

This uses all 8 cores I have available, but remember that finding the right chunk-size is important to optimize speed. Distributing a computation on all cores is not always the best choice.

If you want a deeper insight, you can look at the output of the following

ParallelTable[
  $KernelID, {i, 1, 10}, {j, 1, 10},
  Method -> "FinestGrained"] // Column

(*
{8,8,8,8,8,8,8,8,8,8}
{7,7,7,7,7,7,7,7,7,7}
{6,6,6,6,6,6,6,6,6,6}
{5,5,5,5,5,5,5,5,5,5}
{4,4,4,4,4,4,4,4,4,4}
{3,3,3,3,3,3,3,3,3,3}
{2,2,2,2,2,2,2,2,2,2}
{1,1,1,1,1,1,1,1,1,1}
{8,8,8,8,8,8,8,8,8,8}
{7,7,7,7,7,7,7,7,7,7}
*)

That seems to indicate that your outer iteration needs to be at least 32 to use all cores.

Edit

If you know what you are doing (because finding the sweet-spot in parallelization is not always trivial), you can parallelize to whatever level you like. One solution is to use only a 1-dim loop and recover the indices inside the loop. Here is an example that is equivalent of {i,1,4} and {j,1,4} but doing every element in parallel:

ParallelDo[Print[{QuotientRemainder[x, 4] + 1, $KernelID}], {x, 0, 7}]

img

It uses all 8 kernels I have. If you have a more complex iterator pattern, you can create the iterators upfront

iter = Flatten[Table[{i, j, k}, {i, 3}, {j, 4}, {k, 5}], 2];
ParallelDo[Print[{it, $KernelID}], {it, iter}]

Again, each element is processed in parallel.

Improve this answer
$\endgroup$
  • $\begingroup$ I did run your code on a 32 core machine it uses only 10 cores.... $\endgroup$ – user1740587 May 27 '18 at 13:16
  • $\begingroup$ I understand that but my counters are {i,1,4} and {j,1,5}, the machine has 32 cores. Again the question is how can one use multiple counters while using all cores. Your answer suggest a solution that works only for a small subset of cases. $\endgroup$ – user1740587 May 27 '18 at 14:39
  • $\begingroup$ By using an out Table that calls a 1-dim ParallelTable: Table[ ParallelTable[$KernelID, {j, 1, 10}, Method -> "FinestGrained"], {i, 1, 10} ] $\endgroup$ – halirutan♦ May 27 '18 at 14:44
  • $\begingroup$ If i understand correctly your solution for "Table" translated to "Do" is Do[ ParallelDo[Print[$KernelID], {i, 1, 2}], {j, 1, 2} ] which has the same problem, it uses only 2 cores $\endgroup$ – user1740587 May 27 '18 at 14:51
  • $\begingroup$ for the issue to arise the number of cores must be greater than the maximum of the first counter. $\endgroup$ – user1740587 May 27 '18 at 14:58
1
$\begingroup$

Parallel computing is based on splitting the computation into independent subtasks, that can be executed by (independent) subkernels.

Consider a computation with two counters, say {i,1,2},{j,1,8}.

The outer counter allows only a splitting into two independent subtasks: i=1; {j,1,8}, i=2;{j,1,8}. So these subtasks will be executed on two subkernels. Each of the subkernels has to execute a command with a counter {j,1,8}. Since a subkernel cannot control parallel computations, no other subkernels can be used for the execution of this command. Thus, in this example it is impossible to use more than two subkkernels for the computation.

More general, when using multiple counters, the maximum number of subtasks is the number of elements determined by first counter and therefore it is impossible to use more subkernels than this number. All examples in the other answers demonstrate this.

If you really want to use more subkernels (be aware of the overhead), you have to rewrite your task, e.g. as done by Halirutan.

Edit

Let us consider the situation where the range for the inner counter is a little bit larger.

On my quadcore computer, I can launch 8 subkernels:

LaunchKernels[8];

The following command uses only two subkernels:

ParallelDo[{i,j, i+j}, {i,1,2},{j,1,10^6}] // AbsoluteTiming
(* {0.49775,Null} *)

The outer loop is done in parallel and on each subkernel the inner loop is executed as a normal Do. Since the inner loop is pretty much work, we want to do that in parallel. Then the outer loop must be executed with Do:

Do[ParallelDo[{i,j, i+j}, {j,1,10^6}], {i,1,2}] // AbsoluteTiming
(* {0.409451,Null} *)

This is not much better, due to the overhead of parallel computing. We can reduce this overhead:

Do[ParallelDo[{i,j, i+j}, {j,1,10^6}, Method->"CoarsestGrained"], {i,1,2}] // AbsoluteTiming
(* {0.265565,Null} *)

Halirutan's solution works nice for small examples. It has the disadvantage that all points are created by the main kernel, have to be stored in the memory and have to be sent to the subkernels. 

(iter=Flatten[Table[{i,j},{i,1,2},{j,1,10^6}],1];
   ParallelDo[{i,j,i+j},{it,iter}, Method->"CoarsestGrained"]) // AbsoluteTiming
(* {13.4777,Null} *)

So this should not be the default method.

Improve this answer
$\endgroup$
  • $\begingroup$ I agree with you that "parallel computing is based on splitting the computation into independent subtasks", but in your answer you assume that the only logical way to divide the computation is by splitting the outer counter first but imo it is equally possible to split the inner counter (j) first, which in your case would give a better result. So why does mathematica work in such a strange way? In the documentation I didn't find where it was mentioned. If it was as you say why doesn't mathematica split the counter with the highest number of tics? $\endgroup$ – user1740587 May 28 '18 at 10:35
  • $\begingroup$ @user1740587. For me, this is the straightforward and natural way. The order of the counters counts. For example, consider {i,1,2},{j,1, 8+i}. Do you want that Mathematica automatically rewrites it as {j,1,10},{i, Max[1,j-8], 2}? Such a functionality does not exist in any of this sort of commands, like Table, Do, Integrate. Only in the situation that the second counter does not depend on the first counter, you can simply reverse the two counters. But that is so simple that you can do it yourself as well when entering the command. $\endgroup$ – Fred Simons May 28 '18 at 11:38
  • $\begingroup$ you make a good point, and I agree with you, but why not create a list of vectors {i,j} (like halirutan made manually) and iterate over them by default? $\endgroup$ – user1740587 May 28 '18 at 18:31
  • $\begingroup$ @user1740587. To answer this question, I edited my answer. $\endgroup$ – Fred Simons May 29 '18 at 6:55
0
$\begingroup$

Try this

LaunchKernels@8; 
ParallelDo[Print["i=" <> ToString[i] <> "; " <> "j=" <> ToString[j]], {i, 1,  10}, {j, 1, 10}]
CloseKernels;
Improve this answer
$\endgroup$
  • $\begingroup$ I did run this on a 32 core machine it uses 8 cores. $\endgroup$ – user1740587 May 27 '18 at 13:16
  • $\begingroup$ so use LaunchKernels@32; $\endgroup$ – Kimou May 27 '18 at 13:20
  • $\begingroup$ I also did run this on a 4 core machine with the counters maximum value changed to 2 and "LaunchKernels@8" to "LaunchKernels@4", it uses only 2 cores $\endgroup$ – user1740587 May 27 '18 at 13:20
  • $\begingroup$ on the 32 core one with LaunchKernels@32 it uses only 10 cores $\endgroup$ – user1740587 May 27 '18 at 13:22
  • $\begingroup$ look at your OS if it limits the use of cores $\endgroup$ – Kimou May 27 '18 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.