How to parallelize the most outer loop only?

Question

I have two Doloops. The outer loop iterate over the inner loop. The inner loop contains a set of step by step instructions which each next step depends on the previous one so the inner loop must not be parallelized. However, the parallelization of the outer loop is desirable. How can I just parallelize the outer loop only? Does the code below is parallelized in both loops? If the code below is just parallelized on the outer loop only, then the other problem is that without using SetSharedVariable[a, b] I can't get the values of a and b and if I use it it runs much more slowly than the unparallelized version.

SetSharedVariable[a, b]
AbsoluteTiming[
 ParallelDo[
  {
   a = Table[i + j, {i, 1.0, 100.0}, {j, 1.0, 100.0}];
   Do[
    {
     b = a.a;
     c = If[Tr[b] > 2000, b = a, Break[]];
     }
    , {iterator1, 1, 500}];
   }, {iterator2, 1, 500}]]

Edit

Consider the code below :

a = Table[Null, {i, 1, 1000}];
b = Table[Null, {i, 1, 10}];
SetSharedVariable[a, b]
ParallelDo[Do[a[[iterator1]] = RandomReal[], {iterator1, 1, 1000}];
 b[[iterator2]] = a;, {iterator2, 1, 10}]

Now, a and b are outside of the loops and they are different for each iteration of outer loop. Indeed, b gets bigger and bigger with each outer loop iteration. Without SetSharedVariable after execution the code, a and b aren't accessible and if I use it, it is incredibly more slower than usual. How can I have a and b without losing speed?

Answer 1

I just cleaned up your code a little, added a Print command and lowered the maximum value of the iterators:

ParallelDo[
  a = Table[i + j, {i, 1.0, 100.0}, {j, 1.0, 100.0}];
  Do[b = a.a;
    Print["Kernel ", $KernelID, ", iterator1 ", iterator1, ", iterator2 ", iterator2];
    c = If[Tr[b] > 2000, b = a, Break[]],
    {iterator1, 1, 2}
  ],
  {iterator2, 1, 8}]

Kernel 4, iterator1 1, iterator2 1
Kernel 3, iterator1 1, iterator2 3
Kernel 2, iterator1 1, iterator2 5
Kernel 1, iterator1 1, iterator2 7
Kernel 4, iterator1 2, iterator2 1
Kernel 3, iterator1 2, iterator2 3
Kernel 2, iterator1 2, iterator2 5
Kernel 1, iterator1 2, iterator2 7
Kernel 4, iterator1 1, iterator2 2
Kernel 3, iterator1 1, iterator2 4
Kernel 2, iterator1 1, iterator2 6
Kernel 1, iterator1 1, iterator2 8
Kernel 4, iterator1 2, iterator2 2
Kernel 3, iterator1 2, iterator2 4
Kernel 2, iterator1 2, iterator2 6
Kernel 1, iterator1 2, iterator2 8

From this output you can see that the inner loop is not parallelized.
You can add other values (like a and b) to Print to explore your code further.
Using SetSharedVariable[a, b] is not useful for this code example and is only needed, if the values of these variables have to be synchronized among all parallel kernels all the time. It should not be used if each kernel should, e.g., run the inner loop with a different value for a and b. If the value for a is not different for each kernel (as it is the case in this example), than it's value should be calculated only once outside of ParallelDo.
Also note the fact, that no global value for, e.g. a is set after executing the code.

Edit

In the example you added in your edit you encounter a situation that is related to my last note. The problem is not, that a and b are unknown to the kernels. The problem is, that the b[[iterator2]] the values are assigned to are localized to the kernels.
You can just share b:

SetSharedVariable[b]

ParallelDo[Do[a[[iterator1]] = RandomReal[], {iterator1, 1, 1000}];
 b[[iterator2]] = a;, {iterator2, 1, 10}]

This gives a ~27-fold speedup on my PC compare to using SetSharedVariable[a, b].
Or you can use

b = ParallelTable[Do[a[[iterator1]] = RandomReal[], {iterator1, 1, 1000}];
      a, {iterator2, 1, 10}];

without using SetSharedVariable.
Or you can implement the code with some other function, that is more suitable than ParallelDo for the task at hand.

But you have to be aware of the fact, that there is a difference in what the code actually does if you use SetSharedVariable[b] instead of SetSharedVariable[a, b]!
Compare the output of

SetSharedVariable[a, b]
ParallelDo[Do[a[[iterator1]] = $KernelID, {iterator1, 1, 1000}]; 
  b[[iterator2]] = a;, {iterator2, 1, 10}]
b

with the output of

UnsetShared[a, b]
SetSharedVariable[b]
ParallelDo[Do[a[[iterator1]] = $KernelID, {iterator1, 1, 1000}];
  b[[iterator2]] = a;, {iterator2, 1, 10}]