# More efficiently access iterator values in subkernels?

I'm looking for more efficient ways of observing values in loops in subkernels. For example, take a standard Do loop:

AbsoluteTiming[s = 0; Do[s += j, {j, 10^7}];] (* ~3.7s *)


One way to access the iterator's value would be Monitor, which adds a tiny bit of overhead:

AbsoluteTiming[i = 0; s = 0; Monitor[Do[s += j, {j, 10^7}], i = j]] (* ~3.9s *)


However you don't actually have a variable and you lose the observation when it completes, so that's not what we want.

The other obvious thing is using either an (atomic) increment or assignment, ok, but that about doubles the time:

AbsoluteTiming[i = 0; s = 0; Do[s += j; i++, {j, 10^7}]]   (* ~7.3s *)
AbsoluteTiming[i = 0; s = 0; Do[s += j; i = j, {j, 10^7}]] (* ~7.4s *)


You can't update once in a while either as that logic adds overhead:

AbsoluteTiming[i = 0; s = 0;
Do[s += j; If[Mod[j, 10000] == 0, i = j], {j, 10^7}]]  (* ~13.0s *)


Things become worse in parallel code:

LaunchKernels[4]; AbsoluteTiming[WaitAll @ Table[ParallelSubmit[{i},
s = 0; Do[s+=j, {j, 1, i}]; s], {i, Table[10^6.5, 8]}];] (* 4.72s *)


As soon as you maintain access to the iterator values everything melts down:

$$monitor = <|1->0,2->0,3->0,4->0|>; SetSharedVariable[$$monitor];
Dynamic[$$monitor, UpdateInterval->1] AbsoluteTiming[ WaitAll @ Table[ParallelSubmit[{i}, s = 0; Do[ AssociateTo[$$monitor,  (* iterator access massive slowdown *)
Mod[$$KernelID,$$KernelCount]+1 -> j];
s += j, {j, 1, i}]; s]
, {i, Table[10^6.5, 8]}];
] (* ...minutes *)


Are there any tricks/patterns to avoid slowdowns when you need to keep a handle on iterator values inside such looping constructs?

• "You can't update once in a while either, because duh, that logic adds overhead:" That's true, but in parallel computations, only the local overhead is increased. But the crucial one there is the global one and that is significantly decreased this way. – Henrik Schumacher Mar 5 '20 at 13:07
• Right, that's what I'm getting at, just updated the question. – M.R. Mar 5 '20 at 14:48

Dynamic[i]
$$$$
`