# Parallelizing nested for loops

I'm currently computing nested for loops.

I count up a few counting variables and compute a value for each set of counting variables, then that value is added to an array. It is a way of adding up the contributions of each of these counting variables.

It is something like this simplified code:

For[h = 1, h < hmax, h++, count1 = ((200/(hmax))*h);

For[j = 1, j < jmax, j++, count2 = ((100/(jmax))*j) ;

Value = ((user_defined_function1[count1-count2])^(2))*user_defined_function2[count2];

IArray =ReplacePart[IArray, index -> (IArray[[index]] + Value)  ]

]]


The issue is that as my calculation has gotten more precise, I have found that I need many more counting variables. My new code has 6 nested for loops, instead of the 2 shown in this example. The code is taking too long.

I want to parallelize the calculation, since I have access to a machine with many slow cpu cores.

My first attempt will be replacing the For[] with Paralleldo[]. I read that Paralleldo[] has some complications if you call functions inside the Paralleldo[]. I don't think I understood the complication completely, so I am asking if there will be a problem if I proceed with replacing the for loops with Paralleldo[].

I think I can also reformulate the problem using ParallelTable[], but then the result will be in a different format and I will have to figure out how to extract the meaningful result. This extra work would not be ideal..

Can I use Paralleldo[] for this application? Are there problems? Are there better alternatives?

Thanks!

• In case you missed it: mathematica.stackexchange.com/questions/134609/… Apr 16, 2019 at 16:06
• Please format the post: put code in code blocks. Apr 16, 2019 at 16:24
• Thanks for the info High Performance Mark! I will read up on it. Apr 16, 2019 at 16:24
• "Can a single list be edited/updated within ParallelTable[]" No, it cannot. In Mathematica, the parallel threads do not share any memory, so they cannot modify the same list. More precisely, accessing the same variable is possible with SetSharedVariable, but it comes with compromises. Every access will require a callback to the main kernel. If your "user defined functions" are very slow, then this might not be a big drawback. If not, then this will kill the performance. Apr 16, 2019 at 20:41
• The best approach is to formulate the algorithm in such a way that you do not need to modify the same variable from different parallel threads. If a current iteration of the loop does not need to access data produces in previous iterations, then this is always possible. For your code, I suggest collecting index -> Value pairs from the Table, then combining them at the very end. You could use GroupBy to collect values for identical indices, sum them up, then use a single call to ReplacePart (or better, use SparseArray) Apr 16, 2019 at 20:44

Looking at your code, it has a similar structure to this small example:

arr = ConstantArray[0, 5];
Do[
Module[{index, value},
index = Mod[i, 5, 1];
value = i^2;
arr = ReplacePart[arr, index -> arr[[index]] + value]
],
{i, 100}
]
arr


Here each iteration of the loop modifies the variable arr. The key to effective parallelization in Mathematica is not to modify the same value from different parallel threads.

The code could be restructure like so:

(* generate index -> value pairs *)
rules = Table[Mod[i, 5, 1] -> i^2, {i, 100}];

(* combine them *)
Normal@SparseArray[
Normal[Total /@ GroupBy[rules, First -> Last]],
{5}
]


The idea is to first generate the index -> value pairs in a Table that can be easily changed to a ParallelTable, then combine them only when the parallel computation has finished.

ParallelCombine follows this principle, though in practice I often use ParallelMap/ParallelTable, then combine manually.

P.S. A slightly simpler, but undocumented way to combine the results is

SetSystemOptions["SparseArrayOptions" -> "TreatRepeatedEntries" -> Total];
Normal@SparseArray[rules]
SetSystemOptions["SparseArrayOptions" -> "TreatRepeatedEntries" -> First];


It is very important that you don't forget to set this back to the default First, otherwise things may break.

• thanks for the in depth answer. Before you wrote that, I think I came up with something similar. Is it correct that the main idea for this process is to use parallel processing to calculate the index->value pairs and then afterwards to perform the ReplacePart[] function separately (not using parallel operations)? Apr 17, 2019 at 17:31