What is the best way to parallelize MapAt? Only ParallelMap is provided, but seems bad when executing functions on a subset of a List. I only found 1 similar topic on SE where the subset was a whole level. (which it is not, in this case)

"minimal" working example:

I want to map the function f over {All, All, 1} of list l.

llength = 3
l = Table[ToString@a[x, y, z] Range@5, {x, llength}, {y, llength}, {z, llength}]
f[l_] := g /@ l

ParallelMap[Join[{f@#[[1]]}, #[[2 ;;]]] &, l, {2}] === MapAt[f, l, {All, All, 1}]

in this specific case a viable option might be using Map on level 2 and then using ParallelMap on element 1, mapping f to its elements. Is there a smarter way? This is not general for all index specifications.

  • $\begingroup$ I think the best method may be to first take part = Part[l, All, All, 1] and then use newPart = ParallelMap[f, part] to compute the values. You can use in-place modification: l[[All, All, 1]] = newPart. You can localize l if necessary. $\endgroup$ – Sjoerd Smit Jun 4 at 13:39
  • $\begingroup$ @SjoerdSmit do you mind writing this as an answer? Otherwise i would take the burden upon me. Do you have a idea about performance/memory cost? This seems very redundant/inefficient in terms of memory usage. $\endgroup$ – Gladaed Jun 4 at 14:17

The first (and simplest) thing having come to my mind is to MapAt tasks for parallel kernels and then to WaitAll them:

list = Array[h, {2, 3, 4}];
g[x_] := (N[Pi, 10^6] x);
   Composition[ParallelSubmit, g],
   {All, All, 1}
] === MapAt[g, list, {All, All, 1}]
  • $\begingroup$ as a function you could write it up like SetAttributes[parallelMapAt, HoldAll]; parallelMapAt[fun_, list_List, spec__] := Module[{}, DistributeDefinitions[fun]; WaitAll[MapAt[Composition[ParallelSubmit, fun], list, spec]]] correct? $\endgroup$ – Gladaed Jun 6 at 14:30

Here is a quick function that does mostly what you want:

 ] := Module[{
    copy = list
   copy[[spec]] = ParallelMap[fun, copy[[spec]]];

However, the position specification follows the format of Part rather than that of Position:

In[400]:= parallelMapAt[Sin, N @ Range[20], {4, 6, 8, 10}] ===  
  MapAt[Sin, N @ Range[20], List /@ {4, 6, 8, 10}]

Out[400]= True

Another example:

In[425]:= parallelMapAt[f, Array[# &, {5, 2}], All, 2]

Out[425]= {{1, f[1]}, {2, f[2]}, {3, f[3]}, {4, f[4]}, {5, f[5]}}

If you're worried that making a copy of your input is inefficient (frankly, I doubt that it will give you much trouble), you can use SetAttributes[parallelMapAt, HoldAll] and modify the input list itself:

SetAttributes[parallelMapAt, HoldAll]
   ] := (
   list[[spec]] = ParallelMap[fun, list[[spec]]]

list = N @ Range[20];
parallelMapAt[Sin, list, {4, 6, 8, 10}]

Out[420]= {-0.756802, -0.279415, 0.989358, -0.544021}

Out[421]= {1., 2., 3., -0.756802, 5., -0.279415, 7., 0.989358, 9., \ -0.544021, 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.}

Of course, you may want to elaborate the code to do some error checking before you end up ruining your data by Part-assigning corrupt data if the ParallelMap fails.


As pointed out in the comments, there is still some work to be done since

parallelMapAt[f, Array[h, {2, 3, 4}], All, All, 1] === MapAt[f, Array[h, {2, 3, 4}], {All, All, 1}]

is False. In this case, you'd need to add the levelspec {2} to ParallelMap to make it work correctly. I suspect that the level spec should be equal to the number of non-flattened levels (e.g., All or ;; 3) in spec, but I can't test that right now.

  • $\begingroup$ parallelMapAt[f, Array[h, {2, 3, 4}], All, All, 1] === MapAt[f, Array[h, {2, 3, 4}], {All, All, 1}] evaluates to False. $\endgroup$ – Anton.Sakovich Jun 5 at 7:33
  • $\begingroup$ @Anton.Sakovich Hmmm, yes. I see why. You have to give ParallelMap the right levelspec for this to work. Right now I don't have the time to figure out how to deduce the right levelspec programmatically. $\endgroup$ – Sjoerd Smit Jun 5 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.