# Add Results of ParallelMap in Parallel

I'm new to coding in parallel, but I really could use it. I need a command to call a function on each element in a huge list, and then add them all up. Currently I use something like this:

Apply[Plus, ParallelMap[f,Huge_List]]


This is faster than just using Map, but it quickly is maxing out my computer's memory. Since each evaluation on the huge list is independent and since the results are simply added, I was wondering if there was a way to get Mathematica to evaluate the sum as it is evaluating the mapping. The idea is for Mathematica to not have to keep the entire result of ParallelMap in memory and to instead just keep the running total as it is evaluating the mapping. Is there a way to do this?

• Look up ParallelCombine. Or break the huge list into parts of appropriate size, then let each subkernel first map f onto a part, then sum it, then return only the result of the summation. Finally sum the partial totals. – Szabolcs Jul 8 '15 at 13:21
• Just saw this dask.pydata.org/en/latest, which I think could also be useful for you. It's an out of core processing library, meaning you can process datasets larger than your physical memory. It's in Python though, but luckily for me comments can't be downvoted:) – Ajasja Aug 20 '15 at 8:14

ParallelCombine solution.

hugeList = Range[0, 2*10^4];
f[x_] := x

ParallelCombine[Total[Map[f, #, 1]] &, hugeList, Total[{#}] &]
(* 38503 *)


Chunks of hugeList will be handed to Total[Map[f, #, 1]] & in parallel. This Total first Maps f onto each element of the list handed to it (a chunk of hugeList) thereby giving a total for the chunk and not keeping the summands. The combine function Total[{#}] & takes all these chunk summands and sums them.

ParallelSum will get you some memory savings. I get a successful evaluation of the following:

In:= hugeList = Range[0, 6*10^3];
MemoryConstrained[
Apply[Plus, ParallelMap[N[Pi, 50]^#/#! &, hugeList]] - N[Pi, 50],
2^20
]
Out= 19.999099979189475767266442984669044496068936843225


But, taking the range up to 7e3 exceeds the 1MB constraint. However, I can go up to 2e4 using ParallelSum:

In:= hugeList = Range[0, 2*10^4];
MemoryConstrained[
ParallelSum[N[Pi, 50]^k/k!, {k, hugeList}] - N[Pi, 50],
2^20
]
Out= 19.999099979189475767266442984669044496068936843225


Unfortunately, the approximation to 20 doesn't get any better :)

If ParallelCombine got a better solution, perhaps you could explain in an answer.