How to avoid duplicating the same data on the master kernel into each subkernel?

Question

I have found the same question

Avoid copying memory for each subkernel

However, there was no proper answer. This question was posted in 2015. Now it has been 6 years. So, I wish to find the solution to this problem in Mathematica.

Now I am performing calculations using the command ParallelTable. Sort of the code is as follows.

<<Analytic_H.txt;
B1 = 210;

Do[Ham[p2, p1] = Ham[p1, p2], {p1, 1, B1}, {p2, p1, B1}];
T = Table[Ham[p1, p2], {p1, 1, B1}, {p2, 1, B1}];
AA = T;

HH = ParallelTable[AA; {Min[Eigenvalues[SetPrecision[AA, 15]]], a1, a2, a3}, {a1, 2.6, 3.1, 0.1}, {a2, 4.8, 5.8, 0.2}, {a3, 4.0, 4.4, 0.1}];

As can be seen in the above code, data file "Analytic_H.txt" is loaded first. At this stage, the memory use is about 20 GB. The problem arises when performing ParallelTable part.

Performing ParallelTable, the subkernels are automatically opened and duplicating the data begins in each subkernel. So, if there are 5 subkernels open, then the total memory usage becomes 5 times the memory used in the master kernel (i.e., 5*20 GB = 100 GB).

I wish to avoid this duplication of the memory. Is there any way to share the data in the master kernel without duplicating the data into each subkernel?

Thank You

Answer 1

I don't yet have enough reputation to post this as a comment -- but try to run both DistributeDefinitions and LinkCreate with your data. The first allows you to ensure you have the definitions of your functions and data etc in each of your subkernels. The second is how to efficiently work with data in memory during parallel operations. You haven't posted a reproducible example so I can't do more at this point.

For more on LinkCreate, please see the details in the second answer at:

Why is ParallelTable slower than Table?

And further details in the documentation here:

https://reference.wolframcloud.com/language/tutorial/UsingWSTPToCommunicateBetweenWolframSystemSessions.html

https://reference.wolframcloud.com/language/ref/c/WSLinkName.html

Answer 2

Any data that the subkernels operate on must obviously be copied to the subkernels. This means that if you have a list data, and four subkernels, then in order to do something like ParallelMap[f, data], at least a quarter of data must be copied to each subkernel, thus the computation will use at least twice as much memory as data takes.

Additionally, the definition of f must be copied to subkernels. In turn, every symbols that is used in the definition of f must also be copied.

If f refers to symbols whose definition takes a lot of memory, they will be copied. In order to save memory, avoid unnecessarily referring to large data in f.

Notice that ParallelMap[f, data] does the same computation as ParallelTable[f[ data[[i]] ], {i, Length[data]}], but there is a crucial difference: the first argument of ParallelTable now refers to data. All symbols from the first argument, as well as their dependencies, must be copied to subkernels in order for the computation to work. The system can't analyse the expression f[ data[[i]] ] and automatically figure out that it only accesses part of data. That is much too complicated a task in order to do reliably.

---

Coming to your specific example: structure the computation in such a way that it does not refer to AA as a whole, only to parts of it. In other words, AA must not appear in the first argument of a ParallelTable / ParallelMap / etc. It may appear in the second argument of a ParalellTable, in which case only as much of it will be copied to subkernels as necessary.

If your computation cannot be structurds in this way, then it is not possible to avoid duplicating AA. This is a limitation not of Mathematica specifically, but of distributed memory parallelism.

Answer 3

@Szabolcs

Thank you for your advice. It is very helpful.

Actually, when I mentioned data, it does not really mean data. In the loaded file Analytic_H.txt, there are expressions of each element of a matrix which I need to diagonalize. Each element is called Ham[p1,p2] and the matrix is called AA=T. Here, each expression of the matrix elements is of a function of a1, a2, a3.

In the part of ParallelTable, I am performing the diagonalization of the matrix for each set of the parameter values {a1, a2, a3}. The whole single matrix takes 20 GB of memory. That is why the memory usage becomes nearly 100 GB during this step.

Now that I have seen your answer, it seems that I have to calculate first the elements after separating them into several pieces and putting each of the pieces into each subkernel.

I will try to do this and ask again if it fails or another problem arises.

Please comment to me if you have any idea or advice. Thank you very much.