# How to avoid duplicating the same data on the master kernel into each subkernel?

I have found the same question

Avoid copying memory for each subkernel

However, there was no proper answer. This question was posted in 2015. Now it has been 6 years. So, I wish to find the solution to this problem in Mathematica.

Now I am performing calculations using the command ParallelTable. Sort of the code is as follows.

<<Analytic_H.txt;
B1 = 210;

Do[Ham[p2, p1] = Ham[p1, p2], {p1, 1, B1}, {p2, p1, B1}];
T = Table[Ham[p1, p2], {p1, 1, B1}, {p2, 1, B1}];
AA = T;

HH = ParallelTable[AA; {Min[Eigenvalues[SetPrecision[AA, 15]]], a1, a2, a3}, {a1, 2.6, 3.1, 0.1}, {a2, 4.8, 5.8, 0.2}, {a3, 4.0, 4.4, 0.1}];


As can be seen in the above code, data file "Analytic_H.txt" is loaded first. At this stage, the memory use is about 20 GB. The problem arises when performing ParallelTable part.

Performing ParallelTable, the subkernels are automatically opened and duplicating the data begins in each subkernel. So, if there are 5 subkernels open, then the total memory usage becomes 5 times the memory used in the master kernel (i.e., 5*20 GB = 100 GB).

I wish to avoid this duplication of the memory. Is there any way to share the data in the master kernel without duplicating the data into each subkernel?

Thank You

• I believe if you use Import to import the data as say "myData", then use SharedVariable[myData], the kernels will share a single copy. However, I'm not sure abut this. I would try that first and see if the memory remains at 20 GB. Mar 22, 2021 at 12:30
• @Dominic SetSharedVariable[data] causes data to be always evaluated on the main kernel. data will still be copied over to subkernels, but not just once, but every single time it is accessed. Never use SetSharedVariable unless you fully understand the consequences. Probably a quarter of parlallelization questions here stem from a misuse of SetSharedVariable/Function. Mar 22, 2021 at 12:44
• Ok, thanks for that. Mar 22, 2021 at 12:47

I don't yet have enough reputation to post this as a comment -- but try to run both DistributeDefinitions and LinkCreate with your data. The first allows you to ensure you have the definitions of your functions and data etc in each of your subkernels. The second is how to efficiently work with data in memory during parallel operations. You haven't posted a reproducible example so I can't do more at this point.

Why is ParallelTable slower than Table?

And further details in the documentation here:

https://reference.wolframcloud.com/language/tutorial/UsingWSTPToCommunicateBetweenWolframSystemSessions.html

Any data that the subkernels operate on must obviously be copied to the subkernels. This means that if you have a list data, and four subkernels, then in order to do something like ParallelMap[f, data], at least a quarter of data must be copied to each subkernel, thus the computation will use at least twice as much memory as data takes.

Additionally, the definition of f must be copied to subkernels. In turn, every symbols that is used in the definition of f must also be copied.

If f refers to symbols whose definition takes a lot of memory, they will be copied. In order to save memory, avoid unnecessarily referring to large data in f.

Notice that ParallelMap[f, data] does the same computation as ParallelTable[f[ data[[i]] ], {i, Length[data]}], but there is a crucial difference: the first argument of ParallelTable now refers to data. All symbols from the first argument, as well as their dependencies, must be copied to subkernels in order for the computation to work. The system can't analyse the expression f[ data[[i]] ] and automatically figure out that it only accesses part of data. That is much too complicated a task in order to do reliably.

Coming to your specific example: structure the computation in such a way that it does not refer to AA as a whole, only to parts of it. In other words, AA must not appear in the first argument of a ParallelTable / ParallelMap / etc. It may appear in the second argument of a ParalellTable, in which case only as much of it will be copied to subkernels as necessary.

If your computation cannot be structurds in this way, then it is not possible to avoid duplicating AA. This is a limitation not of Mathematica specifically, but of distributed memory parallelism.

@Szabolcs

Actually, when I mentioned data, it does not really mean data. In the loaded file Analytic_H.txt, there are expressions of each element of a matrix which I need to diagonalize. Each element is called Ham[p1,p2] and the matrix is called AA=T. Here, each expression of the matrix elements is of a function of a1, a2, a3.

In the part of ParallelTable, I am performing the diagonalization of the matrix for each set of the parameter values {a1, a2, a3}. The whole single matrix takes 20 GB of memory. That is why the memory usage becomes nearly 100 GB during this step.

Now that I have seen your answer, it seems that I have to calculate first the elements after separating them into several pieces and putting each of the pieces into each subkernel.

I will try to do this and ask again if it fails or another problem arises.

Please comment to me if you have any idea or advice. Thank you very much.

• @Szabolcs Thank you for your advice. I don't understand but I couldn't reply to your answer as a comment. So I just wrote this as an answer to my question. Please see this and reply to me. Thanks a lot. Have a nice day. Mar 23, 2021 at 7:32
• You may have duplicated accounts: see here to merge them. The information in this answer should be added as an edit to your question instead. Mar 23, 2021 at 10:45
• @MarcoB Yes, the accounts seem to be duplicated but I couldn't log in with the first account. When I re-logged in few hours later after I made an account at first, I found a message on the top of the page. It said that I didn't finish something related to the registration. So I clicked that message and finished the steps followed by the message. After that, this situation happened. So, I didn't make the second account with another e-mail address and thus I cannot logged-in with both accounts simultaneously. Is there any other way to merge them? Mar 24, 2021 at 0:25