Here is a simplified example illustrating my issue;

G = GaussianMatrix[{1000, 1000}];

f[] := (
  n = RandomInteger[];
  n + Total[G, 2]

ParallelTable[f[], {2}] (* two kernels *)

During the evaluation one needs two times the memory which is used for Table. The variable G is duplicated for each subkernel although G remains unchanged.

How to avoid that?

  • $\begingroup$ Welcome to Mathematica.SE! I suggest that: 1) You take the introductory Tour now! 2) When you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. Also, please remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign! 3) As you receive help, try to give it too, by answering questions in your area of expertise. $\endgroup$ – bbgodfrey Feb 12 '15 at 19:06
  • $\begingroup$ Can you explain in more detail why you need the same data to be transferred to each kernel? In this simplied example the Total could be computed on the main kernel only once and only the result needs to be transferred. I'm sure that your real problem is not that simple, but can you explain in more detail what sort of computation you are doing and precisely what data it needs? $\endgroup$ – Szabolcs Feb 12 '15 at 19:18
  • $\begingroup$ I'm doing a particle tracking on a given vector field G (30gb memory) each subkernel computes one particle path with random initial position in this field G. I want to avoid G being copied since it is not necessary. $\endgroup$ – Leon Feb 12 '15 at 19:24
  • $\begingroup$ @Leon When you respond to comments, please address people using the @ character, as I'm doing here. Otherwise they don't get notified about the response and may never return to check. $\endgroup$ – Szabolcs Feb 12 '15 at 19:58
  • 1
    $\begingroup$ @Leon Mathematica's parallel tools are limited and it is not possible to have even read access to the same data structure without duplicating it across kernels. What you can do is create a function that retrieves only the information you need, and use SetSharedFunction on it. This function will always be evaluated on the main kernel. Calling it will be slow. However, you will avoid data duplication. If your algorithm is such that it doesn't spend most of its time in this function, this approach might work without a slowdown that is too significant. ... $\endgroup$ – Szabolcs Feb 12 '15 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.