Here is a simplified example illustrating my issue;

G = GaussianMatrix[{1000, 1000}];

f[] := (
  n = RandomInteger[];
  n + Total[G, 2]

ParallelTable[f[], {2}] (* two kernels *)

During the evaluation one needs two times the memory which is used for Table. The variable G is duplicated for each subkernel although G remains unchanged.

How to avoid that?

  • $\begingroup$ Welcome to Mathematica.SE! I suggest that: 1) You take the introductory Tour now! 2) When you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. Also, please remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign! 3) As you receive help, try to give it too, by answering questions in your area of expertise. $\endgroup$
    – bbgodfrey
    Commented Feb 12, 2015 at 19:06
  • $\begingroup$ Can you explain in more detail why you need the same data to be transferred to each kernel? In this simplied example the Total could be computed on the main kernel only once and only the result needs to be transferred. I'm sure that your real problem is not that simple, but can you explain in more detail what sort of computation you are doing and precisely what data it needs? $\endgroup$
    – Szabolcs
    Commented Feb 12, 2015 at 19:18
  • $\begingroup$ I'm doing a particle tracking on a given vector field G (30gb memory) each subkernel computes one particle path with random initial position in this field G. I want to avoid G being copied since it is not necessary. $\endgroup$
    – Leon
    Commented Feb 12, 2015 at 19:24
  • $\begingroup$ @Leon When you respond to comments, please address people using the @ character, as I'm doing here. Otherwise they don't get notified about the response and may never return to check. $\endgroup$
    – Szabolcs
    Commented Feb 12, 2015 at 19:58
  • 1
    $\begingroup$ @Leon Mathematica's parallel tools are limited and it is not possible to have even read access to the same data structure without duplicating it across kernels. What you can do is create a function that retrieves only the information you need, and use SetSharedFunction on it. This function will always be evaluated on the main kernel. Calling it will be slow. However, you will avoid data duplication. If your algorithm is such that it doesn't spend most of its time in this function, this approach might work without a slowdown that is too significant. ... $\endgroup$
    – Szabolcs
    Commented Feb 12, 2015 at 20:02


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.