5
$\begingroup$

Short question: what is the proper way to clear a variable on a given kernel ? AND how can I ensure that the memory will indeed be freed subsequently ?

Full question. Let me elaborate this question with the following experiment in a fresh session.

kernel = ParallelEvaluate[$KernelID] (* launch sub kernels *)
bytesToMo[b_] := Round[b/1024.^2, .1];
memoryStatus := TableForm[Join[
    {{"FrontEnd", bytesToMo@MemoryInUse[$FrontEnd]},
    {$KernelID, bytesToMo@MemoryInUse[]}},
ParallelEvaluate[{$KernelID, bytesToMo@MemoryInUse[]}]],
TableHeadings -> {None, {"Kernel", "Memory in Mo"}}
];
Row[{
   memoryStatus, (* initial state *)
   "\t\[Rule]\t",
   ParallelEvaluate[
      If[$KernelID == kernel[[1]],x = Range[10^6]];
      If[$KernelID == kernel[[2]],y = Range[10^7]];
   ];
   memoryStatus
}]

As expected, x exists only on the first kernel (the first kernel memory increases) and y exists only on the second kernel (the second kernel memory increases more, as y is larger). One can check it for example like this:

checkXY := TableForm@Join[{{$KernelID, ReleaseHold[#]}}, 
   ParallelEvaluate[{$KernelID, ReleaseHold[#]}]] &[Hold[
        Length[x], Length[y]
   ]];
checkXY

Now to clear x and y, I parallelize a call to Clear:

ParallelEvaluate[Clear[x, y]]; Column[{checkXY, memoryStatus}]

Here is the overall output (sorry, as this is my first post, they don't let me put more than 2 images so I had to collect them in one):

output when it's ok

Well, it works right ? The variables have been cleared and the memory is free... Well, not quite. Because If I run the whole thing one more time, the variables will be cleared, but the memory will not be freed anymore.

Here is the overall output of the second run:

output with memory not freed at the end

The only solution I found is to share the unset symbol from the Master Kernel, which then effectively erases the memory in all kernels.

Clear[x, y]
SetSharedVariable[x, y];
UnsetShared[x, y];
memoryStatus

My questions are:

  1. Why is Clear[x,y] working the first time, but not subsequently ?
  2. If Clear is not the right way, what is the proper way to unset a symbol in a given kernel ? The SetShared/Unset hack does not sound right to me (what if each kernel has a different x but I only want to get rid of a specific one)...

Thank you.

$\endgroup$
  • $\begingroup$ Sorry, I didn't read through your question, but this might help: since v10.4 for some reason ParallelEvaluate distributes definitions automatically. Use ParallelEvaluate[..., DistributedContexts -> None] to prevent this. IMO making ParallelEvaluate behave this way was a bad decision. $\endgroup$ – Szabolcs Nov 5 '16 at 15:07
  • $\begingroup$ @Szabolcs: thanks for the info, but it does not seem to fix the problem. To ask my question differently: if you have a variable x defined on a subkernel, how would you proceed to clean it ? $\endgroup$ – Francois Vigneron Nov 5 '16 at 16:58
2
$\begingroup$

I am answering the short form of your question, as you stated it in comments as well. I did not fully follow the rest of your question, so perhaps if this does not apply exactly, you could edit the question to present a minimal example.


You can operate on a variable on a specific subkernel by using ParallelEvaluate together with that kernel's identifier.

For instance, let's load some parallel kernels, then create a symbol x on each of them and assign it a value:

LaunchKernels[]
(* Out: {"KernelObject"[1, "local"], "KernelObject"[2, "local"]} *)

ParallelEvaluate[x = 3]
(* Out: {3, 3} *)

Now let's clear x on the first subkernel only:

ParallelEvaluate[Clear[x], First@Kernels[]]

We can check that the value of x has been cleared on the first subkernel only, but is retained on the second one:

ParallelEvaluate[x]
(* Out: {x, 3} *)
$\endgroup$
  • $\begingroup$ thanks for your answer which recalls the proper syntax to clear a variable in a given subkernel. However, the point of my question is that it does not seem to reliably free the memory associated to that variable. When the objects start to represent 25% of the available RAM, an unreliable garbage collection after a clear is problematic. $\endgroup$ – Francois Vigneron Mar 9 '17 at 15:21
  • $\begingroup$ I just edited the short question by adding "How can I ensure that the memory will indeed be freed subsequently ?" $\endgroup$ – Francois Vigneron Mar 9 '17 at 15:32
  • $\begingroup$ @FrancoisVigneron I agree that it might be problematic, but I would caution you against changing the question mid-way; such behavior ("moving target questions") discourages answers, because one never know whether one might be wasting time answering the wrong question. You might be better off asking a new question entirely on that topic. $\endgroup$ – MarcoB Mar 9 '17 at 15:34
  • $\begingroup$ Thanks MarcoB. The precision of your answer made me clarify my own question. I validate your answer. I'll be posting a new one on the internal details of garbage collection. $\endgroup$ – Francois Vigneron Mar 12 '17 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.