6
$\begingroup$

Short question: what is the proper way to clear a variable on a given kernel ? AND how can I ensure that the memory will indeed be freed subsequently ?

Full question. Let me elaborate this question with the following experiment in a fresh session.

kernel = ParallelEvaluate[$KernelID] (* launch sub kernels *)
bytesToMo[b_] := Round[b/1024.^2, .1];
memoryStatus := TableForm[Join[
    {{"FrontEnd", bytesToMo@MemoryInUse[$FrontEnd]},
    {$KernelID, bytesToMo@MemoryInUse[]}},
ParallelEvaluate[{$KernelID, bytesToMo@MemoryInUse[]}]],
TableHeadings -> {None, {"Kernel", "Memory in Mo"}}
];
Row[{
   memoryStatus, (* initial state *)
   "\t\[Rule]\t",
   ParallelEvaluate[
      If[$KernelID == kernel[[1]],x = Range[10^6]];
      If[$KernelID == kernel[[2]],y = Range[10^7]];
   ];
   memoryStatus
}]

As expected, x exists only on the first kernel (the first kernel memory increases) and y exists only on the second kernel (the second kernel memory increases more, as y is larger). One can check it for example like this:

checkXY := TableForm@Join[{{$KernelID, ReleaseHold[#]}}, 
   ParallelEvaluate[{$KernelID, ReleaseHold[#]}]] &[Hold[
        Length[x], Length[y]
   ]];
checkXY

Now to clear x and y, I parallelize a call to Clear:

ParallelEvaluate[Clear[x, y]]; Column[{checkXY, memoryStatus}]

Here is the overall output (sorry, as this is my first post, they don't let me put more than 2 images so I had to collect them in one):

output when it's ok

Well, it works right ? The variables have been cleared and the memory is free... Well, not quite. Because If I run the whole thing one more time, the variables will be cleared, but the memory will not be freed anymore.

Here is the overall output of the second run:

output with memory not freed at the end

The only solution I found is to share the unset symbol from the Master Kernel, which then effectively erases the memory in all kernels.

Clear[x, y]
SetSharedVariable[x, y];
UnsetShared[x, y];
memoryStatus

My questions are:

  1. Why is Clear[x,y] working the first time, but not subsequently ?
  2. If Clear is not the right way, what is the proper way to unset a symbol in a given kernel ? The SetShared/Unset hack does not sound right to me (what if each kernel has a different x but I only want to get rid of a specific one)...

Thank you.

$\endgroup$
2
  • $\begingroup$ Sorry, I didn't read through your question, but this might help: since v10.4 for some reason ParallelEvaluate distributes definitions automatically. Use ParallelEvaluate[..., DistributedContexts -> None] to prevent this. IMO making ParallelEvaluate behave this way was a bad decision. $\endgroup$
    – Szabolcs
    Nov 5, 2016 at 15:07
  • $\begingroup$ @Szabolcs: thanks for the info, but it does not seem to fix the problem. To ask my question differently: if you have a variable x defined on a subkernel, how would you proceed to clean it ? $\endgroup$ Nov 5, 2016 at 16:58

1 Answer 1

3
$\begingroup$

I am answering the short form of your question, as you stated it in comments as well. I did not fully follow the rest of your question, so perhaps if this does not apply exactly, you could edit the question to present a minimal example.


You can operate on a variable on a specific subkernel by using ParallelEvaluate together with that kernel's identifier.

For instance, let's load some parallel kernels, then create a symbol x on each of them and assign it a value:

LaunchKernels[]
(* Out: {"KernelObject"[1, "local"], "KernelObject"[2, "local"]} *)

ParallelEvaluate[x = 3]
(* Out: {3, 3} *)

Now let's clear x on the first subkernel only:

ParallelEvaluate[Clear[x], First@Kernels[]]

We can check that the value of x has been cleared on the first subkernel only, but is retained on the second one:

ParallelEvaluate[x]
(* Out: {x, 3} *)
$\endgroup$
4
  • $\begingroup$ thanks for your answer which recalls the proper syntax to clear a variable in a given subkernel. However, the point of my question is that it does not seem to reliably free the memory associated to that variable. When the objects start to represent 25% of the available RAM, an unreliable garbage collection after a clear is problematic. $\endgroup$ Mar 9, 2017 at 15:21
  • $\begingroup$ I just edited the short question by adding "How can I ensure that the memory will indeed be freed subsequently ?" $\endgroup$ Mar 9, 2017 at 15:32
  • $\begingroup$ @FrancoisVigneron I agree that it might be problematic, but I would caution you against changing the question mid-way; such behavior ("moving target questions") discourages answers, because one never know whether one might be wasting time answering the wrong question. You might be better off asking a new question entirely on that topic. $\endgroup$
    – MarcoB
    Mar 9, 2017 at 15:34
  • $\begingroup$ Thanks MarcoB. The precision of your answer made me clarify my own question. I validate your answer. I'll be posting a new one on the internal details of garbage collection. $\endgroup$ Mar 12, 2017 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.