2
$\begingroup$

Following up on a previous question in which I encountered a memory leak in the JSON importer I have created a SandBox script that will run the Import and analysis on a separate kernel, which be terminated when the analysis is complete.

The problem I have encountered is that the memory growth does not occur on the new kernel, but instead happens on one of the main (?) kernels opened when Mathematica is launched. So, my question is whether someone can explain why the SandBox method below does not contain the memory leak associated with importing JSON files... even better would be a suggestion for another work-around.

The code that I use is as follows: EDIT: Code has been changed to include the crucial missing line.

SandBox[computation_, arguments___] := 
  Block[{result, kernel},
   kernel = LaunchKernels[1][[1]];
   DistributeDefinitions[computation,arguments];(*EDIT: Adding this line solves the issue - see comment from Szabolcs below*)
   result = ParallelEvaluate[computation[arguments], kernel];
   CloseKernels[kernel];
   Return[result];
   ];
CachedComputation[cachePath_String, computation_, arguments___] :=
  Block[{results,kernel},
   If[FileExistsQ[cachePath],
    Print["Loaded: ", cachePath];
    Return[Get[cachePath]];
    ];
   results = computation[arguments];
   Put[results, cachePath];
   Print["Created: ", cachePath];
   Return[results];
   ];
Table[CachedComputation[
  NotebookDirectory[] <> "test" <> ToString[t],
  SandBox[analysis, #] &,
  filepaths[[p]]
  ], {p, 1, Length[filepaths]}]

The analysis method imports a file specified by file paths[[p]]. The cached computation method ensures that when the memory leak from importing causes a crash I can resume analysis. The SandBox method was a suggestion from Nigel Hess of Wolfram for dealing with leaky code.

$\endgroup$

closed as off-topic by MarcoB, halirutan, m_goldberg, Henrik Schumacher, Sektor May 30 '18 at 8:39

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question arises due to a simple mistake such as a trivial syntax error, incorrect capitalization, spelling mistake, or other typographical error and is unlikely to help any future visitors, or else it is easily found in the documentation." – MarcoB, halirutan, m_goldberg, Henrik Schumacher, Sektor
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ I haven't read carefully yet, but on a first glance the most likely cause might be that definitions (of analysis, etc.) are not distributed to the subkernel. Look up DistributeDefinitions. With most high level parallelization functions (ParallelTable, etc.), this is done automatically. A notable exception is ParallelEvaluate, which does not auto-distribute. I guess that's because it's one of those functions meant to give lower-level control. $\endgroup$ – Szabolcs Aug 11 '15 at 6:47
  • $\begingroup$ Szabolcs, you are absolutely right - DistributeDefintions was missing. Adding it above (and also declaring ParallelNeeds for custom packages) ensures that the Import operation runs on the sub-kernel... thereby bottling the memory leak in my previous question. THANK YOU! $\endgroup$ – AngelGabriel Aug 12 '15 at 5:21