Memory leakage in table construction

Question

UPDATED WITH CODE:

So it seems the memory leak is not to do with parallelization as it occurs even without using ParallelTable[]. Here's a dataset,chicagoData.wl, that I've created and the code that I'm running on it.

Get["chicagoData.wl"];
$HistoryLength = 0;

scoringMC[featureList_, tractPolysBD_, pop_, divisor_, 
   method_: "centroid"] := Module[{nf, sample, score},
   nf = Nearest[featureList -> "Distance"];
   sample = 
    Flatten[Table[
      RandomPoint[tractPolysBD[[i]], Ceiling[pop[[i]]/divisor]], {i, 
       1, Length[pop]}], 1];
   If[method == "median",
    score = Mean[nf /@ sample];,
    score = Sqrt[Mean[#^2 & /@ (nf /@ sample)]];];
   Return[score]];
scoringMC::usage = 
  "scoringMC[featureList_,tractPolysBD_,pop_,divisor_,method_]";

randomScoringMC[featureList_, regionBD_, tractPolysBD_, pop_, 
  divisor_, sampleSize_, method_: "centroid"] := 
 Monitor[Module[{nf, sample, randomFeatureList, scores, score},
    scores = Table[
      randomFeatureList = RandomPoint[regionBD, Length[featureList]];
      scoringMC[randomFeatureList, tractPolysBD, pop, divisor, 
        method][[1]]
      , {iteration, 1, sampleSize}];
    score = Mean[scores];
    Return[score]];, iteration]
randomScoringMC::usage = 
  "randomScoringMC[featureList_,regionBD_,tractPolysBD_,pop_,divisor_,\
sampleSize_,method_:\"centroid\"]";

Monitor[randomScoringMC[chicagoMSASupermarkets, chicagoMSABDStereo, 
  chicagoMSAStereoBD, chicagoMSAPop2015, 10, 1000, "median"], 
 MemoryInUse[]]

Bizarrely, when I monitor MemoryInUse[] it seems constant. Yet when I watch the memory used by the kernel in Windows Task Manager, I see it slowly goes up (with some cycling). And if I run several of these computations in a row, my computer runs out of memory and the kernel crashes.

Note: If we are able to work out the issue, I'll create a MWE of it and pull my own data file off of this question, as obviously all of this is far from a MWE!

ORIGINAL QUESTION:

I'm consistently having issues where I run out memory in a parallel computation but I am having trouble diagnosing the issue and/or creating a MWE. Here is a rough description of what I am doing:

I have a large polygon composed of about two thousand smaller polygons. All of the polygons are fairly complex in that they have a large number of vertices. The code is as follows:

1. Draw ~200 points at random from the large polygon with RandomPoint[].
2. Create a Nearest[] function for that set of points.
3. Draw a million points in total, with a fixed proportion of those million from each smaller polygon (again using RandomPoint[] on each smaller polygon), with the proportion depending on the small polygon's size and an exogenous weight.
4. Apply the Nearest[] function to get a distance to nearest point (of the 200) to each of the million points and average those distances.

Steps 3 and 4 are in a module that outputs just the average (one number). Now, I want this average for many different random assignments (at least 100) of the 200 points, so I nest all of the above in a ParallelTable[] in which each item in the outputted list is an average.

I can run through about 35 of these before my 8GB of ram is exhausted by 8 kernels. I think this computation should be feasible--if I can run through 8 of them, the kernels should be able to simply forget everything for the next set of 8 iterations. But they don't--the memory piles up iteration after iteration until it is exhausted. Any ideas on what the issue might be? I'm aware of issues with $HistoryLength but not sure exactly how to pass a remedy to the kernels, nor whether this is the issue in the first place. I could probably close kernels and reopen new ones after each set of 8, but surely there's a better way to clear each kernel's memory (or simply prevent it from accumulating). I'm running 11.1.

Since the only particularly "big" thing I'm doing is random number generation, my guess is that the kernels are storing the generated random numbers from old iterations, but I would have thought that this would not be the case given that steps 3 and 4 are in a module that returns just one scalar. Any advice?

Answer 1

For your $HistoryLength question, you may use either ParallelEvaluate or InitializationValue.

ParallelEvaluate

For version 11.1.1 and earlier:

LaunchKernels[]
ParallelEvaluate[$HistoryLength = 0]

The above will set $HistoryLength to 0 on all launched kernels.

InitializationValue

Starting with version 11.2 you can setup an InitializationValue on $HistoryLength for the "Subkernel" EvaluationEnvironment such that whenever a new subkernel is initialised $HistoryLength will be set to 0.

First close kernels created above since they have already been initialised.

CloseKernels[]

Then setup the InitializationValue.

InitializationValue[$HistoryLength, "Local", EvaluationEnvironment -> "Subkernel"] = 0;

Now any subkernels will have this value set at initialisation.

LaunchKernels[];
ParallelEvaluate[$HistoryLength]

{0, 0, 0, 0}

The InitializationValue can be removed with

DeleteObject@
 Values@InitializationObjects[$HistoryLength, "Local", 
   EvaluationEnvironment -> "Subkernel"]

Hope this helps.