Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I believe AndersonDarlingTest and probably all other tests are keeping a copy of data somewhere and not freeing it:

Quit[]; (*Must be run in different cell*)


$HistoryLength = 0;
memdata = {{MemoryInUse[], MaxMemoryUsed[]}}~Join~Table[
     data = 
      RandomVariate[
       MultinormalDistribution[{0, 0}, {{1, 1/2}, {1/2, 2}}], 1000000];
     dist = MultinormalDistribution[{ux, uy}, {{a, c}, {c, b}}];
     AndersonDarlingTest[data, 
      dist, {"FittedDistributionParameters", "PValue"}];
     Clear@data; Clear@dist;
     ClearSystemCache[];
     {MemoryInUse[], MaxMemoryUsed[]}
     , {10}]/1024.^2;

ListPlot[Transpose@memdata, Joined -> True, 
 PlotLegends -> Placed[{"MemoryInUse", "MaxMemoryUsed"}, Top], 
 AxesLabel -> {"Iteration", "Memory (MB)"}]

example of memory leak

Questions:

  • Am I missing something obvious or is this a bug? Just want to make sure before I report it...
  • Is there a work around? I've been spelunking with the excellent spelunking package, but so far no luck...

Heavy symbols gives some candidates, but clearing the symbols in the loop gives me various errors.

heavySymbols["Statistics`*`"]

{"Statistics`GoodnessOfFitTestingDump`iHypothesisTestFunction", 
 "Statistics`StableEstimationDump`iLogPDF17", 
 "Statistics`StableEstimationDump`iPDF05b1"}

Using MMA 9.0.1 on Win7 64 bit.

share|improve this question
    
I can confirm that this happens to me too (but then, I run MMA on a similar system). –  Sjoerd C. de Vries Jan 17 at 22:23
    
Same result on OS X. –  shrx Jan 17 at 22:26
3  
Not so darling ... –  belisarius Jan 17 at 23:23
1  
Looks like a bug. –  Andy Ross Jan 18 at 6:25
add comment

1 Answer

iHypothesisTestFunction seems to grow each iteration. But I have no idea, how to fix it. If I clear the whole object the tests stop working.

Example of memory abuse

$HistoryLength = 0;

Clear[$globalProperties];
    $globalProperties = {OwnValues, DownValues, SubValues, UpValues, 
   NValues, FormatValues, Options, DefaultValues, Attributes, 
   Messages};

ClearAll[getDefinitions];
SetAttributes[getDefinitions, HoldAllComplete];
getDefinitions[s_Symbol] := 
  Flatten@Through[
    Map[Function[prop, 
       Function[sym, prop[sym], HoldAll]], $globalProperties][
     Unevaluated[s]]];

ClearAll[symbolMemoryUsage];
symbolMemoryUsage[sname_String] := 
  ToExpression[sname, InputForm, 
   Function[s, ByteCount[getDefinitions[s]], HoldAllComplete]];

GetMem[] := {MemoryInUse[], MaxMemoryUsed[], 
   symbolMemoryUsage[
    "Statistics`GoodnessOfFitTestingDump`iHypothesisTestFunction"]};
memdata = {GetMem[]}~Join~
    Table[data = 
      RandomVariate[
       MultinormalDistribution[{0, 0}, {{1, 1/2}, {1/2, 2}}], 1000000];
     AndersonDarlingTest[data];
     Clear@data; Clear@dist;
     ClearSystemCache[];
     GetMem[], {10}]/1024.^2;

ListPlot[Transpose@memdata, Joined -> True, 
 PlotLegends -> 
  Placed[{"MemoryInUse", "MaxMemoryUsed", "iHypothesisTestFunction"}, 
   Top], AxesLabel -> {"Iteration", "Memory (MB)"}]

Aha, it seems iHypothesisTestFunction is doing some caching, and I have to selectively remove the downvalues...

Here is the fix. Don't have time to describe in detail how it works. (basically I'm just resetting the down values -- any suggestions welcome).

ClearAll@CleariHypothesisTestFunctionMemory;
CleariHypothesisTestFunctionMemory[] :=
  DownValues@
    Statistics`GoodnessOfFitTestingDump`iHypothesisTestFunction = 
   Uncompress@
    "1:eJztWktv4zYQtrOvPjZFu4cCfVwK9I1edlGgi/\
bkxnESYJ3Ekbd76SG0RNlEaFElqWT978sRrZclOxyt7UWBXgiTGc4M5/\
HNkMo3E3EVPux0OgqGV0zp8ABmH5vhKuG0TzlZ0CDsZounggeXRGsqI0v5oRmORBQwzcRy\
6Q8zeJpow4356vpEiCCiSl2EA6bH1CxG034yj6/Z6SIWekYVU7A8SCK/\
YJLp4z0zP8xGTYO+\
mUo2SYAoldtLtJgbOb7d8sQMS93UD446zHQQPoDNIKaqT59oUmf8syPjI8I5lWEH9j8yw5\
+cRDf3TFNhj80wJNqfjez8q6rAV2wiiVxcn477fxGeUMxJPTDqeBFT7ylsIvOY07E4Tzi3\
osAQHtXvy4sopx0UphP+TXhYlqZ+dGQUYIhjIpX6yZE4TFm7UgPxC/\
WdIzVxppw4mzQg2pk2pnP1vSPtXexMSqbOpDF1J3Xnqm7DxxkUHIl5LJIoOH4bS0ObB/\
QyRVARZgFmm3kMDE8uBt4HkM8AVTlwZ/qhIs9u/Qgwn8ac+LTHOe6ElWQvZlBCUkD7+/\
fBy+e/uUf4OhYv3UO/\
ag9EHuzOXQBXUFWeWxGg4FlY6NmLlrAGw4W0RffXZj2Mx31JNR0mXLNbIhnRtIysI5z/\
UlEvNot6HW1LEBz2XOhavetmTj+lJMCB+KdL05ZVax8B6UbXFGIx8Ze1+\
6mVdSmpzwAzUPALyXwWhSxielGZ7PMcePbbTBRoB3Lj1ZME+\
rDXkSJzOmqHxlDCbZT90qz1CdXlCLok0ggzfZ/CRXmnAoePXIsKogDvDqS+\
AB9k5z42NNCbiWhI9UyU8Mn4ZFMuo87yMMudHoce22y6NaTVDr9q0m59Bh1g2kZ79J+\
ERj7Ny4f9e95mF+\
sH9zkGAm5oFsiUFvei5cK5sRDyNpAWgJDpEFldu5ltr6hOzB3kMAWHABw0Z8ZctJ4qyxKS\
d8keJA2qw2u4YLVitOK5Vl1K2k3b5mylmctC3sQ7M5BETQ1dxqx79AP44jr1qm269QPmnQ\
vUpTdC3pj9BbDtFjeRQZnqOpYJDZ+\
44hSioW7oLJvWvE8sSwl77flRMbbBBU2sG2K7DQbnkNCn3CTjEVFIte9TtPGBAOuFu7jRC\
w8y5S9iwHcr8MsOXNOjYE0RxODH+sjPk6PBDbvrJlrZjkx3bqe07vl+Iom/\
OBGEt1M0pjtX9FlDYyBkS3V3b9fDsvfrhv2vlh4E9ja8626sS2hEybmtxjAmv6qcal7DhF\
SVVXO8Yt6Y2hbScpO8oRvDvSVCdf52QBjPnuOzzjudrKhaurvneXSeG2PU+OCDi0Nc/\
rRPvfK5UwsXz9UurQpOzU45Ywppte4a7khnUXpX4cbW5oeJgEVvomik38Wv76D6nj3acAd\
Bfjj6/ytRy69E3UzYkJmAj6beoQWCXhxz5pMJf8+u+LrRFaUqvkWnHOzZKbVHBvu4A+\
2Bd8e0P9vnN7syLKXO/dwMjTYf7ekTRA0oVx93MRLWpHQNeNZBERjuDdOzpsvh/\
h4O92T5/IloPJOUBCutEOqYuYGGJLZ8QVI12SGtjt9q0+/pbTm2eKUbMJn5CSRfCmX/\
uWCPlgTLeVxols7wUbIZnlMo+SyDEsjlHE62Cu7dbYD7emXLrcu/n5/PiQ==";

enter image description here

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.