Debugging memory leaks

Question

I've written my incredibly complex, incredibly elegant analysis function, that works great on small test data. But when I run it on my real (bigger) data set it keeps running out of memory. It turns out that the analysis function does not free memory, but I can't imagine why. It takes a large number of points, but returns only several scalar values.

Every time I run this it takes up about 500 MB of memory. (here is another example).

What is the best way to debug memory problems?

I've read the memory management tutorial, turned off caching and verified I have no lingering variables in my contexts and of course I have set $HistoryLength to zero.

Also running Reverse@Sort[{ByteCount[Symbol[#]], #} & /@ Names["`*"]] show no huge memory symbols. Just the data:

{{191816648, "alldata"}, {28184, "before"}, {28184, "after"},
 {24096, "compiledSelectBin"}, {15344, "AppendLeftRight"}, {8840, "compiledSelectBinFunc"}...}

EDIT One can use this code to track memory consumption:

DynamicModule[{pm = {}},
 Dynamic@Refresh[pm = Append[pm, MemoryInUse[]]; 
   If[Length[pm] > 120, pm = Drop[pm, 1]]; 
   ListPlot[pm/1024/1024, AxesLabel -> {"Time [s]", "Memory [MB]"}, 
    PlotRange -> {0, All}], UpdateInterval -> 1, 
   TrackedSymbols :> {}]]

I think I finally have a minimum example. Here it is. Unzip to a folder and evaluate the two cells in LeakP.nb. If you evaluate the second cell multiple times you can watch the memory consumption grow. Could somebody (on win7 64 bit mma 8) confirm this?

EDIT 1

I really hope I have nailed it down. Here is a self contained example:

$HistoryLength = 0;
data = RandomReal[{-1, 1}, {10, 100000, 2}];
data = Developer`ToPackedArray[#] & /@ data;
data = Flatten[data, 1];
Dimensions[data]
HistogramList[data, 30, Automatic];
ClearAll[data]; ClearSystemCache[];

EDIT 2

This is fixed in Mathematica 9.0.0.

Answer 1

Preamble

It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are accumulated but not cleared.

I will show here one rather obscure case of leaking of local Module variables, which happens when the variable is referenced by some object / symbol, external w.r.t. its scope. In such cases, such variables are not garbage-collected even after the symbols referencing them get Remove-d, in case if they get assigned DownValues, SubValues or UpValues (OwnValues are ok).

One subtle case with a memory leak

MemoryInUse[]

 17350016

$HistoryLength = 0;

Module[{g},
  Module[{f},
    g[x_] := f[x];
    Do[f[i] = Range[i], {i, 5000}];
  ];
  g[1]]

 {1}

MemoryInUse[]

 72351376

One way to ensure that this does not happen is to insert Clear[f] at the end of the outer Module, storing the result in a separate variable and returning it afterwards. There are more advanced ways to prevent such things as well. I may elaborate on those at some later time.

Memory leaks associated with UI-building

One common cause of memory leaks which is often ovelooked is when some local symbols make it into UI elements. The problem is that UI elements are Mathematica expressions, which do reference those symbols, and therefore, they are not garbage-collected.

Here is an example I borrowed from this thread

memModule[] := 
  Module[{data, memBefore, mu}, 
     mu := Grid[{{"Memory in use: ", MemoryInUse[]/(2^30.), "GB"}}]; 
     memBefore = mu; 
     data = RandomReal[1, {300000, 20}]; 
     DynamicModule[{d1}, 
        d1 := data[[1]]; 
        Panel[Grid[{{memBefore}, {mu}}]] 
       , UnsavedVariables -> {dl} 
    ] 
 ]; 

Now, every time when it gets executed, more memory is being leaked:

memModule[] 
memModule[] 
memModule[] 

Please see my answer in the linked thread for one way out, in this particular case. Generally, this is something to watch out for.

Monitoring symbols

So, one good place to start is to call

Names["Global`*"]

 {"f", "f$", "f$119", "g", "i", "x", "x$"}

or whatever main context you are using (or other contexts, if you create symbols there), and watch for some symbols with high memory usage. In this particular case, the culprit it f$119.

Here are some utility functions which may help with monitoring symbols:

Clear[$globalProperties];
$globalProperties =
    {OwnValues, DownValues, SubValues, UpValues, NValues, 
     FormatValues, Options, DefaultValues, Attributes, Messages};

ClearAll[getDefinitions];
SetAttributes[getDefinitions, HoldAllComplete];
getDefinitions[s_Symbol] :=
    Flatten@Through[
        Map[
           Function[
              prop, 
              (* Unevaluated needed here just for Options, which is not holding *)
              Function[sym, prop[Unevaluated @ sym], HoldAll]
           ],
           $globalProperties
        ][Unevaluated[s]]
      ];

ClearAll[symbolMemoryUsage];
symbolMemoryUsage[sname_String] :=
   ToExpression[sname, InputForm, 
      Function[s, ByteCount[getDefinitions[s]], HoldAllComplete]
   ];

ClearAll[heavySymbols];
heavySymbols[context_, sizeLim_: 10^6] :=
   Pick[#, UnitStep[# - sizeLim] &@Map[symbolMemoryUsage, #], 1] &@
        Names[context <> "*"];

For example, calling

heavySymbols["Global`"]

returns

 {f$119}

Answer 2

This is a memory leak in HistogramList:

You can reclaim the memory by evaluating

Remove["*`*modelData$*"]