Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I've written my incredibly complex, incredibly elegant analysis function, that works great on small test data. But when I run it on my real (bigger) data set it keeps running out of memory. It turns out that the analysis function does not free memory, but I can't imagine why. It takes a large number of points, but returns only several scalar values.

memory usage after time

Every time I run this it takes up about 500 MB of memory. (here is another example).

What is the best way to debug memory problems?

I've read the memory management tutorial, turned off caching and verified I have no lingering variables in my contexts and of course I have set $HistoryLength to zero.

Also running Reverse@Sort[{ByteCount[Symbol[#]], #} & /@ Names["`*"]] show no huge memory symbols. Just the data:

{{191816648, "alldata"}, {28184, "before"}, {28184, "after"},
 {24096, "compiledSelectBin"}, {15344, "AppendLeftRight"}, {8840, "compiledSelectBinFunc"}...}

EDIT One can use this code to track memory consumption:

DynamicModule[{pm = {}},
 Dynamic@Refresh[pm = Append[pm, MemoryInUse[]]; 
   If[Length[pm] > 120, pm = Drop[pm, 1]]; 
   ListPlot[pm/1024/1024, AxesLabel -> {"Time [s]", "Memory [MB]"}, 
    PlotRange -> {0, All}], UpdateInterval -> 1, 
   TrackedSymbols :> {}]]

I think I finally have a minimum example. Here it is. Unzip to a folder and evaluate the two cells in LeakP.nb. If you evaluate the second cell multiple times you can watch the memory consumption grow. Could somebody (on win7 64 bit mma 8) confirm this?

EDIT 1

I really hope I have nailed it down. Here is a self contained example:

$HistoryLength = 0;
data = RandomReal[{-1, 1}, {10, 100000, 2}];
data = Developer`ToPackedArray[#] & /@ data;
data = Flatten[data, 1];
Dimensions[data]
HistogramList[data, 30, Automatic];
ClearAll[data]; ClearSystemCache[];

EDIT 2

This is fixed in Mathematica 9.0.0.

share|improve this question
2  
the question you ask is impossible to answer in any non speculative way without seeing the code. Having said that and assuming for a second that you did follow the typical suggestions have you tried to explicitly set all variables in your Module/Block to Null right after they are not used any longer? –  user21 May 29 '12 at 10:28
1  
@rubenko No, not yet. Also, I'm not asking where is my problem, I'm asking how to start tracking it down. (Aha, the question in the image is rhetorical:) This is a largish piece of code (about 600 lines) and I'm at a loss where to begin... –  Ajasja May 29 '12 at 10:32
2  
Perhaps it's time to try constructing a minimal working example and share it. If the mistake is indeed in your code (and it's not a bug in some Mathematica function you're using), it would be very worthwhile to learn about it, for all of us. (If you post a working example: please note that some of us only have 2 GB of memory, so scale it down a bit :) –  Szabolcs May 29 '12 at 12:03
5  
Note that ByteCount[sym] will only show the memory used by the OwnValues for the symbol. I introduced the symbolMemoryUsage function in my answer specifically to address other global properties as well. –  Leonid Shifrin May 29 '12 at 13:23
1  
I think @acl gave a very sound suggestion: try to refactor your code so that it is built from really small functions and use as little state as possible. It should then be much easier to locate the place where leak is happening. –  Leonid Shifrin May 30 '12 at 10:12

2 Answers 2

up vote 22 down vote accepted

Preamble

It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are accumulated but not cleared.

I will show here one rather obscure case of leaking of local Module variables, which happens when the variable is referenced by some object / symbol, external w.r.t. its scope. In such cases, such variables are not garbage-collected even after the symbols referencing them get Remove-d, in case if they get assigned DownValues, SubValues or UpValues (OwnValues are ok).

One subtle case with a memory leak

MemoryInUse[]
 17350016
$HistoryLength = 0;

Module[{g},
  Module[{f},
    g[x_] := f[x];
    Do[f[i] = Range[i], {i, 5000}];
  ];
  g[1]]
 {1}
MemoryInUse[]
 72351376

One way to ensure that this does not happen is to insert Clear[f] at the end of the outer Module, storing the result in a separate variable and returning it afterwards. There are more advanced ways to prevent such things as well. I may elaborate on those at some later time.

Memory leaks assocaited with UI-building

One common cause of memory leaks which is often ovelooked is when some local symbols make it into UI elements. The problem is that UI elements are Mathematica expressions, which do reference those symbols, and therefore, they are not garbage-collected.

Here is an example I borrowed from this thread

memModule[] := 
  Module[{data, memBefore, mu}, 
     mu := Grid[{{"Memory in use: ", MemoryInUse[]/(2^30.), "GB"}}]; 
     memBefore = mu; 
     data = RandomReal[1, {300000, 20}]; 
     DynamicModule[{d1}, 
        d1 := data[[1]]; 
        Panel[Grid[{{memBefore}, {mu}}]] 
       , UnsavedVariables -> {dl} 
    ] 
 ]; 

Now, every time when it gets executed, more memory is being leaked:

memModule[] 
memModule[] 
memModule[] 

Please see my answer in the linked thread for one way out, in this particular case. Generally, this is something to watch out for.

Monitoring symbols

So, one good place to start is to call

Names["Global`*"]
 {"f", "f$", "f$119", "g", "i", "x", "x$"}

or whatever main context you are using (or other contexts, if you create symbols there), and watch for some symbols with high memory usage. In this particular case, the culprit it f$119.

Here are some utility functions which may help with monitoring symbols:

Clear[$globalProperties];
    $globalProperties =
    {OwnValues, DownValues, SubValues, UpValues, NValues, 
     FormatValues, Options, DefaultValues, Attributes, Messages};

ClearAll[getDefinitions];
SetAttributes[getDefinitions, HoldAllComplete];
getDefinitions[s_Symbol] :=
    Flatten@Through[
        Map[Function[prop, Function[sym, prop[sym], HoldAll]],
           $globalProperties
        ][Unevaluated[s]]
      ];

ClearAll[symbolMemoryUsage];
symbolMemoryUsage[sname_String] :=
   ToExpression[sname, InputForm, 
      Function[s, ByteCount[getDefinitions[s]], HoldAllComplete]
   ];

ClearAll[heavySymbols];
heavySymbols[context_, sizeLim_: 10^6] :=
   Pick[#, UnitStep[# - sizeLim] &@Map[symbolMemoryUsage, #], 1] &@
        Names[context <> "*"];

For example, calling

heavySymbols["Global`"]

returns

 {f$119}
share|improve this answer
    
+1 Thank you for the example. Now I understand the comment you made here. Unfortunately, there appear to be no heavy symbols. heavySymbols[""], returns {"alldata", "GeoProjectionData"} which is that data I loaded at the beginning and an always present built-in. –  Ajasja May 29 '12 at 11:55
    
@Ajasja Well, then I am at a loss. Do you create symbols in other contexts? Or, do you return any expression involving local variables, or embed it into UI, for example? UI-s are another cause of memory leaks, see e.g. my answer in this thread –  Leonid Shifrin May 29 '12 at 12:00
1  
@Ajasja I also had a situation with the kernel eating up gigabytes of memory, linearly increasing in time even though I never kept more than 1GB of data in symbols. I also had checked and no symbols had anywhere near the memory consumed. in the end I rewrote my program in a completely different way and now it doesn't eat memory. have no clue what it was. –  acl May 29 '12 at 12:13
    
@LeonidShifrin I don't create symbols in any other context. What I'm doing a lot is assigning to the same Module variable multiple times. For example data=rawdata; data=GetDiffrences[data]; data=Mean[data]... Return[data] But I belive this shuld be safe. I'm putting memory print statements all over my code as we speak... –  Ajasja May 29 '12 at 12:14
    
@acl Brr... that would be a worst case scenario! I spent two weeks on this code and now I can't get a single result... –  Ajasja May 29 '12 at 12:16

This is a memory leak in HistogramList:

You can reclaim the memory by evaluating

Remove["*`*modelData$*"]
share|improve this answer
1  
Using the previous answer with heavySymbols["*`"] allows to find this. Great case study ! –  Faysal Aberkane May 30 '12 at 21:12
1  
Isn't Remove["System`HistogramListDump`modelData$*"] safer? –  Ajasja May 31 '12 at 8:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.