Aborting evaluation when the memory exceeds a certain limit

Question

Sometimes when generating a lot of data the memory usage shoots beyond its physical realm and my laptop freezes. There is no way to stop Mathematica when this happens. Task manager (or its equivalent in other OSes) won't even start.

I was wondering if it's possible to set a threshold somewhere in Mathematica to stop it automatically after a certain amount of memory.

Answer 1

Since one may not always accurately predict when MemoryContrained is needed, I recommend setting up a watch-dog task. Belisarius described how to do this here in answer to my question. I will reproduce it below as answers that are merely links are discouraged.

---

In Mathematica 8 you could start a memory watchdog, something along the lines of:

maxMemAllowed        = 1.3 1024^3; (*1.3 GB*);
intervalBetweenTests = 1; (*seconds*)
iAmAliveSignal       = 0;
Dynamic[iAmAliveSignal]
RunScheduledTask[
       If[MemoryInUse[] > maxMemAllowed , Quit[], iAmAliveSignal++],      
       intervalBetweenTests];

Remember to run

RemoveScheduledTask[ScheduledTasks[]];

to disable it.

Answer 2

In addition to Mr.Wizard's answer.

In many cases it is very practical to stop the evaluation when the actual amount of free physical memory in your system becomes less than specified threshold. You can get the amount of free physical memory very efficiently via NETLink call to GlobalMemoryStatusEx function of kernel32.dll (which is available both under 32 bit and 64 bit Windows systems). Here I define getFreePhysMem[] function which returns the current amount of free physical memory:

Needs["NETLink`"];
getFreePhysMem::internalError = 
  "globalMemoryStatusEx[memorystatusex] has not returned True.";
If[$OperatingSystem === "Windows", 
  memorystatusex = Symbol["LoadedNETTypes"][];
  globalMemoryStatusEx = 
   Symbol["DefineDLLFunction"][
    "[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Auto)]
            public class MEMORYSTATUSEX
            {public uint dwLength;
            public uint dwMemoryLoad;
            public ulong ullTotalPhys;
            public ulong ullAvailPhys;
            public ulong ullTotalPageFile;
            public ulong ullAvailPageFile;
            public ulong ullTotalVirtual;
            public ulong ullAvailVirtual;
            public ulong ullAvailExtendedVirtual;
            public MEMORYSTATUSEX()
            {this.dwLength = (uint) 
            Marshal.SizeOf(typeof( MEMORYSTATUSEX ));}}
            [return: MarshalAs(UnmanagedType.Bool)]
            [DllImport(\"kernel32.dll\", CharSet=CharSet.Auto,     \
SetLastError=true)]
            public static extern bool GlobalMemoryStatusEx([In, Out]  \
   MEMORYSTATUSEX lpBuffer);"];
  memorystatusex = 
   Complement[Symbol["LoadedNETTypes"][], memorystatusex][[1, 1]];
  memorystatusex = memorystatusex <> "+MEMORYSTATUSEX";
  memorystatusex = Symbol["NETNew"][memorystatusex];
  getFreePhysMem[] := 
   If[TrueQ[globalMemoryStatusEx[memorystatusex]], 
    memorystatusex@ullAvailPhys, 
    Message[getFreePhysMem::internalError]; Abort[]; $Failed]];

Timings:

Do[getFreePhysMem[], {100}] // AbsoluteTiming
(*=> {0.0312500, Null}*)

More information on this function: "Calling kernel.dll from Mathematica. 1", "Calling kernel.dll from Mathematica. 2".

Answer 3

You can make use of either TimeConstrained or MemoryConstrained to terminate evaluation when it runs out of time or memory respectively.

For example, if you have a function that has a reasonable memory footprint, but takes time to evaluate, you can abort evaluation after a certain amount of time (in seconds) has elapsed, as:

TimeConstrained[Eigenvalues@RandomReal[1, {5000, 5000}], 1]
Out[1]= $Aborted

On the other hand, if memory is the one that blows up first, you can wrap it in MemoryConstrained to abort when a certain amount of bytes have been consumed:

MemoryConstrained[Range /@ Range[100], 1000]
Out[2]= $Aborted

Answer 4

You can put your calculation inside TimeConstrained. However in your case, probably the better idea is to restrict the used memory. That's done with MemoryConstrained.

If you don't want to figure out the available memory yourself, see here for how to do it automatically.

For example this terminates a calculation if the calculation needs more than 1 GB of memory:

MemoryConstrained[Simplify[complicatedExpression],1024^3]

This only constrains the simplification, but gives back the original expression if the evaluation fails:

MemoryConstrained[Simplify[#],1024^3,#]&[complicatedExpression]

This stops evaluation after one hour:

TimeConstrained[Simplify[complicatedExpression],3600]

Answer 5

I will provide two function in my init.m for the answer.This code can help your memory remain at least 3% to avoid crash:

$Pre = Function[expr, 
  MemoryConstrained[expr, 
   Round[Last[
       ReadList[
        "!wmic OS get FreePhysicalMemory"]]*1024 - .03 $SystemMemory],
    Print[Style["\[WarningSign]", 28, Red], 
    Style["Memory limit exceeded and Computation aborted!", 15, 
     Red]]], {HoldAll}]

And this function can make your Notebook like as a baby(Thanks C. E. for this post) or use Internal`Restart[]:

System`Restart := 
 Block[{$ContextPath}, Needs["Utilities`CleanSlate`"]; 
  Quiet[Remove["`*"]]; 
  Utilities`CleanSlate`CleanSlate[Verbose -> False]; 
  FrontEndTokenExecute /@ {"SelectAll", "Clear"};]