# Paging RAM in case of memory shortage issue

Is there some way in Mathematica where I can issue execution of some command with letting Mathematica use my hard disk in case it is short of RAM. It will drastically reduce the speed but I am fine with it as long as I get appropriate results.

-
I believe this is the default behavior. Certainly you can look at the "page in" and "page out" statistics when running a big calculation and see that they become active when the RAM is fully active. And yes, it causes a significant decrease in performance when this happens. –  bill s Jul 3 at 7:00
Paging is controlled by the OS, not (usually) the application. Mathematica's behavior is just the same as for any other application that doesn't do anything special in this respect. –  Oleksandr R. Jul 3 at 7:32
I'm not sure this would make sense for a program like Mathematica. Yes, programs like Photoshop do this separately from the OS. But for Photoshop it makes sense: it can put the undo history in a file on disk because it does not need to access this information often. Unlike Photoshop, Mathematica is a programming language and the system can't predict what's going to be used and what isn't any better than the OS can. I think you're looking for a solution to not enough memory in the wrong place. –  Szabolcs Jul 3 at 8:08
Take a look here instead. –  Szabolcs Jul 3 at 8:09
You can use Export to write (fast to read) MX files. It is often more convenient than DumpSave for data (not code). –  Szabolcs Jul 3 at 10:36

To illustrate my comment, the following function allows the user to store definitions on the hard disk, clearing them from RAM, and retrieve them on-demand.

To keep things orderly, I first create a directory.

CreateDirectory[FileNameJoin[{$TemporaryDirectory, "DumpSaves"}]]  Then this defines the function. toHD is short for to harddrive/hard disk. SetAttributes[toHD, HoldAllComplete] toHD[var_] := With[ { fnP = FileNameJoin[{$TemporaryDirectory, "DumpSaves",
ToString[Unevaluated[var]] <> ".mx"}]
},
DumpSave[fnP, var];
ClearAll[Unevaluated[var]];
SetDelayed[
var,
ClearAll[Unevaluated[var]]; Get[fnP]; var
]
]


We then have

list = {1, 2, 3};
toHD[list];
list//Definition


--> list := Get[ FileNameJoin[{\$TemporaryDirectory, "DumpSaves", ToString[Unevaluated[list]] <> ".mx"}]]

We can then do

Append[list, 4]


-> {1,2,3,4}

I am actually baffled that it works in the following case

toHD[list];
AppendTo[list, 4]


-> {1,2,3,4}

list//Definition


-> list = {1, 2, 3, 4}

More examples

f[x_] := x^2;
f[0, 1] = 2;
f//Definition


-> { {f[0, 1] = 2}, { }, {f[x_] := x^2} }

Then,

toHD[f]
f[3]


-> 9

Here, doing

toHD[f]
f[0, 2]:=3


...will actually also not break it

f // Definition


f[0, 1] = 2
f[0, 2] := 3
f[x_] := x^2

Wow I can't seem to beat it :)

In case something breaks

Maybe we could somehow put additional definitions added to a stored symbol on a queue or something. In this way we might fix additional definitions made by Set and SetDelayed. This is possible, because of the following. We can do

h /: SetDelayed[h, _] := Print["first"]


Then

h := 3


Does not define h to be 3, but rather prints "first". In this way we could intercept such statements and add them to some queue. Then it might be more trouble to get correct behavior for TagSet (if that is at all possible), but at least that is a less common function.

-
Todo: better formatting, testing for TagSet, Set etc, shorter out-of-office definition :P. –  Jacob Akkerboom Jul 3 at 10:35
@Blackbird, Thank you. I think it turned out to be quite a useful function, I'm glad I made it :). –  Jacob Akkerboom Jul 3 at 10:36
I could help you with formatting :) If you wish, I can edit your toHD with what my code formatter outputs for it. If you don't like it, you could roll back. –  Leonid Shifrin Jul 3 at 10:40
Somewhat related to your toHD is a function defineCompressed discussed in this answer (under the section Symbol's auto-loading / self-uncompression), and in the post linked by this one, where it was originally defined. –  Leonid Shifrin Jul 3 at 10:46
@JacobAkkerboom "Did your formatter pretty-print the result automatically?" - yes, that's exactly what it does. In case you have not seen,here is a description. It is now quite easy to use for SE posts, and I already used it for a number of my recent answers. It does contain a number of bugs though, which I hope to address soon, but you can give it a try. –  Leonid Shifrin Jul 3 at 12:11