# How to save all data in all variables so that loading it is fast?

I've got this CSV file I've imported that has tens of millions of lines in it. It takes around 20 minutes to import. I've been working with it for a while and have the processed data spread out in a bunch of variables.

Now Windows is bugging me that I need to restart the computer.

I thought about gathering all the data up in a table and then export and import it, but that would be a lot of hassle and take ages. I also thought about just saving the notebook and re-evaluate it, but with this amount of data that will also take a long time.

I wonder what is the best way to save all the data so that I can get it back after having restarted the computer? Something fast and with minimum of hassle would be great.

PS. I have no idea how to tag this thing. There is apparently no big-data tag.

-

Assuming you haven't placed your variables in a non-standard context you can save them all at once using DumpSave's second syntax form, which saves everything in the indicated context.

Quit[] (* start a fresh kernel *)

x = 1; (* define some symbols *)
y = 2;
z[x_] := x^2

Names["Global*"] (* Check they're there *)

(* ==> {"x", "y", "z"}  *)

(* Save everything in the context *)
DumpSave["C:\\Users\\Sjoerd\\Desktop\\dump.mx", "Global"];

Quit[] (* kill kernel to simulate a new start *)

Names["Global*"] (* Are we clean? *)
(* ==> {} *)

(* Get the save symbols *)
<< "C:\\Users\\Sjoerd\\Desktop\\dump.mx"

(* Are they there? *)
Names["Global*"]
(* ==> {"x", "y", "z"} *)

z[y]
(* ==> 4 *)

-
Works wonderfully, thanks! –  Mr Alpha May 12 '12 at 21:28

You can export to the MX format:

Export["out.mx", data, "MX"]


Be aware that this format is not portable between different computer architectures (e.g. 32/64 bit).

Import using

data = Import["out.mx"];


This is the fastest available format in Mathematica. Most likely you can't do better than this, unless you write an interface to a specialized external library.

@Yves I feel we need a "canonical question" for this, but I wasn't able to find an existing question which would be suitable without a lot of editing. Maybe eventually we can repurpose this one. This question is quite straightforward, without many extra details, but it'll be necessary to write a good answer which explains the caveats of MX, the difference between DumpSave and Export and mentions Export["out.mz", Compress[expr], "String"] ... –  Szabolcs May 11 '12 at 12:52
@OleksandrR. Of course, it would be very nice if we could have a supported interface (or, rather, a specification) for creation of .mx files (I mean, not from within Mathematica). This would allow one to write very fast converters from a given external format to .mx, which would load way faster (and save tones of memory as well) than whatever we can now write using the Import-Export framework. –  Leonid Shifrin May 11 '12 at 14:41