36
$\begingroup$

I've got this CSV file I've imported that has tens of millions of lines in it. It takes around 20 minutes to import. I've been working with it for a while and have the processed data spread out in a bunch of variables.

Now Windows is bugging me that I need to restart the computer.

I thought about gathering all the data up in a table and then export and import it, but that would be a lot of hassle and take ages. I also thought about just saving the notebook and re-evaluate it, but with this amount of data that will also take a long time.

I wonder what is the best way to save all the data so that I can get it back after having restarted the computer? Something fast and with minimum of hassle would be great.

PS. I have no idea how to tag this thing. There is apparently no big-data tag.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
46
$\begingroup$

Assuming you haven't placed your variables in a non-standard context you can save them all at once using DumpSave's second syntax form, which saves everything in the indicated context.

Quit[] (* start a fresh kernel *)

x = 1; (* define some symbols *)
y = 2;
z[x_] := x^2

Names["Global`*"] (* Check they're there *)

(* ==> {"x", "y", "z"}  *)

(* Save everything in the context *)
DumpSave["C:\\Users\\Sjoerd\\Desktop\\dump.mx", "Global`"];    

Quit[] (* kill kernel to simulate a new start *)

Names["Global`*"] (* Are we clean? *)
(* ==> {} *)

(* Get the save symbols *)
<< "C:\\Users\\Sjoerd\\Desktop\\dump.mx"

(* Are they there? *)
Names["Global`*"]    
(* ==> {"x", "y", "z"} *)

z[y]    
(* ==> 4 *)
Improve this answer
$\endgroup$
2
  • $\begingroup$ Works wonderfully, thanks! $\endgroup$
    – Mr Alpha
    May 12, 2012 at 21:28
  • $\begingroup$ Smooth, thanks for the solutions! $\endgroup$
    – Physics_maths
    Dec 2, 2015 at 19:00
36
$\begingroup$

You can export to the MX format:

Export["out.mx", data, "MX"]

Be aware that this format is not portable between different computer architectures (e.g. 32/64 bit).

Import using

data = Import["out.mx"];

This is the fastest available format in Mathematica. Most likely you can't do better than this, unless you write an interface to a specialized external library.

Related reading:

Improve this answer
$\endgroup$
14
  • 5
    $\begingroup$ @Yves I feel we need a "canonical question" for this, but I wasn't able to find an existing question which would be suitable without a lot of editing. Maybe eventually we can repurpose this one. This question is quite straightforward, without many extra details, but it'll be necessary to write a good answer which explains the caveats of MX, the difference between DumpSave and Export and mentions Export["out.mz", Compress[expr], "String"] ... $\endgroup$
    – Szabolcs
    May 11, 2012 at 12:52
  • $\begingroup$ ... as a cross-platform alternative (something like Mr. Wizard's answer, made a bit more beginner friendly). $\endgroup$
    – Szabolcs
    May 11, 2012 at 12:53
  • $\begingroup$ This still means I have to do it separately for each variable or collect all the data up into one table? $\endgroup$
    – Mr Alpha
    May 11, 2012 at 13:23
  • $\begingroup$ A .mx file is relatively compact, but it isn't compressed. Export["file.mx.gz", expr] or Export["file.mx", Compress[expr]] are also allowable. I think the .mx.gz should be fastest to import, while the smallest file size seems to result from Export["file.mx", Compress[expr, Method -> {"Version" -> 2}]]. $\endgroup$
    – Oleksandr R.
    May 11, 2012 at 13:24
  • 2
    $\begingroup$ @OleksandrR. Of course, it would be very nice if we could have a supported interface (or, rather, a specification) for creation of .mx files (I mean, not from within Mathematica). This would allow one to write very fast converters from a given external format to .mx, which would load way faster (and save tones of memory as well) than whatever we can now write using the Import-Export framework. $\endgroup$
    – Leonid Shifrin
    May 11, 2012 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.