Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I've got this CSV file I've imported that has tens of millions of lines in it. It takes around 20 minutes to import. I've been working with it for a while and have the processed data spread out in a bunch of variables.

Now Windows is bugging me that I need to restart the computer.

I thought about gathering all the data up in a table and then export and import it, but that would be a lot of hassle and take ages. I also thought about just saving the notebook and re-evaluate it, but with this amount of data that will also take a long time.

I wonder what is the best way to save all the data so that I can get it back after having restarted the computer? Something fast and with minimum of hassle would be great.

PS. I have no idea how to tag this thing. There is apparently no big-data tag.

share|improve this question

2 Answers 2

up vote 24 down vote accepted

Assuming you haven't placed your variables in a non-standard context you can save them all at once using DumpSave's second syntax form, which saves everything in the indicated context.

Quit[] (* start a fresh kernel *)

x = 1; (* define some symbols *)
y = 2;
z[x_] := x^2

Names["Global`*"] (* Check they're there *)

(* ==> {"x", "y", "z"}  *)

(* Save everything in the context *)
DumpSave["C:\\Users\\Sjoerd\\Desktop\\dump.mx", "Global`"];    

Quit[] (* kill kernel to simulate a new start *)

Names["Global`*"] (* Are we clean? *)
(* ==> {} *)

(* Get the save symbols *)
<< "C:\\Users\\Sjoerd\\Desktop\\dump.mx"

(* Are they there? *)
Names["Global`*"]    
(* ==> {"x", "y", "z"} *)

z[y]    
(* ==> 4 *)
share|improve this answer
    
Works wonderfully, thanks! –  Mr Alpha May 12 '12 at 21:28

You can export to the MX format:

Export["out.mx", data, "MX"]

Be aware that this format is not portable between different computer architectures (e.g. 32/64 bit).

Import using

data = Import["out.mx"];

This is the fastest available format in Mathematica. Most likely you can't do better than this, unless you write an interface to a specialized external library.

Related reading:

share|improve this answer
1  
+1 for the link collection! –  Yves Klett May 11 '12 at 12:49
4  
@Yves I feel we need a "canonical question" for this, but I wasn't able to find an existing question which would be suitable without a lot of editing. Maybe eventually we can repurpose this one. This question is quite straightforward, without many extra details, but it'll be necessary to write a good answer which explains the caveats of MX, the difference between DumpSave and Export and mentions Export["out.mz", Compress[expr], "String"] ... –  Szabolcs May 11 '12 at 12:52
    
... as a cross-platform alternative (something like Mr. Wizard's answer, made a bit more beginner friendly). –  Szabolcs May 11 '12 at 12:53
    
This still means I have to do it separately for each variable or collect all the data up into one table? –  Mr Alpha May 11 '12 at 13:23
2  
@OleksandrR. Of course, it would be very nice if we could have a supported interface (or, rather, a specification) for creation of .mx files (I mean, not from within Mathematica). This would allow one to write very fast converters from a given external format to .mx, which would load way faster (and save tones of memory as well) than whatever we can now write using the Import-Export framework. –  Leonid Shifrin May 11 '12 at 14:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.