I have a decently large amount of data that I'm performing a bunch of computation on. My typical M.O. in Mathematica is to use
Table to generate a nested array to store my results, and then refer back to the specific columns for additional computation/plotting. Currently, generating the array takes ~40 min. on a pretty beefy personal machine, using
ParallelTable for the computation. While prototyping additional analysis I will frequently encounter crashes or hangs which require me to restart the kernel, and perform that 40 min of computation again (which is pretty frustrating).
Ideally, I would be able to export the output of that computation as some kind of file that I could then import into Mathematica instead of having to perform the computation repeatedly. I usually use
.csv but this data doesn't work well for that as it has nested array elements which can be vary in size (depending on the initial raw data).
Is there a file format or storage solution that would allow me to keep my computed data as a nested array so that it can be easily imported again? For example,
matlab has the
save command which basically takes a snapshot of the runtime which makes recovering from a crash easy.
Alternatively, is there a better or more idiomatic solution for storing the results of my computation that I should be using?
As pointed out in the comments, there are at least 3 functions for saving Mathematica objects,
DumpSave results in a much smaller file size compared to
Save (~700Mb vs ~1800Mb).
It appears that the main bottleneck is retrieving the data, but it's an order of magnitude faster than repeating the computation every time the kernel crashes.