1
$\begingroup$

This is a question about proper ways to save and retrieve TimeSeries.

I am processing data from two separate sources and compile around 150 TimeSeries from each source; both sources contain information about the same system but they differ in coverage and accuracy; each TimeSeries is endowed with relevant MetaInformation that I want preserved.

The assembly of the TimeSeries takes some time and I want to save them for later use.

The solution I used was to bundle them on a dataset and dump them on a MX file. The file is around 22Mb. The exact layout of the Dataset is

Dataset[
  Association[
    "id_str"->Association[
      "col_1"->TimeSeries[], "col_2"->TimeSeries[]
     ]
   ],
   ... 
  ]

Is there a better way to store TimeSeries in terms of storage space and loading times? Is the "MX" suitable for such purposes? Can anyone present an alternative method, other than exporting the processed data to a .csv or an .xlsx and recreating the TimeSeries from scratch (importing the data along with the metadata and using TimeSeries) when needed?

$\endgroup$
  • 1
    $\begingroup$ Using Associations in the way you did looks like a good approach to me. There is only few memory overhead if the TimeSeries have their data stored as PackedArrays, internally (I am not sure about the internals of TimeSeries). Plus, there is few to do on your side to dump or to retrieve data. When used meaningful keys, Associations are pretty selfdocumenting. Note that the MX format might not be ideal for exchanging data between different OSes and different versions of Mathematica or for achiving data. You should also consider using the WDX format. $\endgroup$ – Henrik Schumacher Apr 22 '18 at 20:06
  • $\begingroup$ I do agree with Henrik, MX is a good format to store and retrieve such data on the same machine for later reuse. If you need to read on another machine or with another software (including another version of Mathematica) other formats are a better choice. My experience with wdx is not very promising, I found it to be slow but that is a few version ago. What works good for Mathematica to Mathematica is compressed strings of inputform (you'll find answers on this site about that). If you need other software to read the data you have to go the hard way and regenerate the timeseries... $\endgroup$ – Albert Retey Apr 23 '18 at 19:49
  • $\begingroup$ @HenrikSchumacher WDX can be useful for some things, but arbitrary data exchange is not one in my experience. There’s now a link for HDF5 data I believe. I think it’s a backend for Dataset storage. That might be the way to go? $\endgroup$ – b3m2a1 Apr 23 '18 at 23:27
  • $\begingroup$ Since the comment of @Henrik Schumacher, I've experimented a bit with "WDX"; saving a single TimeSeries is around 30% more costly in space than it is when using a " ZIP" file to hold the values; obviously, this has to do with the fact that a TimeSeries object is more than just rows of data in a text file. Both choices seem to be equally fast and equally responsive with the "WDX" choice feeling somehow more... robust-of cource that's totally subjective. $\endgroup$ – yosimitsu kodanuri Apr 24 '18 at 6:12
  • $\begingroup$ I am trying to compile a small data repository and I would like it to be persistent over time and accessible from other machines although I don't expect other software different than Mathematica to have access to it. I will try and research both @Albert Retey's suggestion-"compressed strings of InputForm"- and @b3m2a1's suggestion-"HDF5" compression. $\endgroup$ – yosimitsu kodanuri Apr 24 '18 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.