I have a simulation written in another language which currently just dumps all its data (doubles, lists of doubles, lists of lists of doubles) in lines in ASCII, and then I read it into Mathematica by writing a list a "header" listing what each line corresponds to and then AssociationThreading it like so


I realise that this is a pretty dirty way to do things and it's causing some problems. In particular the specification changes frequently and this method is also quite slow (for hundreds of MB of data). I realise that I could just read the header strings from the file, but I thought it might be better in the long run to use a proper format e.g. JSON, CDF (the NASA one, not the Wolfram one), or HDF.

Can anyone offer some advice from experience on what choice to take here? (I like the idea of having some sort of standard format so I can use tools other than mma in the future too).

  • 3
    $\begingroup$ If you like the idea of a standard format, then HDF is surely a good way to go? $\endgroup$ Nov 17, 2015 at 13:29
  • $\begingroup$ I've used netCDF in the past, is pretty versatile $\endgroup$
    – Jason B.
    Nov 17, 2015 at 13:39
  • $\begingroup$ interesting there are few hdf questions on this site, either there are no problems or few are using it. My concerns are jumping in and learning that the mathematica implementation is slow and/or incomplete. Of course issues with maintaining a library for whatever the other language is are a consideratino as well. $\endgroup$
    – george2079
    Nov 17, 2015 at 16:44
  • $\begingroup$ I also would suggest HDF5, I have used it with no problems and think it would work well for your use case, and much better than text files and the alternatives (I think newer NetCDF versions are based on HDF5, so I don't think there is a big difference there). The Mathematica implementation is decent, but could be faster and especially more memory efficient, but that only becomes relevant for data in the order of several GB with current hardware. It is also not very complete (full HDF5 is quite complex), but easily covers what you seem to need. $\endgroup$ Nov 18, 2015 at 11:34
  • 2
    $\begingroup$ @SeanD Albert Retey already pointed out 3 years ago that HDF might be a good choice for your situation but you have not reacted to it. A valid question without an appropriate answer is useless for future visitors. The Q&A that Marco suggested as dup makes sense to me and Albert provided some very detailed answers there. If it is marked as dub, others will find the answer easily. As Kuba said, if you disagree, then edit your question and make your point clear. You yourself suggested HDF as a possibility. $\endgroup$
    – halirutan
    Jun 6, 2018 at 17:40