Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I would like to export data in HDF5 format with Mathematica 9. The Mathematica manual about Import and Export of this format can be read here.

As I'm dealing with large arrays, export often lead to 100MB+ files. I'd like to use the HDF5 "DataEncoding" to compress the exported data, but I can't find a way to use it.

For example, let's consider an array called

datapourrie = RandomInteger[99, {100000, 2, 5}]

First, I export it in simple ASCII format, for the purpose of comparison:

Export["expl.dat", datapourrie]

expl.dat is correctly written, with a size of 4.1 MB.

Now, let's try in HDF5 format:

Export["expl.h5", datapourrie, {"Datasets", "mydata"}]

The file is correctly written in HDF5 format. It's 5.0 MB (the data + metadata). I tested it with a HDF5 viewer outside of Mathematica: everything's OK.

Now, I'd like to compress it, because one of the main advantages of HDF5 format is that it can be used as a binary (compressed) file format. From the Mathematica 9 manual, I read:

""DataEncoding" can be none, "GZIP" or "SZIP""

My problem is that I can't find a way to use this DataEncoding! I would expect the correct sentence to be:

 Export["explgzip.h5", datapourrie, {"Datasets", "mydata"}, "DataEncoding" -> "GZIP"]

but it produces exactly the same file as without the DataEncoding tag. What am I doing wrong?

share|improve this question
    
"Datasets" seems to be an import setting, not an export setting. –  Sjoerd C. de Vries Jan 28 '13 at 14:59
    
@SjoerdC.deVries "Datasets" is an export and import setting as it gives a name to a datasubset. Thus, you can Import a given subset only when you know their dataset name. –  max Jan 28 '13 at 15:49
add comment

2 Answers

up vote 3 down vote accepted

In version 7 halirutan's export method does not produce a file that is recognized by Import.

However, one can write:

Export["matrix2.h5.gz", datapourrie, {"GZIP", "HDF5"}]

And then:

d2 = Import["matrix2.h5.gz", {"Datasets", "/Dataset1"}];

datapourrie == d2

True

share|improve this answer
add comment

The documentation is misleading here. On one hand, the only export option is "Append" which can be found under the Options tab. On the other hand, the general documentation reads

enter image description here

I really wonder, why it is necessary to put Import only behind an option value when "DataEncoding" isn't an export option at all.

Anyway, I have the same behaviour in MacOSX as you have: No data compression. Although, using

ExportString[{{1, 2}, {3, 4}}, "HDF5", "DataEncoding" -> "GZIP"]

and changing "GZIP" to None changes something in the output, but it does not compress the array.

Partial solution

One possible solution is to gzip the "HDF5" files. This seems to be recognised by Mathematica automatically. So either you manually gzip the files, or you use something along these lines to do everything in Mathematica

Export["matrix.h5.gz", ExportString[datapourrie, "HDF5"], "GZIP"]

For your test data this runs in no time, everything else needs probably a benchmarking and tweaking. To re-import your data you can simply do

Import["matrix.h5.gz", {"HDF5", "Datasets", "/Dataset1"}]
share|improve this answer
    
Thanks for this partial solution. But DataEncoding is not an import-only setting. Datasets neither. See for instance this Mathematica manual example: Export["m1.h5", {{1, 2}, {2, 3}}, {"Datasets", "m1"}] –  max Jan 28 '13 at 16:11
1  
"Datasets" is not an option, but an element specifier. When you look on other export help pages like the one of GIF, you see, that under Options you find which option can be used for in- and which for export. But honestly, I find the situation here very weird too. –  halirutan Jan 28 '13 at 16:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.