Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I would like to export data in HDF5 format with Mathematica 9. The Mathematica manual about Import and Export of this format can be read here.

As I'm dealing with large arrays, export often lead to 100MB+ files. I'd like to use the HDF5 "DataEncoding" to compress the exported data, but I can't find a way to use it.

For example, let's consider an array called

datapourrie = RandomInteger[99, {100000, 2, 5}]

First, I export it in simple ASCII format, for the purpose of comparison:

Export["expl.dat", datapourrie]

expl.dat is correctly written, with a size of 4.1 MB.

Now, let's try in HDF5 format:

Export["expl.h5", datapourrie, {"Datasets", "mydata"}]

The file is correctly written in HDF5 format. It's 5.0 MB (the data + metadata). I tested it with a HDF5 viewer outside of Mathematica: everything's OK.

Now, I'd like to compress it, because one of the main advantages of HDF5 format is that it can be used as a binary (compressed) file format. From the Mathematica 9 manual, I read:

"DataEncoding" can be none, "GZIP" or "SZIP"

My problem is that I can't find a way to use this DataEncoding! I would expect the correct sentence to be:

 Export["explgzip.h5", datapourrie, {"Datasets", "mydata"}, "DataEncoding" -> "GZIP"]

but it produces exactly the same file as without the "DataEncoding" tag. What am I doing wrong?

share|improve this question
    
"Datasets" seems to be an import setting, not an export setting. –  Sjoerd C. de Vries Jan 28 '13 at 14:59
    
@SjoerdC.deVries "Datasets" is an export and import setting as it gives a name to a datasubset. Thus, you can Import a given subset only when you know their dataset name. –  max Jan 28 '13 at 15:49
add comment

3 Answers 3

up vote 3 down vote accepted

In version 7 halirutan's export method does not produce a file that is recognized by Import.

However, one can write:

Export["matrix2.h5.gz", datapourrie, {"GZIP", "HDF5"}]

And then:

d2 = Import["matrix2.h5.gz", {"Datasets", "/Dataset1"}];

datapourrie == d2

True

share|improve this answer
1  
This isn't always a workable solution though. What I need right now is internal compression for HDF5 files, not just separately compressing the HDF5 file with gzip. I need to produce a non-zipped HDF5 file that uses internal compression and can be directly read by other programs that support the HDF5 format. I tagged this as a bug because it seems that Mathematica simply doesn't honour the DataEncoding setting. –  Szabolcs May 14 at 18:34
    
Something is still bothering me here: I suspect we're not reading the docs correctly. It says that "DataEncoding" is a "meta-information element", not an option. So it should be given as an element. It's not 100% clear to me how this is done properly but I tried: Export["~/Desktop/d.h5", {"Datasets" -> {data}, "DataEncoding" -> "GZIP"}, "Rules"] based on the Export documentation and some Export examples for other formats. This still doesn't produce a compressed file though. –  Szabolcs May 14 at 19:19
add comment

The documentation is misleading here. On one hand, the only export option is "Append" which can be found under the Options tab. On the other hand, the general documentation reads

enter image description here

I really wonder, why it is necessary to put Import only behind an option value when "DataEncoding" isn't an export option at all.

Anyway, I have the same behaviour in MacOSX as you have: No data compression. Although, using

ExportString[{{1, 2}, {3, 4}}, "HDF5", "DataEncoding" -> "GZIP"]

and changing "GZIP" to None changes something in the output, but it does not compress the array.

Partial solution

One possible solution is to gzip the "HDF5" files. This seems to be recognised by Mathematica automatically. So either you manually gzip the files, or you use something along these lines to do everything in Mathematica

Export["matrix.h5.gz", ExportString[datapourrie, "HDF5"], "GZIP"]

For your test data this runs in no time, everything else needs probably a benchmarking and tweaking. To re-import your data you can simply do

Import["matrix.h5.gz", {"HDF5", "Datasets", "/Dataset1"}]
share|improve this answer
    
Thanks for this partial solution. But DataEncoding is not an import-only setting. Datasets neither. See for instance this Mathematica manual example: Export["m1.h5", {{1, 2}, {2, 3}}, {"Datasets", "m1"}] –  max Jan 28 '13 at 16:11
2  
"Datasets" is not an option, but an element specifier. When you look on other export help pages like the one of GIF, you see, that under Options you find which option can be used for in- and which for export. But honestly, I find the situation here very weird too. –  halirutan Jan 28 '13 at 16:19
add comment

One workaround is to compress the HDF5 file after it has been exported from Mathematica, using the HDF5 command line tools.

Note: on OS X the command line tools can be easily installed using MacPorts using port install h5utils.

The command to recompress the data is

h5repack -v -f GZIP=1 infile.h5 outfile.h5

This can indeed achieve a significant reduction in size.

For convenience you might want to invoke this from within Mathematica using Run.

share|improve this answer
2  
I sent a support request asking about how to use "DataEncoding" properly, because I need this too. I'll report back when I receive a reply. –  Szabolcs May 14 at 19:03
    
Thanks for this partial solution. I am still interested in finding a solution, so your report is welcome. –  max May 15 at 7:30
1  
@max The reply I received says that while the correct syntax should be Export["newtesthdf5.h5", data, {"GZIP", "DataEncoding"}], this produces a file that's not readable by HDFView so a bug report has been filed. I'll re-tag it as a bug based on this. –  Szabolcs May 16 at 19:04
    
Is it just me or is the "correct syntax" strange? –  Mr.Wizard May 16 at 20:14
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.