15
$\begingroup$

I would like to export data in HDF5 format with Mathematica 9. The Mathematica manual about Import and Export of this format can be read here.

As I'm dealing with large arrays, export often lead to 100MB+ files. I'd like to use the HDF5 "DataEncoding" to compress the exported data, but I can't find a way to use it.

For example, let's consider an array called 

datapourrie = RandomInteger[99, {100000, 2, 5}]

First, I export it in simple ASCII format, for the purpose of comparison:

Export["expl.dat", datapourrie]

expl.dat is correctly written, with a size of 4.1 MB.

Now, let's try in HDF5 format:

Export["expl.h5", datapourrie, {"Datasets", "mydata"}]

The file is correctly written in HDF5 format. It's 5.0 MB (the data + metadata). I tested it with a HDF5 viewer outside of Mathematica: everything's OK.

Now, I'd like to compress it, because one of the main advantages of HDF5 format is that it can be used as a binary (compressed) file format. From the Mathematica 9 manual, I read:

"DataEncoding" can be none, "GZIP" or "SZIP"

My problem is that I can't find a way to use this DataEncoding! I would expect the correct sentence to be:

 Export["explgzip.h5", datapourrie, {"Datasets", "mydata"}, "DataEncoding" -> "GZIP"]

but it produces exactly the same file as without the "DataEncoding" tag. What am I doing wrong?

Improve this question
$\endgroup$
2
  • $\begingroup$ "Datasets" seems to be an import setting, not an export setting. $\endgroup$
    – Sjoerd C. de Vries
    Jan 28, 2013 at 14:59
  • $\begingroup$ @SjoerdC.deVries "Datasets" is an export and import setting as it gives a name to a datasubset. Thus, you can Import a given subset only when you know their dataset name. $\endgroup$
    – max
    Jan 28, 2013 at 15:49

4 Answers 4

Reset to default
12
$\begingroup$

The documentation is misleading here. On one hand, the only export option is "Append" which can be found under the Options tab. On the other hand, the general documentation reads

enter image description here

I really wonder, why it is necessary to put Import only behind an option value when "DataEncoding" isn't an export option at all.

Anyway, I have the same behaviour in MacOSX as you have: No data compression. Although, using

ExportString[{{1, 2}, {3, 4}}, "HDF5", "DataEncoding" -> "GZIP"]

and changing "GZIP" to None changes something in the output, but it does not compress the array.

Partial solution

One possible solution is to gzip the "HDF5" files. This seems to be recognised by Mathematica automatically. So either you manually gzip the files, or you use something along these lines to do everything in Mathematica

Export["matrix.h5.gz", ExportString[datapourrie, "HDF5"], "GZIP"]

For your test data this runs in no time, everything else needs probably a benchmarking and tweaking. To re-import your data you can simply do

Import["matrix.h5.gz", {"HDF5", "Datasets", "/Dataset1"}]
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for this partial solution. But DataEncoding is not an import-only setting. Datasets neither. See for instance this Mathematica manual example: Export["m1.h5", {{1, 2}, {2, 3}}, {"Datasets", "m1"}] $\endgroup$
    – max
    Jan 28, 2013 at 16:11
  • 2
    $\begingroup$ "Datasets" is not an option, but an element specifier. When you look on other export help pages like the one of GIF, you see, that under Options you find which option can be used for in- and which for export. But honestly, I find the situation here very weird too. $\endgroup$
    – halirutan ♦
    Jan 28, 2013 at 16:19
8
$\begingroup$

One workaround is to compress the HDF5 file after it has been exported from Mathematica, using the HDF5 command line tools.

Note: on OS X the command line tools can be easily installed using MacPorts using port install h5utils.

The command to recompress the data is

h5repack -v -f GZIP=1 infile.h5 outfile.h5

This can indeed achieve a significant reduction in size.

For convenience you might want to invoke this from within Mathematica using Run.

Improve this answer
$\endgroup$
7
  • 4
    $\begingroup$ I sent a support request asking about how to use "DataEncoding" properly, because I need this too. I'll report back when I receive a reply. $\endgroup$
    – Szabolcs
    May 14, 2014 at 19:03
  • $\begingroup$ Thanks for this partial solution. I am still interested in finding a solution, so your report is welcome. $\endgroup$
    – max
    May 15, 2014 at 7:30
  • 2
    $\begingroup$ @max The reply I received says that while the correct syntax should be Export["newtesthdf5.h5", data, {"GZIP", "DataEncoding"}], this produces a file that's not readable by HDFView so a bug report has been filed. I'll re-tag it as a bug based on this. $\endgroup$
    – Szabolcs
    May 16, 2014 at 19:04
  • 1
    $\begingroup$ Is it just me or is the "correct syntax" strange? $\endgroup$
    – Mr.Wizard
    May 16, 2014 at 20:14
  • $\begingroup$ Any updates on the bug report? yashar asked a question about HDF5 encoding, and then I found this question, and was wondering if the issue has been resolved. Also, what's the correct syntax for specifying both compression (say, "GZIP") and a data format (say, "Real32")? I tried the analogous Export["randmat.h5", RandomReal[{0, 1}, {1000, 1000}], {"GZIP", "DataEncoding"}, {"Real32", "DataFormat"}] but just got error messages. $\endgroup$
    – DumpsterDoofus
    Jan 7, 2015 at 19:24
6
$\begingroup$

In version 7 halirutan's export method does not produce a file that is recognized by Import.

However, one can write:

Export["matrix2.h5.gz", datapourrie, {"GZIP", "HDF5"}]

And then:

d2 = Import["matrix2.h5.gz", {"Datasets", "/Dataset1"}];

datapourrie == d2

True

Improve this answer
$\endgroup$
2
  • 5
    $\begingroup$ This isn't always a workable solution though. What I need right now is internal compression for HDF5 files, not just separately compressing the HDF5 file with gzip. I need to produce a non-zipped HDF5 file that uses internal compression and can be directly read by other programs that support the HDF5 format. I tagged this as a bug because it seems that Mathematica simply doesn't honour the DataEncoding setting. $\endgroup$
    – Szabolcs
    May 14, 2014 at 18:34
  • $\begingroup$ Something is still bothering me here: I suspect we're not reading the docs correctly. It says that "DataEncoding" is a "meta-information element", not an option. So it should be given as an element. It's not 100% clear to me how this is done properly but I tried: Export["~/Desktop/d.h5", {"Datasets" -> {data}, "DataEncoding" -> "GZIP"}, "Rules"] based on the Export documentation and some Export examples for other formats. This still doesn't produce a compressed file though. $\endgroup$
    – Szabolcs
    May 14, 2014 at 19:19
1
$\begingroup$

You can see:

https://github.com/scotmartin1234/HDF5Mathematica

The package has the functionality you need for writing at low level. HOWEVER, the high level functions are designed more for reading. The package could be expanded as open source project to develop the writing part with the encoding; the encoding is part of the standard C library that is accessed by the interface in Mathematica. It's straight forward to assemble the low level functions into a more Mathematica-friendly format (i.e., the higher level functions), but it would take a volunteer to do so. My efforts and work have been on developing the reading side of the equation into a Mathematica friendly format.

This is version 2.00 (August 2016) of the package that was originally provided as version 1.00 in July 2011.

Improve this answer
$\endgroup$
1
  • $\begingroup$ does this work in mac? $\endgroup$
    – elbOlita
    Aug 23, 2017 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.