How can I export to HDF5 with a specific Datatype (“DataFormat”)

Question

I am trying to export some largish integer arrays to HDF5 and know that every entry in them would fit into an unsigned 8 bit integer array. As the default "DataFormat" for export that Mathematica is using is a 32bit integer array the resulting files are unnecessarily large. Does anyone know the correct syntax to export such an integer array as an 8 bit integer dataset?

I found the following suggestions in this answer and a comment to this answer:

Export["int8.h5",
  RandomInteger[{2^8 - 1}, {100, 100}],
  "DataFormat" -> {"UnsignedInteger8"}
]

Export["int8.h5",
  RandomInteger[{2^8 - 1}, {100, 100}],
  {"UnsignedInteger8", "DataFormat"}
]

but they both seem to not work at all or not create the expected data representation in the files, which can be checked by either HDFView (which I consider authorative) or Mathematica itself with:

Import["int8.h5","DataFormat"]

Answer 1

The following seems to work, but I'd also be interested in other, possibly simpler or clearer versions...

Export["int8.h5",
  {
    "Datasets" -> {"/data" -> RandomInteger[{2^8 - 1}, {100, 100}]},
    "DataFormat" -> {"UnsignedInteger8"}
  },
  "Rules"
]

Two notes:

1. Unfortunately I could not create a generalization of these which make the (both documented but obviously broken) features compression ("DataEncoding" element) or attributes ("Attributes" element) work. So if someone has a solution for those, I'd appreciate and accept answers which solve those.
2. I now found that in a comment to this answer Szabolcs has the working syntax (for the not working "DataEncoding" element), but unfortunately that is quite hard to find so I hope this question (and answers) is acceptable anyway...

Answer 2

Thie syntax has slightly changed in Mathematica 12. Now it can be done like this:

In[2]:= Export["test.h5", "path/to/dataset" -> {
   "Data" -> RandomInteger[{2^8 - 1}, {100, 100}],
   "DataFormat" -> "UnsignedInteger8"
   }]

Out[2]= "test.h5"

In[4]:= Import["test.h5", {{"DataFormat", "Dimensions"}, "path/to/dataset"}]

Out[4]= {"UnsignedInteger8", {100, 100}}

For large data, when possible, I recommend using NumericArray (new in 12). In that case information about the type is included in the NumericArray object and is respected by the HDF5 Export:

In[7]:= Export["test.h5", NumericArray @ RandomInteger[{2^8 - 1}, {100, 100}]]

Out[7]= "test.h5"

In[8]:= Import["test.h5", {{"DataFormat", "Dimensions"}, "Dataset1"}]

Out[8]= {"UnsignedInteger8", {100, 100}}

In the example above NumericArray type was automatically deduced as "UnsignedInteger8" but you can also specify it manually as the second argument.

Answer 3

You can see:

https://github.com/scotmartin1234/HDF5Mathematica

The package has the functionality you need for writing at low level. The high level functions are designed more for reading. The package could be expanded as open source project to develop the writing part. It's straight forward to assemble the low level functions into a more Mathematica-friendly format (i.e., the higher level functions), but it would take a volunteer to do so. My efforts and work have been on developing the reading side of the equation into a Mathematica friendly format.

This is version 2.00 (August 2016) of the package that was originally provided as version 1.00 in July 2011