Problem importing from HDF5

Question

I'm trying to import an item from a valid HDF5 file, but I get this error:

I can read the metadata, but cannot import the actual tensor dataset:

And yet it's not an issue with the file's data - as importing (with h5py) in Python works just fine:

Perhaps there's a way of importing pieces of the HDF5 file that works?

Details:

• Here's a link to the h5 dataset to try it yourself
• I'm using MMA 11.3 on latest MacOS
• I've already tried using HDF5Tools and GeneralUtilities`MLStorage without any luck.

Answer 1

Let's examine the dataset metadata with h5dump:

$ h5dump -p -H --dataset="/input_image" file.h5

DATASET "input_image" {
      DATATYPE  H5T_STD_U8LE
      (...)
      FILTERS {
         PREPROCESSING SHUFFLE
         USER_DEFINED_FILTER {
            FILTER_ID 32000
            COMMENT lzf
            PARAMS { 4 261 32768 }
         }
      }
   (...)
   }

In case of unknown filters Mathematica is a little less informative, as it does not give the name of the filter ("LZF") only the ID:

In[15]:= Import["file.h5", {"DataEncoding", "/input_image"}]

Out[15]= {"Shuffle", "Unknown(32000)"}

HDF5 comes with a number of predefined filters:

H5Z_FILTER_DEFLATE - The gzip compression, or deflation, filter
H5Z_FILTER_SZIP - The SZIP compression filter
H5Z_FILTER_NBIT - The N-bit compression filter
H5Z_FILTER_SCALEOFFSET - The scale-offset compression filter
H5Z_FILTER_SHUFFLE - The shuffle algorithm filter
H5Z_FILTER_FLETCHER32 - The Fletcher32 checksum, or error checking, filter

As you can see, LZF is not one of them. It turns out that h5py adds its own filters to whatever is shipped with HDF5. This is how h5py documents the LZF filter:

"lzf"
Custom compression filter for h5py. This filter is much, much faster than gzip (roughly 10x in compression vs. gzip level 4, and 3x faster in decompressing), but at the cost of a worse compression ratio. Use this if you want cheap compression and portability is not a concern.

In other words, you have a file which uses non-standard compression filter which Mathematica does not understand so it cannot decode the data.

That being said, the error message from Mathematica should be more precise.