1
$\begingroup$

I'm trying to import an item from a valid HDF5 file, but I get this error:

enter image description here

I can read the metadata, but cannot import the actual tensor dataset:

enter image description here

And yet it's not an issue with the file's data - as importing (with h5py) in Python works just fine:

enter image description here

Perhaps there's a way of importing pieces of the HDF5 file that works?

Details:

  • Here's a link to the h5 dataset to try it yourself
  • I'm using MMA 11.3 on latest MacOS
  • I've already tried using HDF5Tools and GeneralUtilities`MLStorage without any luck.
|improve this question
$\endgroup$
  • $\begingroup$ The h5 file is about a 1gb. If anyone wants to try processing it I can post it to dropbox... $\endgroup$ – M.R. Sep 19 '18 at 0:18
  • $\begingroup$ Please provide a link to the file. Which version of Mathematica are you using? $\endgroup$ – rafalc Sep 19 '18 at 6:35
  • $\begingroup$ @rafalc Ok, I added the link and I'm using 11.3 $\endgroup$ – M.R. Sep 19 '18 at 15:14
1
$\begingroup$

Let's examine the dataset metadata with h5dump:

$ h5dump -p -H --dataset="/input_image" file.h5

DATASET "input_image" {
      DATATYPE  H5T_STD_U8LE
      (...)
      FILTERS {
         PREPROCESSING SHUFFLE
         USER_DEFINED_FILTER {
            FILTER_ID 32000
            COMMENT lzf
            PARAMS { 4 261 32768 }
         }
      }
   (...)
   }

In case of unknown filters Mathematica is a little less informative, as it does not give the name of the filter ("LZF") only the ID:

In[15]:= Import["file.h5", {"DataEncoding", "/input_image"}]

Out[15]= {"Shuffle", "Unknown(32000)"}

HDF5 comes with a number of predefined filters:

H5Z_FILTER_DEFLATE - The gzip compression, or deflation, filter
H5Z_FILTER_SZIP - The SZIP compression filter
H5Z_FILTER_NBIT - The N-bit compression filter
H5Z_FILTER_SCALEOFFSET - The scale-offset compression filter
H5Z_FILTER_SHUFFLE - The shuffle algorithm filter
H5Z_FILTER_FLETCHER32 - The Fletcher32 checksum, or error checking, filter

As you can see, LZF is not one of them. It turns out that h5py adds its own filters to whatever is shipped with HDF5. This is how h5py documents the LZF filter:

"lzf"
Custom compression filter for h5py. This filter is much, much faster than gzip (roughly 10x in compression vs. gzip level 4, and 3x faster in decompressing), but at the cost of a worse compression ratio. Use this if you want cheap compression and portability is not a concern.

In other words, you have a file which uses non-standard compression filter which Mathematica does not understand so it cannot decode the data.

That being said, the error message from Mathematica should be more precise.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for this informative answer. Is there anyway to re-export it from python to make it importable (get rid of the LZF filter)? $\endgroup$ – M.R. Sep 19 '18 at 17:27
  • $\begingroup$ In HDF5 you cannot change filters on the existing dataset. But you can always import a dataset with h5py and export the data to a new dataset without the LZF filter. From what I see, all datasets in your file use LZF so you may want to repeat this action in a loop. $\endgroup$ – rafalc Sep 19 '18 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.