3
$\begingroup$

I am obliged to work with existing .HDF files. These contain metadata essential to my research that Mathematica (latest version) appears unable to access. Metadata listed on https://reference.wolfram.com/language/ref/format/HDF.html as being accessible are "Datasets" and "Dimensions", which I have read successfully. However, there are others such as, in my specific case, ""actual_range", "metrics" and "_Fill_Value" amongst others that I haven't been able to access. Is there a means within Mathematica to access all metadata?

HDF5 files are better supported, however project data is unavailable in this format.

$\endgroup$
  • $\begingroup$ Hi David, one option I can think of off the top of my head is to use ExternalEvaluate, Python, and pyhdf (aka python-hdf4) to make this happen... Be warned, though, pyhdf can be hell to install. For anybody interested, a large number of HDF4 files are available here. To be clear, I think you're trying to access the attributes of the HDF dataset, right? $\endgroup$ – Carl Lange Feb 11 at 8:28
  • $\begingroup$ Hi Carl. Thank you so much for your response. In answer to your query, yes, attributes of the HDF4 dataset. $\endgroup$ – David H. Feb 11 at 8:35
  • $\begingroup$ In my limited experience with HDF5 files (not generic HDF) Mathematica's implementation is severely limited. Smuggling the metadata in by some other means (as @CarlLange suggests) may be simplest. $\endgroup$ – evanb Feb 11 at 9:57
  • $\begingroup$ Thanks Evan. I have the spec for HDF4 which details how metadata is handled. These are binary files of course, there are times I pine for ASCII. $\endgroup$ – David H. Feb 11 at 10:13
  • 1
    $\begingroup$ Another thing you can try is to convert your HDF file to HDF5 (for example using h4toh5: support.hdfgroup.org/products/hdf5_tools/h4toh5/index.html) and then use Import. $\endgroup$ – rafalc Feb 11 at 12:34
4
$\begingroup$

The easiest way to do this is by passing off all the actual work to python.

This requires that you have a python environment that has pyhdf installed. Honestly, the bulk of the work is going to be actually installing pyhdf, and I'm afraid that I can't help you there.

Assuming that you do have a python environment with pyhdf installed, the rest of this answer is for you.

What's going to happen is we're going to ask pyhdf for the results using ExternalEvaluate. If you've used python before the code is actually very simple.

First, we need to set up an ExternalEvaluate session:

session = 
 StartExternalSession[<|"System" -> "Python", "Version" -> "3.7.1", 
   "Executable" -> "/usr/local/bin/python3.7"|>]

Here I'm linking the session to my local python install that has pyhdf installed. If you got pyhdf using conda or similar, you'll want to find what the right path is for your executable.

Now that we have a session, we set up the pyhdf library:

ExternalEvaluate[session, "from pyhdf.SD import SD, SDC"]

And now we can for instance read information about the datasets in the file. This returns a native Wolfram Language association.

ExternalEvaluate[session, "hdf.datasets()"]

enter image description here

Now, let's use the Land/SeaMask dataset as an example, and read its attributes:

ExternalEvaluate[session, "hdf.select('Land/SeaMask').attributes()"]

<|"valid_range" -> {0, 7}, "_FillValue" -> 221|>

Fantastic. Well, there's your question answered, really!

We can make this a nice function like so:

readHDFDatasetAttributes[file_, dataset_] :=
 ExternalEvaluate[<|"System" -> "Python", "Version" -> "3.7.1", 
   "Executable" -> "/usr/local/bin/python3.7"|>, StringTemplate["
    from pyhdf.SD import SD, SDC
    hdf = SD('`file`', SDC.READ)
    hdf.select('`dataset`').attributes()
    "][<|"file" -> file, "dataset" -> dataset|>]]

A final note: reading the dataset from python and reading it natively result in exactly the same data, which I find pleasing:

ExternalEvaluate[session, "hdf.select('Land/SeaMask')[:,:]"] == 
 Import[hdffile, {"Datasets", "Land/SeaMask"}]

True

$\endgroup$
  • $\begingroup$ Thank you so much for this inspiring reply. My colleagues are huge on Python. We work mostly with FITS files (astronomy). I don't know of anyone who's touched HDF. Hence, I relish the opportunity to have Mathematica employ Python even as just an exercise but I suspect it will be rewarding in my work too. Best wishes. $\endgroup$ – David H. Feb 12 at 0:54
  • $\begingroup$ @DavidH. I'm glad you find it useful - if it's answered your question, please consider marking it as accepted! Last thing: updated HDF5 and FITS support is coming in version 12. $\endgroup$ – Carl Lange Feb 12 at 7:34
  • $\begingroup$ Thank you Carl, all. $\endgroup$ – David H. Feb 13 at 23:06

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.