3
$\begingroup$

I am obliged to work with existing .HDF files. These contain metadata essential to my research that Mathematica (latest version) appears unable to access. Metadata listed on https://reference.wolfram.com/language/ref/format/HDF.html as being accessible are "Datasets" and "Dimensions", which I have read successfully. However, there are others such as, in my specific case, ""actual_range", "metrics" and "_Fill_Value" amongst others that I haven't been able to access. Is there a means within Mathematica to access all metadata?

HDF5 files are better supported, however project data is unavailable in this format.

Improve this question
$\endgroup$
11
  • $\begingroup$ Hi David, one option I can think of off the top of my head is to use ExternalEvaluate, Python, and pyhdf (aka python-hdf4) to make this happen... Be warned, though, pyhdf can be hell to install. For anybody interested, a large number of HDF4 files are available here. To be clear, I think you're trying to access the attributes of the HDF dataset, right? $\endgroup$
    – Carl Lange
    Commented Feb 11, 2019 at 8:28
  • $\begingroup$ Hi Carl. Thank you so much for your response. In answer to your query, yes, attributes of the HDF4 dataset. $\endgroup$
    – David H.
    Commented Feb 11, 2019 at 8:35
  • $\begingroup$ In my limited experience with HDF5 files (not generic HDF) Mathematica's implementation is severely limited. Smuggling the metadata in by some other means (as @CarlLange suggests) may be simplest. $\endgroup$
    – evanb
    Commented Feb 11, 2019 at 9:57
  • $\begingroup$ Thanks Evan. I have the spec for HDF4 which details how metadata is handled. These are binary files of course, there are times I pine for ASCII. $\endgroup$
    – David H.
    Commented Feb 11, 2019 at 10:13
  • 1
    $\begingroup$ Another thing you can try is to convert your HDF file to HDF5 (for example using h4toh5: support.hdfgroup.org/products/hdf5_tools/h4toh5/index.html) and then use Import. $\endgroup$
    – rafalc
    Commented Feb 11, 2019 at 12:34

1 Answer 1

Reset to default
5
$\begingroup$

The easiest way to do this is by passing off all the actual work to python.

This requires that you have a python environment that has pyhdf installed. Honestly, the bulk of the work is going to be actually installing pyhdf, and I'm afraid that I can't help you there.

Assuming that you do have a python environment with pyhdf installed, the rest of this answer is for you.

What's going to happen is we're going to ask pyhdf for the results using ExternalEvaluate. If you've used python before the code is actually very simple.

First, we need to set up an ExternalEvaluate session:

session = 
 StartExternalSession[<|"System" -> "Python", "Version" -> "3.7.1", 
   "Executable" -> "/usr/local/bin/python3.7"|>]

Here I'm linking the session to my local python install that has pyhdf installed. If you got pyhdf using conda or similar, you'll want to find what the right path is for your executable.

Now that we have a session, we set up the pyhdf library:

ExternalEvaluate[session, "from pyhdf.SD import SD, SDC"]

And now we can for instance read information about the datasets in the file. This returns a native Wolfram Language association.

ExternalEvaluate[session, "hdf.datasets()"]

enter image description here

Now, let's use the Land/SeaMask dataset as an example, and read its attributes:

ExternalEvaluate[session, "hdf.select('Land/SeaMask').attributes()"]

<|"valid_range" -> {0, 7}, "_FillValue" -> 221|>

Fantastic. Well, there's your question answered, really!

We can make this a nice function like so:

readHDFDatasetAttributes[file_, dataset_] :=
 ExternalEvaluate[<|"System" -> "Python", "Version" -> "3.7.1", 
   "Executable" -> "/usr/local/bin/python3.7"|>, StringTemplate["
    from pyhdf.SD import SD, SDC
    hdf = SD('`file`', SDC.READ)
    hdf.select('`dataset`').attributes()
    "][<|"file" -> file, "dataset" -> dataset|>]]

A final note: reading the dataset from python and reading it natively result in exactly the same data, which I find pleasing:

ExternalEvaluate[session, "hdf.select('Land/SeaMask')[:,:]"] == 
 Import[hdffile, {"Datasets", "Land/SeaMask"}]

True

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you so much for this inspiring reply. My colleagues are huge on Python. We work mostly with FITS files (astronomy). I don't know of anyone who's touched HDF. Hence, I relish the opportunity to have Mathematica employ Python even as just an exercise but I suspect it will be rewarding in my work too. Best wishes. $\endgroup$
    – David H.
    Commented Feb 12, 2019 at 0:54
  • $\begingroup$ @DavidH. I'm glad you find it useful - if it's answered your question, please consider marking it as accepted! Last thing: updated HDF5 and FITS support is coming in version 12. $\endgroup$
    – Carl Lange
    Commented Feb 12, 2019 at 7:34
  • $\begingroup$ Thank you Carl, all. $\endgroup$
    – David H.
    Commented Feb 13, 2019 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.