# Where is HDF5Tools documentation?

MMA 12 apparently has a new package, HDF5Tools, which includes desperately needed functions that allow one to open an HDF5 file, get data from it, then close it when done. I don't really know this to be true, but I think it is, because when I type , for instance, "HDF5O" MMA provides me with a list of completions including "HDF5OpenFile (HDF5Tools)", and many other intriguing functions that seem designed to solve a long-standing problem with MMA's HDF5 interface. In previous version you could Import and Export HDF5, but with the documented functions you couldn't open a big HDF5 file, get particular pieces of info from it (or put info to it), then close it when done.

However, I have not been able to find any documentation of this new package. The usual F1 trick yields nothing useful, and a web search for HDF5Tools mathematica is likewise bootless.

Are these functions actually implemented in MMA 12.0? If so, are they documented anywhere?

I believe this package contains internal functions, used by Import when working with HDF5 files. These functions have no documentation, and aren't meant to be called from top level.

If you evaluate

Needs["HDF5Tools"];
?HDF5Tools*


and click any of the symbols, you'll see most have usage messages. So you may be able to figure out what they do, but the documented method is to call Import.

If you need data that is in the HDF5 file that isn't being returned by Import, it would be helpful to send this request to customer support, so that the importing can be improved.

• "If you need data that is in the HDF5 file that isn't being returned by Import"... That's not the issue. Suppose you have a 4 GB HDF5 file. You want to modify one small part, or add another dataset. Importing the whole thing into memory, making the change you want, then exporting it all is horribly inefficient. I guess I'll continue to use h5py... – Leon Avery Aug 19 '19 at 20:59
• @LeonAvery that's a different question from the one you posed, above. Have you tried it in 12? The functionality for importing/exporting pieces of the file has been improved. – rcollyer Aug 19 '19 at 21:02
• @LeonAvery I think that you missunderstand the functionality of Import and Export. It is absolutely possible to only import parts of the data and it also is possible to add new datasets to an existing hdf5 file. There are some issues and some missing functionalities, and I have complaining about these here and at WRI more than once, but what you are asking for should be possible since many years. Can you show an example of what you think does not work? – Albert Retey Aug 20 '19 at 11:32
• @AlbertRetey, I can't fit my example into a comment. – Leon Avery Aug 20 '19 at 12:05
• @rcollyer. Yes, my comment refers to a different question from the one I posed. The question I posed was about documentation. The comment above is a reaction to a separate issue raised by Jason B in his answer. – Leon Avery Aug 20 '19 at 12:10

Considering the question about the documentation Jasons answer makes clear that there basically is none (except for the usage messages) and the HDF5Tools package at this time (2019) unfortunately seems to be meant for internal use only.

# Documented Import/Export Functionality

You have been stating that you would want to use that functionality to read only parts of an HDF5 file and append data to an existing one. I have mentioned in my comments that I think that you might be able to do what you ask for already with documented functionality. The following should work since at least version 11.2:

Create a file with a small and a large dataset:

large = RandomReal[{0, 255}, 10^8];
Export["test.h5", {"large" -> large, "small" -> Range[20]}];


Now (if you want to check memory usage) after quitting the kernel you can get only part of the file content:

small = Import["test.h5", {"Datasets", "small"}];
part = Import["test.h5", {"Datasets", "large"}, "TakeElements" -> {1 ;; 10}];


You can check the both will not import the whole file into the Mathematica kernel. You can also add ("append") new datasets to an existing file (but not change, replace or append to existing datasets, AFAIK):

Export["test.h5", part, {"Datasets", "part"}, "Append" -> True]


Of course I don't know whether this documented functionality is good enough for what you need, but it might be. I understand that there is still a lot of functionality missing in the documented Import/Export functionality. I haven't checked with version 12 yet, but performance might also be not as good as it could be, especially the memory usage in older versions has been unreasonably high. I have complained about that on this site and elsewhere before. So I definitely see that a documentated way to make use of the HDF5Tools would be useful and would appreciate any better answers to the actual question about the documentation.

# Spelunking

First of all it might be interesting to know that the HDFTools package already exists and works in version 11.2 and is not new to 12.0.

Looking at the package context, it looks like there are some higher level functions in the toolbox that start with HDF5 and allow a relatively simple programmatic access to HDF5 files and many are easy enough to use with just the usage messages. But these function probably do not provide much more than what the Import/Export functions also provide. So it seems relatively uninteresting to delve deeper into those at the moment.

## High(er) Level Interface

Here is an example of how you could use them to read parts of a dataset:

Needs["HDF5Tools"]
fh = HDF5OpenFile[filename, H5FACCRDONLY]
dh = HDF5OpenDataset[fh, "/small"]
data = HDF5ReadDataset[dh, "TakeElements" -> Range[5]]
HDF5CloseDataset[dh]
HDF5CloseFile[fh]


## Lower Level Interface

The differences between these Mathematica functions and the documented HDF5-C-Library functions seem to be mainly that these functions return arrays instead of filling allocated buffers given as additional arguments as usual in C.

Here is an example of how to use that lower-level interface, which was more or less a direct translation of a C-LibraryFunction I wrote a few years ago to examine the possibilities of faster access to HDF5 files (here I read the second half of the /small dataset from the above example file):

HDF5ToolsInit[];
fid = h5fopen[filename, H5FACCRDONLY];
did = h5dopen[fid, "small", H5PDEFAULT];
fsid = h5dgetspace[did];
rank = h5sgetsimpleextentndims[fsid];
msid = h5screatesimple[rank, {10}, {10}];
h5sselecthyperslab[msid, H5SSELECTSET, {0}, {1}, {10}, {1}];
h5sselecthyperslab[fsid, H5SSELECTSET, {10}, {1}, {10}, {1}];
data = h5dread[did, H5TNATIVEDOUBLE, msid, fsid, H5PDEFAULT];
h5sclose[msid];
h5sclose[fsid];
h5dclose[did];
h5fclose[fid];
Normal[data]


As you can see, the lower level interface is quite verbose and unconvenient and it needs some familiarity with the HDF5 library to achieve anything nontrivial. As has to be expected for non-documented functionality there have been changes between version 11.2 and 12, e.g. the data is returned as a NumericArray in version 12. On the other hand these differences seem to be small and it looks like the HDF5Tools are implemented as LibraryFunctions which should give good performance without extra copies of data, so I would expect these to show good performance even with larger datasets. They are of course not nearly as convenient to use as the h5py library...

Finally here is a piece of code which does something that would not be possible with the existing Import/Export functionality: we overwrite a part of a dataset with new values:

HDF5ToolsInit[]
fid = h5fopen[filename, H5FACCRDWR]
did = h5dopen[fid, "small", H5PDEFAULT]
fsid = h5dgetspace[did]
rank = h5sgetsimpleextentndims[fsid]
msid = h5screatesimple[rank, {10}, {10}]
h5sselecthyperslab[msid, H5SSELECTSET, {0}, {1}, {10}, {1}];
h5sselecthyperslab[fsid, H5SSELECTSET, {10}, {1}, {10}, {1}];
h5dwrite[did, H5TNATIVEDOUBLE, msid, fsid, H5PDEFAULT, Table[k, {k, 100, 109}]]
h5sclose[msid];
h5sclose[fsid];
h5dclose[did];
h5fclose[fid];


One can check with e.g.

Import[filename, {"Datasets", "/small"}]


that this has indeed worked as expected.

I hope that this demonstrates that with some familiarity with the HDF5 library it would probably be relatively straightforward to handle HDF5 files in avery general way and very efficiently with the HDF5Tools package...