# How can I work with larger-than-memory hdf5 files

I am using Mathematica 11.0 and am trying to work with large .h5 files. Does anyone know if it's possible to work with files that are larger than the amount of available memory? I'm hoping that Mathematica would only read in the parts of the file that it needs. I understand that Mathematica 11.1 improved .h5-related functionality—anything new there?

My files are sets of microscopy images, size ~5 GB, and I would like to work with about 3 of these .h5 files at a time (some are label masks that I use to label particular objects in an image).

Side note—I tried to comment on this related post, but evidently I don't have enough reputation to do so:

HDF5 importing large arrays

• It is possible to read parts of hdf5-files. It has always been possible to only read a single dataset within a file, so if your files contain many dataset that would most probably already solve your problem. Since 11.1 you should be able to also read just hyperslabs of a single dataset. I would expect but have not tested that it now should be possible to read only parts of datasets which wouldn't fit into memory as a whole. I think you would need to give more details about the data in your files and what you try to do with that for a more useful answer... – Albert Retey Nov 24 '17 at 10:56
• Ok, thanks everyone. Unfortunately the output I'm using keeps everything in one dataset in the .h5 file, so it looks like the new 11.1 functionality is appropriate. As another option, I am also looking into trying Scot Martin's HDF5 package (github.com/scotmartin1234/HDF5Mathematica). – jeremyc0 Dec 4 '17 at 5:36

When Importing a single dataset, the whole file is not loaded into memory, so as long as the dataset you're interested in is smaller than the amount of available memory - you should be fine.

In M11.1 new Import option called "TakeElements" was introduced for HDF5 files. Consider the following dataset with dimensions {3, 4}:

In[2]:= Import["ExampleData/sample1.h5", "Integer/UInt8"]

Out[1]= {{12, 146, 220, 2}, {14, 78, 192, 173}, {16, 192, 134, 208}}


There are 3 different ways to use "TakeElements:

1. Specify a sequence of points. Since the dataset is 2D, each point is denoted by a pair. To Import first point of each row you can do:

In[4]:= Import["ExampleData/sample1.h5", "Integer/UInt8",
"TakeElements" -> {{1, 1}, {2, 1}, {3, 1}}]

Out[4]= {12, 14, 16}

2. Specify a span for each dimension. This works similarly to Part. Let's import first 3 elements of last two rows:

In[2]:= Import["ExampleData/sample1.h5", "Integer/UInt8",
"TakeElements" -> {2 ;; 3, 1 ;; 3}]

Out[2]= {{14, 78, 192}, {16, 192, 134}}

3. Using subset specification "native" to HDF5. This is quite complicated but also very powerful. I recommend the official HDF5 tutorial. It has some nice pictures to illustrate what's going on. Suppose we want to skip the first column ("Offset" -> {0,1}), and than take one element ("Block" -> 1) from every two consecutive elements, both row-wise and column-wise ("Stride" -> 2) but make only 2 steps in each direction ("Count" -> 2):

In[1]:= Import["ExampleData/sample1.h5", "Integer/UInt8",
"TakeElements" -> {"Offset" -> {0, 1}, "Stride" -> 2, "Block" -> 1, "Count" -> 2}]

Out[1]= {{146, 2}, {192, 208}}


In all 3 cases only the data that you Import will be loaded from the file to the memory.

• Just wanted to add that for the second method (in which you specify Spans), you can specify a single element by using a Span that starts and ends on the same element. For example, to get only the first element, you can use {1;;1,1;;1}. – jeremyc0 Jan 1 '18 at 5:53