16
$\begingroup$

Sadly the documentation is lacking any useful examples on how to Export more complex data in HDF5. There is only

Export["file.h5",{expr1,...},{"Datasets", {"dataset1",...}}] creates an HDF5 file, storing the data arrays {expr1,...} as separate datasets.

The information on how to import HDF5 is more helpful. Information what can be imported is given in a table reproduced below:

enter image description here

Is there any way to export data in a more structured manner than simply as a flat list of individual datasets? Notice the 4th and 5th entry in the table above for importing. Is there an equivalent feature for exporting as well?

Take for instance data in the form of

data = <|
         "group1" -> <|"subgroup1" -> ..., "subgroup2" -> ...|>,
         "group2" -> <|"subgroup1" -> ..., "subgroup2" -> ...|>
         ...
       |> 
$\endgroup$
10
$\begingroup$

Thanks to @dan7geo's helpful hint to ExportStructuredHDF5 exporting to HDF5 is easier than I thought.

Take for instance

data = <|
         "group1" -> <|"subgroup1" -> "group1/subgroup1", 
                       "subgroup2" -> "group1/subgroup2", 
                       "subgroup3" -> 
                              <|
                              "subsubgroup1" -> "group1/subgroup3/subsubgroup1", 
                              "subsubgroup2" -> "group1/subgroup3/subsubgroup2"
                              |>
                     |>, 
         "group2" -> <|"subgroup1" -> "group2/subgroup1"|>|>

Create a temporary file and export data

file = CreateFile[];
ExportStructuredHDF5[file, data];

Importing the file with Import[file] shows a list of datasets names within the file.

{"/group1/subgroup1", "/group1/subgroup2", 
 "/group1/subgroup3/subsubgroup1", "/group1/subgroup3/subsubgroup2",
 "/group2/subgroup1"}

Note that these are the internal names of the datasets and not the content (I choose the content of data to match these names for testing).

A neat feature is to import the file's structure with Import["/Users/Sascha/Desktop/temp.h5", "StructureGraph"]

graph

Note that layout isn't too smart and the root node is the lightblue dot on the left.


Importing the actual data back as an association is done with

imported = Import[file, "Data"] 
<|"/group1/subgroup1" -> "group1/subgroup1", 
   "/group1/subgroup2" -> "group1/subgroup2", 
   "/group1/subgroup3/subsubgroup1" -> "group1/subgroup3/subsubgroup1", 
   "/group1/subgroup3/subsubgroup2" -> "group1/subgroup3/subsubgroup2", 
   "/group2/subgroup1" -> "group2/subgroup1"|>

As you see the importer isn't too smart either and just gives a flat associaton even though Import is aware of the file's structure as can be seen from the graph above.

To get back the original structure one can use ImportStructuredHDF5 instead.

ImportStructuredHDF5["file"] reads an HDF5 file previously created by ExportStructuredHDF5. The option "ImportAsRawArray" controls whether arrays of numbers are returned as RawArrays.

$\endgroup$
  • $\begingroup$ Sadly I did not find a way to use ExportStructuredHDF5 with compression yet, so file sizes are enormous. For comparison: One of my datasets I exported as is (781.9 MB) vs one where I applied Compress before exporting (1.3 MB). If anyone is aware of a compression method that can be applied here to retain compatibility with external tools, I would really appreciate it. $\endgroup$ – Sascha May 28 '17 at 12:50
  • $\begingroup$ Can you extend this solution to work with Images? $\endgroup$ – M.R. Mar 30 '18 at 19:16
8
+25
$\begingroup$

This might be helpful, although there aren't any details in the documentation.

ExportStructuredHDF5

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.