15
$\begingroup$

I am looking to Import HDF5 data with compound data. It appears, at least from what I read, that Mathematica 8.0.4 (and previous versions) does not support reading HDF5 with compound data.

file = "someData.h5";
Import[file, "Data"]
Import[file, {"Datasets", 1}]

Returns:

{{"Unsupported Datatype Class", "Compound"}, ... }
Import::h5type: "The datatype of the dataset "\foobar" is not currently supported. "

It has been a while, since Oct 2011 when Mathematica HDF5 and compound arrays was last updated. It was update with info about Scot Martin's HDF5 package. Unfortunately that package appears be a solution for Windows which uses two .Net dlls (hdf5dll.dll and HDF5DotNet.dll) and I have a Mac. Just wondering if there has been any progress?

I have looked at using h5dump to create an ASCII file and writing an importer. However there is the overhead of writing the file, reading it, and then parsing it.

|improve this question
$\endgroup$
  • $\begingroup$ Could you upload a simple, example HDF5 file with compound data to experiment with? $\endgroup$ – Markus Roellig Apr 9 '12 at 10:44
  • $\begingroup$ I will see what I can do. There are some h5 files available in Scot Martin's HDF5 package in the HDF5HighLevel ExampleFiles directory. h5ex_t_cmpd.h5 has compound data. $\endgroup$ – mmorris Apr 9 '12 at 15:00
9
$\begingroup$

I have created h5dumpImport, an open source Mathematica Package that provides a platform independent way to import HDF5 (.h5) file's datasets with compound datatypes while hiding much of the HDF5 implementation from the user. The package with documentation, examples, and unit test is located here.

Currently, the h5dumpImport package does not directly import the HDF5 (.h5) file format. The h5dumpImport package imports an ASCII dump of a dataset generated by the h5dump command line tool.

Source code and pre-built binary distributions of the HDF5 Software which includes the h5dump command line tool can be found at the The HDF Group's website.

Basic Example

Needs["h5dumpImport`"]
datasets = Import["testData.h5", {"Datasets"}];
dumpFile = h5dump["/usr/bin/h5dump", "testData.h5", datasets[[1]]];
dumpImport = h5dumpImportNew[h5dumpImport[], dumpFile];
dumpImport.h5dumpImportData[All]
dumpImport.h5dumpImportClose[];

Results:

{{1, 11, 111, 1111, 11111, 111111, 1111111, 1.1, 11.11, "one"},
 {2, 22, 222, 2222, 22222, 222222, 2222222, 2.2, 22.22, "two"},
 {3, 33, 333, 3333, 33333, 333333, 3333333, 3.3, 33.33, "three"}}

Detailed installation instructions, usage information, and documentation, examples, and unit tests can be found here.

|improve this answer
$\endgroup$
4
$\begingroup$

We have written a C++ MathLink HDF5 importer for Mathematica to replace the built-in one. We found the built-in importer to crash or leak memory when dealing with large HDF5 files (with many datasets). Our importer is free software available at

http://sourceforge.net/p/h5mma/home/h5mma/

We don't use compound data ourselves, but it should be possible to enhance this reader to add import of compound data.

|improve this answer
$\endgroup$
3
$\begingroup$

For HDF4 I used a combination of BinaryRead and BinaryReadList to directly access the HDF4 file in order to read VData sets, which are also not yet supported in V8. The HDF5 file data format is relatively welll documented and together with h5dump and some peeks into the source code of the C or fortran interface files you might be able to figure out how to do it on the binary file level. Check out the HDF4 package as a (very crude) example/reference.

|improve this answer
$\endgroup$
3
$\begingroup$

It seems that in version 10 of Mathematica (see Documentation Center), the importation of compound .h5 data is now supported, e.g. :

Import["ExampleData/image.h5", {"Datasets", "/image24bitpixel"}]
|improve this answer
$\endgroup$
  • $\begingroup$ It doesn't seem this is true. Perhaps the 'special stuff' they want to support is but the documentation says "Compound data structures are ignored by Import." And, just checking the aforementioned demo image, I get: blarg:ExampleData> h5dump image.h5 | grep DATATYPE | sed 's/^[ \t]*//;s/[ \t,\{]*$//' | sort | uniq DATATYPE H5T_REFERENCE { H5T_STD_REF_OBJECT } DATATYPE H5T_STD_U8LE DATATYPE H5T_STRING and no mention of H5T_COMPOUND alas. Sigh $\endgroup$ – flip Sep 28 '15 at 23:59
  • $\begingroup$ @flip I agree, and I don't believe this file was created with Mathematica either $\endgroup$ – M.R. May 5 '17 at 17:26
3
$\begingroup$

You can see:

https://github.com/scotmartin1234/HDF5Mathematica

There are some screenshots about compound datatypes, and you could also download the package and use it.

This is version 2.00 (August 2016) of the package that was originally provided as version 1.00 in July 2011.

Here are some examples:

With[
 {filename = FileNameJoin[{NotebookDirectory[], "hdf5_test.h5"}]},
 CompoundDataTypeInformation[filename,"./arrays/Vdata with mixed types"]
 ]

{"NumberOfMembers" -> 7, 
 "MemberName" -> {"Character", "Short", "Integer", "Float", "String", "Integer Array", 
   "Float Array"}, 
 "MemberClassType" -> {"STRING", "INTEGER", "INTEGER", "FLOAT", "STRING", "ARRAY", 
   "ARRAY"}, 
 "MemberSystemType" -> {"System.Byte[]", "System.Int16[]", "System.Int32[]", 
  " System.Single[]", "System.Byte[]", "System.Int32[]", "System.Single[]"}, 
 "MemberSystemTypeCountInOneDatum" -> {1, 1, 1, 1, 10, {4}, {20}}}

With[
 {
  filename = FileNameJoin[{NotebookDirectory[], "h5ex_t_cmpd.h5"}],
  dataSet = "DS1"
  },
 ReadRank1[
   filename, 
   dataSet, 
  "ByteConversionFunction" -> (myFunction[#1, #2, #3, #4] &)]
 ] 

{{1153, "Exterior (static)", 53.23, 24.57}, {1184, "Intake", 55.12, 22.95}, 
 {1027, "Intake manifold", 103.55, 31.23}, {1313, "Exhaust manifold", 1252.89, 84.11}}
|improve this answer
$\endgroup$
  • $\begingroup$ Could could also post it on PackageData for more exposure. $\endgroup$ – Szabolcs Aug 19 '16 at 11:45
  • $\begingroup$ great suggestion, thank you! $\endgroup$ – ScotMartin Aug 19 '16 at 23:36
1
$\begingroup$

Starting from M11.1 Import supports compound datasets in HDF5 files. Each element of compound type is imported as an Association with keys being strings. For example

In[2]:= Import["ExampleData/sample2.h5", "/Compound"]

Out[2]= {
  {
    <|
        "Country" -> "Botswana", 
        "Military" -> {5.3, 4.5}, 
        "Elevation"-> <|"Max" -> 4892, "Min" -> 513, "Highest point" -> "Otse Hill"|>
    |>, <|
        "Country" -> "Chile", 
        "Military" -> {8.8, 3.7}, 
        "Elevation" -> <|"Max" -> 6893, "Min" -> 0, "Highest point" -> "Ojos del Salado"|>
    |>
  }, {
    <|
        "Country" -> "France", 
        "Military" -> {5.3, 3.3}, 
        "Elevation" -> <|"Max" -> 4810, "Min" -> -10, "Highest point" -> "Mont Blanc"|>
    |>, <|
        "Country" -> "Laos", 
        "Military" -> {18.9, 4.3}, 
        "Elevation" -> <|"Max" -> 2817, "Min" -> 70, "Highest point" -> "Phou Bia"|>
    |>
  }
}

Notice that values for "Elevation" are imported as Associations which means that "Elevation" type is also compound. If you want to learn more about the datatype of certain dataset in HDF5 file, you can do it like this:

In[1]:= Import["ExampleData/sample2.h5", {"DataFormat", "/Compound"}]

Out[1]= <|
    "Class" -> "Compound", 
    "Structure" -> <|
        "Country" -> "String", 
        "Military" -> <|"Class" -> "Array", "Dimensions" -> {2}, "DataFormat" -> "Real64"|>, 
        "Elevation" -> <|
            "Class" -> "Compound", 
            "Structure" -> <|"Max" -> "Integer16", "Min" -> "Integer16",  "Highest point" -> "String"|>
        |>
    |>
|>

Sometimes it may not be clear what each value in the compound type stands for (for example what does "Military" -> {18.9, 4.3} mean?). In this case you can try importing Attributes of the dataset:

In[1]:= Import["ExampleData/sample2.h5", {"Attributes", "/Compound"}]

Out[1]= <|"Units" -> <|"Military" -> "Per 1000 capita (active, total)", "Elevation" -> "meters"|>|>
|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.