12

My first question is: is it possible that the import is discarding the timestamp data? The relevant timing data is contained in the attributes of the /Strain/strain dataset. These can be extracted using: H1url = "https://losc.ligo.org/s/events/GW150914/H-H1_LOSC_4_V1-1126259446-32.hdf5"; strainH1 = Import[H1url, {"Datasets", "/strain/Strain"}]; attrsH1 ...


12

The documentation is misleading here. On one hand, the only export option is "Append" which can be found under the Options tab. On the other hand, the general documentation reads I really wonder, why it is necessary to put Import only behind an option value when "DataEncoding" isn't an export option at all. Anyway, I have the same behaviour in MacOSX ...


12

The problem is that Mathematica has no concept of using any datatype more specific than Integer or Real, and I think these are (on a 64bit computer) always 64bit long. So I can't think of a way to import that data into Mathematica in a way that would need less memory in the final state. If there is a limit of slightly more than 2GB in the size mma can import ...


11

HDF5 does not directly support complex numbers. Programs that do seem to be able to export complex numbers (like armadillo) to HDF5 will in reality split them into real and imaginary part and use their own non-standard convention for storing these. This means that while they can sometimes read back their own data, there is no compatibility between ...


10

Thanks to @dan7geo's helpful hint to ExportStructuredHDF5 exporting to HDF5 is easier than I thought. Take for instance data = <| "group1" -> <|"subgroup1" -> "group1/subgroup1", "subgroup2" -> "group1/subgroup2", "subgroup3" -> <| ...


8

This might be helpful, although there aren't any details in the documentation.


8

One workaround is to compress the HDF5 file after it has been exported from Mathematica, using the HDF5 command line tools. Note: on OS X the command line tools can be easily installed using MacPorts using port install h5utils. The command to recompress the data is h5repack -v -f GZIP=1 infile.h5 outfile.h5 This can indeed achieve a significant reduction ...


6

In version 7 halirutan's export method does not produce a file that is recognized by Import. However, one can write: Export["matrix2.h5.gz", datapourrie, {"GZIP", "HDF5"}] And then: d2 = Import["matrix2.h5.gz", {"Datasets", "/Dataset1"}]; datapourrie == d2 True


6

TemporalData has a straightforward way of adding timestamps to data. Using your variables: td = TemporalData[hlosc, {start, finish}] Extracting a 5 second interval around tevent: ListLogPlot[td, PlotRange -> {{tevent - 5, tevent + 5}, Automatic}]


5

This way can export the data imgs = ExampleData /@ RandomChoice[ExampleData["TestImage"], 10]; faces = FindFaces /@ imgs; dummy = <|"images" -> imgs, "faces" -> faces|>; Get["GeneralUtilities`"]; ExportStructuredHDF5["micro-face-dataset.h5", <|"images" -> ImageData /@ imgs, "faces" -> faces|>] Import the data from file Import[...


5

Exporting data To export a string, a scalar and a matrix aString="aString"; aScalar=2017; aMatrix=Table[Random[],{4},{2}]; The syntax is: Export["foo.h5", {"Datasets" -> {"aString" -> aString, "aScalar" -> {aScalar}, "aMatrix" -> aMatrix}, "DataFormat" -> {Automatic,Automatic,...


5

AFAIK, this isn't possible yet with Mathematica straight from the oven, but there is a package created by Scot Martin here. As far as I can see, it implements a lot of the HDF5 functionality, including Compound Types. For attribute creation take a look at the following functions : H5Acreate, H5Awrite, H5Aclose (the A comes from "Attribute") BTW, I think ...


5

Here is another version of the C code. As I have supposed this should be done much more efficient by avoiding a complete copy of the data I have implemented that to proof my point. The code below can only handle lists of integers (rank 1 arrays), but it reads those with an adjustable buffer size. With this version I could successfully read an integer ...


5

I believe this package contains internal functions, used by Import when working with HDF5 files. These functions have no documentation, and aren't meant to be called from top level. If you evaluate Needs["HDF5Tools`"]; ?HDF5Tools`* and click any of the symbols, you'll see most have usage messages. So you may be able to figure out what they do, but the ...


5

Considering the question about the documentation Jasons answer makes clear that there basically is none (except for the usage messages) and the HDF5Tools` package at this time (2019) unfortunately seems to be meant for internal use only. Documented Import/Export Functionality You have been stating that you would want to use that functionality to read only ...


4

In Mathematica 11.1 HDF5 Import was significantly improved. Importing 2.4GB dataset should take just a few seconds (depending on your machine and datatype). Let us generate a sample .h5 file with 2.4GB of random real numbers: In[5]:= Export["matrix.h5", {"Datasets" -> {"/Dataset1" -> RandomReal[{-1, 1}, 300000000]}}, "Rules"] Out[5]= "matrix.h5" In[...


4

Extending an existing dataset is not possible with the standard Mathematica Export, at least not with any version up to 11.0.1. What does work is to add additional datasets one by one like this: filename = FileNameJoin[{$HomeDirectory, "Desktop", "tst.h5"}] Export[filename, {{1, 2, 3}}, {"Datasets", "one"}] Export[filename, {{1, 2, 3}}, {"Datasets", "two"},...


4

From HDF5 User's Guide on SWMR mode: The SWMR feature is available in HDF5 version 1.10.0 and later. Since Mathematica 11.3 uses HDF5 1.8.16 there is no support for SWMR neither in 11.3 nor in any previous release.


3

Without having the data file myself, I can only give a general sort of answer here. But when you look at the documentation under Import and Export, it says Import["file.h5"] returns a list of strings representing the names of all datasets contained in the file. and Import["file.h5",elem] imports the specified element from an HDF5 file. So you can ...


3

You can see: https://github.com/scotmartin1234/HDF5Mathematica There are some screenshots about compound datatypes, and you could also download the package and use it. This is version 2.00 (August 2016) of the package that was originally provided as version 1.00 in July 2011.


3

The following seems to work, but I'd also be interested in other, possibly simpler or clearer versions... Export["int8.h5", { "Datasets" -> {"/data" -> RandomInteger[{2^8 - 1}, {100, 100}]}, "DataFormat" -> {"UnsignedInteger8"} }, "Rules" ] Two notes: Unfortunately I could not create a generalization of these which make the (both ...


3

Thie syntax has slightly changed in Mathematica 12. Now it can be done like this: In[2]:= Export["test.h5", "path/to/dataset" -> { "Data" -> RandomInteger[{2^8 - 1}, {100, 100}], "DataFormat" -> "UnsignedInteger8" }] Out[2]= "test.h5" In[4]:= Import["test.h5", {{"DataFormat", "Dimensions"}, "path/to/dataset"}] Out[4]= {"UnsignedInteger8"...


3

Szabolcs summed it up pretty well but there is also another popular convention to store complex numbers in HDF5 as compound type with two members being real numbers. So starting from M11.1 you can import it like this: In[1]:= Import["ExampleData/sample1.h5", "Complex/Complex64"] Out[1]= <|"re" -> 0.58215809, "im" -> -0.92684859|> [UPDATE for ...


3

This can be done by adding a set of "Attributes" after the datasets. You can add all sorts of attributes and annotations: Export["test.h5", {{{1,2},{2,3}},{{10,11},{12,13}}}, {"Datasets",{"m1","m2"}}, {"Attributes"->{ "m1"->{ "ScalarValue"->{10.0}, "ArrayValues"->{1.0,2.0,3.0}, "Info"->{"\"This is a text string\""}}...


3

As Albert Retey notes in a comment, this seems to be a bug specific to version 7 (or perhaps just 7.0.1), given that this behavior is not observed in versions 6.0.2, 8.0.4, 9.0.1 or 10.0.2. Incidentally, you are absolutely right that the file is not exported correctly. If you use a different version to create it as it should be, version 7 will then be able ...


3

When Importing a single dataset, the whole file is not loaded into memory, so as long as the dataset you're interested in is smaller than the amount of available memory - you should be fine. In M11.1 new Import option called "TakeElements" was introduced for HDF5 files. Consider the following dataset with dimensions {3, 4}: In[2]:= Import["ExampleData/...


2

In Mathematica 12 there are two ways to export Attributes: Either as a subelement to a dataset: In[1]:= Export["test.h5", "path/to/dataset" -> { "Data" -> {1, 2, 3, 4}, "Attributes" -> { "DataDescription" -> "SomeIntegers", "DataLength" -> 4 } } ]; Import["test.h5", {"Attributes", "path/to/dataset"}] Out[2]= &...


2

Try... A1 = Import["http://www.math.mtu.edu/~struther/Courses/4610/Data/A1.hdf", "Data"][[1]]; My assumption (not being able to try this) is that Import is returning you a list of things in the file of which A1 is the first. BTW: Try not to use capital letters (especially single capitals) as some are reserved symbols in Mathematica (like D and E).


2

You can see: https://github.com/scotmartin1234/HDF5Mathematica The package has the functionality you need for writing at low level. The high level functions are designed more for reading. The package could be expanded as open source project to develop the writing part. It's straight forward to assemble the low level functions into a more Mathematica-...


2

Ok, after trying a few more things, the following works: Export["test.h5", {"Datasets" -> {"/m1" -> {{1, 2}, {2, 3}}, "/m2" -> {{10, 11}, {12, 13}}}, "DataFormat" -> {"UnsignedInteger16", "UnsignedInteger8"}, "Attributes" -> {"m1" -> {"m1Attribute" -> "testAttribute"}}}, "Rules"] Then Import["test.h5", "DataFormat"] and Import["test....


Only top voted, non community-wiki answers of a minimum length are eligible