3
$\begingroup$

Is there any way to remove a dataset from an hdf5 file using mathematica? Or, alternatively, is it possible to overwrite a dataset while keeping the other datasets intact?

I've tried the following

Export[fileName, newDATASET1, {"Datasets", "DATASET1"}, "Overwrite" -> "Append", "AppendMode" -> "Overwrite"]

but this erases all the other DataSets.

$\endgroup$
3
$\begingroup$

HDF5 by design does not allow for deleting datasets. It's a limitation of the format itself and not of Mathematica's implementation. See paragraph 5.2 in the official HDF5 documentation: https://support.hdfgroup.org/HDF5/doc1.6/UG/10_Datasets.html

Overwriting, on the other hand, is implemented in Mathematica, but you have to use OverwriteTarget symbol instead of "Overwrite" string. In 12.0 the new dataset must have the same dimensions and datatype as the one that is being overwritten:

In[11]:= Export["file.h5", "dset" -> {{1, 2}, {3, 4}}];

In[12]:= Import["file.h5", "dset"]

Out[12]= {{1, 2}, {3, 4}}

In[13]:= Export["file.h5", 
    "dset" -> {{11, 12}, {12, 13}}, 
    OverwriteTarget -> "Append", 
    "AppendMode" -> "Overwrite"
  ];

In[14]:= Import["file.h5", "dset"]

Out[14]= {{11, 12}, {12, 13}}

In[15]:= Export["file.h5", 
    "dset" -> {0}, 
    OverwriteTarget -> "Append",
    "AppendMode" -> "Overwrite"
  ];

During evaluation of In[15]:= Export::h5dodm: Dataset with dimensions {2,2} cannot store data with dimensions {1}.

This limitation has been removed in version 12.1 (although the new features remain undocumented at the moment), where you can overwrite parts of datasets and extend existing datasets if only they were created as extendable - using "MaxDimensions" subelement:

In[1]:= Export["extendable.h5",
  "dset" -> {
    "Data" -> {{1, 2}, {3, 4}},
    "MaxDimensions" -> {10, ∞}
    }];

In[2]:= Import["extendable.h5", "dset"]

Out[2]= {{1, 2}, {3, 4}}

This created a 2 x 2 dataset that can grow to 10 rows and arbitrarily many columns. Let's overwrite/extend the first row with 5 consecutive integers:

In[3]:= Export["extendable.h5", "dset" -> {Range[5]}, 
  OverwriteTarget -> "Append", "AppendMode" -> "Overwrite"];

In[4]:= Import["extendable.h5", "dset"]

Out[4]= {{1, 2, 3, 4, 5}, {3, 4, 0, 0, 0}}

Notice that the dimensions are now 2 x 5 and new elements in the second row have been initialized with default value (0).

To overwrite an arbitrary part of the dataset, one must use "SpaceSelection" subelement which takes the same values as the documented "TakeElements" option.

Let's overwrite 3rd and 4th columns:

In[5]:= Export["extendable.h5", "dset" -> {
    "Data" -> {{16, 17}, {18, 19}},
    "SpaceSelection" -> {All, 3 ;; 4}
    },
  OverwriteTarget -> "Append",
  "AppendMode" -> "Overwrite"
  ];

In[6]:= Import["extendable.h5", "dset"]

Out[6]= {{1, 2, 16, 17, 5}, {3, 4, 18, 19, 0}}

[update - to address concerns from the comment]

Overwriting a dataset does not affect other elements in the file:

In[1]:= Export["file.h5", {
    "dset1" -> {{1, 2}, {3, 4}}, 
    "dset2" -> "xxx"
  }];

In[2]:= Import["file.h5", "Data"]

Out[2]= <|"/dset1" -> {{1, 2}, {3, 4}}, "/dset2" -> "xxx"|>

In[3]:= Export["file.h5", 
    "dset1" -> {{11, 12}, {12, 13}}, 
    OverwriteTarget -> "Append", 
    "AppendMode" -> "Overwrite"
  ];

In[4]:= Import["file.h5", "Data"]

Out[4]= <|"/dset1" -> {{11, 12}, {12, 13}}, "/dset2" -> "xxx"|>
$\endgroup$
4
  • $\begingroup$ But even using the OverwriteTarget, this will delete all the other datasets that I'd like to leave intact... $\endgroup$
    – AJHC
    Apr 14 '20 at 9:14
  • $\begingroup$ No, the rest will be intact. If you have an example that proves otherwise, please file a bug report. $\endgroup$
    – rafalc
    Apr 14 '20 at 9:18
  • $\begingroup$ In my version 11.3 it deletes all the other datasets! So this is really something that only works for versions >12? $\endgroup$
    – AJHC
    Apr 14 '20 at 11:15
  • $\begingroup$ Yes, >= 12. In 11.3 you can only add new datasets to existing files with "Append" -> True. In 12.0 Export capabilities have been improved and to allow better control over extending and overwriting the "Append" option has been split into OverwriteTarget and "AppendMode". $\endgroup$
    – rafalc
    Apr 14 '20 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.