2
$\begingroup$

When I export a list of 49000 matrices which are 100by100 I get some errors.

Export["data.h5", data];

Returns:

LinkObject::linkw: Unable to write data to closed link LinkObject["C:\Program Files\Wolfram Research\Mathematica\9.0\SystemFiles\Converters\Binaries\Windows-x86-64\HDF5.exe",77553,10]. >>

In the document it is written that:

This message is generated by an attempt to write to a MathLink connection that is not active.

The size of data is at minimum 3.5 GB. I need to export in H5format because it's fast. Any idea how I can overcome this issue?

To reproduce the problem:

SetDirectory[NotebookDirectory[]]

a = Flatten[Table[RandomReal[{0, 1}, {100, 100}], {i, 1, 49}, {j, 1, 1000}], 1];

Export["a.h5", a]
$\endgroup$
12
  • $\begingroup$ Perhaps you do not have enough disk space? $\endgroup$ Jan 7 '15 at 16:27
  • $\begingroup$ I have 700GB disk space on the location I want to save the data. $\endgroup$
    – MOON
    Jan 7 '15 at 16:33
  • $\begingroup$ I tried to save a (simple) random matrix of 100 by 100, which was ok. $\endgroup$ Jan 7 '15 at 16:37
  • $\begingroup$ @ChenStatsYu I put the code which produces the same error on my PC. Could you please try it? $\endgroup$
    – MOON
    Jan 7 '15 at 17:05
  • $\begingroup$ It took a while, but it did save the file, which was 3738 MB. $\endgroup$ Jan 7 '15 at 17:16
1
$\begingroup$

There are some serious issues with the way mathematica implements the hdf5 standard; it is only a small subset. Specifically, data is saved as an attribute (http://www.hdfgroup.org/HDF5/doc/UG/13_Attributes.html) which leads to some pretty strong restrictions as to what one can do, "We generally consider the maximum size of an attribute to be 64K bytes. The library has two ways of storing attributes larger than 64K bytes: in dense attribute storage or in a separate dataset. Using dense attribute storage is described in this section, and storing in a separate dataset is described in the next section." I found out the hard way that this is the case storing only small data sets...

$\endgroup$
1
  • $\begingroup$ While I think you are correct about Mathematica only implementing a small subset of hdf5 (and even that subsets lacks in quality), I think it is not correct that "data is saved as an attribute". If I check a file generated with Mathematica (with a dataset definitely violating the attribute restrictions) e.g. HDFView I see the exported dataset reported as type "HDF5 Scalar Dataset". Can you explain why you think "data is saved as an attribute"? $\endgroup$ Nov 25 '15 at 10:31
0
$\begingroup$

You can see:

https://github.com/scotmartin1234/HDF5Mathematica

I am not sure if it will crash as you noted in your own post, but it is a completely separate approach that the built-in HDF5, so you might be in luck.

This is version 2.00 (August 2016) of the package that was originally provided as version 1.00 in July 2011.

$\endgroup$
1
  • $\begingroup$ does this work for mac? I had a look and it seems that it doesnt $\endgroup$
    – elbOlita
    Aug 23 '17 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.