1
$\begingroup$

The UnsignedInteger64 data format should be able to encode correctly any integer in $[0,2^{64}-1]$. When using this data format with Export in HDF5 format, is exhibits a strange behavior around $2^{63}$.

The Export command I use is (in Mathematica 12)

Export["file.h5","/list"->{"Data" -> x, "DataFormat" -> "UnsignedInteger64"}]

where list is a list of integers. The Import command is

y=Import["file.h5", "/list"]

I used the following cases for x:

x1 = {2^63};
x3 = {2^63 + 1};
x2 = {2^63 - 1};
x4 = {2^63, 2^63 + 1};
x5 = {2^63 - 1, 2^63 - 2};
x6 = {2^63 - 1, 2^63 + 1};

which provide outputs y1, y2, y3, y4, y5 and y6, respectively.

The condition that xi==yi is met for i from 1 to 5, but not for 6. In case 6 the value of y is a list of the same Length as x but where all elements are $2^{63}$.

Q: What is going on?

$\endgroup$
1
$\begingroup$

This is, unfortunately, a bug in HDF5 Export, which we will try to fix in the next release of the Wolfram Language.

A workaround for versions 12.3.1 and below is to use (undocumented) "InputDataFormat" sub-element to specify the input type directly and bypass the type inference code:

In[1]:= x = {2^63 - 1, 2^63 + 1};

In[2]:= Export["file.h5", "/list" -> {
   "Data" -> x,
   "InputDataFormat" -> "UnsignedInteger64",
   "Dimensions" -> {2}
   }]

Out[2]= "file.h5"

In[3]:= Import["file.h5", {"DataFormat", "/list"}]

Out[3]= "UnsignedInteger64"

In[4]:= Import["file.h5", "/list"]

Out[4]= {9223372036854775807, 9223372036854775809}

In[5]:= % === x

Out[5]= True

A minor drawback of using "InputDataFormat" is that you must also provide "Dimensions" explicitly.

The difference between "DataFormat" and "InputDataFormat" is that the former specifies the format in which you want to store data in the file and the latter tells Export in what format the data is provided. By default, "InputDataFormat" is set to Automatic, in which case Export tries to deduce the type on its own.

$\endgroup$
2
  • $\begingroup$ Thank you! Do you know if other data formats besides UnsignedInteger64 are affected by this? $\endgroup$
    – anonymous
    Oct 26 '21 at 7:59
  • $\begingroup$ I believe this is specific to UnsignedInteger64. $\endgroup$
    – rafalc
    Oct 26 '21 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.