Strange behavior exporting UnsignedInteger64 in HDF5 format

Question

The UnsignedInteger64 data format should be able to encode correctly any integer in $[0,2^{64}-1]$. When using this data format with Export in HDF5 format, is exhibits a strange behavior around $2^{63}$.

The Export command I use is (in Mathematica 12)

Export["file.h5","/list"->{"Data" -> x, "DataFormat" -> "UnsignedInteger64"}]

where list is a list of integers. The Import command is

y=Import["file.h5", "/list"]

I used the following cases for x:

x1 = {2^63};
x3 = {2^63 + 1};
x2 = {2^63 - 1};
x4 = {2^63, 2^63 + 1};
x5 = {2^63 - 1, 2^63 - 2};
x6 = {2^63 - 1, 2^63 + 1};

which provide outputs y1, y2, y3, y4, y5 and y6, respectively.

The condition that xi==yi is met for i from 1 to 5, but not for 6. In case 6 the value of y is a list of the same Length as x but where all elements are $2^{63}$.

Q: What is going on?

Answer 1

This is, unfortunately, a bug in HDF5 Export, which we will try to fix in the next release of the Wolfram Language.

A workaround for versions 12.3.1 and below is to use (undocumented) "InputDataFormat" sub-element to specify the input type directly and bypass the type inference code:

In[1]:= x = {2^63 - 1, 2^63 + 1};

In[2]:= Export["file.h5", "/list" -> {
   "Data" -> x,
   "InputDataFormat" -> "UnsignedInteger64",
   "Dimensions" -> {2}
   }]

Out[2]= "file.h5"

In[3]:= Import["file.h5", {"DataFormat", "/list"}]

Out[3]= "UnsignedInteger64"

In[4]:= Import["file.h5", "/list"]

Out[4]= {9223372036854775807, 9223372036854775809}

In[5]:= % === x

Out[5]= True

A minor drawback of using "InputDataFormat" is that you must also provide "Dimensions" explicitly.

The difference between "DataFormat" and "InputDataFormat" is that the former specifies the format in which you want to store data in the file and the latter tells Export in what format the data is provided. By default, "InputDataFormat" is set to Automatic, in which case Export tries to deduce the type on its own.