1
$\begingroup$

Recently, I would like to use HDF5 instead of CSV because of times Import taking. Especially, I have to manage large data with high precision (about 50 degits) and it takes very long time to export such data to CSV. Therefore, HDF5 seems nice to me.

However, when I do Import HDF5 file, I only get data with MachinePrecision. On the other hand, CSV returns data with exact precision which I exported.

Here is a simple sample:

In[1]= data = N[Pi, 50];
In[2]= data // Precision
Out[2]= 50.

Then,

In[3]= Export["testing_precision_HDF5.h5", {data}];
In[4]= Export["testing_precision_csv.csv", data];

Finally I do Import and evaluate their Precision;

In[5]= Import["testing_precision_HDF5.h5", {"Datasets", 
    "/Dataset1"}][[1]] // Precision
Out[5]= MachinePrecision

On the other hand,

In[6]= Import["testing_precision_csv.csv"] // Precision
Out[6]= 49.4971

So, where does this difference come from? Is the reason why HDF5 takes short time to Import HDF5 cannot handle high precision data?

Please tell me how to manage high precision data with HDF5 & Mathematica.

Update: I don't think it is a problem with '''Import''' because I checked the .h5 file with HDFview, but it was same result.

$\endgroup$
3
  • 1
    $\begingroup$ I think your question is not so much about Mathematica but about how numbers are stored. HDF5 stores numbers in a binary format, but you can define which binary format to use. From Mathematica you can only write 64bit floats AFAIK, but even if you could write 128bit floats that would not be enough to store the 50 digits you require. For that many digits exporting to a text format like csv might be the best choice. Can you explain what the data you write is used for? If you only write/read with Mathematica MX is probably the best format to use... $\endgroup$ – Albert Retey Oct 5 '20 at 10:49
  • $\begingroup$ @AlbertRetey Thank you for adivise. I understand what is important. Actually I do only read/write to store numerous data and I didn't know .mx format...I'll try, thank you! $\endgroup$ – Keyspire Oct 6 '20 at 12:31
  • $\begingroup$ The important question is whether you need to read or write with another program. If not, .mx is certainly a good choice. If you need a long term storage /archiving format then maybe something else might be better suited, though. MX is fast and versatile but limited to Mathematica only and not absolutely guaranteed to be stable between mathematica versions, I think... $\endgroup$ – Albert Retey Oct 7 '20 at 13:40
0
$\begingroup$

I tried some solutions, finally I found the best way to do; using ToString at Export and ToExpression at Import. Furtheremore, use FullForm before type casting not to export with exponential notation (then we cannnot import such data in usual way). This enable us to handle with data with any precision.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.