Any solution to transfer data between Python and Mathematica

Question

I think Mathematica handles .mat file not very well, python is pretty good at this kind of jobs. In this site, there are plenty of question related to calling Mathematica from Python, or calling Python from Mathematica. The answers are only valid for very small data, I believe like hundreds of numbers. Currently, I am working with data a little larger than that, a typical size is about (40960, 256). Thus I want to ask whether there is a solution to transfer data from one to another? Or a better .mat file interface?

Following is a sample of what is inside the .mat file.

The data transfer between R and python can be done seamlessly. Check rpy2 for more information. Maybe RLink can be used in this case.

Update

Handling .mat file in Mathematica is not as fast as in Python. However, thanks for linked page suggested by @The Toad, this process is much better.

Following are some tests:

in Python:

%time matlab.whosmat('test.mat')
CPU times: user 152 ms, sys: 10.2 ms, total: 162 ms
Wall time: 164 ms
Out[153]: 
[('exp03', (20480, 2), 'double'),
 ('exp22', (22528, 2), 'double'),
 ('exp09', (22528, 2), 'double'),
 ('exp31', (22528, 2), 'double'),
 ...

in Mathematica:

In[166]:= Timing@Import["~/test.mat", {"MAT", "Labels"}]

Out[166]= {0.851815, {"exp03", "exp22", "exp09", "exp31", "exp30", 
  "exp33", "exp32", "exp35", "exp34", "exp19", "exp18", "exp17", 
  "exp16", "exp15", "exp14", "exp13", "exp12", "exp11", "exp10", 
  "exp37", "exp36", "exp02", "Data", "exp08", "exp23", "exp20", 
  "exp21", "exp26", "exp27", "exp24", "exp25", "exp01", "exp28", 
  "exp29", "exp04", "exp05", "exp06", "exp07"}}

in Python

%time data=matlab.loadmat('test.mat')
CPU times: user 199 ms, sys: 28.2 ms, total: 228 ms
Wall time: 236 ms

in Mathematica:

In[168]:= Timing@Import["~/test.mat", {"MAT", "LabeledData"}];

In[169]:= %[[1]]

Out[169]= 0.838842

For another file:

In Python:

%time data=matlab.loadmat('test2.mat')
CPU times: user 3.55 s, sys: 381 ms, total: 3.93 s
Wall time: 3.98 s

In Mathematica:

In[1]:= Timing@Import["~/test2.mat", {"MAT", "LabeledData"}];

In[2]:= First[%]

Out[2]= 12.558158

Answer 1

As suggested by @Jens, HDF5 can be fast imported and manipulated in Mathematica. The performance of importing HDF5 is as efficient as MAT file in Python and you can read only a part of HDF5 file into memory. From the question, Mathematica is about 3~4 times slower in reading MAT files. The speed of reading HDF5 files are very close to the speed in Python.

Performance of HDF5

As suggested by @Jens, performance of HDF5 has been tested. I have to say performance of HDF5 is impressive, the performance is close to .mat file in Python (Because .mat get more compression, the read speed is a little slower.). Since I do not know much about how to convert .mat file to .mat of version 7.3. I directly convert .mat to .hdf file. Following are some timing results:

For the first file: In[1]:= fname = "test.h5"; dat = Timing@Import[fname]

Out[2]= {0.109260, {"/Data", "/__globals__", "/exp01", "/exp02", 
  "/exp03", "/exp04", "/exp05", "/exp06", "/exp07", "/exp08", 
  "/exp09", "/exp10", "/exp11", "/exp12", "/exp13", "/exp14", 
  "/exp15", "/exp16", "/exp17", "/exp18", "/exp19", "/exp20", 
  "/exp21", "/exp22", "/exp23", "/exp24", "/exp25", "/exp26", 
  "/exp27", "/exp28", "/exp29", "/exp30", "/exp31", "/exp32", 
  "/exp33", "/exp34", "/exp35", "/exp36", "/exp37"}}

In[3]:= dat = Timing@Import[fname, "Data"];
First[dat]

Out[4]= 0.142046

In[5]:= dat = Timing@Import[fname, {"Datasets", "Data"}];

In[6]:= dat[[1]]

Out[6]= 0.036610

In[7]:= Dimensions@dat[[2]]

Out[7]= {20480, 37}

For the second file:

In[1]:= fname = "test2.h5";
dat = Timing@Import[fname]

Out[2]= {0.111127, {"/Data", "/Spec", "/__globals__", "/exp001", 
  "/exp002", "/exp003", "/exp004", "/exp005", "/exp006", "/exp007", 
  "/exp008", "/exp009", "/exp010", "/exp011", "/exp012", "/exp013", 
  "/exp014", "/exp015", "/exp016", "/exp017", "/exp018", "/exp...}}

In[3]:= dat = Timing@Import[fname, "Data"];
First[dat]

Out[4]= 2.565427

In[5]:= dat = Timing@Import[fname, {"Datasets", "Data"}];

In[6]:= dat[[1]]

Out[6]= 0.143031

In[7]:= Dimensions@dat[[2]]

Out[7]= {40960, 246}

Convention of MAT file to HDF5

The problem of the solution is how to convert MAT file to HDF5 and the compatibility with new version Matlab is a plus.

Convention MAT file to HDF5 can be done using h5py in Python. I would like to recommend hdf5storage, which is easy to use and have very similar options as savemat in scipy. The HDF5 file can also be load in Matlab using load command.

import hdf5storage as h
from scipy.io import loadmat
data = loadmat('test.mat')
h.savemat('t.mat',data)

Currently, hdf5storage does not support compression. The issue is reported by the author herself last year. Here is a workaround before it is implemented. Changing the default option in __init__.py L192-L196 will always compress the file.

    # Set the h5py options to use for writing scalars and arrays to
    # blank for now.

    self.scalar_options = dict()
    self.array_options = {'compression':'gzip'}  

I will contact the author to fix this issue ASAP.