15
$\begingroup$

I think Mathematica handles .mat file not very well, python is pretty good at this kind of jobs. In this site, there are plenty of question related to calling Mathematica from Python, or calling Python from Mathematica. The answers are only valid for very small data, I believe like hundreds of numbers. Currently, I am working with data a little larger than that, a typical size is about (40960, 256). Thus I want to ask whether there is a solution to transfer data from one to another? Or a better .mat file interface?

Following is a sample of what is inside the .mat file.

matfile

The data transfer between R and python can be done seamlessly. Check rpy2 for more information. Maybe RLink can be used in this case.

Update

Handling .mat file in Mathematica is not as fast as in Python. However, thanks for linked page suggested by @The Toad, this process is much better.

Following are some tests:

in Python:

%time matlab.whosmat('test.mat')
CPU times: user 152 ms, sys: 10.2 ms, total: 162 ms
Wall time: 164 ms
Out[153]: 
[('exp03', (20480, 2), 'double'),
 ('exp22', (22528, 2), 'double'),
 ('exp09', (22528, 2), 'double'),
 ('exp31', (22528, 2), 'double'),
 ...

in Mathematica:

In[166]:= Timing@Import["~/test.mat", {"MAT", "Labels"}]

Out[166]= {0.851815, {"exp03", "exp22", "exp09", "exp31", "exp30", 
  "exp33", "exp32", "exp35", "exp34", "exp19", "exp18", "exp17", 
  "exp16", "exp15", "exp14", "exp13", "exp12", "exp11", "exp10", 
  "exp37", "exp36", "exp02", "Data", "exp08", "exp23", "exp20", 
  "exp21", "exp26", "exp27", "exp24", "exp25", "exp01", "exp28", 
  "exp29", "exp04", "exp05", "exp06", "exp07"}}

in Python

%time data=matlab.loadmat('test.mat')
CPU times: user 199 ms, sys: 28.2 ms, total: 228 ms
Wall time: 236 ms

in Mathematica:

In[168]:= Timing@Import["~/test.mat", {"MAT", "LabeledData"}];

In[169]:= %[[1]]

Out[169]= 0.838842

For another file:

In Python:

%time data=matlab.loadmat('test2.mat')
CPU times: user 3.55 s, sys: 381 ms, total: 3.93 s
Wall time: 3.98 s

In Mathematica:

In[1]:= Timing@Import["~/test2.mat", {"MAT", "LabeledData"}];

In[2]:= First[%]

Out[2]= 12.558158
$\endgroup$
  • 1
    $\begingroup$ Perhaps related: Export performance $\endgroup$ – Jens Jun 21 '15 at 4:12
  • 1
    $\begingroup$ You can import MAT files in Mathematica. Have you tried that? $\endgroup$ – rm -rf Jun 21 '15 at 4:34
  • 2
    $\begingroup$ Maybe I don't understand what you want, but it sounds like you could really benefit by using HDF5 format as I suggest in the answer linked above. It would just be a replacement for what you're already doing with MAT files, it seems. And I have yet to see any file format that is faster to import and export than HDF5. $\endgroup$ – Jens Jun 21 '15 at 5:23
  • 1
    $\begingroup$ @Jens HDF5 is impressive, I add some timing results in the question. Maybe I should convert all the data files to .h5. $\endgroup$ – Kattern Jun 21 '15 at 6:32
  • 1
    $\begingroup$ you might want to try fits as well? $\endgroup$ – chris Jun 21 '15 at 7:42
17
$\begingroup$

As suggested by @Jens, HDF5 can be fast imported and manipulated in Mathematica. The performance of importing HDF5 is as efficient as MAT file in Python and you can read only a part of HDF5 file into memory. From the question, Mathematica is about 3~4 times slower in reading MAT files. The speed of reading HDF5 files are very close to the speed in Python.

Performance of HDF5

As suggested by @Jens, performance of HDF5 has been tested. I have to say performance of HDF5 is impressive, the performance is close to .mat file in Python (Because .mat get more compression, the read speed is a little slower.). Since I do not know much about how to convert .mat file to .mat of version 7.3. I directly convert .mat to .hdf file. Following are some timing results:

For the first file: In[1]:= fname = "test.h5"; dat = Timing@Import[fname]

Out[2]= {0.109260, {"/Data", "/__globals__", "/exp01", "/exp02", 
  "/exp03", "/exp04", "/exp05", "/exp06", "/exp07", "/exp08", 
  "/exp09", "/exp10", "/exp11", "/exp12", "/exp13", "/exp14", 
  "/exp15", "/exp16", "/exp17", "/exp18", "/exp19", "/exp20", 
  "/exp21", "/exp22", "/exp23", "/exp24", "/exp25", "/exp26", 
  "/exp27", "/exp28", "/exp29", "/exp30", "/exp31", "/exp32", 
  "/exp33", "/exp34", "/exp35", "/exp36", "/exp37"}}

In[3]:= dat = Timing@Import[fname, "Data"];
First[dat]

Out[4]= 0.142046

In[5]:= dat = Timing@Import[fname, {"Datasets", "Data"}];

In[6]:= dat[[1]]

Out[6]= 0.036610

In[7]:= Dimensions@dat[[2]]

Out[7]= {20480, 37}

For the second file:

In[1]:= fname = "test2.h5";
dat = Timing@Import[fname]

Out[2]= {0.111127, {"/Data", "/Spec", "/__globals__", "/exp001", 
  "/exp002", "/exp003", "/exp004", "/exp005", "/exp006", "/exp007", 
  "/exp008", "/exp009", "/exp010", "/exp011", "/exp012", "/exp013", 
  "/exp014", "/exp015", "/exp016", "/exp017", "/exp018", "/exp...}}

In[3]:= dat = Timing@Import[fname, "Data"];
First[dat]

Out[4]= 2.565427

In[5]:= dat = Timing@Import[fname, {"Datasets", "Data"}];

In[6]:= dat[[1]]

Out[6]= 0.143031

In[7]:= Dimensions@dat[[2]]

Out[7]= {40960, 246}

Convention of MAT file to HDF5

The problem of the solution is how to convert MAT file to HDF5 and the compatibility with new version Matlab is a plus.

Convention MAT file to HDF5 can be done using h5py in Python. I would like to recommend hdf5storage, which is easy to use and have very similar options as savemat in scipy. The HDF5 file can also be load in Matlab using load command.

import hdf5storage as h
from scipy.io import loadmat
data = loadmat('test.mat')
h.savemat('t.mat',data)

Currently, hdf5storage does not support compression. The issue is reported by the author herself last year. Here is a workaround before it is implemented. Changing the default option in __init__.py L192-L196 will always compress the file.

    # Set the h5py options to use for writing scalars and arrays to
    # blank for now.

    self.scalar_options = dict()
    self.array_options = {'compression':'gzip'}  

I will contact the author to fix this issue ASAP.

$\endgroup$
  • $\begingroup$ Good to see that it worked for you. $\endgroup$ – Jens Jun 22 '15 at 6:07
  • $\begingroup$ @Jens Yes, thanks a lot for your suggestion! With hdf5storage, I can simply convert my old files to MAT version 7.3. $\endgroup$ – Kattern Jun 22 '15 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.