Fast MATLAB, slow Mathematica: Export a matrix as TSV

Question

I am trying to find the best way of exporting matrices (results of a time-dependent CFD calculation). I found that .h5 is the fastest option, but h5 is not generic, i.e., I can't really open like a text file. When I export as

e = RandomReal[{-10, 10}, {1000, 1000}];
AbsoluteTiming[
Export[vxFolder <> "/vx7.txt", e, "Table"];]

Exporting takes 20 seconds. When I do the same thing in MATLAB it takes only 1 second. Why is there a huge difference between MATLAB and Mathematica?

Answer 1

After using them for several years, I have to admit that many Export routines in Mathematica are either broken or way too slow (a factor of 10 to 25 when compared to a "C++" routine, e.g. for such a simple task as exporting an array).

It is very unfortunate. However, the following can export the data within 1.6 seconds while the OP's version took 15.985 seconds on my machine. It allows for specifying how many digits have to be exported.

colsep = "\t";
rowsep = "\n";
prec = 16;
SetAttributes[doubletostring, Listable];
doubletostring[x_] := Internal`DoubleToString[x, False, prec];
Export[
    "a.txt",
    StringJoin[
      Map[
        row \[Function] StringJoin[Riffle[doubletostring /@ row, colsep], rowsep],
        e]
    ]
    ]; // AbsoluteTiming // First

1.60592

This timing is from Mathematica 11.3 on macOS v10.13.4 (High Sierra).

A hand-written C++ routine linked to Mathematica needs about 0.6 seconds to export the matrix e, so in principle, there is still room for improvement.

Since Internal`DoubleToString is undocumented, I have no idea what its second argument does (it appears to me that it has no effect which would be pretty weird). As ilian told me in this post, the second argument controls whether NumberMark shall be printed (False suppresses it). The third argument seems to specify how many leading digits have to be printed.

Some further speedup can be obtained by using ParallelMap instead of Map. Once the parallel kernels are set up and started, the timing is pretty close to the (unparallelized) C++ timing on my Quad Core machine:

Export["a.txt",
    StringJoin[
     ParallelMap[
      row \[Function] 
       StringJoin[Riffle[doubletostring@row, colsep], rowsep],
      e
      ]
     ]
    ]; // AbsoluteTiming // First

0.875412

Edit:

It is always a bit dangerous to rely on undocumented functions because that can be subject to change. As Domen pointed out in a comment, Internal`DoubleToString does not exist anymore in newer versions of Mathematica. (I am using version 14.1, and it's gone indeed.) But there is a new Internal`MRealToString with the same functionality and syntax. I guess the developers just renamed the function to align with the fact that the word "double" is typically not used in the Mathematica universe. We rather call them "machine real numbers" or just "machine reals".

Answer 2

First rerunning the OP's demo for a baseline timing.

e = RandomReal[{-10, 10}, {1000, 1000}];
AbsoluteTiming[
 Export[FileNameJoin[{$InitialDirectory, "vxFolder", "vx7.txt"}], e, "Table"];
 ]

{34.0312, Null}

As you can see in the code below, most of the time is spent formatting the data but the writing takes only 0.7 seconds. If the OP's matrix can be generated as a string in the right format writing can be done quickly.

AbsoluteTiming[
 str = OpenWrite[FileNameJoin[{$InitialDirectory, "vxFolder", "vx8.txt"}]];
 ]

AbsoluteTiming[
 e2 = StringTake[StringReplace[
     ToString[InputForm@e], {"}, {" -> "\n", ", " -> "\t"}], {3, -3}];
 ]

AbsoluteTiming[
 WriteString[str, e2];
 Close[str];
 ]

{0.00266855, Null}

{27.4393, Null}

{0.684855, Null}

An alternative formatting function takes about the same time.

AbsoluteTiming[
 e2 = StringRiffle[StringRiffle[#, "\t"] & /@ Map[
      StringReplace[Internal`DoubleToString[#], "`" -> ""] &, e, {2}], "\n"];
 ]

{29.589, Null}

Answer 3

This really bugged me so I used the opportunity to learn a bit more about LibraryLink. I have to say that having a close look at the internals of Szabolcs' LTemplate package was significantly more helpful than the actual LibraryLink documentation...

Setting up the C++ code:

Needs["CCompilerDriver`"];
Needs["GeneralUtilities`"];

LibraryLink`BoilerPlate[funname_String] := 
 StringJoin["#include \"math.h\"

  #include \"WolframRTL.h\"

  static WolframCompileLibrary_Functions funStructCompile;

  static mbool initialize = 1;

  DLLEXPORT int Initialize_", funname, "(WolframLibraryData libData)
  {
    if( initialize)
    {
        funStructCompile = libData->compileLibraryFunctions;
        initialize = 0;
    }
    return 0;
  }

  DLLEXPORT void Uninitialize_", funname, "(WolframLibraryData libData)
  {
    if( !initialize)
    {
        initialize = 1;
    }
  }

  "]


funname = "writeRealMatrixToFile";
preamble = "
    #include <iostream>
    #include <fstream>
    #include <boost/lexical_cast.hpp>
    using namespace std;
    using namespace boost;
    ";

code = StringJoin[preamble, "\n\n", LibraryLink`BoilerPlate[funname], 
   "extern \"C\" DLLEXPORT int ", funname, 
   "(WolframLibraryData libData, mint Argc, MArgument * Args, MArgument Res)
      {
       int err = 0;
          const char * file = MArgument_getUTF8String(Args[0]);
          const MTensor a0 = MArgument_getMTensor(Args[1]);
          const mreal * a = libData->MTensor_getRealData(a0);
          const char * prefix = MArgument_getUTF8String(Args[2]);
          const char * infix = MArgument_getUTF8String(Args[3]); 
          const char * suffix = MArgument_getUTF8String(Args[4]);
          const char * linesep = MArgument_getUTF8String(Args[5]);

          mint const* dims;
          dims = libData->MTensor_getDimensions(a0);
       mint i, j;

       ofstream outputstream;

       try
       {
           outputstream.open(file, ofstream::out | ofstream::app);

           for( i=0; i < (dims[0]-1); i++)
           {
               outputstream << prefix;
               for( j=0; j < dims[1]-1; j++)
               {
                   outputstream << lexical_cast<string>(*a) << infix; \

                   a++;
               }
               outputstream << lexical_cast<string>(*a) << suffix;
               a++;
               outputstream << linesep;
           }
           {
               outputstream << prefix;
               for( j=0; j < dims[1]-1; j++)
               {
                   outputstream << lexical_cast<string>(*a) << infix;
                   a++;
               }
               outputstream << lexical_cast<string>(*a) << suffix;
           }

           outputstream.close();
           MArgument_setInteger(Res, err);
       }
       catch (...)
       {
           return LIBRARY_FUNCTION_ERROR;
       }

       return LIBRARY_NO_ERROR;
      }
      "];

Creating the library in a temporary directory. (If you'd like to reuse the library, just specify a value of choice for "TargetDirectory".) Since we require boost and it is not located at a path that can be found automatically by the compiler, I also specify "IncludeDirectories". Of course you have to make sure that boost is installed on your system.

libfile = CreateLibrary[code, funname,
   "Language" -> "C++",
   "TargetDirectory" -> $TemporaryDirectory,
   "IncludeDirectories" -> "/opt/local/include"
   ];

Loading the library function and creating a wrapper function as rudimentary user interface.

If[TrueQ[Head[writeRealMatrixToFile] === LibraryFunction], 
  LibraryFunctionUnload[writeRealMatrixToFile]];
ClearAll[writeRealMatrixToFile];
If[TrueQ[FindLibrary[libfile] =!= $Failed],
  writeRealMatrixToFile = LibraryFunctionLoad[libfile, funname,
    {
     "UTF8String", {Real, 2, "Constant"}, "UTF8String", "UTF8String", 
     "UTF8String", "UTF8String"
     },
    Integer]
  ];

Options[WriteRealMatrixToFile] = {
   "Prefix" -> "",
   "Infix" -> "\t",
   "Suffix" -> "",
   "RowSep" -> "\n"
   };
WriteRealMatrixToFile[file_String, 
   A_?(MatrixQ[#, Developer`MachineRealQ] &), OptionsPattern[]] := 
  writeRealMatrixToFile[file, A,
   OptionValue["Prefix"],
   OptionValue["Infix"],
   OptionValue["Suffix"],
   OptionValue["RowSep"]
   ];

Running the test example.

A = RandomReal[{-1, 1}, {1000, 1000}];
file = "test.txt";
Put[file];
WriteRealMatrixToFile[file, A]; // AbsoluteTiming // First
B = Import[file, "Table"]; // AbsoluteTiming // First
Max[Abs[A - B]]

0.668737

1.5853

0.

This is over 20 times faster than using Export with "Table" as export format:

Export["A.txt", A, "Table"]; // AbsoluteTiming // First

16.0703

Final remark:

I wrote writeRealMatrixToFile such that it appends to existing files (creating a new one if needed). This way, it can be used to export multiple data sets with other chunks of code in between. Typical file formats where this might be helpful are, e.g., povray and obj.

In particular, povray needs lists of vectors specified in the format < x1, y1, z1 >,< x2, y2, z2 >,... and to put < and > is the purpose of the "Prefix" and "Suffix" option. "Infix" specifies the column separator (e.g. ","; default is a tabulator ("\t")) and "RowSep" specifies the row seperators (default is newline ("\n")).

Edit

New version employs boost::lexical_cast on the C++ side for conversion of double to string, preventing a loss of relative precision.

Answer 4

$Version
e = RandomReal[{-10, 10}, {1000, 1000}];
Export["test1.txt", e, "Table"] // AbsoluteTiming

"11.3.0 for Microsoft Windows (64-bit) (March 7, 2018)"

{51.09841768091894, "test1.txt"}

colsep = "\t";
rowsep = "\n";
prec = 16;
SetAttributes[doubletostring, Listable];
doubletostring[x_] := Internal`DoubleToString[x, False, prec];
Export["test2.txt", 
  StringJoin[
   Map[row \[Function] 
     StringJoin[Riffle[doubletostring /@ row, colsep], rowsep], 
    e]]] // AbsoluteTiming

{13.71278727644397, "test2.txt"}

Export["test3.txt", 
  e // Map[Row[#, "\t"] &] // Column // 
    NumberForm[#, 16, ExponentFunction -> (Null &)] & // ToString] // AbsoluteTiming

{2.621917257541675, "test3.txt"}