20
$\begingroup$

I am trying to find the best way of exporting matrices (results of a time-dependent CFD calculation). I found that .h5 is the fastest option, but h5 is not generic, i.e., I can't really open like a text file. When I export as

e = RandomReal[{-10, 10}, {1000, 1000}];
AbsoluteTiming[
Export[vxFolder <> "/vx7.txt", e, "Table"];]

Exporting takes 20 seconds. When I do the same thing in MATLAB it takes only 1 second. Why is there a huge difference between MATLAB and Mathematica?

Enter image description here Enter image description here

$\endgroup$
  • 1
    $\begingroup$ Generally I'd guess Mathematica's trying to be too clever, but since even this takes like 4 seconds: AbsoluteTiming[txtString =StringRiffle[Map[Map@ToString, e]];][[1]] it seems to be a string handling deficiency. $\endgroup$ – b3m2a1 Apr 11 '18 at 9:28
  • $\begingroup$ It's insane. Already Export["a.txt", ExportString[e, "Table"]] is three times faster. I guess the internal implementation of Export uses Write to write data one by one (or at least line by line)... $\endgroup$ – Henrik Schumacher Apr 11 '18 at 10:07
  • $\begingroup$ What is CFD? Computational fluid dynamics? $\endgroup$ – Peter Mortensen Apr 12 '18 at 10:00
  • 1
    $\begingroup$ In 11.3, CSV is able to export the data much faster (5.8s compared to 17.8 on my machine) than Table, and is still a Text format. $\endgroup$ – GenericAccountName Apr 12 '18 at 14:01
19
$\begingroup$

After using them for several years, I have to admit that many Export routines in Mathematica are either broken or way too slow (a factor of 10 to 25 when compared to a "C++" routine, e.g. for such a simple task as exporting an array).

It is very unfortunate. However, the following can export the data within 1.6 seconds while the OP's version took 15.985 seconds on my machine. It allows for specifying how many digits have to be exported.

colsep = "\t";
rowsep = "\n";
prec = 16;
SetAttributes[doubletostring, Listable];
doubletostring[x_] := Internal`DoubleToString[x, False, prec];
Export[
    "a.txt",
    StringJoin[
      Map[
        row \[Function] StringJoin[Riffle[doubletostring /@ row, colsep], rowsep],
        e]
    ]
    ]; // AbsoluteTiming // First

1.60592

This timing is from Mathematica 11.3 on macOS v10.13.4 (High Sierra).

A hand-written C++ routine linked to Mathematica needs about 0.6 seconds to export the matrix e, so in principle, there is still room for improvement.

Since Internal`DoubleToString is undocumented, I have no idea what its second argument does (it appears to me that it has no effect which would be pretty weird). As ilian told me in this post, the second argument controls whether NumberMark shall be printed (False suppresses it). The third argument seems to specify how many leading digits have to be printed.

Some further speedup can be obtained by using ParallelMap instead of Map. Once the parallel kernels are set up and started, the timing is pretty close to the (unparallelized) C++ timing on my Quad Core machine:

Export["a.txt",
    StringJoin[
     ParallelMap[
      row \[Function] 
       StringJoin[Riffle[doubletostring@row, colsep], rowsep],
      e
      ]
     ]
    ]; // AbsoluteTiming // First

0.875412

$\endgroup$
  • $\begingroup$ I found your Map function took 26 seconds and the Export took 0.3 seconds, so most of the time is spend formatting. $\endgroup$ – Chris Degnen Apr 11 '18 at 11:03
  • $\begingroup$ @ChrisDegnen Yes, because I put already all write operations into a single one. But I am quite surprise that this code took so long an your machine. I have no idea why this happens. $\endgroup$ – Henrik Schumacher Apr 11 '18 at 11:05
  • $\begingroup$ @HenrikSchumacher Your last version does get rid-off the backtick and it is at least 5 times faster than the following one. But I am having a problem importing your version as a table or data. If I do export likea = RandomReal[{-10, 10}, {5, 5}]; Export[vxFolder <> "/b.txt", ExportString[a, "Table"]]; When I import this bImp = Import[vxFolder <> "/b.txt", "Table"]; Dimensions[bImp] I get {5,5} which I can use as input matrix. In your version, I get {9} as the dimension. When I look at the text files your version looks unstructured and this version I show, you can see the columns and rows. $\endgroup$ – Erdem Apr 11 '18 at 13:15
  • $\begingroup$ Erdem, thanks for pointing this out. I shoud have used tabulators "\t" as column separators. I fixed it now. Please try again. $\endgroup$ – Henrik Schumacher Apr 11 '18 at 13:25
  • $\begingroup$ Does there exist some similar capability as mex-compilation for mathematica as does for Matlab? $\endgroup$ – mathreadler Apr 11 '18 at 14:13
11
$\begingroup$

First rerunning the OP's demo for a baseline timing.

e = RandomReal[{-10, 10}, {1000, 1000}];
AbsoluteTiming[
 Export[FileNameJoin[{$InitialDirectory, "vxFolder", "vx7.txt"}], e, "Table"];
 ]

{34.0312, Null}

As you can see in the code below, most of the time is spent formatting the data but the writing takes only 0.7 seconds. If the OP's matrix can be generated as a string in the right format writing can be done quickly.

AbsoluteTiming[
 str = OpenWrite[FileNameJoin[{$InitialDirectory, "vxFolder", "vx8.txt"}]];
 ]

AbsoluteTiming[
 e2 = StringTake[StringReplace[
     ToString[InputForm@e], {"}, {" -> "\n", ", " -> "\t"}], {3, -3}];
 ]

AbsoluteTiming[
 WriteString[str, e2];
 Close[str];
 ]

{0.00266855, Null}

{27.4393, Null}

{0.684855, Null}

An alternative formatting function takes about the same time.

AbsoluteTiming[
 e2 = StringRiffle[StringRiffle[#, "\t"] & /@ Map[
      StringReplace[Internal`DoubleToString[#], "`" -> ""] &, e, {2}], "\n"];
 ]

{29.589, Null}

$\endgroup$
7
$\begingroup$

This really bugged me so I used the opportunity to learn a bit more about LibraryLink. I have to say that having a close look at the internals of Szabolcs' LTemplate package was significantly more helpful than the actual LibraryLink documentation...

Setting up the C++ code:

Needs["CCompilerDriver`"];
Needs["GeneralUtilities`"];

LibraryLink`BoilerPlate[funname_String] := 
 StringJoin["#include \"math.h\"

  #include \"WolframRTL.h\"

  static WolframCompileLibrary_Functions funStructCompile;

  static mbool initialize = 1;

  DLLEXPORT int Initialize_", funname, "(WolframLibraryData libData)
  {
    if( initialize)
    {
        funStructCompile = libData->compileLibraryFunctions;
        initialize = 0;
    }
    return 0;
  }

  DLLEXPORT void Uninitialize_", funname, "(WolframLibraryData libData)
  {
    if( !initialize)
    {
        initialize = 1;
    }
  }

  "]


funname = "writeRealMatrixToFile";
preamble = "
    #include <iostream>
    #include <fstream>
    #include <boost/lexical_cast.hpp>
    using namespace std;
    using namespace boost;
    ";

code = StringJoin[preamble, "\n\n", LibraryLink`BoilerPlate[funname], 
   "extern \"C\" DLLEXPORT int ", funname, 
   "(WolframLibraryData libData, mint Argc, MArgument * Args, MArgument Res)
      {
       int err = 0;
          const char * file = MArgument_getUTF8String(Args[0]);
          const MTensor a0 = MArgument_getMTensor(Args[1]);
          const mreal * a = libData->MTensor_getRealData(a0);
          const char * prefix = MArgument_getUTF8String(Args[2]);
          const char * infix = MArgument_getUTF8String(Args[3]); 
          const char * suffix = MArgument_getUTF8String(Args[4]);
          const char * linesep = MArgument_getUTF8String(Args[5]);

          mint const* dims;
          dims = libData->MTensor_getDimensions(a0);
       mint i, j;

       ofstream outputstream;

       try
       {
           outputstream.open(file, ofstream::out | ofstream::app);

           for( i=0; i < (dims[0]-1); i++)
           {
               outputstream << prefix;
               for( j=0; j < dims[1]-1; j++)
               {
                   outputstream << lexical_cast<string>(*a) << infix; \

                   a++;
               }
               outputstream << lexical_cast<string>(*a) << suffix;
               a++;
               outputstream << linesep;
           }
           {
               outputstream << prefix;
               for( j=0; j < dims[1]-1; j++)
               {
                   outputstream << lexical_cast<string>(*a) << infix;
                   a++;
               }
               outputstream << lexical_cast<string>(*a) << suffix;
           }

           outputstream.close();
           MArgument_setInteger(Res, err);
       }
       catch (...)
       {
           return LIBRARY_FUNCTION_ERROR;
       }

       return LIBRARY_NO_ERROR;
      }
      "];

Creating the library in a temporary directory. (If you'd like to reuse the library, just specify a value of choice for "TargetDirectory".) Since we require boost and it is not located at a path that can be found automatically by the compiler, I also specify "IncludeDirectories". Of course you have to make sure that boost is installed on your system.

libfile = CreateLibrary[code, funname,
   "Language" -> "C++",
   "TargetDirectory" -> $TemporaryDirectory,
   "IncludeDirectories" -> "/opt/local/include"
   ];

Loading the library function and creating a wrapper function as rudimentary user interface.

If[TrueQ[Head[writeRealMatrixToFile] === LibraryFunction], 
  LibraryFunctionUnload[writeRealMatrixToFile]];
ClearAll[writeRealMatrixToFile];
If[TrueQ[FindLibrary[libfile] =!= $Failed],
  writeRealMatrixToFile = LibraryFunctionLoad[libfile, funname,
    {
     "UTF8String", {Real, 2, "Constant"}, "UTF8String", "UTF8String", 
     "UTF8String", "UTF8String"
     },
    Integer]
  ];

Options[WriteRealMatrixToFile] = {
   "Prefix" -> "",
   "Infix" -> "\t",
   "Suffix" -> "",
   "RowSep" -> "\n"
   };
WriteRealMatrixToFile[file_String, 
   A_?(MatrixQ[#, Developer`MachineRealQ] &), OptionsPattern[]] := 
  writeRealMatrixToFile[file, A,
   OptionValue["Prefix"],
   OptionValue["Infix"],
   OptionValue["Suffix"],
   OptionValue["RowSep"]
   ];

Running the test example.

A = RandomReal[{-1, 1}, {1000, 1000}];
file = "test.txt";
Put[file];
WriteRealMatrixToFile[file, A]; // AbsoluteTiming // First
B = Import[file, "Table"]; // AbsoluteTiming // First
Max[Abs[A - B]]

0.668737

1.5853

0.

This is over 20 times faster than using Export with "Table" as export format:

Export["A.txt", A, "Table"]; // AbsoluteTiming // First

16.0703

Final remark:

I wrote writeRealMatrixToFile such that it appends to existing files (creating a new one if needed). This way, it can be used to export multiple data sets with other chunks of code in between. Typical file formats where this might be helpful are, e.g., povray and obj.

In particular, povray needs lists of vectors specified in the format < x1, y1, z1 >,< x2, y2, z2 >,... and to put < and > is the purpose of the "Prefix" and "Suffix" option. "Infix" specifies the column separator (e.g. ","; default is a tabulator ("\t")) and "RowSep" specifies the row seperators (default is newline ("\n")).

Edit

New version employs boost::lexical_cast on the C++ side for conversion of double to string, preventing a loss of relative precision.

$\endgroup$
  • $\begingroup$ Please provide minimal instructions on installing boost: is it sufficient just to unpack the archive? Which folder should be specified in "IncludeDirectories"? $\endgroup$ – Alexey Popkov May 15 '18 at 9:08
  • $\begingroup$ You might also have to compile the libraries. But it is quite likely that it is already installed on your system. What's your OS? I use macOS and boost was installed through macports as a dependency for various other packages. For other unixoid systems, you may use apt-get. $\endgroup$ – Henrik Schumacher May 15 '18 at 9:21
  • $\begingroup$ If it is correctly installed, you might not need to specify "IncludeDirectories" at all; usually, the header files for inclusion should be on the compiler's search path. (macports has the tendency to install somewhere else.) For Windows, I am a bit clueless. You might get help over at StackOverflow. $\endgroup$ – Henrik Schumacher May 15 '18 at 9:23
  • $\begingroup$ I've installed boost binaries for Windows and specified path to them but still get cryptic errors when evaluating the CreateLibrary code. :( $\endgroup$ – Alexey Popkov May 15 '18 at 11:54
  • $\begingroup$ What if you skip assigning "IncludeDirectories"? $\endgroup$ – Henrik Schumacher May 15 '18 at 12:21
5
$\begingroup$
$Version
e = RandomReal[{-10, 10}, {1000, 1000}];
Export["test1.txt", e, "Table"] // AbsoluteTiming

"11.3.0 for Microsoft Windows (64-bit) (March 7, 2018)"

{51.09841768091894, "test1.txt"}

colsep = "\t";
rowsep = "\n";
prec = 16;
SetAttributes[doubletostring, Listable];
doubletostring[x_] := Internal`DoubleToString[x, False, prec];
Export["test2.txt", 
  StringJoin[
   Map[row \[Function] 
     StringJoin[Riffle[doubletostring /@ row, colsep], rowsep], 
    e]]] // AbsoluteTiming

{13.71278727644397, "test2.txt"}

Export["test3.txt", 
  e // Map[Row[#, "\t"] &] // Column // 
    NumberForm[#, 16, ExponentFunction -> (Null &)] & // ToString] // AbsoluteTiming

{2.621917257541675, "test3.txt"}

$\endgroup$
  • $\begingroup$ Strange. The last solution is a little slower than Henrik's solution in v9.0.1. (3.28s vs 2.38s. ) In v11.2, my timing is similar to yours. (The naive solution only takes 15s in v9.0.1 BTW.) $\endgroup$ – xzczd Apr 12 '18 at 6:18
  • $\begingroup$ @xzczd 另外我还注意到11.3的ReadList比9.0.1慢太多,不知道11.2是什么情况,试试ReadList["test1.txt",Table[Number,{1000}]];//AbsoluteTiming $\endgroup$ – chyanog Apr 12 '18 at 6:33
  • 1
    $\begingroup$ As to the ReadList sample: 14.35s in v11.2, 0.876s in v9.0.1. $\endgroup$ – xzczd Apr 12 '18 at 7:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.