I am handling large numerical data in Mathematica. In smaller problems everything worked fine using Export and Import with the parameter csv and nothing more.

Now I am facing a much larger data volume and plain Export and Import is way too slow for CSV format.

What I want to do: First, exporting a numerical list of approximately $1400 \cdot 260$. Then I perform some calculations outside of mathematica and finally I import a csv file back using Import.

In this question I read how to improve the speed of Export with CSV.

I tried

  ExportString[Transpose[temp], "CSV", "FieldSeparators" -> ", "], 

for a toy-example of dimension $9 \cdot 6$. This could be

temp = RandomReal[{0, 1000}, {9, 6}];

The problem is that I got additional blank lines in my CSV file. How can I avoid those blank lines? I did not have them using plain Export.

Second part of the question: can I use a similar approach to speed up Import for CSV files?

I work in English locale in Windows 7 and Mathematica 8. My data should be comma-separated.

The result of the above looks like this:







EDIT: A small example of the data that I want to import (2nd step) can be found here. The true problem size is here.

  • $\begingroup$ Can you give a short sample of what your input and output files look like? $\endgroup$
    – RunnyKine
    Commented Nov 4, 2013 at 17:37
  • $\begingroup$ First I would like to takle the export. For this one could use the random number example provided. I would like to combine Export and ExportString in order to speed up the export. If it is correct for the toy example then it should be alright for the big data. Is this enough info for you? $\endgroup$
    – Richi W
    Commented Nov 4, 2013 at 17:40
  • $\begingroup$ I actually was looking to use a different approach especially for the Import aspect since ReadList is much faster than Import. I was going to write something from scratch that will be much faster than anything Import and Export can achieve. So for me to see a sample file will be helpful. $\endgroup$
    – RunnyKine
    Commented Nov 4, 2013 at 17:45
  • 1
    $\begingroup$ Why don't you have a look at this? $\endgroup$ Commented Nov 4, 2013 at 17:57
  • 1
    $\begingroup$ Are you required to use csv? Exchanging data in binary is typically much faster, though obviously your external program needs to be modified to read it. $\endgroup$
    – george2079
    Commented Nov 4, 2013 at 18:46

2 Answers 2


Here's a much faster, purely Mathematica way than using Import to import your data:


As Leonid mentioned the previous code doesn't exactly replicate Import. The truth is I was only trying to retrieve the numerical part. Here's an updated version that tries to replicate the output from Import.

readYourCSV2[file_String?FileExistsQ, n_Integer] := Module[{str = OpenRead[file], data}, 
  data = ReadList[str, Table[Record, {n}], RecordSeparators -> {",", "\n"}]; 
  ReleaseHold[ToExpression[data, InputForm, Hold] /. {Plus[Times[x_, E | e], y_] :> x * 10 ^ y}] 

Here, n is the number of columns.


Now for the Export, here's a fast, again, purely Mathematica way to export in CSV format.

writeYourCSV[file_String, list_List?MatrixQ] := 
 With[{str = OpenWrite[file, PageWidth -> Infinity], len = Length[ list[[1]] ]},
      Scan[Write[str, Sequence @@ (Flatten[Table[{FortranForm[ #[[i]] ], OutputForm[","]}, 
              {i, len - 1}]]) ~ Join ~ { FortranForm[ #[[len]] ] }] &, list]; Close[str];

This takes less than 10 seconds to write your large data back to CSV format:

writeYourCSV["testcsv.csv", databig] // AbsoluteTiming

{9.921969, Null}

  • $\begingroup$ This does not give the same result as Import["~/Downloads/returns_out_small.csv"], even if we compare purely numerical part. $\endgroup$ Commented Nov 4, 2013 at 18:49
  • $\begingroup$ But the large file contains many more columns (1419, to be precise). $\endgroup$ Commented Nov 4, 2013 at 19:09
  • $\begingroup$ I did compare this to my method, calling your code as DeleteCases[readYourCSV["~/Downloads/returns_out.csv", 1419], {(Null | EndOfFile) ..}]. The strange thing was that your result was one row less (missed the very first one, it seems), and also that the results agreed only to 10^-3. $\endgroup$ Commented Nov 4, 2013 at 20:11
  • $\begingroup$ @LeonidShifrin. Thanks for checking it. I think the reason is I forgot to add the \n RecordSeparator. I've updated my code to include this and also include the first row (heading) of the .CSV file $\endgroup$
    – RunnyKine
    Commented Nov 4, 2013 at 20:42
  • $\begingroup$ Yes, I also meant the numerics only. I did not mean the headers - I ignore them too. $\endgroup$ Commented Nov 4, 2013 at 21:53

Here is a Java-based solution, pretty fast but valid only when all your columns are numerical (double) values.

First, grab and run the code for the Java reloader (The linked version should work on Windows and probably Linux, but was reported to have issues for OS X. So, Mac users may try this one instead: Import["https://gist.github.com/lshifr/7307845/raw/SimpleJavaReloader.m"] - not yet tested this version for other platforms). Then, compile this class:

JCompileLoad["public class DoubleParser{
   public static double[] parseDouble(String[] strdub){
      double[] res = new double[strdub.length];
      int i = 0;
      for(;i < strdub.length;i++){
            res[i]= Double.parseDouble(strdub[i]);
         } catch (NumberFormatException e){
            res[i] = 0;
      return res;

Then, here is the Mathematica counterpart:

importDoubleCSV[file_String?FileExistsQ, opts : OptionsPattern[]]:=

For your small file, the result agrees with what you get by using Import, after you remove empty rows:


(* True *)

Your large file gets processed on my machine in about 4 seconds:



which doesn't look bad to me. I wasn't patient enough to wait until Import["~/Downloads/returns_out.csv"] finishes, so did not compare results in this case - but the reader is most welcome to do that (and the timings too).

An added advantage here is that we get the results packed:

Developer`PackedArrayQ @ resLrg

(* True *)

Note that Java parsing code adopts a convention to replace all non-parsable strings with zeros. It is possible to improve on this, by returning also the positions of non-parsable strings, separately. Note also that the UTF-8 encoding is implicitly assumed.

  • $\begingroup$ Wow! No we have Javareloader for Mac! Cool! $\endgroup$
    – Murta
    Commented Nov 5, 2013 at 2:17
  • $\begingroup$ I actually think that the speed achieved here is more or less (up to a factor of 2 or 3) as good as one gets. Further qualitative improvements can only be achieved by switching to a binary format to store the data - which was already suggested in comments. $\endgroup$ Commented Nov 5, 2013 at 3:20
  • $\begingroup$ @Murta You bet :). This post was done using a Mac, and I am on a Mac right now. $\endgroup$ Commented Nov 5, 2013 at 3:23
  • $\begingroup$ I tested the new version (using direct import from github link) in MMA 9 for Mac and in MMA 10 beta for windows. Worked wonderfully! Now I'm more motivated to study some Java in the comfort of mathematica environment. $\endgroup$
    – Murta
    Commented Nov 8, 2013 at 1:30
  • $\begingroup$ @Murta I also do have the syntax highlighter and some other tools for Java, will hopefully publish those some time soon. $\endgroup$ Commented Nov 8, 2013 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.