4
$\begingroup$

I'm trying to export a 1000*100 matrix of numbers to a .dat file using Export (I want the numbers to be separated by a tab) and it's very slow, it takes several minutes, whereas when I export to an Excel file it takes a few seconds.

Is there a way to speed this up ?

Also how can I control the number of decimals I want to export ?

Edit
I found the issue, this happens when I export my dat file on a network drive (at work), the export to an Excel file and a dat file must be implemented in different ways that produce different behaviours. Thank you though for the different interesting answers.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ What command are you using to export the data? Exporting to .dat should take seconds rather than minutes. For example on my system, exporting a 1000x100 of random reals to a .dat file takes about 2.6 seconds. $\endgroup$
    – Heike
    Feb 23, 2012 at 11:11
  • $\begingroup$ mat = RandomReal[1, {1000, 100}]; Export["test.dat", mat, "TSV"]; // AbsoluteTiming runs in 2.3 seconds here. $\endgroup$
    – Szabolcs
    Feb 23, 2012 at 12:51

3 Answers 3

Reset to default
8
$\begingroup$

I suggest you watch the presentation BigData: Demystifying Large Datasets in Mathematica by Nick Lariviere. He discusses the various possibilities quite clearly.

Improve this answer
$\endgroup$
4
$\begingroup$

To answer your second question, to control the number of decimals of the exported data you could change the accuracy of your data using SetAccuracy. For example if you only want to export the first two decimals of each entry you could do something like

data = RandomReal[20, {1000, 100}];
Export["test.dat", SetAccuracy[data, 3]]
Improve this answer
$\endgroup$
2
  • $\begingroup$ Heike, do you know any (reasonable often used) application, where you want to restrict the accuracy beside when you want to look at the numbers? Maybe my answer is not correct, but I cannot think of any. $\endgroup$
    – halirutan ♦
    Feb 23, 2012 at 16:38
  • $\begingroup$ @halirutan It could be that the data is the result of manipulation of experimental data. In that case, the calculated data will have a number of significant digits dependent on e.g. the measurement error of the experimental data. Any digits beyond that won't have any meaning so there's no point saving those. $\endgroup$
    – Heike
    Feb 23, 2012 at 16:57
3
$\begingroup$

I cannot reproduce your behavior here. For a $1000\times 1000$ it requires only some seconds here

data = Table[N[i/j], {j, 1000}, {i, 1000}];
AbsoluteTiming[
 Export["tmp/matrix.dat", data, "Table", "FieldSeparator" -> {"\t"}]]

(*
  {13.442387, "tmp/matrix.dat"}
*)

If your separator for numbers is always a tab, then clearly the way @Szabolcs pointed out in his comment should be the preferred one. 

Export["tmp/matrix.dat", data, "TSV"]

My sample shows, that you can basically use every desired line/field separator you like.

Heike gave the answer to your second question. I would like to point out, that you might lose information because real numbers are usually stored with $MachinePrecision. When you use now SetAccuracy or SetPrecision you cut them off. There is usually only one situation where you want this: When you look at the numbers. I suspect you are looking at a $1000\times 100$ matrix, so why do you want to cut your numbers?

Improve this answer
$\endgroup$
1
  • $\begingroup$ I wanted to be able to visually quickly check some numbers in the exported file as decimals are not important in my case, and didn't know the proper function to truncate decimals. $\endgroup$
    – faysou
    Feb 23, 2012 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.