8
$\begingroup$

I have a large 1D dataset of double precision numbers that are stored in many external database files. In order to load the whole dataset into Mathematica 8 I iterate over the files AppendingTo a List. The problem is that my computer keeps running out of memory even though the memory available should be more than enough to store the whole set. When researching the issue I discovered a strange behaviour (see the code below). The list produced by AppendTo consumes much more memory than would correspond to the size of an integer. I presume that, in the latter List, Mathematica uses some different kind of objects to store the data.

Any suggestions on how to reduce the memory usage?

In[78]:= ByteCount[Range[1, 10000]]
Out[78]= 40168

In[79]:= data = List[];
For[i = 0, i &lt 10000, ++i,
  AppendTo[data, i];
];

In[81]:= ByteCount[data]
Out[81]= 320040
$\endgroup$
  • 2
    $\begingroup$ In addition to Rojo's answer, for better performance avoid AppendTo and do something similar to this: Join @@ (Import /@ FileNames["data*.txt"]). Also avoid For if you can. $\endgroup$ – Szabolcs May 29 '12 at 14:45
15
$\begingroup$

You're looking to pack your array.

It's done with Developer`ToPackedArray

dataPacked = Developer`ToPackedArray[data];
ByteCount[dataPacked]

40168

There's more on this following the link.

The links were taken from the packed array section of this answer.

$\endgroup$
  • 1
    $\begingroup$ You beat me, have an upvote. $\endgroup$ – s0rce May 29 '12 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.