I have a large 1D dataset of double precision numbers that are stored in many external database files. In order to load the whole dataset into Mathematica 8 I iterate over the files
List. The problem is that my computer keeps running out of memory even though the memory available should be more than enough to store the whole set. When researching the issue I discovered a strange behaviour (see the code below). The list produced by
AppendTo consumes much more memory than would correspond to the size of an integer. I presume that, in the latter
List, Mathematica uses some different kind of objects to store the data.
Any suggestions on how to reduce the memory usage?
In:= ByteCount[Range[1, 10000]] Out= 40168 In:= data = List; For[i = 0, i < 10000, ++i, AppendTo[data, i]; ]; In:= ByteCount[data] Out= 320040