I have a large .txt file for which I need to do some data analysis. I've been working on it for a while, and the analysis part has got faster and faster so that most of the evaluation time is now spent reading the data. The data is just a long list of numbers like this:
. . . 2.216769 2.188232 2.178618 2.204103 2.172208 2.178618 2.191284 . . .
where the value of the numbers fluctuates between 5 and -2. There are a few hundred millions of these, and I've been reading and analyzing them in chunks of 20 millions at a time. So far, the most efficient way I found was to open an
InputStream and use
ReadList in the straight forward manner:
data=ReadList[stream, Real, 2*10^7]
but even this method reads only at a speed of about 30 MB/s, much below the speed limit of the hard disk (around 140 MB/s). I tried two more convoluted ways: one was parallelizing reading, which didn't make any difference, and the second was using
BinaryReadList followed by
FromCharacterCode and then
ToExpression, which ended up being 10 times slower.
I'm not sure but it feels like there has to be a much more efficient way to read this data. Any help please?
EDIT: Here is the Google Drive link for a data sample. It consists of 1 million numbers, but the file is just 10 MBs in size.