I have a large .txt file for which I need to do some data analysis. I've been working on it for a while, and the analysis part has got faster and faster so that most of the evaluation time is now spent reading the data. The data is just a long list of numbers like this:
.
.
.
2.216769
2.188232
2.178618
2.204103
2.172208
2.178618
2.191284
.
.
.
where the value of the numbers fluctuates between 5 and -2. There are a few hundred millions of these, and I've been reading and analyzing them in chunks of 20 millions at a time. So far, the most efficient way I found was to open an InputStream
and use ReadList
in the straight forward manner:
data=ReadList[stream, Real, 2*10^7]
but even this method reads only at a speed of about 30 MB/s, much below the speed limit of the hard disk (around 140 MB/s). I tried two more convoluted ways: one was parallelizing reading, which didn't make any difference, and the second was using BinaryReadList
followed by FromCharacterCode
and then ToExpression
, which ended up being 10 times slower.
I'm not sure but it feels like there has to be a much more efficient way to read this data. Any help please?
EDIT: Here is the Google Drive link for a data sample. It consists of 1 million numbers, but the file is just 10 MBs in size.