8
$\begingroup$

I have a large .txt file for which I need to do some data analysis. I've been working on it for a while, and the analysis part has got faster and faster so that most of the evaluation time is now spent reading the data. The data is just a long list of numbers like this:

.
.
.
2.216769
2.188232
2.178618
2.204103
2.172208
2.178618
2.191284
.
.
.

where the value of the numbers fluctuates between 5 and -2. There are a few hundred millions of these, and I've been reading and analyzing them in chunks of 20 millions at a time. So far, the most efficient way I found was to open an InputStream and use ReadList in the straight forward manner:

data=ReadList[stream, Real, 2*10^7]

but even this method reads only at a speed of about 30 MB/s, much below the speed limit of the hard disk (around 140 MB/s). I tried two more convoluted ways: one was parallelizing reading, which didn't make any difference, and the second was using BinaryReadList followed by FromCharacterCode and then ToExpression, which ended up being 10 times slower.

I'm not sure but it feels like there has to be a much more efficient way to read this data. Any help please?

EDIT: Here is the Google Drive link for a data sample. It consists of 1 million numbers, but the file is just 10 MBs in size.

$\endgroup$
  • 1
    $\begingroup$ I unfortunately have no answer but it would be helpful to have an example file (maybe not 1 million lines) so that people could check the timings. $\endgroup$ – anderstood Feb 22 '18 at 0:33
  • 2
    $\begingroup$ what you have is probably the best you can do with mathematica. Of course you only need to run it once and dump the data out in a binary format for future use. (and flog whoever supplied that much data in ascii to begin with) $\endgroup$ – george2079 Feb 22 '18 at 2:40
8
+50
$\begingroup$

Your real problem is

I have a large .txt file

In a txt file, each number is represented by a sequence of bytes with possibly varying length followed by a newline. When you read it back in, the bytes representing the digits, the number-point and the newline need to be converted back to real numbers. This is time-consuming.

On the other hand, in a binary file, each real number has fixed number of bytes and there doesn't need to be some separator between them like a newline. For instance, a "Real32" contains only 4 bytes, while the same number, e.g. 0.827736082 needs the following bytes in your text-file

ToCharacterCode["0.827736082"]
(* {48, 46, 56, 50, 55, 55, 51, 54, 48, 56, 50} *)

Therefore, to turn this back into a number, you need to re-convert each byte into a digit and rebuilt a real number from it. To give you an idea how time-consuming this is, let us create a sample list with 10^6 entries and export it to a list like yours and to a binary file

list = RandomReal[1, 10^6];

asciFile = "~/tmp/ascii.txt";
binFile = "~/tmp/binary.data";

Export[asciFile, list, "Table"];

st = OpenWrite["~/tmp/binary.data", BinaryFormat -> True];
BinaryWrite[st, list, "Real32"];
Close[st];

The first thing you should note is vast difference in file size

FileByteCount /@ {asciFile, binFile}
(* {19270172, 4000000} *)

and after understanding the paragraphs above, it's clear where this difference comes from. The numbers in list will look like this 0.21946835530269415 which consists of 19 bytes. Some will be shorter though. With Real32 number using 4 bytes you clearly see where this difference comes from.

When we look at the timings to read back in each file, we see that the ASCII file needs about 10x the time of reading the binary file.

BinaryReadList[asciFile, "Byte"]; // RepeatedTiming
(* {0.12, Null} *)

BinaryReadList[binFile, "Real32"]; // RepeatedTiming
(* {0.018, Null} *)

Therefore, the only acceptable solution for serially reading your data is to use a binary format for storing your data in the first place. This is supported by the fact that even C code that consists of nothing more than a loop to read in your file is not faster.

If your numbers were of fixed length, it might be worth a shot to read in the data in bytes and convert them in parallel into real numbers, but this is not possible in your current layout.

$\endgroup$
  • $\begingroup$ Thanks for the great explanation! Actually the numbers are of fixed length. I wrote them earlier as they appear in Mathematica, which uses a short notation. Sorry for that. $\endgroup$ – Tofi Feb 25 '18 at 19:41
  • $\begingroup$ @Tofi I was not referring to your question here! I was looking at your sample file which has shorter numbers at the end. The question is, if you have say 5 digits like 1.2345 what happens with the number e.g. 1.1? Is it padded to 1.1000 or left as 1.1? If you can influence the files you read, isn't it possible to store them in binary format in the first place? $\endgroup$ – halirutan Feb 26 '18 at 11:55
2
$\begingroup$

This is not a complete code answer, but you should get the idea.

After running some tests, it seems that reading the data in parallel seems to speed up the process. Unfortunately there are a few hurdles to overcome when splitting up the stream.

Here is what I came up with so far:

filename = "D:\\temp\\list.txt"; (*replace with your file*)
LaunchKernels[];
kernels = Kernels[];
stream = OpenRead[filename];
endPosition = SetStreamPosition[stream, Infinity]; (*obtain the last position in the stream*)
outList = Flatten@ParallelTable[
   list = {};
   listTemp = {};
   stream = OpenRead[filename];
   SetStreamPosition[
    stream, (i - 1)*Round[endPosition/Length[kernels]]];
   Skip[stream, Real];
   While[True, (*this is the part that is not working properly at the end of each stream segment*)
    listTemp = ReadList[stream, Real, 100];
    If[StreamPosition[stream] >= i*Round[endPosition/Length[kernels]],
      Break[]];
    AppendTo[list, listTemp];
    ];
   list
   , {i, $KernelCount}
]; // AbsoluteTiming

The code gets the Length of the stream in characters (note that StreamPosition is the position in characters, not in lines or numbers). So this is also the part where it gets a little tricky and the code above does not read every single number of your file yet. You might come up with a proper solution, the code above should give you the idea.

Overall this code above cut down the reading time to approximately 1/$KernelCount of the time when using ReadList with a single kernel in my experiment.

$\endgroup$
  • $\begingroup$ Hmm, this code isn't even close what I get when I use a simple ReadList. I see your point, but even launching the parallel kernels takes over 10x the time of simply reading the one million values from the sample.txt. $\endgroup$ – halirutan Feb 25 '18 at 1:38
  • $\begingroup$ My measurements did not include any overhead loading parallel kernels, etc.. And I used my own file with one million entries, not the sample one. I figured, that the real case scenario is about way bigger files anyways. When I have time a can do some more digging. Also I do think that things probably are heavily dependent on the setup: I ran it on Mathematica 11.2 on Windows with a 7200 rpm HD (not an SSD). $\endgroup$ – Wizard Feb 25 '18 at 10:20
  • $\begingroup$ Thanks for your answer. In fact I tried parallelization before, and it didn't make any difference. I tried your code with some modifications and it still reads at the same speed. Idk it looks like reading cannot be parallelized. $\endgroup$ – Tofi Feb 25 '18 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.