Performance-driven approach to using Big Data without killing the hard-drive

Question

Situation

A calculation has to be done in which the amount of data is too much to fit into memory. Fortunately, only a part of the overall data set has to be known at a time. The preprocessing is done in C++, the data is exported as binary file (typically between 20-70 GB) and should be used within Mathematica now.

General problem

It is of course much faster to read the needed parts of my data from a computer's RAM (on order of $10^{-6}$ seconds) than to read it from disk (between $10^{-5}$ seconds with BinaryReadList and $10^{-2}$ seconds with Import). To my mind: Reading only a few numbers of the overall 20-70 GB in a step each time and again is extremely IO-consuming and doesn't make sense. So I decided to read 4 GB into my RAM as cache at a time and use the RAM as long as the needed numbers are to be found within the cache.

All operations needed are that time-consuming ($10^{-4}$) that caching isn't very fast. In fact, it's slower than simply reading directly from my SSD via binary files and BinaryReadList.

Question

Considering the above mentioned facts:

a) Is it a good idea to step by step read in data of very small chunks? Will it harm my underlying SSD? Or is it technically equivalent for my SSD to read ten thousand entries in one step or ten entries in thousand steps? How does BinaryReadList work under the hood? Do I do any harm to my harddrive by completely relying on it millions and millions of time?

b) If my current approach to repeatedly use BinaryReadList does harm to my machine: What are (fast!) alternatives?

---

EDIT: Why this question is no duplicate in my eyes

The linked answer from Leonid Shifrin in the comment below answer the following question (let me cite):

Let us say we have a large list already constructed in memory in Mathematica, call it testList. Its elements are lists themselves. What I will do is traverse it element by element. For a given element (sub-list), we will analyze how much memory it occupies, and if this amount exceeds a certain threshold that we specify, we will create a key-value pair for it. The key will be some dummy generated symbol, and the value will be a file name for a file where we will save a contents of this element.

As I have a list that doesn't fit into memory, that wasn't generated by Mathematica for performance reasons and that isn't available in another format than raw binary data I have in my eyes no possibility to make use of Leonid's marvellous concept.

My question is slightly different from the one answered in the linked question in so far that I want to bit by bit read a file (whose structure I have no control over) very fast. The performance of Leonid's approach seems as if a speed-memory-tradeoff has taken place. This is not what I want. So the initial question remains: Can I read the binary data with BinaryReadList in millions of small chunks without doing harm to my harddrive. I admit that in order to answer this question knowledge about the underlying aspects of Wolframs technology is needed.

As requested here is more code to support my problem even if I don't see in how far this might be useful due to the fact that I can't share the original about 50 GB large file here:

(* Write example data in binary format. *)
path = "path_to_data.dat";
BinaryWrite[path, Range[10^9], "Integer64"];
str = OpenRead[path, BinaryFormat -> True];
(* Repeatedly read very small chunk and travel through InputStream. *)
n = 1; 
While[n < 100000, BinaryReadList[str, "Integer64", 20]; n++]

Answer 1

So, a large part of this question seems to be concerned with whether or not BinaryReadList can cause meaningful damage to an underlying drive (SSD or HDD). From what I can see, the answer to this should generally be no, regardless of internal implementations.

Regarding SSDs: this Super User answer seems to state that reads on SSDs are essentially free. No matter how many separate reads you make, and no matter how large the reads are, there's probably no real reason to worry about damaging the drive.

Regarding HDDs: reads are not precisely free on HDDs, since they require the head to move. However, given that reads are probably less strenuous than writes, and that writes have no obvious correlation with hard drive failures (see this answer on Super User), it's still probably not worth worrying about.

In either case, I would be extremely surprised to learn that Mathematica has low-level access enough to any drive to cause damage by performing reads on it, so there's no particular reason to suspect that the implementation of BinaryReadList is exceptionally important.

Thus, I would be comfortable assuming that you are not doing excess harm to your machine.

---

Regarding speed, it's likely bad for performance to fetch less than the block size of the device at any one point. However, since the file handler is being kept open between reads, the associated file system overhead should only be paid once.

To test the overall speed of various block sizes for BinaryReadList, I created a 1 GB file of random integers and did some comparison tests off of a couple of drives of greatly varying speeds:

testfile = RandomInteger[{0, 2^64 - 1}, 134217728];

Export[file, testfile, "UnsignedInteger64"]; (* where file is a file on the disk of choice *)

test[n_, file_] := (fdf = OpenRead[file, BinaryFormat -> True];
   i = 0;
   res = First@
     AbsoluteTiming[
      dat = ReleaseHold[
         ConstantArray[
          Hold[BinaryReadList[fdf, "UnsignedInteger64", n]], 
          1000000/n]];
      ];
   Close[fdf];
   Clear[fdf, i];
   res);

Table[test[10^i, file], {i, 0, 6}]

This will, for a given file, test how long it takes to read 1000000 unsigned 64-bit integers at a block size of n. ConstantArray was the fastest method I could find for generating the appropriate set of BinaryReadList calls, but it still represents a significant amount of the processing time. The validity of the test call can be ascertained by executing Flatten[dat] == testfile[[1;;1000000]] afterwards.

On a flash drive I happen to have laying around, it evens out greatly for n larger than 100, producing a table output as follows. (The tables are logarithmic in n, so they correspond to n={1,10,100,1000,...})

{1.46397, 0.552859, 0.423608, 0.410021, 0.408184, 0.40834, 0.408047}

On a typical consumer hard drive, the result is basically the same:

{1.50543, 0.551514, 0.425884, 0.411355, 0.410407, 0.409781, 0.411309}

On a high end solid state drive, similarly:

{1.50259, 0.549351, 0.423322, 0.412367, 0.409349, 0.408443, 0.407553}

Off of a RAM-disk setup with ImDisk:

{1.51346, 0.552947, 0.425905, 0.411393, 0.408058, 0.407433, 0.409391}

These numbers are actually quite surprising to me, as they indicate that the read-write speed of the disk has essentially no correlation with the speed of BinaryReadList.

Based on these numbers, I guess the fastest thing to do with BinaryReadList is read at least 10 numbers at a time. That said, if you're optimizing for a specific use case, please benchmark against that use case.