Situation
A calculation has to be done in which the amount of data is too much to fit into memory. Fortunately, only a part of the overall data set has to be known at a time. The preprocessing is done in C++, the data is exported as binary file (typically between 20-70 GB) and should be used within Mathematica now.
General problem
It is of course much faster to read the needed parts of my data from a computer's RAM (on order of $10^{-6}$ seconds) than to read it from disk (between $10^{-5}$ seconds with BinaryReadList
and $10^{-2}$ seconds with Import
). To my mind: Reading only a few numbers of the overall 20-70 GB in a step each time and again is extremely IO-consuming and doesn't make sense. So I decided to read 4 GB into my RAM as cache at a time and use the RAM as long as the needed numbers are to be found within the cache.
All operations needed are that time-consuming ($10^{-4}$) that caching isn't very fast. In fact, it's slower than simply reading directly from my SSD via binary files and BinaryReadList
.
Question
Considering the above mentioned facts:
a) Is it a good idea to step by step read in data of very small chunks? Will it harm my underlying SSD? Or is it technically equivalent for my SSD to read ten thousand entries in one step or ten entries in thousand steps? How does BinaryReadList
work under the hood? Do I do any harm to my harddrive by completely relying on it millions and millions of time?
b) If my current approach to repeatedly use BinaryReadList
does harm to my machine: What are (fast!) alternatives?
EDIT: Why this question is no duplicate in my eyes
The linked answer from Leonid Shifrin in the comment below answer the following question (let me cite):
Let us say we have a large list already constructed in memory in Mathematica, call it testList. Its elements are lists themselves. What I will do is traverse it element by element. For a given element (sub-list), we will analyze how much memory it occupies, and if this amount exceeds a certain threshold that we specify, we will create a key-value pair for it. The key will be some dummy generated symbol, and the value will be a file name for a file where we will save a contents of this element.
As I have a list that doesn't fit into memory, that wasn't generated by Mathematica for performance reasons and that isn't available in another format than raw binary data I have in my eyes no possibility to make use of Leonid's marvellous concept.
My question is slightly different from the one answered in the linked question in so far that I want to bit by bit read a file (whose structure I have no control over) very fast. The performance of Leonid's approach seems as if a speed-memory-tradeoff has taken place. This is not what I want. So the initial question remains: Can I read the binary data with BinaryReadList
in millions of small chunks without doing harm to my harddrive. I admit that in order to answer this question knowledge about the underlying aspects of Wolframs technology is needed.
As requested here is more code to support my problem even if I don't see in how far this might be useful due to the fact that I can't share the original about 50 GB large file here:
(* Write example data in binary format. *)
path = "path_to_data.dat";
BinaryWrite[path, Range[10^9], "Integer64"];
str = OpenRead[path, BinaryFormat -> True];
(* Repeatedly read very small chunk and travel through InputStream. *)
n = 1;
While[n < 100000, BinaryReadList[str, "Integer64", 20]; n++]
BinaryReadList
what I currently use - the lack of performance renders it useless for my purpose. Secondly, my question already proposes a direct and simple solution but asks for technical details onBinaryReadList
and making extended use thereof. $\endgroup$ – pbx Mar 15 '17 at 14:59BinaryReadList
and its effect on your hard-drive I think you'll want to contact WRI directly. Certainly post here whatever they tell you, but I'm not certain anyone here will be able or allowed to tell you what you want to know. $\endgroup$ – b3m2a1 Mar 15 '17 at 15:14