I've been using BinaryReadList
to load reasonably large 100-500MB files and its been working well.
However, I've never had a way to watch the progress. Based @Leonid's answer I hacked together a ProgressIndicator
for the function.
In my case my large data files consist of sets of 4 32bit (big endian) floats. Here I read the file in 100 steps. Timing is similiar to reading the file all at once.
ImportProgress[filename_] := Module[{str, data, n},
(*open stream*)
str = OpenRead[filename, BinaryFormat -> True];
data = {}; n = 1;
(*display the dynamic progress indicator*)
Print[ProgressIndicator[Dynamic[n/100]]];
(*read data*)
While[n < 100,
AppendTo[data,
BinaryReadList[str, {"Real32", "Real32", "Real32", "Real32"},
Ceiling[FileByteCount[filename]/16/100], ByteOrdering -> +1]];
n++];
(*close stream*)
Close[str];
(*return the data*)
data
]
(*read the data*)
data = ImportProgress["R06_14539-v01.pos"];
Here is a more elegant alternative using Monitor
where the ProgressIndicator
disappears after the file finishes loading:
ImportProgressMonitor[filename_] := Monitor[
Module[{str, data},
(*open stream*)
str = OpenRead[filename, BinaryFormat -> True];
data = {};
(*read data*)
n = 1;
While[n < 100,
AppendTo[data,
BinaryReadList[str, {"Real32", "Real32", "Real32", "Real32"},
Ceiling[FileByteCount[filename]/16/100], ByteOrdering -> +1]];
n++];
(*close stream*)
Close[str];
data
],
(*monitor the progress*)
ProgressIndicator[n/100]
]
Get
? $\endgroup$Get:bigfile
error usingGet[]
. $\endgroup${{...}, {...}}
where...
is a long sequence of expressions. $\endgroup$