I just wanted to add an all Mathematica solution as it probably indicates how similar problems can be tackled where for whatever reason something like the Java approach isn't possible or necessary. I've used a similar approach to e.g. read only small parts of huge XML-files. This isn't faster as the memory intensive read-all-at-once but at least isn't slower and should beund is much more memory efficient (I'd expect it to use about twice the memory of the final result, as we need one copy).
It basically reads the file in chunks of adjustable size, extracts the part of interest of each chunk and collects everything. Here is the code:
AbsoluteTiming[
str = OpenRead[filename, BinaryFormat -> True];
chunksize = 500;
data = {};
res = {1};
While[Or[data == {},
And[res[[-1, -1]] =!= EndOfFile, Length[res] > 0]],
data = {data,
res =
BinaryReadList[
str, {"Real32", "Real32", "Real32", "Real32", "Real32",
"Real32", "Real32", "Real32", "Real32", "Integer32",
"Integer32"}, chunksize, ByteOrdering -> +1][[All, {4, 11}]]
};
];
Close[str];
data = Partition[Flatten[data], 2];
]
A slightly more elegant way using Sow and Reap with localized variables and packed as a function is this:
readFile[filename_, chunksize_Integer: 500] := Module[{
str = OpenRead[filename, BinaryFormat -> True],
res = {{1}},
data
},
data = Reap[
While[Or[And[res[[-1, -1]] =!= EndOfFile, Length[res] > 0]],
Sow[res =
BinaryReadList[
str, {"Real32", "Real32", "Real32", "Real32", "Real32",
"Real32", "Real32", "Real32", "Real32", "Integer32",
"Integer32"}, chunksize, ByteOrdering -> +1][[
All, {4, 11}]]];
]
][[2]];
Close[str];
Flatten[data[[1]], 1]
]
I did expect this to use about twice the memory of the final result, as we need one copy and depending on the chunksize the additional memory for the intermediate data (that is all columns except for 4 and 11) which is "thrown away" should become neglectable for small chunk sizes.
I have now done some measuring and one interesting thing is that memory usage doesn't necessarily become much larger than the final total datasize, which indicates that Mathematica doesn't make a full copy for the rearranged final result. Another interesting feature is that the memory usage of this approach for small chunksizes seems to decrease with increasing chunksizes. Only for very large chunk sizes (larger than about 100 000 for this example with a 600MB file on my computer) the behavior is "as expected": the memory consumption will then increase as the chunk size increases. My interpretation of this is that Mathematica is doing some internal optimization/buffering for BinaryReadList which mainly influences the situation for small chunk sizes. The result is that very small chunksizes are slow and use more memory than larger values, presumably a side effect of the large number of iterations and thus evaluations of the loop-body. Then for junksizes larger than about 50 neither runtime nor memory usage seem to vary very much until chunk sizes get relatively large, presumably comparable to the internal buffer sizes. The precise explanation for all details of this behaviour is beyond my current knowledge, all in all I see a great decrease in memory usage for this approach, but max. memory usage for large files is always in the order of sizeOfFinalResult + O[100MB] for medium sized chunks and can be kept at roughly that value without much cost in run time even for very large files. With this approach it is possible to read files too large to be read all-at-once. Your mileage may vary depending on data, chunksize and hardware, so some testing-measuring when using this approach is certainly a good idea...
