# importing large data files with a constraint

As I have learned in this community, one way to import a hefty data files (e.g. millions of rows of {x1, y1} data) if you don't care about the individual elements (maybe just want to obtain some statistics) is to use the following:

s = OpenRead["path/test.dat"];
p = Sort[RandomSample[Range[5 10^5], 100000]];
dataTable = ((If[#1 != 1, Skip[s, Record, #1 - 1], Null];
ReadList[s, Number, 2]) &) /@ Differences[Prepend[p, 0]];


which assumes that the length of "test.dat" is 500000 and we randomly sample only 100000 items.

I am now in want of a slightly different function. I'd like to take just the ordered pairs of "test.dat" such that x1 < x0 but this time take all of them and not random ones.

I can do this by importing the whole table and then using something like Cases to get the desired range but importing the entire table is exactly what I want to avoid since the tables are very very large.

-
May I suggest running a text processing utility like AWK, Perl, or whatever filter you want instead of using Mathematica for this task? –  belisarius Nov 21 '13 at 17:25
If I understand your problem correctly you can't avoid looking at all lines of the file. If your file is large and performance is a matter, I'd strongly suggest to read the file in chunks of appropriate size instead of line by line and then treat each chunk in memory. See e.g. my answer (here)[mathematica.stackexchange.com/a/15216/169] about how you can do that. If you don't mind the extra efforts Leonid's answer to the same question provides an even faster method to read files using Java, but you'd have to adopt the Java code for your problem... –  Albert Retey Nov 22 '13 at 9:57
I do not have a file to test on, but the following code shows the main idea: Use Sow
keepOnlyOrdered = (If[OrderedQ[#], Sow[#]]; #) &;