Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

As I have learned in this community, one way to import a hefty data files (e.g. millions of rows of {x1, y1} data) if you don't care about the individual elements (maybe just want to obtain some statistics) is to use the following:

s = OpenRead["path/test.dat"];
p = Sort[RandomSample[Range[5 10^5], 100000]];
dataTable = ((If[#1 != 1, Skip[s, Record, #1 - 1], Null]; 
  ReadList[s, Number, 2]) &) /@ Differences[Prepend[p, 0]];

which assumes that the length of "test.dat" is 500000 and we randomly sample only 100000 items.

I am now in want of a slightly different function. I'd like to take just the ordered pairs of "test.dat" such that x1 < x0 but this time take all of them and not random ones.

I can do this by importing the whole table and then using something like Cases to get the desired range but importing the entire table is exactly what I want to avoid since the tables are very very large.

share|improve this question
1  
May I suggest running a text processing utility like AWK, Perl, or whatever filter you want instead of using Mathematica for this task? –  belisarius Nov 21 '13 at 17:25
    
If I understand your problem correctly you can't avoid looking at all lines of the file. If your file is large and performance is a matter, I'd strongly suggest to read the file in chunks of appropriate size instead of line by line and then treat each chunk in memory. See e.g. my answer (here)[mathematica.stackexchange.com/a/15216/169] about how you can do that. If you don't mind the extra efforts Leonid's answer to the same question provides an even faster method to read files using Java, but you'd have to adopt the Java code for your problem... –  Albert Retey Nov 22 '13 at 9:57
add comment

1 Answer

I do not have a file to test on, but the following code shows the main idea: Use Sow

keepOnlyOrdered = (If[OrderedQ[#], Sow[#]]; #) &; 
Reap[ While[keepOnlyOrdered[ReadList[s, Number, 2]] != EndOfFile,]]
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.