2
$\begingroup$

As I have learned in this community, one way to import a hefty data files (e.g. millions of rows of {x1, y1} data) if you don't care about the individual elements (maybe just want to obtain some statistics) is to use the following:

s = OpenRead["path/test.dat"];
p = Sort[RandomSample[Range[5 10^5], 100000]];
dataTable = ((If[#1 != 1, Skip[s, Record, #1 - 1], Null]; 
  ReadList[s, Number, 2]) &) /@ Differences[Prepend[p, 0]];

which assumes that the length of "test.dat" is 500000 and we randomly sample only 100000 items.

I am now in want of a slightly different function. I'd like to take just the ordered pairs of "test.dat" such that x1 < x0 but this time take all of them and not random ones.

I can do this by importing the whole table and then using something like Cases to get the desired range but importing the entire table is exactly what I want to avoid since the tables are very very large.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ May I suggest running a text processing utility like AWK, Perl, or whatever filter you want instead of using Mathematica for this task? $\endgroup$
    – Dr. belisarius
    Commented Nov 21, 2013 at 17:25
  • $\begingroup$ If I understand your problem correctly you can't avoid looking at all lines of the file. If your file is large and performance is a matter, I'd strongly suggest to read the file in chunks of appropriate size instead of line by line and then treat each chunk in memory. See e.g. my answer (here)[mathematica.stackexchange.com/a/15216/169] about how you can do that. If you don't mind the extra efforts Leonid's answer to the same question provides an even faster method to read files using Java, but you'd have to adopt the Java code for your problem... $\endgroup$
    – Albert Retey
    Commented Nov 22, 2013 at 9:57

1 Answer 1

Reset to default
4
$\begingroup$

I do not have a file to test on, but the following code shows the main idea: Use Sow

keepOnlyOrdered = (If[OrderedQ[#], Sow[#]]; #) &; 
Reap[ While[keepOnlyOrdered[ReadList[s, Number, 2]] != EndOfFile,]]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.