# Read and filter a text file without importing

I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data. I want to read the first and last line of B from each section (to have a list of 2 x million lines at the end). Import doesn't work since my file is too big to be read into memory. Is there a command like scan and filter for this?

-
You may try for example SetStreamPosition[], but ... why don't you just preprocess the file with any capable language for text processing (AWK, perl, sed, whatever) –  belisarius Feb 11 '13 at 14:40
Have a look at this question. There are a number of solutions shown there, which you might be able to adopt to your situation. All of them will however require to write some custom code. –  Leonid Shifrin Feb 11 '13 at 15:05
Possibly a duplicate of this question. The link is certainly relevant to the current question. –  m_goldberg Feb 11 '13 at 16:58
Was my answer useful to you? –  rcollyer Feb 22 '13 at 23:16

You're looking for Skip. This does not check for an EndOfFile condition, and should use something like BlockStream to handle aborts, but the following should work:

strm = OpenRead["filename"];

(* Repeat the following until done *)

Skip[strm, String, A];

(*
String just gets the entire line. If the number of elements is
constant for each row, then it can be replaced in the Read by
ConstantArray[String, numels]~Join~{String}.
*)

Skip[strm, String, B - 2];