Read and filter a text file without importing

Question

I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data. I want to read the first and last line of B from each section (to have a list of 2 x million lines at the end). Import doesn't work since my file is too big to be read into memory. Is there a command like scan and filter for this?

Answer 1

You're looking for Skip. This does not check for an EndOfFile condition, and should use something like BlockStream to handle aborts, but the following should work:

strm = OpenRead["filename"];

(* Repeat the following until done *)

Skip[strm, String, A];    
AppendTo[ results, Read[strm, String]]

(* 
  String just gets the entire line. If the number of elements is 
  constant for each row, then it can be replaced in the Read by   
  ConstantArray[String, numels]~Join~{String}. 
*)

Skip[strm, String, B - 2];
AppendTo[ results, Read[strm, String]]

(* close the stream *)
Close[ strm ];