I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data. I want to read the first and last line of B from each section (to have a list of 2 x million lines at the end). Import doesn't work since my file is too big to be read into memory. Is there a command like scan and filter for this?

  • 1
    $\begingroup$ You may try for example SetStreamPosition[], but ... why don't you just preprocess the file with any capable language for text processing (AWK, perl, sed, whatever) $\endgroup$ Feb 11, 2013 at 14:40
  • 1
    $\begingroup$ Have a look at this question. There are a number of solutions shown there, which you might be able to adopt to your situation. All of them will however require to write some custom code. $\endgroup$ Feb 11, 2013 at 15:05
  • $\begingroup$ Possibly a duplicate of this question. The link is certainly relevant to the current question. $\endgroup$
    – m_goldberg
    Feb 11, 2013 at 16:58
  • $\begingroup$ Was my answer useful to you? $\endgroup$
    – rcollyer
    Feb 22, 2013 at 23:16

1 Answer 1


You're looking for Skip. This does not check for an EndOfFile condition, and should use something like BlockStream to handle aborts, but the following should work:

strm = OpenRead["filename"];

(* Repeat the following until done *)

Skip[strm, String, A];    
AppendTo[ results, Read[strm, String]]

  String just gets the entire line. If the number of elements is 
  constant for each row, then it can be replaced in the Read by   
  ConstantArray[String, numels]~Join~{String}. 

Skip[strm, String, B - 2];
AppendTo[ results, Read[strm, String]]

(* close the stream *)
Close[ strm ];

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.