Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data. I want to read the first and last line of B from each section (to have a list of 2 x million lines at the end). Import doesn't work since my file is too big to be read into memory. Is there a command like scan and filter for this?

share|improve this question
1  
You may try for example SetStreamPosition[], but ... why don't you just preprocess the file with any capable language for text processing (AWK, perl, sed, whatever) –  belisarius Feb 11 '13 at 14:40
1  
Have a look at this question. There are a number of solutions shown there, which you might be able to adopt to your situation. All of them will however require to write some custom code. –  Leonid Shifrin Feb 11 '13 at 15:05
    
Possibly a duplicate of this question. The link is certainly relevant to the current question. –  m_goldberg Feb 11 '13 at 16:58
    
Was my answer useful to you? –  rcollyer Feb 22 '13 at 23:16

1 Answer 1

You're looking for Skip. This does not check for an EndOfFile condition, and should use something like BlockStream to handle aborts, but the following should work:

strm = OpenRead["filename"];

(* Repeat the following until done *)

Skip[strm, String, A];    
AppendTo[ results, Read[strm, String]]

(* 
  String just gets the entire line. If the number of elements is 
  constant for each row, then it can be replaced in the Read by   
  ConstantArray[String, numels]~Join~{String}. 
*)

Skip[strm, String, B - 2];
AppendTo[ results, Read[strm, String]]

(* close the stream *)
Close[ strm ];
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.