3
$\begingroup$

I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data. I want to read the first and last line of B from each section (to have a list of 2 x million lines at the end). Import doesn't work since my file is too big to be read into memory. Is there a command like scan and filter for this?

$\endgroup$
  • 1
    $\begingroup$ You may try for example SetStreamPosition[], but ... why don't you just preprocess the file with any capable language for text processing (AWK, perl, sed, whatever) $\endgroup$ – Dr. belisarius Feb 11 '13 at 14:40
  • 1
    $\begingroup$ Have a look at this question. There are a number of solutions shown there, which you might be able to adopt to your situation. All of them will however require to write some custom code. $\endgroup$ – Leonid Shifrin Feb 11 '13 at 15:05
  • $\begingroup$ Possibly a duplicate of this question. The link is certainly relevant to the current question. $\endgroup$ – m_goldberg Feb 11 '13 at 16:58
  • $\begingroup$ Was my answer useful to you? $\endgroup$ – rcollyer Feb 22 '13 at 23:16
8
$\begingroup$

You're looking for Skip. This does not check for an EndOfFile condition, and should use something like BlockStream to handle aborts, but the following should work:

strm = OpenRead["filename"];

(* Repeat the following until done *)

Skip[strm, String, A];    
AppendTo[ results, Read[strm, String]]

(* 
  String just gets the entire line. If the number of elements is 
  constant for each row, then it can be replaced in the Read by   
  ConstantArray[String, numels]~Join~{String}. 
*)

Skip[strm, String, B - 2];
AppendTo[ results, Read[strm, String]]

(* close the stream *)
Close[ strm ];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.