Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I want to do some large data analysis, and i want to be able to import files in a fast way. I tried Import, but it is quite slow; ReadList takes half the time. This is the code I was using:

files = FileNames["Path\\*.dat"];
a = Table[Delete[Import[file, "Table", "HeaderLines" -> 38], {{-1}, {-2}}], {file,files}];

They are files with 38 header lines and 2 foot lines which i dont want. My question is how to import the same files using ReadList and ignoring those unwanted lines, or another approach to improve import performance.

share|improve this question
add comment

1 Answer 1

up vote 3 down vote accepted

This is a common problem. The header lines are straightforward, but the footer lines require postprocessing:

strm = OpenRead["filename"];
(* Header *)
Skip[strm, String, 38];
data = ReadList[strm][[;;-3]]; (* <- Part does post-process removal of footer*)
Close[strm];

But, if you are going to be doing this with large files, I highly recommend you do something to protect against Abort potentially leaving a stream open. This can be done by using BlockStream from a previous answer of mine, which results in shorter code:

readMyFile[filename_, header_:0, footer_:0] :=
BlockStream[{strm, filename},
  Skip[strm, String, header];
  ReadList[strm][[ ;; -(footer + 1) ]]
]

Edit: my initial code used

Read[strm, ConstantArray[String, header]]

but it was pointed out that Skip does this much better.

share|improve this answer
    
You can also Skip rather than Read the data you are going to ignore. –  chuy Mar 18 at 15:36
    
@chuy Skip has been in there since v2, and I've never used it. But, looking it over, yes you are correct it has a much simpler interface than using Read. Changing my answer ... –  rcollyer Mar 18 at 16:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.