ReadList with header and foot lines

Question

I want to do some large data analysis, and i want to be able to import files in a fast way. I tried Import, but it is quite slow; ReadList takes half the time. This is the code I was using:

files = FileNames["Path\\*.dat"];
a = Table[Delete[Import[file, "Table", "HeaderLines" -> 38], {{-1}, {-2}}], {file,files}];

They are files with 38 header lines and 2 foot lines which i dont want. My question is how to import the same files using ReadList and ignoring those unwanted lines, or another approach to improve import performance.

Answer 1

This is a common problem. The header lines are straightforward, but the footer lines require postprocessing:

strm = OpenRead["filename"];
(* Header *)
Skip[strm, String, 38];
data = ReadList[strm][[;;-3]]; (* <- Part does post-process removal of footer*)
Close[strm];

But, if you are going to be doing this with large files, I highly recommend you do something to protect against Abort potentially leaving a stream open. This can be done by using BlockStream from a previous answer of mine, which results in shorter code:

readMyFile[filename_, header_:0, footer_:0] :=
BlockStream[{strm, filename},
  Skip[strm, String, header];
  ReadList[strm][[ ;; -(footer + 1) ]]
]

---

Edit: my initial code used

Read[strm, ConstantArray[String, header]]

but it was pointed out that Skip does this much better.