5
$\begingroup$

I want to do some large data analysis, and i want to be able to import files in a fast way. I tried Import, but it is quite slow; ReadList takes half the time. This is the code I was using:

files = FileNames["Path\\*.dat"];
a = Table[Delete[Import[file, "Table", "HeaderLines" -> 38], {{-1}, {-2}}], {file,files}];

They are files with 38 header lines and 2 foot lines which i dont want. My question is how to import the same files using ReadList and ignoring those unwanted lines, or another approach to improve import performance.

$\endgroup$
  • $\begingroup$ Are you able to answer this question here? I am not sure whether it is related: I have header with four lines where unique identifier is specified by two lines. $\endgroup$ – hhh Nov 11 '14 at 3:16
6
$\begingroup$

This is a common problem. The header lines are straightforward, but the footer lines require postprocessing:

strm = OpenRead["filename"];
(* Header *)
Skip[strm, String, 38];
data = ReadList[strm][[;;-3]]; (* <- Part does post-process removal of footer*)
Close[strm];

But, if you are going to be doing this with large files, I highly recommend you do something to protect against Abort potentially leaving a stream open. This can be done by using BlockStream from a previous answer of mine, which results in shorter code:

readMyFile[filename_, header_:0, footer_:0] :=
BlockStream[{strm, filename},
  Skip[strm, String, header];
  ReadList[strm][[ ;; -(footer + 1) ]]
]

Edit: my initial code used

Read[strm, ConstantArray[String, header]]

but it was pointed out that Skip does this much better.

$\endgroup$
  • $\begingroup$ You can also Skip rather than Read the data you are going to ignore. $\endgroup$ – chuy Mar 18 '14 at 15:36
  • $\begingroup$ @chuy Skip has been in there since v2, and I've never used it. But, looking it over, yes you are correct it has a much simpler interface than using Read. Changing my answer ... $\endgroup$ – rcollyer Mar 18 '14 at 16:06
  • $\begingroup$ Perhaps you can understand my problem here: I have a header file on four lines and unique identifiers are specified by two lines. I could do string manipulation but there must be some ready tools to process streams in Mathematica like in R, can you help with this Q? Are there some ready tools to process files with headers without errorsome string-editing? $\endgroup$ – hhh Nov 11 '14 at 3:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.