6
$\begingroup$

I want to do some large data analysis, and i want to be able to import files in a fast way. I tried Import, but it is quite slow; ReadList takes half the time. This is the code I was using:

files = FileNames["Path\\*.dat"];
a = Table[Delete[Import[file, "Table", "HeaderLines" -> 38], {{-1}, {-2}}], {file,files}];

They are files with 38 header lines and 2 foot lines which i dont want. My question is how to import the same files using ReadList and ignoring those unwanted lines, or another approach to improve import performance.

$\endgroup$
1
  • $\begingroup$ Are you able to answer this question here? I am not sure whether it is related: I have header with four lines where unique identifier is specified by two lines. $\endgroup$
    – hhh
    Nov 11, 2014 at 3:16

1 Answer 1

8
$\begingroup$

This is a common problem. The header lines are straightforward, but the footer lines require postprocessing:

strm = OpenRead["filename"];
(* Header *)
Skip[strm, String, 38];
data = ReadList[strm][[;;-3]]; (* <- Part does post-process removal of footer*)
Close[strm];

But, if you are going to be doing this with large files, I highly recommend you do something to protect against Abort potentially leaving a stream open. This can be done by using BlockStream from a previous answer of mine, which results in shorter code:

readMyFile[filename_, header_:0, footer_:0] :=
BlockStream[{strm, filename},
  Skip[strm, String, header];
  ReadList[strm][[ ;; -(footer + 1) ]]
]

Edit: my initial code used

Read[strm, ConstantArray[String, header]]

but it was pointed out that Skip does this much better.

$\endgroup$
3
  • $\begingroup$ You can also Skip rather than Read the data you are going to ignore. $\endgroup$
    – chuy
    Mar 18, 2014 at 15:36
  • $\begingroup$ @chuy Skip has been in there since v2, and I've never used it. But, looking it over, yes you are correct it has a much simpler interface than using Read. Changing my answer ... $\endgroup$
    – rcollyer
    Mar 18, 2014 at 16:06
  • $\begingroup$ Perhaps you can understand my problem here: I have a header file on four lines and unique identifiers are specified by two lines. I could do string manipulation but there must be some ready tools to process streams in Mathematica like in R, can you help with this Q? Are there some ready tools to process files with headers without errorsome string-editing? $\endgroup$
    – hhh
    Nov 11, 2014 at 3:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.