Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I want to do some large data analysis, and i want to be able to import files in a fast way. I tried Import, but it is quite slow; ReadList takes half the time. This is the code I was using:

files = FileNames["Path\\*.dat"];
a = Table[Delete[Import[file, "Table", "HeaderLines" -> 38], {{-1}, {-2}}], {file,files}];

They are files with 38 header lines and 2 foot lines which i dont want. My question is how to import the same files using ReadList and ignoring those unwanted lines, or another approach to improve import performance.

share|improve this question
    
Are you able to answer this question here? I am not sure whether it is related: I have header with four lines where unique identifier is specified by two lines. –  hhh Nov 11 at 3:16

1 Answer 1

up vote 3 down vote accepted

This is a common problem. The header lines are straightforward, but the footer lines require postprocessing:

strm = OpenRead["filename"];
(* Header *)
Skip[strm, String, 38];
data = ReadList[strm][[;;-3]]; (* <- Part does post-process removal of footer*)
Close[strm];

But, if you are going to be doing this with large files, I highly recommend you do something to protect against Abort potentially leaving a stream open. This can be done by using BlockStream from a previous answer of mine, which results in shorter code:

readMyFile[filename_, header_:0, footer_:0] :=
BlockStream[{strm, filename},
  Skip[strm, String, header];
  ReadList[strm][[ ;; -(footer + 1) ]]
]

Edit: my initial code used

Read[strm, ConstantArray[String, header]]

but it was pointed out that Skip does this much better.

share|improve this answer
    
You can also Skip rather than Read the data you are going to ignore. –  chuy Mar 18 at 15:36
    
@chuy Skip has been in there since v2, and I've never used it. But, looking it over, yes you are correct it has a much simpler interface than using Read. Changing my answer ... –  rcollyer Mar 18 at 16:06
    
Perhaps you can understand my problem here: I have a header file on four lines and unique identifiers are specified by two lines. I could do string manipulation but there must be some ready tools to process streams in Mathematica like in R, can you help with this Q? Are there some ready tools to process files with headers without errorsome string-editing? –  hhh Nov 11 at 3:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.