Importing several huge files with headers & footers

Question

having here several files in ASCII format

http://ge.tt/91rv0Jf/v/0

I want to import, while removing headers and footers and then join the several lists.

strm = OpenRead[".omf"]
Skip[strm, Record, 38, NullRecords  -> True];
mydata = ReadList[strm, Number, 7500*3, RecordLists -> True]
Close[strm];

This correctly removes headers and footers, so I don't know to combine this to a

alldata = Map[Import[ ], FileNames[]]

like command to import all files in the directory.

data = Import[#, "Table", HeaderLines -> 38, NumData ->  7500*3, 
AllNumeric -> False, TrailerLines -> 2] & /@ FileNames[]

This works for importing all files, but the footers (or trailers?) are not removed, output:

{{ 755205.327 391367.272 126872.268},{ 638270.884 407054.202 408065.135},{ 621838.43 593464.262 -26779.3974},{ 792199.887 301929.715 -144422.249},{ 827047.904 221093.933 -81909.9342},<<7493>>,{ 638270.884 407054.202 408065.135},
{ 755205.327 391367.272 126872.268},{# End: Data Text},{# End: Segment}}

HeaderLines isn't even mentioned in the Math7.0 Doc. In principle I only need the 3 column in every single .omf file

column3 = mydata[[All, 3]];

and those columns from every single file should than be arranged to a columnlenght X filenumber matrix list. Is it possible to apply this already when doing Import or OpenRead to save memory as this is just a example and the real files will be in the 2 digit GB range? Thanks for your tips

Answer 1

You can wrap your Skip and Readlist in a Module and make it a function:

importfile[name_] := Module[{strm, mydata},
  strm = OpenRead[name];
  Skip[strm, Record, 38, NullRecords -> True];
  mydata = ReadList[strm, Number, 7500*3, RecordLists -> True];
  Close[strm];
  mydata
]

importfile@"f1-Oxs_TimeDriver-Magnetization-000000-0000028.omf"

This will also likely be faster then import.

Alternatively, if you want to use Import, or in the case that your files are different lengths, you can import and then use Drop to remove the last 2 elements.

data = Drop[
    Import[#, "Table", HeaderLines -> 38, NumData -> 7500*3, 
     AllNumeric -> False, TrailerLines -> 2], -2] & /@ FileNames[]

If you want only the third column of very large files see: Reading periodic elements from a large file

Answer 2

A simplistic,if slightly inefficient, solution might be:

data = Import[#,"Table"][[39;;-3]]&/@Filenames[];

Other options might involve, 'Cases', 'Choices', Drop, Take.

For example:

data = Drop[Drop[Import[#,"Table"],38],-2]&/@Filenames[];

If you are using a Linux derivative or something like cygwin on Windows, then this is an efficient solution for reading only the third field from your files ( assumes whitespace separated data, other delimiters can be specified ).

data= ReadList["!head --lines=-2 " <>#<> " | tail --lines=+39 | cut -f 3 "]& /@ FileNames[]