Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

having here several files in ASCII format

http://ge.tt/91rv0Jf/v/0

I want to import, while removing headers and footers and then join the several lists.

strm = OpenRead[".omf"]
Skip[strm, Record, 38, NullRecords  -> True];
mydata = ReadList[strm, Number, 7500*3, RecordLists -> True]
Close[strm];

This correctly removes headers and footers, so I don't know to combine this to a

alldata = Map[Import[ ], FileNames[]]

like command to import all files in the directory.

data = Import[#, "Table", HeaderLines -> 38, NumData ->  7500*3, 
AllNumeric -> False, TrailerLines -> 2] & /@ FileNames[]

This works for importing all files, but the footers (or trailers?) are not removed, output:

{{ 755205.327 391367.272 126872.268},{ 638270.884 407054.202 408065.135},{ 621838.43 593464.262 -26779.3974},{ 792199.887 301929.715 -144422.249},{ 827047.904 221093.933 -81909.9342},<<7493>>,{ 638270.884 407054.202 408065.135},
{ 755205.327 391367.272 126872.268},{# End: Data Text},{# End: Segment}}

HeaderLines isn't even mentioned in the Math7.0 Doc. In principle I only need the 3 column in every single .omf file

column3 = mydata[[All, 3]];

and those columns from every single file should than be arranged to a columnlenght X filenumber matrix list. Is it possible to apply this already when doing Import or OpenRead to save memory as this is just a example and the real files will be in the 2 digit GB range? Thanks for your tips

share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

You can wrap your Skip and Readlist in a Module and make it a function:

importfile[name_] := Module[{strm, mydata},
  strm = OpenRead[name];
  Skip[strm, Record, 38, NullRecords -> True];
  mydata = ReadList[strm, Number, 7500*3, RecordLists -> True];
  Close[strm];
  mydata
]

importfile@"f1-Oxs_TimeDriver-Magnetization-000000-0000028.omf"

This will also likely be faster then import.

Alternatively, if you want to use Import, or in the case that your files are different lengths, you can import and then use Drop to remove the last 2 elements.

data = Drop[
    Import[#, "Table", HeaderLines -> 38, NumData -> 7500*3, 
     AllNumeric -> False, TrailerLines -> 2], -2] & /@ FileNames[]

If you want only the third column of very large files see: Reading periodic elements from a large file

share|improve this answer
add comment

A simplistic,if slightly inefficient, solution might be:

data = Import[#,"Table"][[39;;-3]]&/@Filenames[];

Other options might involve, 'Cases', 'Choices', Drop, Take.

For example:

data = Drop[Drop[Import[#,"Table"],38],-2]&/@Filenames[];

If you are using a Linux derivative or something like cygwin on Windows, then this is an efficient solution for reading only the third field from your files ( assumes whitespace separated data, other delimiters can be specified ).

data= ReadList["!head --lines=-2 " <>#<> " | tail --lines=+39 | cut -f 3 "]& /@ FileNames[]
share|improve this answer
    
Thanks, I posted a follow up question to this mathematica.stackexchange.com/questions/24289/… –  James Last Apr 29 '13 at 12:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.