Is there a way to make ReadList
ignore certain "comment lines" (say, all lines that start with #
, or in general all lines that match a string pattern)? This would be very useful for many types of files.
1 Answer
george2079's answer works great for ignoring comment lines, but it uses Import
to read in the whole file at once. Sometimes you have really large files that you can't Import
in one go, you have to read them line by line. Of course, for files where you can use Import, that will be faster because you are applying the test to the one big string instead of to each line one after the other.
So should it ever be needed, here is this slower method,
readTableIgnoreComments[fname_, comment_] :=
Module[{stream, input, list},
stream = OpenRead[fname];
input = ReadLine[stream];
list = Reap[
While[input =!= EndOfFile,
input = StringReplace[input, StartOfLine ~~ "#" ~~ __ -> ""];
If[input =!= "",
Sow[ImportString[input, "Table"][[1]]];
];
input = ReadLine[stream];
];
][[2, 1]];
Close[stream];
list];
Say you have a data file, like this one.
readTableIgnoreComments["AFmW2BKQ.txt", "#"]
(*{{13.2177,6.30967,5.20213,5.93021},
{0.350016,13.4168,4.70314,6.16283},{3.61517,2.2622,13.4662,8.27606},
{9.82748,9.46039,0.894798,2.29597},{11.3524,5.98368,1.55609,2.20832},
{9.59792,3.70512,13.2309,12.156},{3.11005,11.4154,3.06589,11.9867},
{8.46262,9.44805,3.81819,12.8846},{13.7071,3.44253,4.13362,1.14301},
{0.353064,11.238,7.46061,7.04745},{5.24699,10.2069,4.79834,7.86099},
{7.90967,1.95548,6.44391,6.85132},{12.2419,7.94127,12.8604,10.4504},
{5.58408,6.49862,10.0892,10.2229},{5.5434,4.1264,13.0629,12.3711},
{5.35179,10.4674,1.39775,10.0056},{5.90251,12.3466,1.8162,5.9312},
{10.0368,3.42365,13.5114,10.938},{9.3393,4.55733,7.98305,1.01929},
{9.68279,9.00243,6.19094,0.482091},{8.86858,5.26325,2.35884,7.05454},
{4.37432,8.51505,3.90883,0.380504},{4.86367,9.19055,3.04116,10.9041},
{13.1294,7.39576,8.72494,1.72672},{12.5197,7.75693,9.2014,6.95952},
{8.95084,5.61415,12.6574,9.6697},{0.707468,3.96087,1.08438,10.7936},
{13.5005,3.14536,4.87679,6.00281},{6.74514,11.6024,2.23439,4.34998},
{7.34057,5.99825,1.02762,3.7478},{4.14484,6.2788,13.2522,4.1299},
{1.467,8.05903,8.09584,7.55979},{1.68094,2.35345,6.66405,4.4644},
{2.51467,8.88769,5.72158,6.80248},{13.6002,3.71197,2.81909,1.05188}}*)
As george2079 and Szabolcs pointed out, if you are lucky enough to be running an operating system with sed
, then by far the fastest way to do this is to use sed
to replace the commented lines,
readListIgnoreComments[fname_, comment_] := Module[{list},
Run["sed -e '/^" <> comment <> ".*/d' " <> fname <> " > TEMP_" <>
fname];
list = ReadList["TEMP_" <> fname];
DeleteFile["TEMP_" <> fname];
list
]
ReadList
, reading in the whole file at once viaReadList["file"]
or reading in a certain number of lines each time viaReadList["file",types,n]
? $\endgroup$ReadList["file"]
, but maybeReadList["file",types,n]
is more convenient to ignore comments? $\endgroup$ReadList["file"]
, then can't you simply remove the elements that match the pattern? $\endgroup$ReadList
supports this. Two ideas: 1. read the whole file as a string, filter the comments, convert the string to a stream, read from there 2. usesed
to strip the comments, pipe the output to Mathematica (i.e.ReadList["!sed ..."]
where you'll need to look up the correct arguments tosed
). $\endgroup$