5
$\begingroup$

Is there a way to make ReadList ignore certain "comment lines" (say, all lines that start with #, or in general all lines that match a string pattern)? This would be very useful for many types of files.

$\endgroup$
9
  • $\begingroup$ How are you using ReadList, reading in the whole file at once via ReadList["file"] or reading in a certain number of lines each time via ReadList["file",types,n]? $\endgroup$
    – Jason B.
    Commented Jan 25, 2016 at 13:14
  • $\begingroup$ @JasonB Up to now I've done ReadList["file"], but maybe ReadList["file",types,n] is more convenient to ignore comments? $\endgroup$
    – a06e
    Commented Jan 25, 2016 at 13:16
  • $\begingroup$ So if you've read in the whole file using ReadList["file"], then can't you simply remove the elements that match the pattern? $\endgroup$
    – Jason B.
    Commented Jan 25, 2016 at 13:18
  • $\begingroup$ I don't think ReadList supports this. Two ideas: 1. read the whole file as a string, filter the comments, convert the string to a stream, read from there 2. use sed to strip the comments, pipe the output to Mathematica (i.e. ReadList["!sed ..."] where you'll need to look up the correct arguments to sed). $\endgroup$
    – Szabolcs
    Commented Jan 25, 2016 at 14:04
  • $\begingroup$ possible dup : mathematica.stackexchange.com/q/50718/2079 $\endgroup$
    – george2079
    Commented Jan 25, 2016 at 14:25

1 Answer 1

5
$\begingroup$

george2079's answer works great for ignoring comment lines, but it uses Import to read in the whole file at once. Sometimes you have really large files that you can't Import in one go, you have to read them line by line. Of course, for files where you can use Import, that will be faster because you are applying the test to the one big string instead of to each line one after the other.

So should it ever be needed, here is this slower method,

readTableIgnoreComments[fname_, comment_] := 
  Module[{stream, input, list},
   stream = OpenRead[fname];
   input = ReadLine[stream];
   list = Reap[
      While[input =!= EndOfFile,
        input = StringReplace[input, StartOfLine ~~ "#" ~~ __ -> ""];

        If[input =!= "",
         Sow[ImportString[input, "Table"][[1]]];
         ];
        input = ReadLine[stream];
        ];
      ][[2, 1]];
   Close[stream];
   list];

Say you have a data file, like this one.

readTableIgnoreComments["AFmW2BKQ.txt", "#"]

(*{{13.2177,6.30967,5.20213,5.93021},
{0.350016,13.4168,4.70314,6.16283},{3.61517,2.2622,13.4662,8.27606},
{9.82748,9.46039,0.894798,2.29597},{11.3524,5.98368,1.55609,2.20832},
{9.59792,3.70512,13.2309,12.156},{3.11005,11.4154,3.06589,11.9867},
{8.46262,9.44805,3.81819,12.8846},{13.7071,3.44253,4.13362,1.14301},
{0.353064,11.238,7.46061,7.04745},{5.24699,10.2069,4.79834,7.86099},
{7.90967,1.95548,6.44391,6.85132},{12.2419,7.94127,12.8604,10.4504},
{5.58408,6.49862,10.0892,10.2229},{5.5434,4.1264,13.0629,12.3711},
{5.35179,10.4674,1.39775,10.0056},{5.90251,12.3466,1.8162,5.9312},
{10.0368,3.42365,13.5114,10.938},{9.3393,4.55733,7.98305,1.01929},
{9.68279,9.00243,6.19094,0.482091},{8.86858,5.26325,2.35884,7.05454},
{4.37432,8.51505,3.90883,0.380504},{4.86367,9.19055,3.04116,10.9041},
{13.1294,7.39576,8.72494,1.72672},{12.5197,7.75693,9.2014,6.95952},
{8.95084,5.61415,12.6574,9.6697},{0.707468,3.96087,1.08438,10.7936},
{13.5005,3.14536,4.87679,6.00281},{6.74514,11.6024,2.23439,4.34998},
{7.34057,5.99825,1.02762,3.7478},{4.14484,6.2788,13.2522,4.1299},
{1.467,8.05903,8.09584,7.55979},{1.68094,2.35345,6.66405,4.4644},
{2.51467,8.88769,5.72158,6.80248},{13.6002,3.71197,2.81909,1.05188}}*)

As george2079 and Szabolcs pointed out, if you are lucky enough to be running an operating system with sed, then by far the fastest way to do this is to use sed to replace the commented lines,

readListIgnoreComments[fname_, comment_] := Module[{list},
  Run["sed -e '/^" <> comment <> ".*/d' " <> fname <> " > TEMP_" <> 
    fname];
  list = ReadList["TEMP_" <> fname];
  DeleteFile["TEMP_" <> fname];
  list
  ]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.