# Removing commented lines of text in a string

Suppose that I have a string str. str contains headings -- which are written in commented lines -- and data. For example:

str = "# Heading 1
Data 1
Data 2
Data 3";


where # and @ denote commented lines.

Now, I would like to write a function removeComments to remove all lines in str that are commented. That is, I would like to remove all lines in str that start with the character # or the character @.

I have come up with one way to do this, I think. My procedure is to first split str into lines using StringSplit with the "\n" delimiter; then use DeleteCases to delete all lines in that list which begin with either # or @; and, finally, convert the list back to a string using Riffle and StringJoin. So, I have the following:

removeComments[str_String] := Module[{result},
result = StringSplit[str, "\n"];
result =
DeleteCases[
result, _?(StringMatchQ[#, ("#" ~~ ___) | ("@" ~~ ___)] &)];
result = StringJoin[Riffle[result, "\n"]];
result
]


which gives the following correct output (a string):

Data 1
Data 2
Data 3


However, now suppose that one of my data items (for example, a single space in front of Data 2) has one or more leading spaces:

str = "# Heading 1
Data 1
Data 2
Data 3";



In this case, I get the incorrect output:

Data 1
Data 3


That is, Data 2 is omitted altogether. Something is clearly wrong with my approach, but I haven't been able to find my mistake (although it is probably obvious). Could you please help me find what is wrong? Thanks for your time.

-
Do you wish to keep the leading space before "Data 2" or do you wish to discard it? – Mr.Wizard Aug 30 '13 at 21:50
@Mr.Wizard Sorry; I was not clear. I would like to keep the leading space before Data 2. – Andrew Aug 30 '13 at 21:51
@Mr.Wizard No problem! :-) Thank you for your time and help. – Andrew Aug 30 '13 at 22:31
Andrew, I notice that you didn't Accept my answer. Is that an oversight or do you find it lacking? Can I improve it? – Mr.Wizard Sep 4 '13 at 11:21
@Mr.Wizard Sorry! It was just an oversight on my part; I have used your help but forgot to Accept. Your answer has been extremely helpful to me! Thank you! – Andrew Sep 4 '13 at 13:07

My original answer (see edit history) was not correct, unless all of your data lines are contiguous.

str = "# Heading 1\n# Heading 2\n@ Heading 3\n@ Heading 4\nData 1\n Data 2\nData 3\n@ \

StringReplace[str, Shortest[StartOfLine ~~ "@" | "#" ~~ ___ ~~ "\n"] :> ""]

"Data 1\n Data 2\nData 3\nData 4"


Or:

"" <> StringSplit[str, Shortest[StartOfLine ~~ "@" | "#" ~~ ___ ~~ "\n"]]

"Data 1\n Data 2\nData 3\nData 4"


And the RE equivalent:

First @ StringPatternPatternConvert[Shortest[StartOfLine ~~ "@" | "#" ~~ ___ ~~ "\n"]]

"(?ms)^[@#].*?\n"

"" <> StringSplit[str, RegularExpression["(?ms)^[@#].*?\n"]]

"Data 1\n Data 2\nData 3\nData 4"

-
Thanks! Yes, this works as desired, in that all data is retained (including the leading space before Data 2). – Andrew Aug 30 '13 at 21:55
@Andrew Okay, great. I added the RE equivalent if you enjoy such things. – Mr.Wizard Aug 30 '13 at 21:56
Although it hasn't been specified clearly I think it might be more robust to use ___ instead of just __ to also discard empty comment lines, which otherwise well might cause problems when trying to interpret the remaining lines. The RE equivalent would be .* instead of .+ – Albert Retey Sep 1 '13 at 11:38
@Albert Good point; I'll change it. – Mr.Wizard Sep 1 '13 at 11:39

This answer only discusses the implied question of why the OP's code fails. I think this should be made clear because it puzzled me until Mr.Wizard and rm -rf enlightened me. I would like keep others from falling into the same trap.

I was studying the code posted in this question, trying to learn why it failed, when I ran into the following.

StringMatchQ[" ", "@"]


True

I found this very surprising and posted a question on it for a very short period. Mr.Wizard and rm -rf quickly pointed out that @ was a wild card character and that my code should be

StringMatchQ[" ", "\\@"]


False

This same mistake also occurs in the code posted in the question above. With the following minor change, the OP's code works.

removeComments[str_String] :=
Module[{result},
result = StringSplit[str, "\n"];
result =
DeleteCases[result, _?(StringMatchQ[#, ("#" ~~ ___) | ("\\@" ~~ ___)] &)];
StringJoin[Riffle[result, "\n"]]]


$\$Data 2
Incidentally the pattern could be condesed like this: StringMatchQ[#, "#" | "\\@" ~~ ___]` – Mr.Wizard Aug 31 '13 at 1:38