# How to write a function to remove comments from a .m source file preserving formatting such as line wrapping reasonably?

How to write a function to remove comments from a .m source file preserving formatting such as line wrapping reasonably?

This means if I have

foo;
(* comment 1 *)
bar[
baz (* comment 2 *)
(* comment 3 *)
];


I'd ideally end up getting

foo;
bar[
baz
];

-
Use another tool.There are Legion better suited to the job. en.wikipedia.org/wiki/AWK –  belisarius Jan 12 '13 at 8:01

Here is an alternative "first principle" approach, which does not use string patterns as a main tool, but instead makes use of the fact that comments have a simple structure and can only be escaped when they appear inside strings. Therefore, we can write a very simple parser which only parses strings and comments. Here is the tokenizer:

ClearAll[expr, parse, string, comments,tokenize, $commentPattern,$stringPattern];

tokenize[s_String] := StringSplit[s, t : "\\\"" | "\"" | "(*" | "*)" :> t];


Here are auxiliary patterns we will need:

$stringPattern = PatternSequence["\"", middle___, "\""] /; ! MemberQ[{middle}, "\""];$commentPattern =
PatternSequence["(*", middle___, "*)"] /;
Count[{middle}, "(*"] == Count[{middle}, "*)"];


Here is the parser:

parse[left___, s : $stringPattern, right___] := expr[parse[left], string[s], parse[right]]; parse[left___, c :$commentPattern, right___] :=

parse[{tokens___}] := parse[tokens];
parse[tokens___] := expr[tokens];


The heads expr, string and comments are inert heads.

Finally, here is the function to remove comments from a string of code:

ClearAll[removeComments];
StringJoin[
expr | string -> Sequence
]


Applying this to the initial string of code str as removeComments[str] returns the expected answer.

This parser won't be particularly fast. The reason I like this approach is that it does not rely on some external things such as specific forms of Import, so it will only be wrong if the principles are wrong (e.g. I missed some other forms where comments can be escaped, etc), but I consider string manipulations rather fragile for parsing purposes generally. Interestingly, this seems to be one of the simplest problems I know of which illustrates that regexps are not sufficient to parse code representing recursive (nested) expressions / statements.

-
Parsing sure is better than pattern matching. Is it possible to handle escaped citation marks, for instance: "a=\"string1\\\"(* Not a comment *)\";\n(* Comment *)\nb=\"string2\";" –  ssch Jan 12 '13 at 12:28
@ssch Yes, sure. I only had to slightly modify the tokenizer, see in the code above. –  Leonid Shifrin Jan 12 '13 at 12:35

Below is the cell expression corresponding to your example cell. It should not be too hard to find the expression programmatically, for example by giving it a cell tag and using Cells and NotebookRead. Anyway, let's work with this for the moment.

cell = Cell[BoxData[{
RowBox[{
RowBox[{"foo", ";"}], "\[IndentingNewLine]",
RowBox[{"(*",
RowBox[{"comment", " ", "1"}], "*)"}]}], "\n",
RowBox[{
RowBox[{"bar", "[", "baz", " ",
RowBox[{"(*",
RowBox[{"comment", " ", "2"}], "*)"}],
"\[IndentingNewLine]",
RowBox[{"(*",
RowBox[{"comment", " ", "3"}], "*)"}], "]"}], ";"}]}],
"Input"];


We can then simply do

DeleteCases[cell, RowBox[{"(*", ___, "*)"}], Infinity] // CellPrint


giving

(*the lines below are the contents of the cell printed by the code above*)

foo;

bar[baz
];


I hope to play around with Leonids parser a great deal, I find such things extremely interesting. But I thought this would be a refreshing alternative.

-
Sorry, this answer is really not about .m files, unless you of course generate those .m files from notebooks. Anyway, as I know of no Q&A really dealing with preserving formatting and deleting comments or something similar I will keep it. –  Jacob Akkerboom Jun 2 '13 at 23:43
Related Q&A. –  Jacob Akkerboom Jul 11 '13 at 22:44

As observed by ssh, the simple string replacements will not work if comment delimiters are contained as parts of string arguments in your source. For example, this could be a file you're looking at:

txt = Import["t.m", "Text"]


with the content:

foo["(* Don't *) erase!
"];
(* comment 1 *)
bar[
baz (* comment 2 *)
(* comment 3 *)
];


Then a very simple way to get rid of the comments is this:

ToExpression[txt, StandardForm, HoldForm]


foo[(* Don't *) erase!

];bar[baz];

Granted, this doesn't preserve the line breaks and indentation, but it's quick and foolproof (or fooproof ?).

Edit

Here is a slightly improved version that preserves line breaks:

StringJoin[Riffle[DeleteCases[
StringSplit[
ToString[
With[{a = StringReplace[txt, "\n" :> "$newlinecharacter;"]}, ToExpression[a, StandardForm, HoldForm] ]], "$newlinecharacter;"], " "], "\n"]]


Edit 2

The outputs above look as if they don't have the quotation marks of the string - but they are there, you just have to do

Export["t2.m", %]


on the result. Or you can copy the output above and paste it into an input cell. The " will show up then.

-
ToString is being a bit of an annoyance breaking ": ToString[HoldForm[s="1+1"]] // FullForm gives "s = 1 + 1" –  ssch Jan 12 '13 at 2:08
@ssh It does look ugly, but in my edit 2 I mention that sometimes the export is OK (not for your example unfortunately). Anyway, my solution doesn't preserve the format completely, so you're right that I need an alternative to ToString... –  Jens Jan 12 '13 at 2:36
@ssch try: ToString[HoldForm[s = "1+1"], InputForm] // FullForm –  Mr.Wizard Jan 12 '13 at 13:41

Update Combining mine and Rolfs answer gives a pretty robust way:

str="foo;
(* bar *)
s=\"(*\"
bar[
baz (* comment 2 *)
(* comment 3
(* With nested (* comment 4 *) *)
*)
];"; (* Import["t.m","Text"]; *)

(* Change to: Import["t.m",... *)
rule = Flatten[{
StartOfLine ~~ " " ... ~~ "(*" ~~ Whitespace ... ~~ # ~~
Whitespace ... ~~ "*)\n" -> "",
"(*" ~~ Whitespace ... ~~ # ~~ Whitespace ... ~~ "*)" -> ""
StringReplace[str, rule]

(* out:
foo;
s="(*"
bar[
baz
];
*)


It extracts the specific comments and looks for a string like "(* That comment *)" Ignoring any whitespace between (* and the comment since that is stripped by Import["t.m", {"Package", "Comments"}].

end of update

I look for comment-only lines separately to get rid of the extra spacing and newline. Other than that it's just simple string patterns.

Caution: Since this is just simple pattern matching there's bound to be some edge cases where it doesn't work. For instance if the .m has s="(*" somewhere this will break for sure

str = "foo;
(* comment 1 *)
bar[
baz (* comment 2 *)
(* comment 3 *)
];";
StringReplace[str,
{
StartOfLine ~~ " " ... ~~ "(*" ~~ Shortest[___] ~~ "*)\n" -> "",
"(*" ~~ Shortest[___] ~~ "*)" -> ""
}]
(* out:
"foo;
bar[
baz
];"
*)

-
+1 nice way to look for the comment patterns - that should make it easier to rule out the last obstacle: it still fails when you have s=\"(* comment 2 *)\" in the code where the comment shouldn't be removed because it's in a string... –  Jens Jan 12 '13 at 3:42
@Jens Good catch, I can't figure out any sane way to resolve that within the limitations of pattern matching :) –  ssch Jan 12 '13 at 16:31

Basically you can do this with string patterns. So t.m is

foo;
(* comment 1 *)
bar[
baz (* comment 2 *)
(* comment 3 *)
];


Then definining a file fix2 with

Export["t2.m",
StringReplace[
StringReplace[Import["t.m","Text"] ,
"(*"~~ Whitespace ~~"*)" -> ""],
"Text"]

FilePrint@"t2.m"


we get

In[1]:= <<fix2

foo;

bar[
baz

];

-
If add "*)"~~("\n"|"")~~s2__ it will also remove the newline as OP did in his example. –  jVincent Jan 11 '13 at 23:56
@jVincent you only want to remove the newline if the line is just a comment, perhaps: { StartOfLine ~~ " " ... ~~ "(*" ~~ Shortest[___] ~~ "*)\n" -> "", "(*" ~~ Shortest[___] ~~ "*)" -> "" } –  ssch Jan 12 '13 at 0:05
What does !!t.m mean and do? –  Problemania Jan 12 '13 at 0:11
!! is like FilePrint, on Linux. –  Rolf Mertig Jan 12 '13 at 0:24
@ssch Nice, but it soon gets hard to read. I think using "Comments" of Import is simpler. But of course, your solution is also fine. –  Rolf Mertig Jan 12 '13 at 0:26