# How to find pieces of code responsible for syntax errors?

Especially: How to find the piece of code responsible for a missing parenthesis problem caused by a misplaced SuperScriptBox

For last 30 minutes (and I was lucky today, sometimes I spend 5 times as much time) I have been removing code little by little from my large one cell, (have to be one cell, it is Manipulate demo stylesheet) in order to find where the syntax error is (since Mathematica, with all its might and power, can't tell the user which line number the syntax error is located at). Each time I see the red line show up on the right side of the notebook, indicating a syntax error, I know that I will now waste another hour at least looking for the syntax error.

Why is it that hard for Mathematica to tell the user where the syntax error is?

I zoomed down to this little 5 character code, and I copy it to new notebook to examine it:

So, there is a hidden bad character there. Sometimes, with my big fingers and my small keyboard (can't find big keyboards any more, everything is so tiny), I must have hit wrong key combination, which must have injected some bad character to the notebook, and this happens.

I will keep this cell on its own in order to find what the problem is. I tried to select the cell, and do convert to input form, but nothing happened. no output came of it.

Then I did cell->ShowExpression, and this came up

Cell[BoxData[
RowBox[{"(",
RowBox[{"k", "h"}],
SuperscriptBox[")", "2"]}]], "Input",
CellChangeTimes->{3.536893041212217*^9},
EmphasizeSyntaxErrors->True]


So, what does all this mean? Do you see from the above what is the problem?

I think I entered the superscript as I always do, using Ctrl+^ then 2.

I think the notebook interface is powerful and allows many things not possible otherwise, but because of problems such the above, I wish sometimes I am using plain text editor like emacs, where I can see what I have and not worry about hidden things I do not see, but I tried that once, and it is not practical overall setup to work with.

Btw, originally, I had lots of code like the above inside strings, i.e. inside " " for formatting, and now I am removing all the strings to do some other way of formatting. I do not know if this has anything to do with it. But as you can see, now there is no string around the expression. Here is a screen shot of the earlier version of the notebook, from the same location of the code, showing how the above was inside a string, and I simply was removing the outside string quotes, that is all:

-
Instead of using strings for displaying equations - it is probably best to use HoldForm. –  Simon Jan 30 '12 at 7:04
Btw - there might be a problem in the ToExpression code that your example reveals. ToExpression["(h k)^2"] works fine, but the version with a superscript, $\text{ToExpression["}(h\ k)^2\text{"]}$, produces an error. For reasons very similar to the example that you gave: The string with a superscript is interpretted using the string representation of boxes "(k h\!$$\*SuperscriptBox[\()$$, $$2$$]\)" –  Simon Jan 30 '12 at 7:11
Out of curiosity, why do you think all your code has to be in a single cell for demonstrations? (I've split out utility functions into separate cells for demonstrations I've written in the past.) –  Brett Champion Jan 31 '12 at 5:07
You can have multiple initialization cells. See for example LinesThroughPointsInThePoincareDisk –  Brett Champion Jan 31 '12 at 14:59
I'm not certain how this structure got created, but the FE definitely should be highlighting the mismatched delimiters here. And even legitimate structures which you could easily type (e.g., in a new cell type a right paren followed by a superscript) don't show mismatched delimiter highlighting. I consider that a bug which I just fixed for the next release of Mathematica. –  John Fultz Jul 21 '13 at 5:40

There was actually an old post on mathgroup, Strange Syntax problem, that looks like it was exactly the same problem you're experiencing. A power/superscript being associated with the closing parenthesis instead of the whole parenthesized expression (see the box forms presented in my original answer below for more details). Unfortunately, this problem was not resolved in the the mathgroup post.

The best I can suggest at the moment is to simply press the + button
to get the expanded form

This should help narrow down the problem. Note the Syntax::bktmcp warning is talking about non-matching brackets, yet in the visible expression, all brackets are properly matched. So, the problem lies in the underlying box expression.

Alternatively, if it is this particular problem, you could Show Expression, then search for the problematic code snippet SuperscriptBox[")". Then you can modify the box structure directly. This is normally easiest if you past it into a new input cell so that the Extend Selection mechanism works.

There is no "hidden bad character" in the cell that you posted. Rather, the box structure is crazy and can't be interpreted by Mathematica - I have trouble imagining how you managed to create it...

Cell[BoxData[
RowBox[{"(",
RowBox[{"k", "h"}],
SuperscriptBox[")", "2"]}]], "Input"]


and here is how it should look

Cell[BoxData[
SuperscriptBox[
RowBox[{"(",
RowBox[{"k", " ", "h"}], ")"}], "2"]], "Input"]


So, your question does not really a work with the example you provided.

However, a simple way to maybe clean your code (at least in this case) is to copy it as plain text or input text, both of which produce (k h)^2 when pasted.

-
@Nasser: As I said in my answer, there are no "bad characters" in your code - just bad box structures. –  Simon Jan 30 '12 at 7:48
And as I've said to you before, you should divide up you code and move things into initialization cells (as you are allowed to do in demonstrations). 10,000 lines of code is just getting ridiculous! This really would make your life easier, since the line breaking + indenting algorithm of cells becomes slow as they get large. Another option (that you already discarded, but I can't remember why) is to use "Code" cells or to (temporarily) turn off AutoIndent. –  Simon Jan 30 '12 at 7:50
@Nasser: What it is called is important to this site. The primary beneficiary of questions on this site is neither the questioner nor the answerer(s), but rather the many people who have the same problem and will find these discussions via google. –  Simon Jan 30 '12 at 8:07
As for moving code to the initialization cells, here's a demonstration where the first argument of Manipulate is just makePicture[...]. Nice and simple. I know that what you're doing is more complicated and that you've written many demonstrations, but I have trouble believing that you can't make a reasonable separation of your code. –  Simon Jan 30 '12 at 8:10
@Simon I've faced many similar problems with closing parantheses, and would like to propose the addition of the keyword Syntax::bktmcp in your answer (this does not appear as searchable text in your answer, even though there is an image). Would you be ok with that? –  Vincent Tjeng Jul 16 '13 at 2:40

It happens to me quite often while working with huge notebooks. It is quite irritating because not always it is obvious what have caused an error and where to find it.

This is part of bigger code. As You see an error is transferring with Crtl+C -> Crtl+V. Usually rewritting problematic parts (if You've found them) helps but it is not convenient at all.

## Edit

I have succesfully tested this method on notebook that contains ~40000 characters. It preserves notebook formatting etc.

The notebook You are trying to use it on cannot contain those orange-marked cells. Just copy the content and -> delete marked cell -> paste clipboard and save the notebook. Then it is ready for this procedure.

I'm going to show an approach to solve this issue using important remark of Simon:

Cell[BoxData[
RowBox[{"(",
RowBox[{"k", "h"}],
SuperscriptBox[")", "2"]}]], "Input"]


and here is how it should look

Cell[BoxData[
SuperscriptBox[
RowBox[{"(",
RowBox[{"k", " ", "h"}], ")"}], "2"]], "Input"]


However, it is not so simple to resctructure this cell, Mathematica refuses to show even BoxData of this cell, unless You Shift+Crtl+E. I'm going to use brute force of strings.

SetDirectory@NotebookDirectory[];
test = Import["beforeTest.nb", "Text"];
tt = StringReplace[test,
"RowBox[" ~~ x : RegularExpression["((?!RowBox).)*"] ~~ "RowBox[" ~~
y : RegularExpression["((?!RowBox).)*"] ~~
"SuperscriptBox[\")\"," ~~ z : Except["]"] .. ~~ "]}]"
:>
"SuperscriptBox[RowBox[{\"(\", RowBox[" ~~ y ~~ " \")\"}]," ~~
z ~~"]"
];

Export["test.txt", tt];
RenameFile["test.txt", "test.nb"];


Well, not very sophisticated but it works:

It needs testing of course:

• I do not know if the problem of crazy structured cell appears in other context than this with Superscript because I am no able to creat such on demand. However, if my memory serves me, there wasn't any other case.

• I'm not sure if this method will preserve notebook formatting created earlier.

• I'm not sure if StringPattern I've created is universal and bulletproof enough.

I appreciate any help with improving this pattern and method in general and with testing this as it seems to be a critical problem sometimes.

-
You can get the notebook cell expressions, even if there is an error, by saving the notebook and using Import["mydir/KubaParenthesisQ.nb", "Notebook"]. So you do not have to search in the string :). –  Jacob Akkerboom Jul 16 '13 at 13:56
@JacobAkkerboom I encourage you to write something working this way. Maybe even on current notebook via pallette. I have no time for that now and I don't want to waste the bounty :) –  Kuba Jul 19 '13 at 6:11

I think using control+f to find the error, after you have done "show cell expression", is easy enough. However, if we ever collect code that can automatically check for various syntax errors, this may be a nice addition.

## Easy Procedure

Make a new notebook and evaluate the following

nb = EvaluationNotebook[];


as well as

With[{sB = SuperscriptBox @@ {")", "2"}},
Cell[BoxData[RowBox[{"(", RowBox[{"k", "h"}], sB}]], "Input",
CellChangeTimes -> {3.536893041212217*^9},
EmphasizeSyntaxErrors -> True] // CellPrint;
]


so that this is the notebook in which you have the broken cell. Make sure to leave the generated (broken) cell in it's "fancy" form. Do not use "Show Cell Expression", as this somehow makes NotebookRead unable to read the CellObject later.

Then make a new notebook, in which to put the code below.

First, let's consider some simple functions to find the cell where the error is.

badCellNumber[nb_] :=
Length[Cases[#, SuperscriptBox[")", _], Infinity]] == 0 &] + 1;

cellObjectByNumber[nb_, no_] :=
Part[Cells[nb], no]


and set

badNumber = badCellNumber[nb];


Now we select all bad code in nb as follows. The code below assumes that you are not showing the cell expression of the bad cell. Otherwise you can ommit the second and third line below. The first just serves to show you what is going on.

SetSelectedNotebook[nb]
FrontEndExecute[FrontEndToken@"ToggleShowExpression"]
NotebookFind[nb, "SuperscriptBox[\")\""]


This code looks for a very specific pattern. If you have a similar case where it does not work, please let me know.

## Text file procedure

This part of the answer is based on Kuba's answer very much. All of this is not practical yet. Again, maybe somebody can someday use it to check for multiple syntax errors programatically. But it is really an overkill for now.

Make sure you have not generated any cell groupings in nb, and in particular make sure you have not generated any output in that notebook.

Now we would like to get the string expression corresponding to the cell where the error is. Then we might be able to find out where exactly inside the cell the error is. However, getting this string is not so easy. Kuba's answer shows that we can also just replace code that causes errors by good code, but that seems a bit dangerous.

Below is a function that would tell us the position of the erroneous code in a string. You can play around with ToString on the CellObject, but the strings generated this way do not suit our purposes. Here is the function

With[{rNWC = RepeatedNull[WhitespaceCharacter]},
positionOfTroubleIn[str_] :=
StringPosition[str,
Shortest[
"SuperscriptBox[" ~~ rNWC ~~ "\")\"" ~~ rNWC ~~ "," ~~
rNWC ~~ ___ ~~ rNWC ~~ "]"]] // First
]


Probably the rNWC additions do not help. I think most of time a space followed by a ")" will be made into a RowBox anyway.

Before executing the code below, make sure you have saved the notebook (nb) with bad cell in it. Also, evaluate (anywhere)

nFN = NotebookFileName[nb];


The code below will create a textFile version of that notebook in the directory where you have saved it. Unfortunately I did not see a way around copying the file here.

textName = StringJoin[StringDrop[nFN, -3], ".txt"];
CopyFile[nFN, textName]
fileText = Import[textName];


The code below finds out where the "real code" begins and ends

beginning =
StringPosition[fileText, "(* Beginning of Notebook Content *)"][[1,
2]]
end = StringPosition[fileText, "(* End of Notebook Content *)"][[1,
1]]
fileText2 = StringTake[fileText, {beginning + 2, end - 2}];


Parser

To find the string that corresponds the cell, we have to parse the text file. The parser I have made for this is quite alright I'd say, except that it does not consider comments. A nice feature is that cares about strings and does not consider brackets inside strings for parsing.

openingBrackets = {1, First@#} & /@ StringPosition[fileText2, "["];
closingBrackets = {-1, First@#} & /@ StringPosition[fileText2, "]"];
quotes = {2, First@#} & /@ StringPosition[fileText2, "\""];

orderedBracketPositions =
SortBy[Join[openingBrackets, closingBrackets, quotes ], Last];


Now we find the brackets that are not inside strings

quoteState = True;
positionsOfRelevantBrackets =
Reap[
Block[{res},
Do[
res = orderedBracketPositions[[i]];
If[
res[[1]] == 2,
quoteState = ! quoteState,
If[
quoteState,
Sow[res]
]

]
,
{i, Length[orderedBracketPositions]}
]
]
][[2, 1]];


We are then ready to find the closing brackets of all the cells, by using the fact that when a Cell bracket closes, all but one cell bracket are matched (the cell bracket corresponding to Notebook is not matched).

positionsOfCellClosingBrackets =
Last /@ Part[positionsOfRelevantBrackets,
Flatten@Rest[
Position[Accumulate[First /@ positionsOfRelevantBrackets], 1]]];


With some magic numbers, we also find the starts of the cell :)

cellStarts = Prepend[Most[positionsOfCellClosingBrackets] + 4, 12];


Now lets put the starting positions and closing positions side by side

cellDelims = {cellStarts, positionsOfCellClosingBrackets} //
Transpose;


We can finally find the string

badCellString = StringTake[fileText2, cellDelims[[badNumber]]];


and the "position of trouble"

poT = positionOfTroubleIn[badCellString];


and the string causing the trouble

badString = StringTake[badCellString, poT];


Selecting the string inside the cell

Now we select all the bad code in the broken notebook as follows. The code below assumes that you are not showing the cell expression of the bad cell. Otherwise you can ommit the second and third line below. The first just serves to show you what is going on.

SetSelectedNotebook[nb]

@Kuba I've updated this. I was looking for something like NotebookFind before and I think I even looked at that function, misunderstanding it. Anyway, it is great, you can also search in normal cells with it. –  Jacob Akkerboom Jul 25 '13 at 14:46