How to find pieces of code responsible for syntax errors?

Question

Especially: How to find the piece of code responsible for a missing parenthesis problem caused by a misplaced SuperScriptBox

For last 30 minutes (and I was lucky today, sometimes I spend 5 times as much time) I have been removing code little by little from my large one cell, (have to be one cell, it is Manipulate demo stylesheet) in order to find where the syntax error is (since Mathematica, with all its might and power, can't tell the user which line number the syntax error is located at). Each time I see the red line show up on the right side of the notebook, indicating a syntax error, I know that I will now waste another hour at least looking for the syntax error.

Why is it that hard for Mathematica to tell the user where the syntax error is?

I zoomed down to this little 5 character code, and I copy it to new notebook to examine it:

So, there is a hidden bad character there. Sometimes, with my big fingers and my small keyboard (can't find big keyboards any more, everything is so tiny), I must have hit wrong key combination, which must have injected some bad character to the notebook, and this happens.

I will keep this cell on its own in order to find what the problem is. I tried to select the cell, and do convert to input form, but nothing happened. no output came of it.

Then I did cell->ShowExpression, and this came up

Cell[BoxData[
 RowBox[{"(", 
  RowBox[{"k", "h"}], 
  SuperscriptBox[")", "2"]}]], "Input",
 CellChangeTimes->{3.536893041212217*^9},
 EmphasizeSyntaxErrors->True]

So, what does all this mean? Do you see from the above what is the problem?

I think I entered the superscript as I always do, using Ctrl+^ then 2.

I think the notebook interface is powerful and allows many things not possible otherwise, but because of problems such the above, I wish sometimes I am using plain text editor like emacs, where I can see what I have and not worry about hidden things I do not see, but I tried that once, and it is not practical overall setup to work with.

Btw, originally, I had lots of code like the above inside strings, i.e. inside " " for formatting, and now I am removing all the strings to do some other way of formatting. I do not know if this has anything to do with it. But as you can see, now there is no string around the expression. Here is a screen shot of the earlier version of the notebook, from the same location of the code, showing how the above was inside a string, and I simply was removing the outside string quotes, that is all:

Answer 1

New answer

There was actually an old post on mathgroup, Strange Syntax problem, that looks like it was exactly the same problem you're experiencing. A power/superscript being associated with the closing parenthesis instead of the whole parenthesized expression (see the box forms presented in my original answer below for more details). Unfortunately, this problem was not resolved in the the mathgroup post.

The best I can suggest at the moment is to simply press the + button
to get the expanded form

This should help narrow down the problem. Note the Syntax::bktmcp warning is talking about non-matching brackets, yet in the visible expression, all brackets are properly matched. So, the problem lies in the underlying box expression.

Alternatively, if it is this particular problem, you could Show Expression, then search for the problematic code snippet SuperscriptBox[")". Then you can modify the box structure directly. This is normally easiest if you paste it into a new input cell so that the Extend Selection mechanism works.

Original answer

There is no "hidden bad character" in the cell that you posted. Rather, the box structure is crazy and can't be interpreted by Mathematica - I have trouble imagining how you managed to create it...

Here's your cell

Cell[BoxData[
 RowBox[{"(", 
  RowBox[{"k", "h"}], 
  SuperscriptBox[")", "2"]}]], "Input"]

and here is how it should look

Cell[BoxData[
 SuperscriptBox[
  RowBox[{"(", 
   RowBox[{"k", " ", "h"}], ")"}], "2"]], "Input"]

So, your question does not really a work with the example you provided.

However, a simple way to maybe clean your code (at least in this case) is to copy it as plain text or input text, both of which produce (k h)^2 when pasted.

Answer 2

It happens to me quite often while working with huge notebooks. It is quite irritating because not always it is obvious what have caused an error and where to find it.

This is part of bigger code. As You see an error is transferring with Crtl+C -> Crtl+V. Usually rewritting problematic parts (if You've found them) helps but it is not convenient at all.

Edit

I have succesfully tested this method on notebook that contains ~40000 characters. It preserves notebook formatting etc.

The notebook You are trying to use it on cannot contain those orange-marked cells. Just copy the content and -> delete marked cell -> paste clipboard and save the notebook. Then it is ready for this procedure.

I'm going to show an approach to solve this issue using important remark of Simon:

Here's your cell

Cell[BoxData[
 RowBox[{"(", 
  RowBox[{"k", "h"}], 
  SuperscriptBox[")", "2"]}]], "Input"]

and here is how it should look

Cell[BoxData[
 SuperscriptBox[
  RowBox[{"(", 
   RowBox[{"k", " ", "h"}], ")"}], "2"]], "Input"]

However, it is not so simple to resctructure this cell, Mathematica refuses to show even BoxData of this cell, unless You Shift+Crtl+E. I'm going to use brute force of strings.

SetDirectory@NotebookDirectory[];
test = Import["beforeTest.nb", "Text"];
tt = StringReplace[test, 
                   "RowBox[" ~~ x : RegularExpression["((?!RowBox).)*"] ~~ "RowBox[" ~~
                   y : RegularExpression["((?!RowBox).)*"] ~~ 
                   "SuperscriptBox[\")\"," ~~ z : Except["]"] .. ~~ "]}]" 
                                 :> 
                   "SuperscriptBox[RowBox[{\"(\", RowBox[" ~~ y ~~ " \")\"}]," ~~ 
                   z ~~"]"
                  ];

 Export["test.txt", tt];
 RenameFile["test.txt", "test.nb"];

Well, not very sophisticated but it works:

It needs testing of course:

• I do not know if the problem of crazy structured cell appears in other context than this with Superscript because I am no able to creat such on demand. However, if my memory serves me, there wasn't any other case.

• I'm not sure if this method will preserve notebook formatting created earlier.

• I'm not sure if StringPattern I've created is universal and bulletproof enough.

I appreciate any help with improving this pattern and method in general and with testing this as it seems to be a critical problem sometimes.

Answer 3

About

I think using control+f to find the error, after you have done "show cell expression", is easy enough. However, if we ever collect code that can automatically check for various syntax errors, this may be a nice addition.

Easy Procedure

Make a new notebook and evaluate the following

nb = EvaluationNotebook[];

as well as

With[{sB = SuperscriptBox @@ {")", "2"}},
 Cell[BoxData[RowBox[{"(", RowBox[{"k", "h"}], sB}]], "Input", 
    CellChangeTimes -> {3.536893041212217*^9}, 
    EmphasizeSyntaxErrors -> True] // CellPrint;
 ]

so that this is the notebook in which you have the broken cell. Make sure to leave the generated (broken) cell in it's "fancy" form. Do not use "Show Cell Expression", as this somehow makes NotebookRead unable to read the CellObject later.

Then make a new notebook, in which to put the code below.

Finding the bad cell

First, let's consider some simple functions to find the cell where the error is.

badCellNumber[nb_] :=      
  LengthWhile[NotebookRead@Cells[nb], 
    Length[Cases[#, SuperscriptBox[")", _], Infinity]] == 0 &] + 1;

cellObjectByNumber[nb_, no_] :=
 Part[Cells[nb], no]

and set

badNumber = badCellNumber[nb];
badCellObject = cellObjectByNumber[nb, badNumber];

Now we select all bad code in nb as follows. The code below assumes that you are not showing the cell expression of the bad cell. Otherwise you can ommit the second and third line below. The first just serves to show you what is going on.

SetSelectedNotebook[nb]
SelectionMove[badCellObject, All, Cell]
FrontEndExecute[FrontEndToken@"ToggleShowExpression"]
NotebookFind[nb, "SuperscriptBox[\")\""]

This code looks for a very specific pattern. If you have a similar case where it does not work, please let me know.

Text file procedure

About

This part of the answer is based on Kuba's answer very much. All of this is not practical yet. Again, maybe somebody can someday use it to check for multiple syntax errors programatically. But it is really an overkill for now.

Make sure you have not generated any cell groupings in nb, and in particular make sure you have not generated any output in that notebook.

Now we would like to get the string expression corresponding to the cell where the error is. Then we might be able to find out where exactly inside the cell the error is. However, getting this string is not so easy. Kuba's answer shows that we can also just replace code that causes errors by good code, but that seems a bit dangerous.

Below is a function that would tell us the position of the erroneous code in a string. You can play around with ToString on the CellObject, but the strings generated this way do not suit our purposes. Here is the function

With[{rNWC = RepeatedNull[WhitespaceCharacter]},
 positionOfTroubleIn[str_] :=      
  StringPosition[str, 
    Shortest[
     "SuperscriptBox[" ~~ rNWC ~~ "\")\"" ~~ rNWC ~~ "," ~~ 
      rNWC ~~ ___ ~~ rNWC ~~ "]"]] // First
 ]

Probably the rNWC additions do not help. I think most of time a space followed by a ")" will be made into a RowBox anyway.

Before executing the code below, make sure you have saved the notebook (nb) with bad cell in it. Also, evaluate (anywhere)

nFN = NotebookFileName[nb];

The code below will create a textFile version of that notebook in the directory where you have saved it. Unfortunately I did not see a way around copying the file here.

textName = StringJoin[StringDrop[nFN, -3], ".txt"];
CopyFile[nFN, textName]
fileText = Import[textName];

The code below finds out where the "real code" begins and ends

beginning = 
 StringPosition[fileText, "(* Beginning of Notebook Content *)"][[1, 
   2]]
end = StringPosition[fileText, "(* End of Notebook Content *)"][[1, 
   1]]
fileText2 = StringTake[fileText, {beginning + 2, end - 2}];

Parser

To find the string that corresponds the cell, we have to parse the text file. The parser I have made for this is quite alright I'd say, except that it does not consider comments. A nice feature is that cares about strings and does not consider brackets inside strings for parsing.

openingBrackets = {1, First@#} & /@ StringPosition[fileText2, "["];
closingBrackets = {-1, First@#} & /@ StringPosition[fileText2, "]"];
quotes = {2, First@#} & /@ StringPosition[fileText2, "\""];

orderedBracketPositions = 
  SortBy[Join[openingBrackets, closingBrackets, quotes ], Last];

Now we find the brackets that are not inside strings

quoteState = True;
(*commentState=True;*)
positionsOfRelevantBrackets =
  Reap[
    Block[{res},
     Do[
      res = orderedBracketPositions[[i]];
      If[
       res[[1]] == 2, 
       quoteState = ! quoteState,
       If[
        quoteState,
        Sow[res]
        ]

       ]
      ,
      {i, Length[orderedBracketPositions]}
      ]
     ]
    ][[2, 1]];

We are then ready to find the closing brackets of all the cells, by using the fact that when a Cell bracket closes, all but one cell bracket are matched (the cell bracket corresponding to Notebook is not matched).

positionsOfCellClosingBrackets = 
  Last /@ Part[positionsOfRelevantBrackets, 
    Flatten@Rest[
      Position[Accumulate[First /@ positionsOfRelevantBrackets], 1]]];

With some magic numbers, we also find the starts of the cell :)

cellStarts = Prepend[Most[positionsOfCellClosingBrackets] + 4, 12];

Now lets put the starting positions and closing positions side by side

cellDelims = {cellStarts, positionsOfCellClosingBrackets} // 
   Transpose;

We can finally find the string

badCellString = StringTake[fileText2, cellDelims[[badNumber]]];

and the "position of trouble"

poT = positionOfTroubleIn[badCellString];

and the string causing the trouble

badString = StringTake[badCellString, poT];

Selecting the string inside the cell

Now we select all the bad code in the broken notebook as follows. The code below assumes that you are not showing the cell expression of the bad cell. Otherwise you can ommit the second and third line below. The first just serves to show you what is going on.

SetSelectedNotebook[nb]
SelectionMove[ badCellObject, All, Cell]
FrontEndExecute[FrontEndToken@"ToggleShowExpression"]
NotebookFind[nb, badString]