Consider the following expression with bad syntax:


enter image description here

How can I detect the error and retrieve the error message programmatically, so I can check for it in unit tests?

  • $\begingroup$ A poor man's equivalent: Testing Framework applications in the FrontEnd area $\endgroup$
    – Kuba
    Feb 11 '17 at 10:53
  • 1
    $\begingroup$ related: 29086 and related option: HighlightFormattingErrors $\endgroup$
    – Kuba
    Feb 11 '17 at 10:54
  • $\begingroup$ Lacking a better solution, what I have done is wrapping my test with Hash[] and comparing with the expected value. $\endgroup$ Feb 13 '17 at 21:00
  • $\begingroup$ @GustavoDelfino Hash only works so far, and is very brittle to innocuous changes to the Graphics structure, e.g. this sort of shift is not uncommon: Graphics[{primitives}, opts] to Graphics[{{primitives}}, opts], and it does nothing to the end state. Other than FE error detection, if you are comparing Graphics objects, the only meaningful comparison is their end states, i.e. you need a parser. $\endgroup$
    – rcollyer
    Feb 24 '17 at 14:53

This has been partially answered before, so here I will highlight some of its evolution since that previous answer. VerificationTest and its MUnit` antecedents do not have a mechanism for introducing new error types, so we need a function to do that for us. For this, I use a function called checkGraphicsRendering (outlined at the end) with the general use being the two argument form, as follows

  {testResult, "RenderingErrors" -> {FEError ...}}

where testFunction operates on the result of expressionToTest generating testResult while simultaneously checking the FrontEnd for any FEErrors (pink boxes). When there are a large number of tests with identical testFunctions, I will usually set

checkGraphicsRendering`$defaultTest = testFunction

so that I can employ the one argument form. Additionally, there are some cases where executing expressionToTest within a notebook is essential, e.g. checking that PlotTheme -> Automatic picks up the stylesheet theme, so there are two options for checkGraphicsRendering: "NotebookEvaluate" which takes boolean values and "NotebookOptions" which are passed to CreateDocument and default to checkGraphicsRendering`$defaultNotebookOptions. So, if you want to have additional options in addition to the default, just use

"NotebookOptions" -> {opts ..., checkGraphicsRendering`$defaultNotebookOptions}

and they will be flattened out internally. Lastly, when dealing with automatically generated tests, I generally alter the output of checkGraphicsRendering to

Hold[expressionToTest] -> {testResult, "RenderingErrors" -> {FEError ...}}

as it allows the user to simply execute what expression the test actually ran.

Here I will discuss the package itself. As mentioned above, there are two user settable constants: $defaultNotebookOptions and $defaultTest which the package checks prior to setting the default, so they are not overridden. Usually, I set the NotebookFileName as you do not need to have a notebook named Untitled-2045 when run from the FE, but it comes with a drawback: the notebook is created on disk. So, I often find FEMessages.nb notebooks scattered all over the place. The most important note, though, is NotebookEvaluate does not work when the FE is not the $ParentLink. So, I workaround this by Blocking $ParentLink when needed.



ClearAll[checkGraphicsRendering, linkedNotebookEvaluate];
SetAttributes[checkGraphicsRendering, HoldAll];

$defaultNotebookOptions = If[ValueQ@$defaultNotebookOptions,
  {Visible -> False, NotebookFileName -> "FEMessages.nb"}

$defaultTest = If[ValueQ@$defaultTest, $defaultTest, Identity];

Options[checkGraphicsRendering] := {"NotebookEvaluate" -> False, 
   "NotebookOptions" :> $defaultNotebookOptions};

linkedNotebookEvaluate[expr_] := 
 If[$Linked && Cases[$FrontEnd, _LinkObject, -1] =!= {$ParentLink}
   Block[{$ParentLink}, NotebookEvaluate[expr, InsertResults->True]]
   NotebookEvaluate[expr, InsertResults->True]

checkGraphicsRendering[test_, expr_, OptionsPattern[]]:=
Block[{nb, res, pinks, evalFlag},
  evalFlag= TrueQ@OptionValue["NotebookEvaluate"];
    nb = CreateDocument[
        ExpressionCell[Defer[expr], "Input"]
        ExpressionCell[res = expr, "Output"]
    ]& @@ Flatten[{OptionValue["NotebookOptions"]}]; 
    res = If[evalFlag, linkedNotebookEvaluate[nb], res];
    SelectionMove[nb, All, Cell];
    pinks = MathLink`CallFrontEnd[FrontEnd`GetErrorsInSelectionPacket[nb]];
  {test@res, "Rendering Errors" -> pinks}

checkGraphicsRendering[expr:Except[_?OptionQ], opts:OptionsPattern[]] := 
  checkGraphicsRendering[$defaultTest, expr, opts]

  • $\begingroup$ Thank you, very useful! Regarding the $ParentLink problem, also very useful. I asked about it here and even WRI people couldn't answer: community.wolfram.com/groups/-/m/t/971065 Perhaps you could post an answer there too. BTW this problem with EvaluateNotebook does not seem to be present in 10.3 and later. $\endgroup$
    – Szabolcs
    Feb 12 '17 at 10:46
  • $\begingroup$ @Szabolcs I discovered it quite by accident, trying to run tests within WB. I'll look at it again, but I'm fairly sure it still exists in some form. $\endgroup$
    – rcollyer
    Feb 12 '17 at 12:55
  • 2
    $\begingroup$ @Szabolcs: NotebookEvaluate when run in a standalone or sub-kernel is fraught with issues. We've run into them internally when creating new testing tools. The FE devs tell us that this is a very tricky problem to solve. $\endgroup$
    – Stefan R
    Mar 10 '17 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.