Unit tests for Graphics or FE boxes

Question

Consider the following expression with bad syntax:

Graphics[{foo}]

How can I detect the error and retrieve the error message programmatically, so I can check for it in unit tests?

Answer 1

This has been partially answered before, so here I will highlight some of its evolution since that previous answer. VerificationTest and its MUnit` antecedents do not have a mechanism for introducing new error types, so we need a function to do that for us. For this, I use a function called checkGraphicsRendering (outlined at the end) with the general use being the two argument form, as follows

VerificationTest[
  checkGraphicsRendering[
    testFunction
    ,
    expressionToTest
  ]
  ,
  {testResult, "RenderingErrors" -> {FEError ...}}
  ,
  testOptions
]

where testFunction operates on the result of expressionToTest generating testResult while simultaneously checking the FrontEnd for any FEErrors (pink boxes). When there are a large number of tests with identical testFunctions, I will usually set

checkGraphicsRendering`$defaultTest = testFunction

so that I can employ the one argument form. Additionally, there are some cases where executing expressionToTest within a notebook is essential, e.g. checking that PlotTheme -> Automatic picks up the stylesheet theme, so there are two options for checkGraphicsRendering: "NotebookEvaluate" which takes boolean values and "NotebookOptions" which are passed to CreateDocument and default to checkGraphicsRendering`$defaultNotebookOptions. So, if you want to have additional options in addition to the default, just use

"NotebookOptions" -> {opts ..., checkGraphicsRendering`$defaultNotebookOptions}

and they will be flattened out internally. Lastly, when dealing with automatically generated tests, I generally alter the output of checkGraphicsRendering to

Hold[expressionToTest] -> {testResult, "RenderingErrors" -> {FEError ...}}

as it allows the user to simply execute what expression the test actually ran.

---

Here I will discuss the package itself. As mentioned above, there are two user settable constants: $defaultNotebookOptions and $defaultTest which the package checks prior to setting the default, so they are not overridden. Usually, I set the NotebookFileName as you do not need to have a notebook named Untitled-2045 when run from the FE, but it comes with a drawback: the notebook is created on disk. So, I often find FEMessages.nb notebooks scattered all over the place. The most important note, though, is NotebookEvaluate does not work when the FE is not the $ParentLink. So, I workaround this by Blocking $ParentLink when needed.

BeginPackage["checkGraphicsRendering`"];

checkGraphicsRendering;
$defaultNotebookOptions;
$defaultTest;

Begin["`Private`"];
ClearAll[checkGraphicsRendering, linkedNotebookEvaluate];
SetAttributes[checkGraphicsRendering, HoldAll];

$defaultNotebookOptions = If[ValueQ@$defaultNotebookOptions,
  $defaultNotebookOptions, 
  {Visible -> False, NotebookFileName -> "FEMessages.nb"}
];

$defaultTest = If[ValueQ@$defaultTest, $defaultTest, Identity];

Options[checkGraphicsRendering] := {"NotebookEvaluate" -> False, 
   "NotebookOptions" :> $defaultNotebookOptions};

linkedNotebookEvaluate[expr_] := 
 If[$Linked && Cases[$FrontEnd, _LinkObject, -1] =!= {$ParentLink}
   , 
   Block[{$ParentLink}, NotebookEvaluate[expr, InsertResults->True]]
   , 
   NotebookEvaluate[expr, InsertResults->True]
];

checkGraphicsRendering[test_, expr_, OptionsPattern[]]:=
Block[{nb, res, pinks, evalFlag},
  evalFlag= TrueQ@OptionValue["NotebookEvaluate"];
  UsingFrontEnd[
    nb = CreateDocument[
      If[evalFlag
        ,
        ExpressionCell[Defer[expr], "Input"]
        ,
        ExpressionCell[res = expr, "Output"]
      ]
      , 
      ##
    ]& @@ Flatten[{OptionValue["NotebookOptions"]}]; 
    res = If[evalFlag, linkedNotebookEvaluate[nb], res];
    SelectionMove[nb, All, Cell];
    pinks = MathLink`CallFrontEnd[FrontEnd`GetErrorsInSelectionPacket[nb]];
    NotebookClose[nb]
  ];
  {test@res, "Rendering Errors" -> pinks}
]

checkGraphicsRendering[expr:Except[_?OptionQ], opts:OptionsPattern[]] := 
  checkGraphicsRendering[$defaultTest, expr, opts]

End[];
EndPackage[];