Unit tests for Graphics or FE boxes

Consider the following expression with bad syntax:

Graphics[{foo}]


How can I detect the error and retrieve the error message programmatically, so I can check for it in unit tests?

• A poor man's equivalent: Testing Framework applications in the FrontEnd area – Kuba Feb 11 '17 at 10:53
• related: 29086 and related option: HighlightFormattingErrors – Kuba Feb 11 '17 at 10:54
• Lacking a better solution, what I have done is wrapping my test with Hash[] and comparing with the expected value. – Gustavo Delfino Feb 13 '17 at 21:00
• @GustavoDelfino Hash only works so far, and is very brittle to innocuous changes to the Graphics structure, e.g. this sort of shift is not uncommon: Graphics[{primitives}, opts] to Graphics[{{primitives}}, opts], and it does nothing to the end state. Other than FE error detection, if you are comparing Graphics objects, the only meaningful comparison is their end states, i.e. you need a parser. – rcollyer Feb 24 '17 at 14:53

This has been partially answered before, so here I will highlight some of its evolution since that previous answer. VerificationTest and its MUnit antecedents do not have a mechanism for introducing new error types, so we need a function to do that for us. For this, I use a function called checkGraphicsRendering (outlined at the end) with the general use being the two argument form, as follows

VerificationTest[
checkGraphicsRendering[
testFunction
,
expressionToTest
]
,
{testResult, "RenderingErrors" -> {FEError ...}}
,
testOptions
]


where testFunction operates on the result of expressionToTest generating testResult while simultaneously checking the FrontEnd for any FEErrors (pink boxes). When there are a large number of tests with identical testFunctions, I will usually set

checkGraphicsRendering$defaultTest = testFunction  so that I can employ the one argument form. Additionally, there are some cases where executing expressionToTest within a notebook is essential, e.g. checking that PlotTheme -> Automatic picks up the stylesheet theme, so there are two options for checkGraphicsRendering: "NotebookEvaluate" which takes boolean values and "NotebookOptions" which are passed to CreateDocument and default to checkGraphicsRendering$defaultNotebookOptions. So, if you want to have additional options in addition to the default, just use

"NotebookOptions" -> {opts ..., checkGraphicsRendering$defaultNotebookOptions}  and they will be flattened out internally. Lastly, when dealing with automatically generated tests, I generally alter the output of checkGraphicsRendering to Hold[expressionToTest] -> {testResult, "RenderingErrors" -> {FEError ...}}  as it allows the user to simply execute what expression the test actually ran. Here I will discuss the package itself. As mentioned above, there are two user settable constants: $defaultNotebookOptions and $defaultTest which the package checks prior to setting the default, so they are not overridden. Usually, I set the NotebookFileName as you do not need to have a notebook named Untitled-2045 when run from the FE, but it comes with a drawback: the notebook is created on disk. So, I often find FEMessages.nb notebooks scattered all over the place. The most important note, though, is NotebookEvaluate does not work when the FE is not the $ParentLink. So, I workaround this by Blocking $ParentLink when needed. BeginPackage["checkGraphicsRendering"]; checkGraphicsRendering;$defaultNotebookOptions;
$defaultTest; Begin["Private"]; ClearAll[checkGraphicsRendering, linkedNotebookEvaluate]; SetAttributes[checkGraphicsRendering, HoldAll];$defaultNotebookOptions = If[ValueQ@$defaultNotebookOptions,$defaultNotebookOptions,
{Visible -> False, NotebookFileName -> "FEMessages.nb"}
];

$defaultTest = If[ValueQ@$defaultTest, $defaultTest, Identity]; Options[checkGraphicsRendering] := {"NotebookEvaluate" -> False, "NotebookOptions" :>$defaultNotebookOptions};

If[$Linked && Cases[$FrontEnd, _LinkObject, -1] =!= {$ParentLink} , Block[{$ParentLink}, NotebookEvaluate[expr, InsertResults->True]]
,
NotebookEvaluate[expr, InsertResults->True]
];

checkGraphicsRendering[test_, expr_, OptionsPattern[]]:=
Block[{nb, res, pinks, evalFlag},
evalFlag= TrueQ@OptionValue["NotebookEvaluate"];
UsingFrontEnd[
nb = CreateDocument[
If[evalFlag
,
ExpressionCell[Defer[expr], "Input"]
,
ExpressionCell[res = expr, "Output"]
]
,
##
]& @@ Flatten[{OptionValue["NotebookOptions"]}];
pinks = MathLinkCallFrontEnd[FrontEndGetErrorsInSelectionPacket[nb]];
checkGraphicsRendering[$defaultTest, expr, opts] End[]; EndPackage[];  • Thank you, very useful! Regarding the $ParentLink problem, also very useful. I asked about it here and even WRI people couldn't answer: community.wolfram.com/groups/-/m/t/971065 Perhaps you could post an answer there too. BTW this problem with EvaluateNotebook does not seem to be present in 10.3 and later. – Szabolcs Feb 12 '17 at 10:46
• @Szabolcs: NotebookEvaluate` when run in a standalone or sub-kernel is fraught with issues. We've run into them internally when creating new testing tools. The FE devs tell us that this is a very tricky problem to solve. – Stefan R Mar 10 '17 at 20:19