# How do I correctly embed VerificationTests in my packages?

I like to write my tests before I code and write them inline when developing a package, and when loading a package those VerificationTests should run and the load should fail and print a report if any of the tests fail.

I don't want a separate testing notebook. I don't know how to achieve this functionality, the difficulty is that when the package is run all the tests are as well and I don't know how to accumulate the failures and stop the load and present a report.

So here's a simplified example:

BeginPackage["MusicTools"];
note::usage = "a note";
noteQ::usage = "test for a note";
Begin["Private"];
noteQ[note[x_] /; NumericQ[x]] := True
noteQ[_] := False

VerificationTest[noteQ["test"],False]

End[ ]
EndPackage[]


The code to generate an alert might look like this:

CreateDocument[TestReport[VerificationTest[noteQ["45"],False]]["TestResults"]]

• Are you sure you want to use VerificationTest here? Could you achieve your objective by using Assert? (And setting On["Assert"], since they're turned off by default). Mar 25 '15 at 22:20
• @stefan Then we wouldn't get the benefits of being able to use TestReport.
– M.R.
Mar 27 '15 at 3:21
• @M.R. Can you elaborate why you don't want to use a Testing notebook? TestReport was built in part to take advantage of the Testing Notebook framework. Its pretty standard programming practise to have your test harness separate to your main UI. Mar 27 '15 at 10:05
• I was thinking about this problem just today. I always put a test block after each function inside my pack. Today my final package has these functions as comments, and I fell it's not the best practice, but I would like to keep it in the same pack, but not as comments. Maybe a more experienced programmer can share some experience. Mar 28 '15 at 2:19
• I'm afraid it is very unlikely, that more experienced programmers are able to give a detailed canonical answer, because this question is not widely applicable IMO. There are good reasons, why most common (I don't know any counter example) programming languages separate their unit-tests from the source code and even put them into a complete different package/directory. Apr 1 '15 at 14:44

Would be helpful for you to use the following?

By setting On[AbortMessages[test::"notpassed"]] the system will stop loading the package when the message test::"notpassed" is generated.

E.g. with:

On[AbortMessages[test::"nopass"]];
Do[Print[x]; Message[test::"nopass"], {100}];


only one x will be printed.

ClearAll[AbortMessages];
AbortMessages::"usage" = "On[AbortMessages[symbol::\"tag\", ...]] and Off[AbortMessage[]] extend the Mathematica exception handling capacity.
On[AbortMessages[]] sets the environment so that any Message[symbol::\"tag\"] printed by Mathematica aborts calculations immediately.
On[AbortMessages[symbol::\"tag\", ...]] sets the environment to abort immediately but only after any of the symbol::\"tag\", ... messages is generated.
Off[AbortMessages[]] restores the exception handling capacity to the Mathematica default.
On[symbol::\"tag\", ...] and Off[symbol::\"tag\", ...] take precedence over On[AbortMessages[symbol:: \"tag\", ...]], i.e., (Off[s::t];On[AbortMessages[s::t]]) will neither generate the Message[s::t] nor abort the calculations.";

SetAttributes[AbortMessages, HoldAll];

On[AbortMessages[msgNames___MessageName]] ^:= (
Unprotect[Message, $MessagesAbortQ];$MessagesAbortQ = False;
Message[args___] :=
Check[
Block[{$MessagesAbortQ = False}, Message[args]], Abort[], msgNames ] /;$MessagesAbortQ;
$MessagesAbortQ = True; Protect[Message,$MessagesAbortQ];
);

Off[AbortMessages[]] ^:= (
(*mPrint["\"Off[AbortMessages[]]\""];*)
Unprotect[Message, $MessagesAbortQ];$MessagesAbortQ = False;
Clear[Message];
Protect[Message, \$MessagesAbortQ];
);


You could use Condition /; with your predefined tests on definition of new symbols and functions within your package. So the definition of a Symbol is only loaded, if the test yields True. Add a message if the test goes wrong.

Maybe this is what you are looking for:

BeginPackage["MusicTools"];
note::usage = "a note";
noteQ::usage = "test for a note";
noteQ::TestResults = "The argument 1 is not a note.";
Octave::usage = "Octave[note] generates the octave of a note.";
Begin["Private"];
noteQ[note[x_] /; NumericQ[x]] := True;
noteQ[arg___] := (Message[noteQ::"TestResults", arg]; False);

Octave[noteIn_] /; noteQ[noteIn] := note[2*noteIn〚1〛];

End[]
EndPackage[]


In:

 Octave[note[440]]


Out:

 note[880]


In:

 Octave[note["A"]]


Out:

"noteQ::TestResults: The argument note[A] is not a note."
Octave[note["A"]]

• Thanks @sacatus, but I need it to be all or nothing and I know about Condition but I want to leverage the new suite of functionality in version 10 with VerificationTest and friends. Basically, if all the tests pass the package loads, but if any fail, the load aborts and a test report detailing all failures should be generated.
– M.R.
Mar 28 '15 at 15:59