# Best practices for using the Testing Framework

I am looking for best practices and recommended workflows for using the Testing Framework. For example, lets say I am developing a Mathematica package for other users, it is hosted on GitHub and I wish to make it easy for other developers to contribute to repository.

The project includes a suite of (unit) tests that I would like to run as often as necessary. I would like to run them within FrontEnd and conveniently inspect the results (timings, which tests failed and why, etc), save them to file and also further automate testing in the future. I can think of few possible workflows but none of them ticks all the boxes.

• "Testing Notebook", Pro: It gives a clear overview of tests and looks nice. It can be run in FrontEnd and also programmatically with TestReport. Con: Contains even more metadata than the normal notebook (all those buttons) so it is less suitable to put under version control. Somehow the evaluation of tests is slower in testing notebook than in normal notebook.

• ".wlt file", Pro: This is the documented way to use plain text files with test and run them programmatically from another notebook or command line. Con: I can't open .wlt files in FrontEnd!? So I have to open them in other text editor without nice features of FrontEnd (autocomplete, code coloring, etc).

• ".wl file", Pro: I can easily edit it in FrontEnd as any other package. Con: This workflow is not documented. Should I put all tests in some specific Context?

So the question is, what is your recommended workflow for testing? While browsing other people's GitHub repositories I have seen some custom made approaches, but I would like rely on something that is documented and easy to explain to other developers who will contribute to the project.

I am also curious what is the purpose of BeginTestSection and EndTestSection in .wlt files?

• In his "Mathematica Cookbook" Sal Mangano hints at some advanced features regarding the MUnit tests here. Using BeginTestSection[ "SectionName", $sectionSwitch ] should enable one to turn off and on a section using the Boolean variable $sectionSwitch.
– gwr
Dec 21, 2021 at 14:41
• Update to my comment: You can see what BeginTestSection does by doing ?MUnit'BeginTestSection. That functionality does not seem work in the new VerificationTest framework. To me the function thus looks useless?
– gwr
Dec 21, 2021 at 15:47

I think @Jason B.'s answer is a good workflow. Here is an alternative that I have adopted over the years. Most of my code I write in *.m / *.mt / *.wl / *.wlt files using IntelliJ Idea with the Mathematica Plugin, so version control is straightforward.

My projects are written as paclets (see How to distribute Mathematica packages as paclets? for an introduction). Paclets allow you to specify resource files in the PacletInfo.m file. For every (major) function, I write a test-file <function-name>.wlt or <function-name>.mt and add it to the Paclet. Here is an example from my Multiplets paclet:

Paclet[
Name -> "Multiplets",
Version -> "0.1",
MathematicaVersion -> "11.0+",
Description -> "Defines multiplets and Young tableaux.",
Thumbnail -> "multiplets.png",
Creator -> "Johannes E. M. Mosig",
Extensions -> {
{"Kernel", Root -> ".", Context -> "Multiplets"},
{"Resource", Root -> "Testing", Resources -> {
"MultipletDimension.mt",
"MultipletQ.mt",
"MultipletReduce.mt",
"Tableau.mt",
"TableauAppend.mt",
"TableauClear.mt",
"TableauDimension.mt",
"TableauFirst.mt",
"TableauFromMultiplet.mt",
"TableauQ.mt",
"TableauRest.mt",
"TableauSimplify.mt",
"TableauToMatrix.mt",
"TableauToMultiplet.mt"
}
}
}
]


In Mathematica, I can then load my paclet

<< Multiplets


and check if any given function works as expected

TestReport@PacletResource["Multiplets", "MultipletReduce.mt"]


I can also open the test file within Mathematica, using

NotebookOpen@PacletResource["Multiplets", "MultipletReduce.mt"]


The front end opens *.mt and *.wlt files as plain text files, however, so if the notebook interface is important to you, then you may want to safe them as *.wl files instead.

In addition, one may add a test-script that just calls all other test scripts, for convenience.

• Great answer! See my updates here with regard to the new paclet framework.
– gwr
Dec 21, 2021 at 15:44

Testing notebooks or .wlt test files? I use both. I create the tests in a notebook, but keep a .wlt file for automated test runs.

This is my workflow:

• I start by creating a new "Testing Notebook" from the file menu
• I add in my testing inputs. For each test, I put all inputs in one cell, and return a single expression.
• Select all the cells and use the "Convert Selection" button to create the tests.
• Assign TestIDs. Obviously having meaningful names here is better, but I tend to just use the randomly generated ones.
• Save the testing notebook in the Tests subdirectory of my package. Then select "Save As .wlt" from the menu, save with the same name as the notebook.

Now I have 2 files in the directory for every test, but that's fine. When I make some code changes, before I commit them I just run the command

TestReport /@ FileNames["~/Path_to_package/*.wlt"]


If the result looks like this,

then everything is good. If I get a failure, then I can copy that TestReportObject and query it directly, asking for "TestsFailed", which returns a list of TestResultObject(s), and I can find the failing test right there. In practice, when I get a test failure I tend to just reopen the notebook, and run the tests interactively to find the cause.

Then when I update the tests in the notebook, I just save as a .wlt file, overwriting the old file.

If you have a build script for any libraries, or paclet build, then you can add this line to the script

successFlag = SameQ[
Total[
Map[TestReport[#]["TestsFailedCount"]&,
FileNames @ "~/Path_to_package/*.wlt"
]
],
0
];


And then abort the build if successFlag is False.

With this method you aren't really getting around the issues of file duplication, large files in version control. For my purposes, this isn't a big deal since the notebooks I use aren't that large. This just seems like the workflow that the framework is designed for - we have easily editable testing notebooks, and .wlt files that are less readable, much smaller, and run faster.

I'm actually very new to testing in general, the above is what I've been able to make work so far.

• Nice to see that someone actually uses the Testing Framework. Seems it'll be worth it to put in the effort to get over the activation barrier to actually start using it. Apr 13, 2018 at 0:18
• I often need to have tests parametrized. Can you think about something better than Block[{envVar = 1}, TestReport...]?
– Kuba
Apr 13, 2018 at 9:16
• @Jason B. Thank you. So you don't mind to have large notebooks in version control system? Did you know that you can pass notebooks to TestReport? It seems that .wlt files then become kind of redundant in your workflow? Apr 13, 2018 at 9:50
• @Pinti in some sense, yes, but I'd actually more call them your IDE, while the true source could just as well be considered to be the WLT files. I do this for my own package development--I have a set of notebooks I develop in, but I only ever put the .m files that I export from them under VC. It actually works quite nicely. Apr 16, 2018 at 6:39
• @M.R. I think it should be pretty easy to combine them into a Dataset for presentation, these report objects are probably just inert wrappers on Associations. May 21, 2018 at 1:19

After I have spent some time developing my package MeshTools and experimenting with testing workflows I am coming back to report my experience.

I have put tests (VerificationTest) in a .wl file and this currently satisfies my requirements. I can conveniently edit it in the FrontEnd (see figure below), code is saved in InputForm rather than FullForm and can be easily read in other text editors. Version Control Systems (Git) have no problem with automatic merging and diffs.

During development I regularly run this test file from another development notebook and inspect the results.

(* Assuming I have PacletInfo.m in added directory. *)

Get["MeshTools"]
$testReport=TestReport@FileNameJoin[{NotebookDirectory[],"Tests.wl"}]  These 2 helper function give me a clear overview of the most important information about tests and highlight any failed tests. getTestResults[tr_TestReportObject]:=Module[ {fields,results,abbreviations}, (* Add other querries to this list. *) fields={"TestIndex","Outcome","AbsoluteTimeUsed","MemoryUsed","TestID"}; abbreviations={"TestIndex"->"Idx","AbsoluteTimeUsed"->"Time [s]"}; results=ReplaceRepeated[ Outer[#1[#2]&,Values[tr["TestResults"]],fields], {x_Quantity:>QuantityMagnitude[x],x_Real:>Round[x,0.001]} ]; Join[{fields/.abbreviations},results] ] printTestResults[tr_TestReportObject]:=Module[ {list,indx,time,noTests}, list=getTestResults[tr]; indx=MapIndexed[ If[ MemberQ[#1,"Failure"|"MessagesFailure"|"Error"], First[#2], Nothing ]&, list ]; time=Round[QuantityMagnitude[tr["TimeElapsed"]],0.01]; noTests=Length[tr["TestResults"]]; Print[noTests," tests run in ",time," seconds."]; If[ TrueQ@tr["AllTestsSucceeded"], Print["All tests succeeded!"], Print[tr["TestsFailedCount"]," tests failed!"]; ]; Print@Grid[list, Alignment->Left, Dividers->{None,{2->True}}, Background->{None,Thread[indx->Pink]} ]; tr ] printTestResults[$testReport];


Integration with Git

I have included automatic (unit) testing before every Git commit via hooks. Besides the Tests.wl file the testing folder contains RunTests.wls script with the following content. If any test fails, then script returns value 1 and commit is rejected.

#!/usr/bin/env wolframscript
(* ::Package:: *)

Print["  Running \"MeshTools\" tests...","\n"];

(* Modify paths depending from which directory the script is started.
By default we assume script is started from git root directory. *)
Get["MeshTools"];

Print["  Using Mathematica: ",$VersionNumber]; Module[ {report,time,results,failIdx}, report=TestReport[ FileNameJoin[{Directory[],"Tests","Tests.wl"}] ]; time=Round[ QuantityMagnitude@report["TimeElapsed"], 0.001 ]; results=report["TestResults"]; failIdx=report["TestsFailedIndices"]; Print["\n"," ",Length[results]," tests run in ",time," seconds."]; If[ TrueQ@report["AllTestsSucceeded"] , Print[" All tests succeeded!"]; Quit[0] (* exit code if all test pass *) , Print[" ",Length[failIdx]," tests failed!"]; Do[ Print[" ",i," / ",results[i]["Outcome"]," / ",results[i]["TestID"]], {i,failIdx} ]; Quit[1] (* exit code if any test fails *) ] ]  In pre-commit file I have put only the call to .wls script and test are automatically run before every commit. #!/bin/sh ./Tests/RunTests.wls  In my experience this works well if you keep your unit tests as short as possible (e.g each test <0.1 second) otherwise you may lose patience waiting at each commit. It has already saved me a few times when I was not paying enough attention and some tests have unexpectedly failed. • I really tried to use testing notebooks but they are slow, limited and buggy :( So this is what I do too atm. Anyway, I created a handy template available in v0.8.0 here: mathematica.stackexchange.com/a/164654/5478 Maybe you are interested. – Kuba Jul 6, 2018 at 13:59 • @Kuba I understand your frustration regarding testing notebooks. Looks nice, works so-so. :) Thank you for the link, I have not yet tried your package, but I will do it sometimes soon. Jul 6, 2018 at 15:10 ## Update with regard to new Paclet Framework Using Paclets myself I would like to update the answer given by @JEM_Mosig above since with some revisions of the Paclet framework beginning afaik in v12.1 things have changed a bit, e.g., a Paclet now is a PacletObject and its properties are given in an Association. As of now a simple paclet might have the following PacletInfo.wl file: PacletObject[ <| "Name" -> "MyPackage", "Version" -> "0.1", "WolframVersion" -> "13+", "Creator" -> "Bingo S. Little <bsl@somedomain.de>", "Description" -> "What the package will do for you.", "Extensions" -> { { "Kernel", "Root" -> "Kernel", "Context"-> { "MyPackage" } }, { "Asset", "Root" -> "Tests", "Assets" -> { { "TestDirectory", "." }, (* "/" will also work *) { "TestCoreFunctionality" , "TestCoreFunctionality.wlt"} } }, { "Path" }, { "Documentation" } } |> ]  The file structure accordingly will be: MyPackage +---Documentation +---Kernel | +---MyPackage.wl +---Tests | +---TestCoreFunctionality.wlt +---PacletInfo.wl  If you have not installed the Paclet yet, which is not unlikely should you be writing tests during development, then you need to do: PacletDirectoryLoad[ "directory" ] (* where to find the paclet *)  Should the Paclet be installed that step can be skipped and we can immediately do: Needs["MyPackage"]  A typical test file like TestCoreFunctionality.wlt will be a text file with the following structure: (* Wolfram Language Test File *) VerificationTest[ (* test expression *) , (* expected output *) , (* messages *) {} , TestID -> "CoreFunctionality-H3A2Z6-211221_shortTestDescription" ]  Unfortunately, the "Path" extension seems to be a mixed bag: DirectoryName[ "MyPackage/Tests/*" ] (* D:\MyPackage\Tests *) FileNames[ "*.wlt", "MyPackage/Tests" ] (* {} *)  But working with the PacletObject and AssetLocation works fine: obj = PacletObject[ "MyPackage" ]; obj["AssetLocation", "TestDirectory" ] (* D:\MyPackage\Tests *) $$testFiles = FileNames[ "*.wlt", obj["AssetLocation", "TestDirectory" ] ];$$tests = Dataset @ AssociationThread[ Range[ Length @ $$testFiles] -> FileBaseName /@$$testFiles ]  And then we can run our tests: $testReport = TestReport[ obj["AssetLocation", "TestCoreFunctionality" ]

• For more information on how to work with the revised paclet framework in Workbench have a look at 257190.
– gwr
Dec 21, 2021 at 15:39

## Edit

The function described below is available in the function repository, and can be used in version 11.3+ as, for example, ResourceFunction["WriteUnitTest"]["~/mytestfile.wl", 1/0 + 3]

This answer gives a workflow using the "Testing Notebook" to create test files. I no longer use this workflow as I had encountered some instability with the testing notebooks as well as the fact that they do not deal with symbols outside of the Global and System contexts.

My current workflow is to write unit tests quickly in a normal notebook and have them written in a readable way to a file. I write the tests as soon as I get the functionality how I want it, I write the tests often and try to make them always self-contained.

Here is an example of a test for FindMoleculeSubstructure. In a normal notebook I would evaluate

WriteUnitTest["MytestFile.mt",
With[{mol = Molecule[Interpreter["Chemical"]["buckyball"]],
fivememberedRing =
MoleculePattern[{_, _, _, _, _},
Bond /@ Partition[Range[5], 2, 1, 1]]},
Length[FindMoleculeSubstructure[mol, fivememberedRing, All]]]]


and I am presented with this dialog

when I press the "Yes" button the test is written to file in a very readable form. This is important because later when a test fails I don't want to waste any time staring at the test figuring out what it does.

The code for WriteUnitTest is as follows:

SetAttributes[prettyInputString, HoldAllComplete];
prettyInputString[expr_] := Module[
{res},
res = GeneralUtilitiesToPrettyString[Unevaluated[expr]];
res // StringReplace["\r\n" -> "\n"] //
StringReplace["\n" -> "\n\t"] //
StringReplace["Global" -> ""] //
StringReplace["\\[LeftDoubleBracket]"~~Shortest[part__]~~"\\[RightDoubleBracket]" :> "[["<>part<>"]]"]
]

Attributes[WriteUnitTest] = HoldRest
Options[WriteUnitTest] = {"ConfirmResults" -> True}
WriteUnitTest[filename_String, input_, append_String:"",opts:OptionsPattern[]] := DynamicModule[
{
inputString = prettyInputString[input], writeToFile,
output, outputString, newFile, evaluationData, messages, messageString,
fileText
},

evaluationData = EvaluationData[input];

output = evaluationData["Result"];
messages = evaluationData["MessagesExpressions"];

If[!FileExistsQ[filename],
newFile = True;
PrintTemporary["File to be created: ", filename],
PrintTemporary["File to be modified: ", filename]
];

outputString = With[{eval=output},prettyInputString[eval]];

(* the testing framework prefers HoldForm here *)
messageString = StringReplace[
ToString[messages,InputForm,PageWidth->80],
"Hold[" -> "HoldForm["
];

writeToFile[] := (
If[
newFile,
Export[
filename,
"BeginTestSection[\""<>FileBaseName[filename]<>"\"]\n\n\nEndTestSection[]\n",
"Text"
]
];

Block[{id = FileBaseName[filename]},
id = id <> "_" <> StringJoin[(If[# > 9, "", "0"] <>ToString[#]) & /@ Date[][[1;;3]]] <> "-" <> ToUpperCase[IntegerString[RandomInteger[{0, 36^^1000000}], 36, 6]] <> append;

outputString = StringJoin[
"VerificationTest[\n\t",
inputString,
"\n\t,\n\t",
outputString,
"\n\t,\n\t",
messageString,
"\n\t,\n\tTestID->\"" <> id <> "\"\n]\n\n"
];
];

fileText = Import[filename, "Text"];
fileText = If[
StringContainsQ[
fileText,
"EndTestSection[]"
]
,
StringReplace[
fileText,
"EndTestSection[]" :> outputString <> "\nEndTestSection[]\n"
],
fileText<> "\n" <> outputString
];
Export[filename, fileText, "Text"];
Print @ filename;

SelectionMove[ButtonNotebook[], All, ButtonCell];
NotebookDelete[ButtonNotebook[]];
);

If[
OptionValue["ConfirmResults"]
,

Column[
{
Row[{"Input: ", inputString}],
Row[{"Result: ", outputString}],
Row[{"Messages: ", messageString}],
Row[
{
Button[
Text[Style["Yes"]],
writeToFile[];
,
"BaseStyle" -> Darker[Green]
]
,
Button[Text[Style["No"]],
Print["Aborting without writing to ", filename];
SelectionMove[ButtonNotebook[], All, ButtonCell];
NotebookDelete[ButtonNotebook[]];
, "BaseStyle" -> Darker[Red]]
},
"  "
]
}
]

,
writeToFile[];
filename
]
]


It was hacked together, so any suggestions for improving it are most appreciated.

• Thanks, I learned a lot by inspecting the function WriteUnitTest. But I have to admit that I am reluctant to rely on workflows involving such "hacky" functions. Isn't just writing directly to a testing file, opened side-by-side with a development notebook, simpler? May 9, 2019 at 5:06
• Simpler maybe, but not very scalable. For me, it's important to be able to make tests easily, quickly, and have them always work as expected - when I implement a new feature I can add the test using this method easily, and the result is very readable. Just looking in the tests directory of the chemistry paclet I've accumulated ~2500 tests in the past year. May 9, 2019 at 14:01