# Best practices for using the Testing Framework

I am looking for best practices and recommended workflows for using the Testing Framework. For example, lets say I am developing a Mathematica package for other users, it is hosted on GitHub and I wish to make it easy for other developers to contribute to repository.

The project includes a suite of (unit) tests that I would like to run as often as necessary. I would like to run them within FrontEnd and conveniently inspect the results (timings, which tests failed and why, etc), save them to file and also further automate testing in the future. I can think of few possible workflows but none of them ticks all the boxes.

• "Testing Notebook", Pro: It gives a clear overview of tests and looks nice. It can be run in FrontEnd and also programmatically with TestReport. Con: Contains even more metadata than the normal notebook (all those buttons) so it is less suitable to put under version control. Somehow the evaluation of tests is slower in testing notebook than in normal notebook.

• ".wlt file", Pro: This is the documented way to use plain text files with test and run them programmatically from another notebook or command line. Con: I can't open .wlt files in FrontEnd!? So I have to open them in other text editor without nice features of FrontEnd (autocomplete, code coloring, etc).

• ".wl file", Pro: I can easily edit it in FrontEnd as any other package. Con: This workflow is not documented. Should I put all tests in some specific Context?

So the question is, what is your recommended workflow for testing? While browsing other people's GitHub repositories I have seen some custom made approaches, but I would like rely on something that is documented and easy to explain to other developers who will contribute to the project.

I am also curious what is the purpose of BeginTestSection and EndTestSection in .wlt files?

Testing notebooks or .wlt test files? I use both. I create the tests in a notebook, but keep a .wlt file for automated test runs.

This is my workflow:

• I start by creating a new "Testing Notebook" from the file menu
• I add in my testing inputs. For each test, I put all inputs in one cell, and return a single expression.
• Select all the cells and use the "Convert Selection" button to create the tests.
• Assign TestIDs. Obviously having meaningful names here is better, but I tend to just use the randomly generated ones.
• Save the testing notebook in the Tests subdirectory of my package. Then select "Save As .wlt" from the menu, save with the same name as the notebook.

Now I have 2 files in the directory for every test, but that's fine. When I make some code changes, before I commit them I just run the command

TestReport /@ FileNames["~/Path_to_package/*.wlt"]


If the result looks like this,

then everything is good. If I get a failure, then I can copy that TestReportObject and query it directly, asking for "TestsFailed", which returns a list of TestResultObject(s), and I can find the failing test right there. In practice, when I get a test failure I tend to just reopen the notebook, and run the tests interactively to find the cause.

Then when I update the tests in the notebook, I just save as a .wlt file, overwriting the old file.

If you have a build script for any libraries, or paclet build, then you can add this line to the script

successFlag = SameQ[
Total[
Map[TestReport[#]["TestsFailedCount"]&,
FileNames @ "~/Path_to_package/*.wlt"
]
],
0
];


And then abort the build if successFlag is False.

With this method you aren't really getting around the issues of file duplication, large files in version control. For my purposes, this isn't a big deal since the notebooks I use aren't that large. This just seems like the workflow that the framework is designed for - we have easily editable testing notebooks, and .wlt files that are less readable, much smaller, and run faster.

I'm actually very new to testing in general, the above is what I've been able to make work so far.

• Nice to see that someone actually uses the Testing Framework. Seems it'll be worth it to put in the effort to get over the activation barrier to actually start using it. – b3m2a1 Apr 13 at 0:18
• I often need to have tests parametrized. Can you think about something better than Block[{envVar = 1}, TestReport...]? – Kuba Apr 13 at 9:16
• @Jason B. Thank you. So you don't mind to have large notebooks in version control system? Did you know that you can pass notebooks to TestReport? It seems that .wlt files then become kind of redundant in your workflow? – Pinti Apr 13 at 9:50
• @Pinti in some sense, yes, but I'd actually more call them your IDE, while the true source could just as well be considered to be the WLT files. I do this for my own package development--I have a set of notebooks I develop in, but I only ever put the .m files that I export from them under VC. It actually works quite nicely. – b3m2a1 Apr 16 at 6:39
• @M.R. I think it should be pretty easy to combine them into a Dataset for presentation, these report objects are probably just inert wrappers on Associations. – Jason B. May 21 at 1:19

I think @Jason B.'s answer is a good workflow. Here is an alternative that I have adopted over the years. Most of my code I write in *.m / *.mt / *.wl / *.wlt files using IntelliJ Idea with the Mathematica Plugin, so version control is straightforward.

My projects are written as paclets (see How to distribute Mathematica packages as paclets? for an introduction). Paclets allow you to specify resource files in the PacletInfo.m file. For every (major) function, I write a test-file <function-name>.wlt or <function-name>.mt and add it to the Paclet. Here is an example from my Multiplets paclet:

Paclet[
Name -> "Multiplets",
Version -> "0.1",
MathematicaVersion -> "11.0+",
Description -> "Defines multiplets and Young tableaux.",
Thumbnail -> "multiplets.png",
Creator -> "Johannes E. M. Mosig",
Extensions -> {
{"Kernel", Root -> ".", Context -> "Multiplets"},
{"Resource", Root -> "Testing", Resources -> {
"MultipletDimension.mt",
"MultipletQ.mt",
"MultipletReduce.mt",
"Tableau.mt",
"TableauAppend.mt",
"TableauClear.mt",
"TableauDimension.mt",
"TableauFirst.mt",
"TableauFromMultiplet.mt",
"TableauQ.mt",
"TableauRest.mt",
"TableauSimplify.mt",
"TableauToMatrix.mt",
"TableauToMultiplet.mt"
}
}
}
]


In Mathematica, I can then load my paclet

<< Multiplets


and check if any given function works as expected

TestReport@PacletResource["Multiplets", "MultipletReduce.mt"]


I can also open the test file within Mathematica, using

NotebookOpen@PacletResource["Multiplets", "MultipletReduce.mt"]


The front end opens *.mt and *.wlt files as plain text files, however, so if the notebook interface is important to you, then you may want to safe them as *.wl files instead.

In addition, one may add a test-script that just calls all other test scripts, for convenience.

After I have spent some time developing my package MeshTools and experimenting with testing workflows I am coming back to report my experience.

I have put tests (VerificationTest) in a .wl file and this currently satisfies my requirements. I can conveniently edit it in the FrontEnd (see figure below), code is saved in InputForm rather than FullForm and can be easily read in other text editors. Version Control Systems (Git) have no problem with automatic merging and diffs.

During development I regularly run this test file from another development notebook and inspect the results.

(* $workingDir is package development root directory *) Get["MeshTools.wl", Path ->$workingDir]
tr = TestReport[FileNameJoin[{\$workingDir, "Tests", "Tests.wl"}]]


These 2 helper function give me a clear overview of the most important information about tests and highlight any failed tests.

getTestResults[tr_TestReportObject]:=Module[
{fields,results},
(* Add other querries to this list. *)
fields={"TestIndex","Outcome","TestID","AbsoluteTimeUsed","MemoryUsed"};
results=Outer[#1[#2]&,Values[tr["TestResults"]],fields];
results=ReplaceAll[results,x_Quantity:>QuantityMagnitude[x]];
Join[{fields},results]
]

printTestResults[list_]:=With[{
indx=MapIndexed[If[MemberQ[#1,"Failure"],First@#2,Nothing]&,list]
},
Grid[list,
Alignment->Left,
Dividers->{None,{2->True}},
]
]

printTestResults@getTestResults[tr]


Integration with Git

I have included automatic (unit) testing before every Git commit via hooks. Besides the Tests.wl file the testing folder contains RunTests.wls script with the following content. If any test fails, then script returns value 1 and commit is rejected.

#!/usr/bin/env wolframscript
(* ::Package:: *)

Print["  Running unit tests..."];
(* This script should be run from Git repository root directory. *)
Get["MeshTools.wl",Path->Directory[]];

With[{
tr=TestReport[FileNameJoin[{Directory[],"Tests","Tests.wl"}]]
},
Exit@If[
TrueQ[tr["AllTestsSucceeded"]],
Print["  All tests passed!"];
0, (* exit code if all test pass *)
Print["  Tests failed! ID: ",tr["TestsFailedIndices"]];
1 (* exit code if any test fail *)
]
]


In pre-commit file I have put only the call to .wls script and test are automatically run before every commit.

#!/bin/sh
./Tests/RunTests.wls


In my experience this works well if you keep your unit tests as short as possible (e.g each test <0.1 second) otherwise you may lose patience waiting at each commit. It has already saved me a few times when I was not paying enough attention and some tests have unexpectedly failed.

• I really tried to use testing notebooks but they are slow, limited and bugg :( So this is what I do too atm. Anyway, I create a handy template available in v0.8.0 here: mathematica.stackexchange.com/a/164654/5478 Maybe you are interested. – Kuba Jul 6 at 13:59
• @Kuba I understand your frustration regarding testing notebooks. Looks nice, works so-so. :) Thank you for the link, I have not yet tried your package, but I will do it sometimes soon. – Pinti Jul 6 at 15:10