Best practices for using the Testing Framework

Question

I am looking for best practices and recommended workflows for using the Testing Framework. For example, lets say I am developing a Mathematica package for other users, it is hosted on GitHub and I wish to make it easy for other developers to contribute to repository.

The project includes a suite of (unit) tests that I would like to run as often as necessary. I would like to run them within FrontEnd and conveniently inspect the results (timings, which tests failed and why, etc), save them to file and also further automate testing in the future. I can think of few possible workflows but none of them ticks all the boxes.

• "Testing Notebook", Pro: It gives a clear overview of tests and looks nice. It can be run in FrontEnd and also programmatically with TestReport. Con: Contains even more metadata than the normal notebook (all those buttons) so it is less suitable to put under version control. Somehow the evaluation of tests is slower in testing notebook than in normal notebook.

• ".wlt file", Pro: This is the documented way to use plain text files with test and run them programmatically from another notebook or command line. Con: I can't open .wlt files in FrontEnd!? So I have to open them in other text editor without nice features of FrontEnd (autocomplete, code coloring, etc).

• ".wl file", Pro: I can easily edit it in FrontEnd as any other package. Con: This workflow is not documented. Should I put all tests in some specific Context?

So the question is, what is your recommended workflow for testing? While browsing other people's GitHub repositories I have seen some custom made approaches, but I would like rely on something that is documented and easy to explain to other developers who will contribute to the project.

I am also curious what is the purpose of BeginTestSection and EndTestSection in .wlt files?

Answer 1

I think @Jason B.'s answer is a good workflow. Here is an alternative that I have adopted over the years. Most of my code I write in *.m / *.mt / *.wl / *.wlt files using IntelliJ Idea with the Mathematica Plugin, so version control is straightforward.

My projects are written as paclets (see How to distribute Mathematica packages as paclets? for an introduction). Paclets allow you to specify resource files in the PacletInfo.m file. For every (major) function, I write a test-file <function-name>.wlt or <function-name>.mt and add it to the Paclet. Here is an example from my Multiplets paclet:

Paclet[
  Name -> "Multiplets",
  Version -> "0.1",
  MathematicaVersion -> "11.0+",
  Description -> "Defines multiplets and Young tableaux.",
  Thumbnail -> "multiplets.png",
  Creator -> "Johannes E. M. Mosig",
  Extensions -> {
    {"Kernel", Root -> ".", Context -> "Multiplets`"},
    {"Resource", Root -> "Testing", Resources -> {
        "MultipletDimension.mt",
        "MultipletQ.mt",
        "MultipletReduce.mt",
        "Tableau.mt",
        "TableauAppend.mt",
        "TableauClear.mt",
        "TableauDimension.mt",
        "TableauFirst.mt",
        "TableauFromMultiplet.mt",
        "TableauQ.mt",
        "TableauRest.mt",
        "TableauSimplify.mt",
        "TableauToMatrix.mt",
        "TableauToMultiplet.mt"
      }
    }
  }
]

In Mathematica, I can then load my paclet

<< Multiplets`

and check if any given function works as expected

TestReport@PacletResource["Multiplets", "MultipletReduce.mt"]

I can also open the test file within Mathematica, using

NotebookOpen@PacletResource["Multiplets", "MultipletReduce.mt"]

The front end opens *.mt and *.wlt files as plain text files, however, so if the notebook interface is important to you, then you may want to safe them as *.wl files instead.

In addition, one may add a test-script that just calls all other test scripts, for convenience.

Answer 2

Testing notebooks or .wlt test files? I use both. I create the tests in a notebook, but keep a .wlt file for automated test runs.

This is my workflow:

• I start by creating a new "Testing Notebook" from the file menu
• I add in my testing inputs. For each test, I put all inputs in one cell, and return a single expression.
• Select all the cells and use the "Convert Selection" button to create the tests.
• Assign TestIDs. Obviously having meaningful names here is better, but I tend to just use the randomly generated ones.
• Save the testing notebook in the Tests subdirectory of my package. Then select "Save As .wlt" from the menu, save with the same name as the notebook.

Now I have 2 files in the directory for every test, but that's fine. When I make some code changes, before I commit them I just run the command

TestReport /@ FileNames["~/Path_to_package/*.wlt"]

If the result looks like this,

then everything is good. If I get a failure, then I can copy that TestReportObject and query it directly, asking for "TestsFailed", which returns a list of TestResultObject(s), and I can find the failing test right there. In practice, when I get a test failure I tend to just reopen the notebook, and run the tests interactively to find the cause.

Then when I update the tests in the notebook, I just save as a .wlt file, overwriting the old file.

If you have a build script for any libraries, or paclet build, then you can add this line to the script

successFlag = SameQ[
    Total[
        Map[TestReport[#]["TestsFailedCount"]&,
            FileNames @ "~/Path_to_package/*.wlt"
        ]
    ],
    0
];

And then abort the build if successFlag is False.

With this method you aren't really getting around the issues of file duplication, large files in version control. For my purposes, this isn't a big deal since the notebooks I use aren't that large. This just seems like the workflow that the framework is designed for - we have easily editable testing notebooks, and .wlt files that are less readable, much smaller, and run faster.

I'm actually very new to testing in general, the above is what I've been able to make work so far.

Answer 3

After I have spent some time developing my package MeshTools and experimenting with testing workflows I am coming back to report my experience.

I have put tests (VerificationTest) in a .wl file and this currently satisfies my requirements. I can conveniently edit it in the FrontEnd (see figure below), code is saved in InputForm rather than FullForm and can be easily read in other text editors. Version Control Systems (Git) have no problem with automatic merging and diffs.

During development I regularly run this test file from another development notebook and inspect the results.

(* Assuming I have PacletInfo.m in added directory. *)
PacletDirectoryAdd@ParentDirectory[NotebookDirectory[]]

Get["MeshTools`"]
$testReport=TestReport@FileNameJoin[{NotebookDirectory[],"Tests.wl"}]

These 2 helper function give me a clear overview of the most important information about tests and highlight any failed tests.

getTestResults[tr_TestReportObject]:=Module[
  {fields,results,abbreviations},

   (* Add other querries to this list. *)
  fields={"TestIndex","Outcome","AbsoluteTimeUsed","MemoryUsed","TestID"};
  abbreviations={"TestIndex"->"Idx","AbsoluteTimeUsed"->"Time [s]"};

  results=ReplaceRepeated[
    Outer[#1[#2]&,Values[tr["TestResults"]],fields],
    {x_Quantity:>QuantityMagnitude[x],x_Real:>Round[x,0.001]}
  ];

  Join[{fields/.abbreviations},results]
]

printTestResults[tr_TestReportObject]:=Module[
  {list,indx,time,noTests},

  list=getTestResults[tr];
  indx=MapIndexed[
    If[
        MemberQ[#1,"Failure"|"MessagesFailure"|"Error"],
        First[#2],
        Nothing
    ]&,
    list
  ];
  time=Round[QuantityMagnitude[tr["TimeElapsed"]],0.01];
  noTests=Length[tr["TestResults"]];

  Print[noTests," tests run in ",time," seconds."];

  If[
    TrueQ@tr["AllTestsSucceeded"],
    Print["All tests succeeded!"],
    Print[tr["TestsFailedCount"]," tests failed!"];
  ];

  Print@Grid[list,
    Alignment->Left,
    Dividers->{None,{2->True}},
    Background->{None,Thread[indx->Pink]}
  ];
  tr
]



printTestResults[$testReport];

---

Integration with Git

I have included automatic (unit) testing before every Git commit via hooks. Besides the Tests.wl file the testing folder contains RunTests.wls script with the following content. If any test fails, then script returns value 1 and commit is rejected.

#!/usr/bin/env wolframscript
(* ::Package:: *)

Print["  Running \"MeshTools\" tests...","\n"];

(* Modify paths depending from which directory the script is started.
By default we assume script is started from git root directory. *)
PacletDirectoryAdd@Directory[];
Get["MeshTools`"];

Print["  Using Mathematica: ",$VersionNumber];

Module[
    {report,time,results,failIdx},

    report=TestReport[
        FileNameJoin[{Directory[],"Tests","Tests.wl"}]
    ];
    time=Round[
        QuantityMagnitude@report["TimeElapsed"],
        0.001
    ];
    results=report["TestResults"];
    failIdx=report["TestsFailedIndices"];

    Print["\n","  ",Length[results]," tests run in ",time," seconds."];

    If[
        TrueQ@report["AllTestsSucceeded"]
        ,
        Print["  All tests succeeded!"];
        Quit[0] (* exit code if all test pass *)
        ,
        Print["  ",Length[failIdx]," tests failed!"];
        Do[
            Print["  ",i," / ",results[i]["Outcome"]," / ",results[i]["TestID"]],
            {i,failIdx}
        ];
        Quit[1] (* exit code if any test fails *)
    ]
]

In pre-commit file I have put only the call to .wls script and test are automatically run before every commit.

#!/bin/sh
./Tests/RunTests.wls

In my experience this works well if you keep your unit tests as short as possible (e.g each test <0.1 second) otherwise you may lose patience waiting at each commit. It has already saved me a few times when I was not paying enough attention and some tests have unexpectedly failed.

Answer 4

Update with regard to new Paclet Framework

Using Paclets myself I would like to update the answer given by @JEM_Mosig above since with some revisions of the Paclet framework beginning afaik in v12.1 things have changed a bit, e.g., a Paclet now is a PacletObject and its properties are given in an Association.

As of now a simple paclet might have the following PacletInfo.wl file:

PacletObject[
    <| 
        "Name" -> "MyPackage",
        "Version" -> "0.1",
        "WolframVersion" -> "13+",
        "Creator" -> "Bingo S. Little <[email protected]>",
        "Description" -> "What the package will do for you.",
        "Extensions" -> {
           { "Kernel", "Root" -> "Kernel", "Context"-> { "MyPackage`" } },
           { "Asset", "Root" -> "Tests", "Assets" -> { 
                 { "TestDirectory", "." }, (* "/" will also work *)
                 { "TestCoreFunctionality" , "TestCoreFunctionality.wlt"} 
               }
           },
           { "Path" },
           { "Documentation" } 
        }
    |>
]

The file structure accordingly will be:

MyPackage
+---Documentation
+---Kernel
|   +---MyPackage.wl
+---Tests
|   +---TestCoreFunctionality.wlt
+---PacletInfo.wl

If you have not installed the Paclet yet, which is not unlikely should you be writing tests during development, then you need to do:

PacletDirectoryLoad[ "directory" ] (* where to find the paclet *)

Should the Paclet be installed that step can be skipped and we can immediately do:

Needs["MyPackage`"]

A typical test file like TestCoreFunctionality.wlt will be a text file with the following structure:

(* Wolfram Language Test File *)

VerificationTest[
    (* test expression *)
    ,
    (* expected output *)
    ,
    (* messages *)
    {}
    ,
    TestID -> "CoreFunctionality-H3A2Z6-211221_shortTestDescription"
]

Unfortunately, the "Path" extension seems to be a mixed bag:

DirectoryName[ "MyPackage/Tests/*" ]
(* D:\MyPackage\Tests *)

FileNames[ "*.wlt", "MyPackage/Tests" ]
(* {} *)

But working with the PacletObject and AssetLocation works fine:

obj = PacletObject[ "MyPackage" ];
obj["AssetLocation", "TestDirectory" ]
(* D:\MyPackage\Tests *) 

$testFiles = FileNames[ "*.wlt", obj["AssetLocation", "TestDirectory" ] ];
$tests = Dataset @ AssociationThread[ 
             Range[ Length @ $testFiles] -> FileBaseName /@ $testFiles 
         ]

And then we can run our tests:

$testReport = TestReport[ obj["AssetLocation", "TestCoreFunctionality" ]

Answer 5

Edit

The function described below is available in the function repository, and can be used in version 11.3+ as, for example, ResourceFunction["WriteUnitTest"]["~/mytestfile.wl", 1/0 + 3]

Original answer

This answer gives a workflow using the "Testing Notebook" to create test files. I no longer use this workflow as I had encountered some instability with the testing notebooks as well as the fact that they do not deal with symbols outside of the Global and System contexts.

My current workflow is to write unit tests quickly in a normal notebook and have them written in a readable way to a file. I write the tests as soon as I get the functionality how I want it, I write the tests often and try to make them always self-contained.

Here is an example of a test for FindMoleculeSubstructure. In a normal notebook I would evaluate

WriteUnitTest["MytestFile.mt",
 With[{mol = Molecule[Interpreter["Chemical"]["buckyball"]], 
   fivememberedRing = 
    MoleculePattern[{_, _, _, _, _}, 
     Bond /@ Partition[Range[5], 2, 1, 1]]},
  Length[FindMoleculeSubstructure[mol, fivememberedRing, All]]]]

and I am presented with this dialog

when I press the "Yes" button the test is written to file in a very readable form. This is important because later when a test fails I don't want to waste any time staring at the test figuring out what it does.

The code for WriteUnitTest is as follows:

SetAttributes[prettyInputString, HoldAllComplete];
prettyInputString[expr_] := Module[
    {res},
    res = GeneralUtilities`ToPrettyString[Unevaluated[expr]];
    res // StringReplace["\r\n" -> "\n"] // 
        StringReplace["\n" -> "\n\t"] // 
        StringReplace["Global`" -> ""] //
        StringReplace["\\[LeftDoubleBracket]"~~Shortest[part__]~~"\\[RightDoubleBracket]" :> "[["<>part<>"]]"]
]

Attributes[WriteUnitTest] = HoldRest
Options[WriteUnitTest] = {"ConfirmResults" -> True}
WriteUnitTest[filename_String, input_, append_String:"",opts:OptionsPattern[]] := DynamicModule[
    {
        inputString = prettyInputString[input], writeToFile,
        output, outputString, newFile, evaluationData, messages, messageString,
        fileText
    },

    evaluationData = EvaluationData[input];

    output = evaluationData["Result"];
    messages = evaluationData["MessagesExpressions"];


    If[!FileExistsQ[filename],
        newFile = True;
        PrintTemporary["File to be created: ", filename],
        PrintTemporary["File to be modified: ", filename]
    ];



    outputString = With[{eval=output},prettyInputString[eval]];

    (* the testing framework prefers HoldForm here *)
    messageString = StringReplace[
        ToString[messages,InputForm,PageWidth->80],
        "Hold[" -> "HoldForm["
    ];



    writeToFile[] := (
        If[ 
            newFile,
            Export[
                filename,
                "BeginTestSection[\""<>FileBaseName[filename]<>"\"]\n\n\nEndTestSection[]\n",
                "Text"
            ]
        ];

        Block[{id = FileBaseName[filename]},
            id = id <> "_" <> StringJoin[(If[# > 9, "", "0"] <>ToString[#]) & /@ Date[][[1;;3]]] <> "-" <> ToUpperCase[IntegerString[RandomInteger[{0, 36^^1000000}], 36, 6]] <> append;

            Print["Adding test: ", id];
            outputString = StringJoin[
                "VerificationTest[\n\t",
                inputString, 
                "\n\t,\n\t",
                outputString,
                "\n\t,\n\t",
                messageString,
                "\n\t,\n\tTestID->\"" <> id <> "\"\n]\n\n"
            ];
        ];

        fileText = Import[filename, "Text"];
        fileText = If[
            StringContainsQ[
                fileText,
                "EndTestSection[]"
            ]
            ,
            StringReplace[
                fileText,
                "EndTestSection[]" :> outputString <> "\nEndTestSection[]\n"
            ],
            fileText<> "\n" <> outputString
        ];
        Export[filename, fileText, "Text"];
        Print @ filename;

        SelectionMove[ButtonNotebook[], All, ButtonCell]; 
        NotebookDelete[ButtonNotebook[]];
    );

    If[
        OptionValue["ConfirmResults"]
        ,


        Column[
            {
                Row[{"Input: ", inputString}],
                Row[{"Result: ", outputString}],
                Row[{"Messages: ", messageString}],
                Row[
                    {
                        Button[
                            Text[Style["Yes"]],
                            writeToFile[];
                            , 
                            "BaseStyle" -> Darker[Green]
                        ]
                    , 
                    Button[Text[Style["No"]], 
                        Print["Aborting without writing to ", filename];
                        SelectionMove[ButtonNotebook[], All, ButtonCell]; 
                        NotebookDelete[ButtonNotebook[]];
                    , "BaseStyle" -> Darker[Red]]
                    }, 
                    "  "
                ]
            }
        ]

        ,
        writeToFile[];
        filename
    ]
]

It was hacked together, so any suggestions for improving it are most appreciated.