How to use the Testing Notebook to test a package without shadowing?

Question

I think an important purpose of the new Testing Notebook in version 10 should be to test user packages (instead to test the builtin functions in the System context). However, what's the proper way to load a package into the testing notebook? I didn't find an example in documentation.

What I have tried is to simply load it as an input. But it doesn't seem to work. For example, I have a file test.m in the system directory, with

BeginPackage["test`"]
testFunc
EndPackage[]

Then I start Mathematica, open File->New->Testing Notebook, and input the following as separate input cells:

<< test.m (* also tried Needs["test`"] without difference *)

and

testFunc

If I evaluate each cell by hand, there is no problem. But if I quit the kernel (to start over) and then hit the run button, I sometimes got shadowing messages: testFunc::shdw. By sometimes I mean I did a few tests, each time quit the kernel and start over, and the result is either as left or right of the attached figure

I got really confused here. Also, in the expected output of the right panel, the context becomes explicit. This is quite annoying for a long expression with a long context. I was wondering if it is normal.

Answer 1

From my understanding of what's going on, Mathematica does the following when you hit "Run":

1. Parse the entire test notebook to look for input test cells and get the corresponding cell ids.
2. Run the tests and collect the outcomes (using the cell ids from step 1).
3. Generate test stats (total tests run, successes, failures) and provide links to jump to the next failed test (using the cell ids from Step 1).

Step 1 is the cause for the problem you see, because symbols are generated in the appropriate context at parse time, not when they're evaluated. In this case, the current context at parse time is Global` not test `. To see what I mean, try running the following test notebook:

It will fail at the 4th test. As a workaround, you can do "Evaluate Notebook" instead of "Run" — trying this on the above notebook (after quitting) will fail the 3rd test, but pass the 4th as we want. A major drawback is that you don't get the test statistics and cannot quickly jump to failure points, which can be useful for large test suites.

As a matter of good practice and working within the limitations of the testing framework, I would suggest using the full context for your package symbols in the test cells so that there is no ambiguity.

Answer 2

What R.M. answer means is that Get as a part of the test will not help. The reason is that the content is parsed before Get is evaluated so needed contexts are not present yet.

Additionally the test is run inside Block[{$Context},...] so manual Get before hitting Run won't fix that either because Block leaves only System` and Global` visible during content parsing.

---

• General solution

  Rule of thumb

  First of all, you should think about running a Test Notebook as an evaluation of a single expression, as opposed to evaluation of a list of cells in a notebook.

  It is so because the whole test notebook is parsed at once before the test is evaluated. Would be nice to have that mentioned in documentation as this makes a major difference.

  Setup

  So we need at least two test notebooks.

  • package loading test
  • functions testing notebook|s

  
  And to make it work we have to be sure that after running the first one we will have MyPackage` available on $ContextPath so that the "whatever test" notebook contents will be parsed with knowledge about MyPackage` symbols.

  To achieve that we need to run the "load test" outside the Block[{$ContextPath}... which is what Run button currently does. We also need to strip it from the latter test.

  Here is a simple function which will fix that if you evaluate TestNotebookUpdate[] in both those notebooks:

TestNotebookUpdate[] :=TestNotebookUpdate[EvaluationNotebook[]];

TestNotebookUpdate[nb_NotebookObject] := 
  (
    CurrentValue[nb, DockedCells] = Replace[
        CurrentValue[nb, DockedCells]
      , ButtonBox[
            lbl_?(Not@*FreeQ["Run-Label"])
          , opts1___
          , ButtonFunction :> _
          , opts2___
        ] :> ButtonBox[
            lbl /. DynamicBox[_["MUnitStrings", "Run-Label"], ___] ->   "Run_unblocked"
          , opts1
          , ButtonFunction :> (Block[{$ContextPath}
              , Needs["MUnit`"]]; MUnit`PaletteRun[InputNotebook[]]
              )
          , opts2
        ],
    Infinity
]);

Answer 3

This still seems to be an issue in 2023. I use the following workaround.

1. I load the package I want to test.
2. Instead of using the RunAll button, I highlight all test cells and use shift-Enter.

In this way one does not need to think about ContextPath and one does obtain the test statistics.

Highlighting all cells is easy using a section header.