36
$\begingroup$

I think an important purpose of the new Testing Notebook in version 10 should be to test user packages (instead to test the builtin functions in the System context). However, what's the proper way to load a package into the testing notebook? I didn't find an example in documentation.

What I have tried is to simply load it as an input. But it doesn't seem to work. For example, I have a file test.m in the system directory, with

BeginPackage["test`"]
testFunc
EndPackage[]

Then I start Mathematica, open File->New->Testing Notebook, and input the following as separate input cells:

<< test.m (* also tried Needs["test`"] without difference *)

and

testFunc

If I evaluate each cell by hand, there is no problem. But if I quit the kernel (to start over) and then hit the run button, I sometimes got shadowing messages: testFunc::shdw. By sometimes I mean I did a few tests, each time quit the kernel and start over, and the result is either as left or right of the attached figure

enter image description here

I got really confused here. Also, in the expected output of the right panel, the context becomes explicit. This is quite annoying for a long expression with a long context. I was wondering if it is normal.

$\endgroup$
  • $\begingroup$ How about putting the "<< test.m" in a regular initialization cell? Then you just need to remember to "Evaluate Initialization Cells" before pressing the run button. Unfortunately Mathematica does not ask "do you want to run initialization cells first?" when you press the run button. $\endgroup$ – Gustavo Delfino Jan 19 '15 at 9:35
21
$\begingroup$

From my understanding of what's going on, Mathematica does the following when you hit "Run":

  1. Parse the entire test notebook to look for input test cells and get the corresponding cell ids.
  2. Run the tests and collect the outcomes (using the cell ids from step 1).
  3. Generate test stats (total tests run, successes, failures) and provide links to jump to the next failed test (using the cell ids from Step 1).

Step 1 is the cause for the problem you see, because symbols are generated in the appropriate context at parse time, not when they're evaluated. In this case, the current context at parse time is Global` not test `. To see what I mean, try running the following test notebook:

enter image description here

It will fail at the 4th test. As a workaround, you can do "Evaluate Notebook" instead of "Run" — trying this on the above notebook (after quitting) will fail the 3rd test, but pass the 4th as we want. A major drawback is that you don't get the test statistics and cannot quickly jump to failure points, which can be useful for large test suites.

As a matter of good practice and working within the limitations of the testing framework, I would suggest using the full context for your package symbols in the test cells so that there is no ambiguity.

$\endgroup$
  • $\begingroup$ I've haven't been able to get the Testing Notebook to work satisfactorily for packages. While you can work around the Context by manually setting the ContextPath inside a test, which changes it for all tests, I haven't been able to get around the Testing Notebook failing on Print[] even when Print[] is in a comment, i.e. (* Print["wth"] *) will prevent the test runner from running. That is, it seems if you have a Print[] anywhere in a package or the test nb it seems to cause the runner in a Testing Notebook to fail. $\endgroup$ – pjc42 Aug 21 '14 at 21:44
  • 1
    $\begingroup$ @pjc42 Evaluate this before you click Run: Block[{$ContextPath}, Needs["MUnit`"]; MUnit`$formCheck = False;] $\endgroup$ – Kuba Jan 31 '17 at 8:03
13
$\begingroup$

What R.M. answer means is that Get as a part of the test will not help. The reason is that the content is parsed before Get is evaluated so needed contexts are not present yet.

Additionally the test is run inside Block[{$Context},...] so manual Get before hitting Run won't fix that either because Block leaves only System` and Global` visible during content parsing.


  • General solution

    Rule of thumb

    First of all, you should think about running a Test Notebook as an evaluation of a single expression, as opposed to evaluation of a list of cells in a notebook.

    It is so because the whole test notebook is parsed at once before the test is evaluated. Would be nice to have that mentioned in documentation as this makes a major difference.

    Setup

    So we need at least two test notebooks.

    • package loading test
    • functions testing notebook|s


    And to make it work we have to be sure that after running the first one we will have MyPackage` available on $ContextPath so that the "whatever test" notebook contents will be parsed with knowledge about MyPackage` symbols.

    To achieve that we need to run the "load test" outside the Block[{$ContextPath}... which is what Run button currently does. We also need to strip it from the latter test.

    Here is a simple function which will fix that if you evaluate TestNotebookUpdate[] in both those notebooks:

TestNotebookUpdate[] :=TestNotebookUpdate[EvaluationNotebook[]];

TestNotebookUpdate[nb_NotebookObject] := 
  (
    CurrentValue[nb, DockedCells] = Replace[
        CurrentValue[nb, DockedCells]
      , ButtonBox[
            lbl_?(Not@*FreeQ["Run-Label"])
          , opts1___
          , ButtonFunction :> _
          , opts2___
        ] :> ButtonBox[
            lbl /. DynamicBox[_["MUnitStrings", "Run-Label"], ___] ->   "Run_unblocked"
          , opts1
          , ButtonFunction :> (Block[{$ContextPath}
              , Needs["MUnit`"]]; MUnit`PaletteRun[InputNotebook[]]
              )
          , opts2
        ],
    Infinity
]);

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.