Automated testing for compatibility with older Mathematica versions

Question

I have several packages which I actively develop and maintain. I try to stay up to date with new releases of Mathematica and usually update within a couple of months of a new version coming out. As a result, I can always be sure that my packages work with the latest version of Mathematica. However, not all users of my packages will want or even be able to update when new releases come out. I would like to support these users, but doing so is difficult when all my development and testing happens with only the latest version.

Is there an automated way of checking which version of Mathematica is required by my packages? In particular, I would like to know if I am using functionality which was only introduced in a specific version of Mathematica. Some specific examples I have encountered are:

• DeleteDuplicates only appeared in version 7.0
• SeriesCoefficient had a change in behaviour in some cases in version 7.0.
• Wavelet support was added in version 8.0

I am aware that the documentation usually says when a feature is new in a specific version, but some of my packages are quite large and it would not be feasible to manually check the reference pages for each function which is used. The NewInXXAlphabeticalListing guides would be less effort to check, but I would prefer an automated solution.

Once I know which version is required I can add a check against $VersionNumber when my package is loaded and output an warning if the version being used is to old.

Answer 1

The only reliable way seems to have a good set of unit test suites, and run them in earlier versions of Mathematica (I mention this here since the answer and comments mentioning this were deleted). However, having explicit rules for when functions were introduced and / or last changed, extracted from the docs, seems to me a good thing, which may help reduce some work, give hints, etc. So, in addition to the suggestions in other answers / comments (I particularly support the unit testing suggestion), the following code can be executed to extract the versioning information from the documentation:

ClearAll[getVersionSince];
getVersionSince::fail = "Unable to extract version information for function `1`";
getVersionSince[file_String?FileExistsQ] :=
  With[{nb = NotebookOpen[file, Visible -> False]},
    With[{res = 
       Cases[
         NotebookGet[nb],
         Cell[s_String, "History", ___]:>
          StringCases[StringTrim@s,
           {"New in" ~~ (Whitespace | "") ~~ d : ((DigitCharacter | ".") ..) ~~ __ ~~  
             "Last modified in" ~~ (Whitespace | "") ~~  m : (((DigitCharacter | ".") ..)) :>
                 {d, m},           
            "New in" ~~ (Whitespace | "") ~~ d : ((DigitCharacter | ".") ..) :> d}
         ],
         Infinity]},
      NotebookClose[nb];
     First@res /; res =!= {}] /; nb =!= $Failed];

getVersionSince[file_String?FileExistsQ] :=
   (Message[getVersionSince::fail, Style[FileBaseName[file], Red]]; $Failed);

Here is the setup I used:

$docdir = 
  FileNameJoin[{$InstallationDirectory, "Documentation", "English", 
      "System", "ReferencePages", "Symbols"}];
$functions = FileNames["*.nb", {$docdir}];

To produce the rules for the functions, you can use something like

rules = Table[FileBaseName[f] -> getVersionSince[f],{f, $functions}]

I actually used

j = 0;
Monitor[
 functionRules = 
   abortableTableAlt[(j++; FileBaseName[f] -> getVersionSince[f]), {f, $functions}], 
 j]

where the abortable table function abortableTableAlt is described at the bottom of this answer. The process was time and memory-consuming, so I saved the result to a file, available from this gist. The result has this format:

{"AbelianGroup" -> {"8"}, "AbortKernels" -> {"7"}, "Abort" -> {"2"}, 
  "AbortProtect" -> {"2"}, "Abs" -> {"1"}, "AbsoluteCurrentValue" -> {"6"}, 
 "AbsoluteDashing" -> {{"2", "6"}}, "AbsoluteFileName" -> {"7"}, << 3400 >>,
 "$UserName" -> {"3"},  "$Version" -> {"1"}, "$VersionNumber" -> {"2"}, "$$Media" -> {"1"}}

Where a single number (string) is the earliest version when the function (or at least the doc.page) was introduced, and the format like {{"2","6"}} means that it was introduced in v.2 and last modified in v.6. In total, 3412 notebooks were processed. The only function for which this code failed was KConnectedComponents.

You can import the file from the mentioned gist, and I'd recommend to use Dispatch to speed up the rule application. Alternatively, you can use Save to attach these rules to whatever function (symbol) you want. As was suggested in another answer, this can be combined with some tool to extract dependencies, which would give you a list of system symbols used in your code. One such tool was developed David Wagner way back in 1996 and desribed in his article in the Mathematica Journal. I developed a similar tool here, but it is a work in progress and contains some bugs as of now.

Answer 2

Here is a way to see what is new or changed relative to an older version. Assuming English documentation,

newSince[oldversion_] := Module[{new},
  new[version_] := 
    StringReplace[Cases[
      Import[FileNameJoin[{$InstallationDirectory, "Documentation", "English", "System", "Guides", "NewIn" <> ToString@version <> "0AlphabeticalListing.nb"}]], 
          x_String /; StringMatchQ[x, "paclet:ref/" ~~ __], \[Infinity]
      ], "paclet:ref/" -> ""];
  Union[Flatten[new /@ Range[oldversion + 1, Round@$VersionNumber]]]
 ]

You could check your notebook source file for occurrences of strings that are members of the string list returned by newSince[]:

Clear[notebookCompatibility]; 
notebookCompatibility[notebookFilename_, oldversion_: 6] := 
  Module[{file, newlist, words},
   file = Import[notebookFilename, "Text"];
   newlist = newSince[oldversion];
   Cases[Union@StringSplit[file, {Whitespace, "\"", "\\"}], 
     x_String /; MemberQ[newlist, x]]
 ]

I ran this on a recent notebook, and got

notebookCompatibility[<path to notebook>]

(*  {"TextRecognize"}  *)

You could remove functions from the list, depending on whether they give you problems in practice or not.