# How to find out what files are touched when Get is called?

Is there a way to capture all package files touched whenever a package is called via Get, Needs, Beginpackage or similar?

While FindFile["package"] can point to the first file that is called, the problem is not solved yet. The first file might contain additional Get (or related) calls that could point to files with completely different names at completely different places. These dependencies can only be resolved at runtime.

An idea is hinted in the documentation of Get: whenever Get is called, it sets $InputFileName to the file it reads from. A possible way to capture the temporarily set $InputFileName would be to redefine Get via the Villegas-Gayley wrapper. Unfortunately, it is not possible, as any further Get called from inside a Get won't have access to the outermost layer of the redefined Get. If someone can do it nevertheless, it would be really useful.

Note that in a normal situation, an already loaded package won't be loaded again, so any repeated call would not capture any files. To overcome this, the function must be such that it always calls each dependent package/file. For that, it must not leave (too much) footprint on globals (like $Packages, $ContextPath, etc.).

-

Here I offer the safe version of Get that can be used successively to collect all the source files and contexts of packages without polluting the memory (too much).

## What it does

I have practically reverse-engineered all the necessary functions (Get, Needs, BeginPackage, Begin, EndPackage and End) so that I could inject the monitoring code for introspection. $Context and $ContextPath stacks are correctly set and reset, needs and get behave exactly as their built-in counterparts (at least in my extensive testing). During execution, all contexts created and files encountered (even if not opened by Needs being already in $Package) are collected. At the end, $Context, $ContextPath and $Packages are reset to their original values, and every symbol of every visited context (even private ones) are removed as an attempt to recreate the before-call state of memory as best as possible.

contextJoin[s: {__String}] := StringReplace[StringJoin[#<>""& /@s], "".. -> ""]
packageButton[file_String] :=
Button[FileNameTake@file, NotebookOpen@file, Appearance -> "Palette"];

safeGet[pkg_String, arg___, opts : OptionsPattern[]] := Module[{
bp, ep, begin, end, get, needs, contexts = {}, files = {},
assoc = {}, edges = {}, all, $input, from = "Global", cStack = {$Context}, cpStack = {$ContextPath}}, Block[{$Packages = $Packages,$ContextPath = $ContextPath,$Context = $Context}, Off[General::shdw]; Options[get] = Options@Get; (* TODO: key for encoding... *) get[package_String, key___String, getopts : OptionsPattern[]] := Block[{$Path = $Path, SystemPrivate$InputFileName},
If[FreeQ[$Path, #], AppendTo[$Path, #]] & /@ OptionValue@Path;
SystemPrivate$InputFileName = FindFile@package; AppendTo[edges, from -> SystemPrivate$InputFileName];
files = Union[files, {SystemPrivate$InputFileName}];$input = package;
Block[{from = SystemPrivate$InputFileName, stream, last, temp}, stream = OpenRead[SystemPrivate$InputFileName];
While[(temp = Read[stream, Hold@Expression]) =!= EndOfFile,
last = ReleaseHold@(temp /. {HoldPattern@$Input ->$input})];
Close@stream;
last]];
needs[package_String] := needs[package, FindFile@package];
needs[package_String, file_String] := Module[{res},
If[FreeQ[$ContextPath, package], PrependTo[$ContextPath, package]];
(* Sic! Added irregardless whether package was really loaded or not. *)
If[FreeQ[$Packages, package], PrependTo[$Packages, package];
If[FreeQ[files, file], AppendTo[files, file]];
res = get@file;
If[FreeQ[$Packages, package], Message[Needs::nocont, package]]; (* Sic! Only$Packages are checked but not $ContextPath. *) res ]]; bp[ctx_] := bp[ctx, {}]; bp[ctx_, needed_List] := ( AppendTo[cStack,$Context];
AppendTo[cpStack, $ContextPath];$ContextPath = DeleteDuplicates@Join[{ctx}, needed, {"System"}];
(* DeleteDuplicates keeps order, unlike Union. *)
$Packages = DeleteDuplicates@Prepend[$Packages, ctx];
$Context = ctx; assoc = Union[assoc, {$Context -> packageButton@SystemPrivate$InputFileName}]; contexts = Union[contexts, {$Context}];
needs /@ needed; (* Public import as it should be for BeginPackage *)
ctx
);
begin[ctx_] := (
AppendTo[cStack, $Context];$Context = contextJoin@{$Context, ctx}; assoc = Union[assoc, {$Context -> packageButton@SystemPrivate$InputFileName}]; contexts = Union[contexts, {$Context}];
ctx
);
ep[] := ({$ContextPath, cpStack} = {Last@cpStack, Most@cpStack}; {$Context, cStack} = {Last@cStack, Most@cStack};);
end[] := ({$Context, cStack} = {Last@cStack, Most@cStack};); Block[{BeginPackage = bp, EndPackage = ep, Begin = begin, End = end, Needs = needs, Get = get}, Get[pkg, arg]];(* Main call. *) all = # <> "*" & /@ contexts; Unprotect /@ all; Quiet[Remove /@ all]; On[General::shdw]; { "Package" -> pkg, "Contexts" -> contexts, "Files" -> (packageButton /@ files), "Associations" -> (assoc), "Names" -> (Names /@ all), "DependencyGraph" -> Graph[edges, EdgeStyle -> Arrowheads@Medium, GraphLayout -> {"LayeredEmbedding", "RootVertex" -> edges[[1, 1]], LayerSizeFunction -> (.4 &), "LeafDistance" -> 1}, VertexShapeFunction -> (Inset[Style[packageButton@#2, Black], #1, #3] &), ImageSize -> 1000] } ]];  It returns the following things: 1. The package name 2. All the contexts (even private ones) created during the call 3. All the files visited during the call 4. An association list of contexts -> files, so that later any symbol (i.e. their source file) can be easily tracked. 5. A dependency graph of the related package files. Let's try it with a complex package call: safeGet@"OpenCLLink"   {"Package" -> "OpenCLLink", "Contexts" -> {"CCompilerDriver", "CCompilerDriverCCompilerDriverBase", ... "LibraryLink", "LibraryLinkPrivate", "OpenCLLink", "OpenCLLinkPrivate", "SymbolicC", "SymbolicCPrivate"}, "Files" -> {"CCompilerDriverBase.m", ..., "WGLPrivate.m"}, "Names" -> {{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {"LibraryLink$LibraryError"}, {}, {}, {}, {}, {}},
"DependencyGraph" -> Graph[...]}


The graph is the most informative:

Note that all the contexts touched during the call (stored in contexts) and their symbols are correctly removed, so that at the end, no symbols remain to be listed by Names. The only exception is LibraryLink$LibraryError: it is probably used by some other function/package already loaded. Another example: "DependencyGraph" /. safeGet@"PacletManager"  ## Some details of the code • Files read directly with OpenRead or Import (i.e. omitting Get or Needs) are not captured! • Since $Input has attribute Locked, it cannot be modified locally like $InputFileName, so the Hold/ReleaseHold is required to make the replacement structurally by hand. $Input is used e.g. in the PacletManagerPacletManager.m file; $InputFileName is used extensively in many packages (e.g. CUDALink). • Iterative reading is necessary (ReadList won't work) because any context called with BeginPackage must be created before context-symbols are parsed, otherwise new symbols would be created in the Global context. • A list of non-safe functions can be suppressed to minimize the footprint the package makes by adding them to the replacement list where $Input is replaced inside get (e.g. Print|Save|... -> Hold).
• Options Method and CharacterEncoding of Get are not implemented, but I did not find any case where they are actually used.
• No error messages when an invalid context is called (BeginPackage::cxt and Begin::cxt).
• Symbols created (and protected) by the package directly in the Global context during the call cannot be recreated again in a second call, so the standar error will be generated Mathematica not being able to overwrite the symbol (Set::write and SetDelayed::write). These symbols could perhaps be captured and suppressed via the $NewSymbol. - This looks nice! Could you explain the last bullet point (I can't try your code now to investigate myself)? I think I understand what you're trying to say, but I'm not sure where it differs from the normal Get... if the developer has an Unprotect at the beginning of their code, then shouldn't that solve problems? – R. M. Feb 17 '14 at 18:23 @rm-rf I guess it does not differ from Get, but my original aim was to make safeGet as safe as possible, by even preventing such modifications that are directed toward other contexts, unwanted Save[...] or anything that could be harmful when the user assumes that by calling safeGet, the state of the memory does not change (too much). – István Zachar Feb 17 '14 at 18:56 +1 for the hard work. I did similar things in the past, and will later look for my code and may be post it too, if it is not too similar to yours. Too bad I have zero time now to study your code in detail, but I will revisit this later. – Leonid Shifrin Feb 17 '14 at 20:00 Thanks @Leonid, I would appreciate any testing or insigth. I think I owe you a lot, many details I gathered from here-and-there that originate from you. – István Zachar Feb 24 '14 at 17:01 @IstvánZachar As soon as I get some time on my hands, I will definitely have a closer look. I do have some similar developments, we could take the best parts from both (assuming that your code leave any room for improvements), and set up a project, e.g. on GitHub. I think this may potentially be very useful, but I have no idea when I get enough time to work on this. I will surely keep this in mind, though. – Leonid Shifrin Feb 24 '14 at 20:16 This is not a complete answer, but here I will suggest a somewhat different way to extract the dependencies, than in Istvan's answer. I think my method is somewhat more economical. The idea is to load the package of interest in a dynamic environment where $Packages variable is reset to {} initially. This will automatically prompt all packages to be loaded afresh, even those which have been loaded already.

Here is the code for such an environment (formatting done using the code formatting palette):

ClearAll[makeCustomLoadingEnvironment];
Module[{seen,depth},
Function[
code,
Block[{$Packages={},seen,depth=0}, seen[___]:= False; InternalInheritedBlock[ {Get}, Unprotect[Get]; call:Get[args___]/;!seen[args]:= Module[{}, seen[args]=True; depth++; argF[{$InputFileName,depth},args];
call;
depth--;
];
Protect[Get];
code
]
],
HoldAll
]
];


This constructs the dynamic environment where Get is overloaded, such that an arbitrary function gets applied to its arguments prior to the actual call. Note that since Get is called recursively, the standard Villegas - Gayley technique is not enough, and we actually need a cached of already called argument lists.

Here is a function which actually extracts the dependencies:

ClearAll[getDependencies];
getDependencies[context_String]:=
Module[{tag,env},
(If[#1==={},{},First[#1]]&)[Reap[env[Get[context]],tag,#2&][[2]]]
];


when actually called as, for example,

getDependencies["OpenCLLink"] // Short[#, 6] &


it returns something like:, where each sublist has this structure

{{filename, depth}, {dependencies}}


and each element of dependencies sequence can be either a file name or a context name.

From this result, one can reconstruct the dependency tree / graph. I still have a few glitches in my code which does that, so will post it when I fix those. But my main point here was in any case the method described above, while the dependency graph extraction (or other means of transforming the list of dependencies above to some convenient data structure) is a somewhat separate topic.

-
Nice, I knew you would be able to tackle the recursive VG! Concerning the empty \$Packages at start, I deliberately decided not to do that in my code (though can be easily done in the Block`)..... but I forgot why I did so. I'm already in chat, so if you have the time, we might continue there (the room for Szabolcs & me). – István Zachar Feb 25 '14 at 17:46
@IstvánZachar Thanks. But what is that room? Could you give a link? – Leonid Shifrin Feb 25 '14 at 17:48
– István Zachar Feb 25 '14 at 17:50
@IstvánZachar I've left some notes on that module system stuff for you in the chat room. – Leonid Shifrin Feb 25 '14 at 20:27