Import package with correct symbol contexts

Question

I am looking to do some automated analysis on packages, e.g. automatically check for common mistakes. Mathematica makes it relatively easy to manipulate code as data, and Import can get the expressions from a package, so I was hoping that this would be possible.

The biggest obstacle so far is that having the proper context for each symbol is often critical, and Import doesn't handle this.

Is there an existing solution that can infer the correct context of each symbol when importing a package?

Example:

code = "
 BeginPackage[\"Foo`\"]
 MyFoo::usage = \"MyFoo[] is a function\";
 Begin[\"`Private`\"]

 foo[] := 3 (* helper function *)
 MyFoo[] := foo[]^2 (* my function *)

 End[]
 EndPackage[]
 ";

ImportString[code, "HeldExpressions"]

{HoldComplete[BeginPackage["Foo`"]], 
 HoldComplete[MyFoo::usage = "MyFoo[] is a function";], 
 HoldComplete[Begin["`Private`"]],
 HoldComplete[foo[] := 3], 
 HoldComplete[MyFoo[] := foo[]^2],
 HoldComplete[End[]], 
 HoldComplete[EndPackage[]]}

I want HoldComplete[Foo`MyFoo::usage = "MyFoo[] is a function";], HoldComplete[Foo`Private`foo[] := 3], etc.

I know that creating something that always works is plainly impossible without rewriting the evaluator itself. What I am hoping for is something that works well enough to be practically useful and handles most common package structures. And something that hopefully already exists.

Answer 1

code = "
  BeginPackage[\"Foo`\"]
  MyFoo::usage = \"MyFoo[] is a function\";
  Begin[\"`Private`\"]

  foo[] := 3 (* helper function *)
  MyFoo[] := foo[]^2 (* my function *)
 bar[]:=DynamicModule[{x},{Dynamic[x],x}];

  End[]
  EndPackage[]
  "

Added a line with DynamicModule to show obstacles.

1. Printing with full names:

   • As Boxes

     We can use FullForm with context for each symbol? to show contexts in read expressions representations.

     But it will not prevent expressions with FormatValues (MakeBoxes upvalues) from being applied, e.g.

     Block[{$ContextPath = {}, $Context = "System`"}, 
      RawBoxes@ToBoxes@Dynamic[x]
     ]
     

     x  (*DynamicBox[...]*)
     

     There is DynamicBox inside but one would expect Dynamic[Global`x] anyway.

     So let's use code from Unchanged Box form of an arbitrary expression

      ToInputFormBoxes = MathLink`CallFrontEnd[
        FrontEnd`UndocumentedTestFEParserPacket[#, True]
      ][[1, 1]] &;
     
      withContext = Block[{$ContextPath = {}, $Context = "System`"}, 
        RawBoxes @ ToInputFormBoxes @ ToString[ #, InputForm ]
      ] &;
     
      withContext[Dynamic[x]]
     

      Dynamic[Global`x]
     

   • As a String

     Alternatively, one just use ToString and not care but I prefer boxes as syntax highlighting is respected, etc.

     Block[{$ContextPath = {}, $Context = "System`"}, 
       ToString @ Dynamic[x]
     ]
     

     "Dynamic[Global`x]"

2. Reading package step by step

   Now we are ready we can read expression by expression and return formatted output. The key is an ability of Read to do Read[..., Hold[Expression]].

   This example is created for a string but StringToStream can be replaced with OpenRead.

   GetAndRead[code_String] := Module[{str, expr, list},
     str = StringToStream[code];
     list = Reap @ While[
        expr = Read[str, HoldComplete[Expression]]; 
        expr =!= EndOfFile
        ,
        Sow[ Interpretation @@ {withContext[expr], expr} ];
        ReleaseHold[expr];
     ];
     Close[str];
     list[[-1, 1]]
   ]
   
   GetAndRead[code] // Column