14
$\begingroup$

I am looking to do some automated analysis on packages, e.g. automatically check for common mistakes. Mathematica makes it relatively easy to manipulate code as data, and Import can get the expressions from a package, so I was hoping that this would be possible.

The biggest obstacle so far is that having the proper context for each symbol is often critical, and Import doesn't handle this.

Is there an existing solution that can infer the correct context of each symbol when importing a package?

Example:

code = "
 BeginPackage[\"Foo`\"]
 MyFoo::usage = \"MyFoo[] is a function\";
 Begin[\"`Private`\"]

 foo[] := 3 (* helper function *)
 MyFoo[] := foo[]^2 (* my function *)

 End[]
 EndPackage[]
 ";

ImportString[code, "HeldExpressions"]
{HoldComplete[BeginPackage["Foo`"]], 
 HoldComplete[MyFoo::usage = "MyFoo[] is a function";], 
 HoldComplete[Begin["`Private`"]],
 HoldComplete[foo[] := 3], 
 HoldComplete[MyFoo[] := foo[]^2],
 HoldComplete[End[]], 
 HoldComplete[EndPackage[]]}

I want HoldComplete[Foo`MyFoo::usage = "MyFoo[] is a function";], HoldComplete[Foo`Private`foo[] := 3], etc.

I know that creating something that always works is plainly impossible without rewriting the evaluator itself. What I am hoping for is something that works well enough to be practically useful and handles most common package structures. And something that hopefully already exists.

$\endgroup$
  • $\begingroup$ If you want your analysis to be completely static, you need to have all the source files for dependencies and their dependencies available as well. Is that the case for your indented use cases of this? $\endgroup$ – Leonid Shifrin Aug 23 '16 at 18:00
  • $\begingroup$ @Leonid Yes, but as a start I would be content with a single-file solution. Also, the analysis doesn't need to be completely static in the sense that if there is a way to record the expressions from the package while the package is being loaded, that is fine, even preferable. But I don't think it's possible. Is there some undocumented way to hook into the parser? Also, I realize that such a tool must to some extent be influenced by the current environment and what symbols already exists in contexts in the context path. $\endgroup$ – Szabolcs Aug 24 '16 at 7:57
  • $\begingroup$ @Szabolcs so at the end it can load the package except of importing it? $\endgroup$ – Kuba Aug 24 '16 at 8:08
16
+300
$\begingroup$
code = "
  BeginPackage[\"Foo`\"]
  MyFoo::usage = \"MyFoo[] is a function\";
  Begin[\"`Private`\"]

  foo[] := 3 (* helper function *)
  MyFoo[] := foo[]^2 (* my function *)
 bar[]:=DynamicModule[{x},{Dynamic[x],x}];

  End[]
  EndPackage[]
  "

Added a line with DynamicModule to show obstacles.

  1. Printing with full names:

    • As Boxes

      We can use FullForm with context for each symbol? to show contexts in read expressions representations.

      But it will not prevent expressions with FormatValues (MakeBoxes upvalues) from being applied, e.g.

      Block[{$ContextPath = {}, $Context = "System`"}, 
       RawBoxes@ToBoxes@Dynamic[x]
      ]
      
      x  (*DynamicBox[...]*)
      

      There is DynamicBox inside but one would expect Dynamic[Global`x] anyway.

      So let's use code from Unchanged Box form of an arbitrary expression

       ToInputFormBoxes = MathLink`CallFrontEnd[
         FrontEnd`UndocumentedTestFEParserPacket[#, True]
       ][[1, 1]] &;
      
       withContext = Block[{$ContextPath = {}, $Context = "System`"}, 
         RawBoxes @ ToInputFormBoxes @ ToString[ #, InputForm ]
       ] &;
      
       withContext[Dynamic[x]]
      
       Dynamic[Global`x]
      
    • As a String

      Alternatively, one just use ToString and not care but I prefer boxes as syntax highlighting is respected, etc.

      Block[{$ContextPath = {}, $Context = "System`"}, 
        ToString @ Dynamic[x]
      ]
      

      "Dynamic[Global`x]"

  2. Reading package step by step

    Now we are ready we can read expression by expression and return formatted output. The key is an ability of Read to do Read[..., Hold[Expression]].

    This example is created for a string but StringToStream can be replaced with OpenRead.

    GetAndRead[code_String] := Module[{str, expr, list},
      str = StringToStream[code];
      list = Reap @ While[
         expr = Read[str, HoldComplete[Expression]]; 
         expr =!= EndOfFile
         ,
         Sow[ Interpretation @@ {withContext[expr], expr} ];
         ReleaseHold[expr];
      ];
      Close[str];
      list[[-1, 1]]
    ]
    
    GetAndRead[code] // Column
    

    enter image description here

$\endgroup$
  • 5
    $\begingroup$ I did not know Read[..., Hold[Expression]]. This is really the key to the solution. $\endgroup$ – Szabolcs Aug 24 '16 at 11:05
  • $\begingroup$ This is much better than what I expected to be possible. Do you foresee any problems? $\endgroup$ – Szabolcs Aug 24 '16 at 11:10
  • 3
    $\begingroup$ Read[..., HoldComplete[Expression]] is safer. $\endgroup$ – Szabolcs Aug 24 '16 at 11:12
  • 2
    $\begingroup$ Actually, even Read[..., f[Number]] works. f is auto-applied. This is documented as "You can use Read to get objects to insert into any expression structure, not necessarily a list." "List" here refers to the {...} in Read[..., {Number, Number}]. The point is that Read[..., f[Number, Number]] is valid too, and so is Read[..., f[g[Number], h[Number]] or any other combination. I did not get this from that one sentence I quoted above. $\endgroup$ – Szabolcs Aug 24 '16 at 11:48
  • 2
    $\begingroup$ same thing is also used in the MUnit package, which is delivered as source code... $\endgroup$ – Albert Retey Aug 24 '16 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.