What is the most convenient way to read definitions of in-memory symbols when we don't have the source files?

Question

Note: I put Simon's implementation on GitHub. Contributions welcome!

---

When trying to read the definition of already defined (package or built-in) symbols using Information or FullDefinition, the biggest inconvenience is that lots of distracting private context names appear in front of all symbol names.

Currently I am using a little function contextFreeDefinition[] to avoid this problem. It will attempt to hide the most frequently appearing context name in the definition. contextFreeDefinition[] is based on this answer.

Compare for example ClearAttributes[RunThrough, ReadProtected]; Information[RunThrough] and contextFreeDefinition[RunThrough]. The latter is a lot less cluttered because the System`Dump` context is hidden in the definition. (I usually paste the output of this function into Workbench and re-indent it using the Source -> Format context menu item for better readability)

---

Unfortunately contextFreeDefinition[] does not always successfully hide contexts, for example try the following:

ImportString["1", "List"]; (* force Stub symbols to be loaded *)

System`Convert`TableDump`ImportList // contextFreeDefinition

and notice that several symbols (especially patterns) still have System`Convert`TableDump` prepended. For example, I see the following in the FullDefinition it prints:

protectRegEx[System`Convert`TableDump`s_String] := 
     StringReplace[System`Convert`TableDump`s, $ProtectedCharacterRules]

The symbol System`Convert`TableDump`s still has the context name prepended even though the function tried to hide exactly this context.

Question: How can contextFreeDefinition[] be fixed so it always hides the context, or what other alternative approaches are there to read the definitions of in-memory symbols?

---

The code of contextFreeDefinition[].

Clear[commonestContexts, contextFreeDefinition]

commonestContexts[sym_Symbol, n_: 1] := Quiet[
  Commonest[
   Cases[Level[DownValues[sym], {-1}, HoldComplete], 
    s_Symbol /; FreeQ[$ContextPath, Context[s]] :> Context[s]], n],
  Commonest::dstlms]

contextFreeDefinition::contexts = "Not showing the following contexts: `1`";

contextFreeDefinition[sym_Symbol, contexts_List] := 
 (If[contexts =!= {}, Message[contextFreeDefinition::contexts, contexts]];
  Internal`InheritedBlock[{sym}, ClearAttributes[sym, ReadProtected];
   Block[{$ContextPath = Join[$ContextPath, contexts]}, 
    Print@InputForm[FullDefinition[sym]]]])

contextFreeDefinition[sym_Symbol, context_String] := 
 contextFreeDefinition[sym, {context}]

contextFreeDefinition[sym_Symbol] := 
 contextFreeDefinition[sym, commonestContexts[sym]]

---

Understanding and using the function:

commonestContexts[sym, n] will find the n most frequently used contexts that are not in $ContextPath in the definition of symbol sym.

contextFreeDefinition[sym] will print the FullDefinition of sym, hiding the commonest context that would appear there. It will also issue a message with the name of the context being hidden.

contextFreeDefinition[sym, {"Context1`", "Context2`", ...}] will try to hide an explicitly given list of contexts.

Answer 1

Link to the code on GitHub

---

I have been using this. It's mostly Leonid's code from the stackoverflow question you linked to, but it uses Definition instead of DownValues. Symbol names are printed without any context, but the full symbol name is put into a Tooltip so you can always find out what context a symbol is in.

Update

FullDefinition[symbol] claims to "print the definitions given for symbol, and all symbols on which these depend", but sometimes one wants to explore deeper than the first level of dependency. Here is a version of Spelunk which uses plain Definition instead of FullDefinition, but allows you to click on symbols in the definition to get their definition. So you can dig right down into the dependency chain.

Update 2

The code now copes with definitions containing strings with backticks in, and cases where Definition throws an error.

Also, it now works for symbols which have OwnValues, e.g. Internal`$VideoEncodings.

BeginPackage["Spelunk`"];

Spelunk::usage = "Spelunk[symbol]";

Begin["`Private`"];

defboxes[symbol_Symbol] := Hold[symbol] /. _[sym_] :>
        If[MemberQ[Attributes[sym], Locked], "Locked",
          Internal`InheritedBlock[{sym},
            Unprotect[sym]; ClearAttributes[sym, ReadProtected];
            Quiet@Check[ToBoxes[Definition@sym], "DefError"] /. 
            InterpretationBox[a_, b___] :> a ]];

defboxes[s_String] := defboxes[#] &@ToExpression[s, InputForm, Unevaluated]

prettyboxes[boxes_] := 
  boxes /. {" "} -> {"\n-----------\n"} //. {RowBox[{left___, ";", 
       next : Except["\n"], right___}] :> 
     RowBox[{left, ";", "\n", "\t", next, right}], 
    RowBox[{sc : ("Block" | "Module" | "With"), "[", 
       RowBox[{vars_, ",", body_}], "]"}] :> 
     RowBox[{sc, "[", RowBox[{vars, ",", "\n\t", body}], "]"}]};

fancydefinition[symbol_Symbol] :=
  Cell[BoxData[
    prettyboxes[
     defboxes[symbol] /. 
      s_String?(StringMatchQ[#, __ ~~ "`" ~~ __] &) :> 
       First@StringCases[s, 
         a : (__ ~~ "`" ~~ b__) :> processsymbol[a, b]]]], "Output", 
   Background -> RGBColor[1, 0.95, 0.9],
   CellGroupingRules->"OutputGrouping",
   GeneratedCell->True,
   CellAutoOverwrite->True,
   ShowAutoStyles->True,
   LanguageCategory->"Mathematica",
   FontWeight->"Bold"
];

processsymbol[a_, b_] := Module[{db},
  Which[
   ! StringFreeQ[a, "\""], a,
   ! StringFreeQ[a, "_"] || (db = defboxes[a]) === "Null", 
   TooltipBox[b, a],
   db === "Locked", TooltipBox[b, a <> "\nLocked Symbol"],
   db === "DefError", TooltipBox[b, a <> "\nError getting Definition"],
   True, ButtonBox[TooltipBox[b, a], ButtonFunction :> Spelunk@a, 
    BaseStyle -> {}, Evaluator -> Automatic]]]

Spelunk[symbol_Symbol] := CellPrint[fancydefinition[symbol]];

Spelunk[s_String] := CellPrint[fancydefinition[#] &@ToExpression[s, InputForm, Unevaluated]];

SetAttributes[{defboxes, fancydefinition, Spelunk}, HoldFirst] 

End[];

EndPackage[];

Answer 2

I have not tested this yet but here is one possible approach:

contextFreeDefinition[sym_Symbol, contexts_List] :=
 Internal`InheritedBlock[{sym},
  ClearAttributes[sym, ReadProtected];
  If[contexts =!= {}, Message[contextFreeDefinition::contexts, contexts]];
  Block[{ipf = ToString @ InputForm @ FullDefinition @ sym},
   ipf = MakeExpression@StringSplit[ipf, "\n"] /. HoldComplete[Null] -> "";
   ipf /. x_Symbol /; MemberQ[contexts, Context[x]] :>
     With[{eval = StringDrop[ToString@Unevaluated@x, StringLength@Context@x]},
         eval /; True]
     /. HoldComplete -> HoldForm // Column
   ]
  ]

(sorry for the messy formatting)

Please point out all failings and I shall see if this is redeemable tomorrow.

Answer 3

I would just use strings, for all their fragility:

ClearAll[print];
print[sym_, {conts_String}] :=
 With[{altptrn =  Alternatives @@ Reverse[SortBy[{conts}, StringLength]]},
    Print@StringReplace[ToString[InputForm@FullDefinition@sym],
         (x : (_ | "") ~~ altptrn ~~ y : (_ | "")) /; ! (x === "\"" && y === "\"") :> 
            StringJoin[x, y]]]

contextFreeDefinition[sym_Symbol, contexts_List] :=
 (
    If[contexts =!= {}, Message[contextFreeDefinition::contexts, contexts]];
    print[sym, contexts]
 );

Note that my code to protect against modifications inside strings is not quite robust. But, if needed, it can be easily made more robust by preprocessing the string.