Inserting symbols' definitions during read time of the code

Question

Let's say we've got a package:

data = "
 BeginPackage[\"Test`\"]

 testFunction::usage=\"-\";

 Begin[\"`Private`\"]

 testFunction[]:= {123, explicitvalue, Hold[explicitvalue]};

 End[]
 EndPackage[]
 ";

path = FileNameJoin@{$TemporaryDirectory, "testPackage.m"};

Export[    path,    data,     "Text"]

and let's say

Test`explicitvalue = {1, 1, 1}

is defined before we read it.

New we can get it:

Get[path]

?? testFunction
testFunction[]

testFunction[]:={123,explicitvalue,Hold[explicitvalue]}

{123, {1, 1, 1}, Hold[explicitvalue]}

This is correct and expected but how to construct the behaviour mentioned in the title so that after Get we will have the following definition of testFunction:

testFunction[]:={123,{1, 1, 1},Hold[{1, 1, 1}]}

In context of only the package we could just inject {1, 1, 1} value with With[{explicitvalue = {1, 1, 1}}....

But that's not the case.

Is this possible without modifying the package contents?

Answer 1

We can replace something, in all definitions associated with a symbol, using function like this:

ClearAll[replaceExtendedDefinition]
SetAttributes[replaceExtendedDefinition, HoldFirst]
replaceExtendedDefinition[sym_, rules_] :=
    Replace[
        Language`ExtendedDefinition[sym]
        ,
        (rule:Rule | RuleDelayed)[lhs_, rhs_] :>
            With[
                {eval =
                    With[
                        {heldRhs = 
                            Replace[HoldComplete[rhs], rules, {1, Infinity}]}
                        ,
                        Function[Null, rule[lhs, #], HoldAllComplete] @@ heldRhs
                    ]}
                ,
                eval /; True
            ]
        ,
        {4, 5}
    ]

We can easily use it for all symbols from given context:

Scan[
    (Language`ExtendedDefinition[#] =
        replaceExtendedDefinition[#, HoldPattern@explicitvalue -> explicitvalue])&
    ,
    Names["Test`*"]
]

Now we have:

??testFunction
(* - *)
(* testFunction[] := {123, {1, 1, 1}, Hold[{1, 1, 1}]} *)

replaceExtendedDefinition in current form replaces all occurrences of replaced expressions on right hand side of definitions including those in local scopes, which may not be desirable. Its advantage is that it can replace more complicated expressions, not only symbols.

---

Version using Szabolcs's With trick. It avoids replacements of symbols in local scopes, but does not allow replacements of more complicated expressions.

ClearAll[extendedDefinitionWithReplacement]
SetAttributes[extendedDefinitionWithReplacement, HoldAll]
extendedDefinitionWithReplacement[vars : {(_Set | _SetDelayed) ..}, sym_] :=
    ReplacePart[
        Unevaluated @ With[vars, Null],
        {2} -> Language`ExtendedDefinition[sym]
    ]

Test symbol with all possible ...Values:

ClearAll[test, var]
test::usage = "test is a test symbol.";
SetAttributes[test, Listable];
Format[test] = "TEST" <> ToString[var];
test[var][var_] := var;
test[var_] := var;
test[1, var] := Module[{var}, var];
f[test] ^= var;
N[test] = var;
Default[test] = var;
test = var;

Its extended definition:

Language`ExtendedDefinition[test]
(* Language`DefinitionList[TESTvar -> {
    OwnValues -> HoldPattern[TESTvar] :> var,
    SubValues -> {HoldPattern[TESTvar[var][var_]] : >var},
    UpValues -> {HoldPattern[f[TESTvar]]->var},
    DownValues -> {
        HoldPattern[TESTvar[1, var]] :> Module[{var},var],
        HoldPattern[TESTvar[var_]] :> var},
    NValues -> {HoldPattern[N[TESTvar, {MachinePrecision, MachinePrecision}]] -> var},
    FormatValues -> {
        HoldPattern[TESTvar] :> "TEST" <> ToString[var],
        HoldPattern[MakeBoxes[TESTvar, FormatType_]] :> Format["TEST" <> ToString[var], FormatType]},
    DefaultValues -> {HoldPattern[Default[TESTvar]] -> var},
    Messages->{HoldPattern[TESTvar::usage] -> test is a test symbol.},
    Attributes->{Listable}
}] *)

And definition with replacement:

extendedDefinitionWithReplacement[{var = value}, test]
(* Language`DefinitionList[TESTvar -> {
    OwnValues -> HoldPattern[TESTvar] :> value,
    SubValues -> {HoldPattern[TESTvar[value][var_]] :> var},
    UpValues -> {HoldPattern[f[TESTvar]] -> value},
    DownValues -> {
        HoldPattern[TESTvar[1, value]] :> Module[{var}, var],
        HoldPattern[TESTvar[var_]] :> var
    },
    NValues -> {HoldPattern[N[TESTvar, {MachinePrecision, MachinePrecision}]] -> value},
    FormatValues -> {
        HoldPattern[TESTvar] :> "TEST" <> ToString[value],
        HoldPattern[MakeBoxes[TESTvar, FormatType$_]] :> Format["TEST" <> ToString[value], FormatType$]},
    DefaultValues -> {HoldPattern[Default[TESTvar]] -> value},
    Messages -> {HoldPattern[TESTvar::usage] -> test is a test symbol.},
    Attributes->{Listable}
}] *)

Answer 2

This is messy and imperfect, but it will work in simple cases:

Import the package contents:

pack = ImportString[data, {"Package", "HeldExpressions"}]

(*
{HoldComplete[BeginPackage["Test`"]], 
 HoldComplete[testFunction::usage = "-";], 
 HoldComplete[Begin["`Private`"]], 
 HoldComplete[testFunction[] := {123, explicitvalue, Hold[explicitvalue]};], 
 HoldComplete[End[]], HoldComplete[EndPackage[]]}
*)

Import won't case about the Begin and BeginPackage, so the symbol we get is now Global`explicitvalue, not Test`explicitvalue. This may cause problems if the package refers to more symbols named explicitvalue in multiple contexts, without using a fully qualified name. If it doesn't (as is usually the case), we can simply replace it:

pack = pack /. HoldPattern[explicitvalue] -> {1, 1, 1}

(* {HoldComplete[BeginPackage["Test`"]], 
 HoldComplete[testFunction::usage = "-";], 
 HoldComplete[Begin["`Private`"]], 
 HoldComplete[testFunction[] := {123, {1, 1, 1}, Hold[{1, 1, 1}]};], 
 HoldComplete[End[]], HoldComplete[EndPackage[]]} *)

Now we can re-export the package into a file and load it:

Export[..., pack, {"Package", "HeldExpressions"}]

It is important to load it from a file with Get or Needs if it uses functionality such as $InputFileName (which only work in packages).

---

Of course one big problem is that you might not want to replace every single occurrence of explicitvalue to respect localization in Module and similar functions.

A partial solution to this is using With instead of ReplaceAll.

Compare

expr = Hold[Module[{a = 1}, a^2], a^2, HoldComplete[a], a::usage = "-", f[a_] := a^2]

expr /. a -> 42

(* Hold[Module[{42 = 1}, 42^2], 42^2, HoldComplete[42], 42::"usage" = "-", f[Pattern[42, _]] := 42^2] *)

with

Unevaluated@With[{a = 42}, e] /. e -> expr

(* Hold[Module[{a = 1}, a^2], 1^2, HoldComplete[1], 1::"usage" = "-", f[a_] := a^2] *)

The With avoided replacement in the Module and the function definition, but didn't avoid it in the message definition (which is probably a place where you'd also want to prevent it).