# Making a symbol's new definitions be tried before all previously defined ones

Is there any way to "close" a package (or a symbol, or a context) in that if a user of the package adds definitions to the symbol they will be tried before the package defined ones, just like what happens with built-ins?

-
This is a very important question. This will allow users to wrap package functions. Furthermore, certain "builtins", such as all Parallel* functions, are actually implemented as a package!! –  Szabolcs Mar 14 '12 at 19:37
@Szabolcs, but in those cases their functions don't work "as regular built-ins". Try adding ParallelSubmit[___]:=8 after unprotecting... It gets added at the bottom of the DownValue list and ParallelSubmit[2+2] works "as usual" instead of catching the new definition –  Rojo Mar 14 '12 at 22:27
@Rojo Works with my method for a double blank (although with a slight glitch, an error message is generated - due to a minor bug, but that does not affect the result). Try withUserDefs[ParallelSubmit, {SetAttributes[ParallelSubmit, HoldAllComplete]; ParallelSubmit[__] := 8}, ParallelSubmit[2 + 2]]. Although, tripple blank is the only pattern which my method does not currently cover. –  Leonid Shifrin Mar 15 '12 at 0:09
@LeonidShifrin, your method is great, robust. But It is for cases when you're the user of a package and want to extend it. The other answers are attempts to deal with the case when you are creating the "extensible" package. What I just commented here was that the Parallel* built-ins don't behave as "closed" like other built-ins. You can extend them using your magic, but not right away. So, peeking at how they did it won't help improve those attempts –  Rojo Mar 15 '12 at 0:15
@Rojo I think, Szabolcs didn't mean to peek at how those were written - he just meant that it would be nice to be able to consistently override functions, no matter built-ins or top-level. My mechanism is not perfect, but possesses one important property - it can be nested. Nesting is important, because it gives some level of consistency. –  Leonid Shifrin Mar 15 '12 at 0:24

I will suggest a solution for DownValues - based definitions, but it may be generalized to other types of definitions as well. I will only consider a case of a single symbol, but again, the generalization is rather straightforward. You will also have to execute your code in a special dynamic environment.

A first ingredient of my suggestion is a (slightly modified) symbol-cloning functionality described here:

Clear[GlobalProperties];
GlobalProperties[] :=
{OwnValues, DownValues, SubValues, UpValues,
NValues, FormatValues, Options, DefaultValues, Attributes};

ClearAll[clone];
Attributes[clone] = {HoldAll};
clone[s_Symbol, new_Symbol] :=
With[{clone = new, sopts = Options[Unevaluated[s]]},
With[{setProp = (#[clone] = (#[s] /. HoldPattern[s] :> clone)) &},
Map[setProp, DeleteCases[GlobalProperties[], Options]];
If[sopts =!= {}, Options[clone] = (sopts /. HoldPattern[s] :> clone)];
HoldPattern[s] :> clone]]


Here comes my suggested dynamic environment then:

ClearAll[withUserDefs];
SetAttributes[withUserDefs, HoldAll];
withUserDefs[sym_Symbol, {defs__}, code_] :=
Module[{s, inSym},
clone[sym, s];
Block[{sym},
defs;
sym[args___] /; ! TrueQ[inSym] :=
Block[{sym, inSym = True},
clone[s, sym];
With[{result = sym[args]},
result /; result =!= Unevaluated[sym[args]]
]
];
code]];


What is happening here: first, we clone the original symbol. Then, we Block it, and run the definitions which we want to override the previous ones. Then, we add a catch-all definition which uses the Villegas-Gayley trick, but also Block-s the symbol in question again, and inside the inner Block uses the clone to effectively "unblock" the symbol to its original defs by reverse-cloning it, and run those. The extra trick to use With and a shared local variable result is needed to avoid infinite iteration in cases when neither user-defined nor previous rules apply.

Here comes an example:

ClearAll[f];
f[1] = 10;
f[x_] := x^2;
f[x_, y_] := x + y;


And now:

withUserDefs[f, {f[x_] := x^4}, {f[1], f[2], f[1, 2], f[1, 2, 3]}]

(*
==> {1, 16, 3, f[1, 2, 3]}
*)


you can see that in the first two cases, the modified definitions were used, in the third one the original definition was used, and the last one did not match any and evaluated to itself.

One can nest these constructs, and the inner one will override the outer ones then. For example:

withUserDefs[f, {f[x_] := x^4},
withUserDefs[f, {f[1] := 100},
{f[1], f[2], f[1, 2], f[1, 2, 3]}]]

(*
==> {100, 16, 3, f[1, 2, 3]}
*)


You can ask why I wasn't just using the Villegas-Gayley trick by itself, which is much simpler. The answer is that there is no guarantee that the ordering of definitions will be right with it, even if we manually reorder them, and moreover, cases like f[1] = 10 are immune to all reorderings of DownValues and will always be at the top, so can not be dealt with at all, within a pure Villegas-Gayley approach - but can be successfully dealt with in this more complex one.

The suggested approach is as good as the symbol's cloning procedure is. For SubValues, UpValues etc, it should be modified. The full solution involving all possible global rules and multiple symbols will likely look more complex, but I just wanted to illustrate the idea in the simplest possible setting. Also, we probably can not count on such generalization to be fully robust.

-

If we are the ones writing the package in question, then we could proceed as follows. First, we define a public version of the function that delegates all calls to a private version:

ClearAll[publicFn]
publicFn[args___] := privateFn[args]


Then we define the functionality we desire on the private function:

ClearAll[privateFn]
privateFn[x_] /; x < 0 := 100 x
privateFn[x_] := 10 x


Before users make any modifications, the public function behaves in the way we have defined:

publicFn /@ Range[-5, 5]


{-500,-400,-300,-200,-100,0,10,20,30,40,50}

Now, if a user comes along and adds a definition to the public function...

publicFn[n_?EvenQ] := 0


... then the corresponding behaviour is overridden, but the other definitions remain in place.

publicFn /@ Range[-5, 5]


{-500,0,-300,0,-100,0,10,0,30,0,50}

Since the public function is defined with the broadest possible definition, all more-specific user definitions will be tried before the pre-defined rules. Of course, if the user redefines the public function's one definition exactly, then all bets are off.

Handling Partial Functions

If the function is only partially defined, we may want the public function to return unevaluated when the arguments are outside its domain. To do this, we need a slightly more elaborate version of the public function:

ClearAll[publicFn]
publicFn[args___] := Module[{v = privateFn[args]}, v /; !MatchQ[v, _privateFn]]


Unfortunately, Mathematica no longer considers this definition to be "broad" due to the complex MatchQ condition. Consequently, this definition will no longer be applied after subsequent user-supplied definitions. What we need to do is take over the management of the down-values of publicFn so that our "broad" definition is always last in the list. This is accomplished by installing a new SetDelayed/Set definition as an up-value on publicFn:

publicFn /: (SetDelayed|Set)[h_publicFn, def_] :=
(DownValues[publicFn] =
Insert[
DeleteCases[DownValues[publicFn], Verbatim@HoldPattern[h] :> _]
, HoldPattern[h] :> def
, -2
];)


We'll change privateFn so that it is only partially defined:

ClearAll[privateFn]
privateFn[x_] /; x < 0 := 100 x

publicFn /@ Range[-5, 5]


{-500, -400, -300, -200, -100, publicFn[0], publicFn[1], publicFn[2], publicFn[3], publicFn[4], publicFn[5]}

Users can then add further partial definitions:

publicFn[n_?EvenQ] := 0
publicFn[5] = "will be replaced by a subsequent definition";
publicFn[5] = "five";


{-500, 0, -300, 0, -100, 0, publicFn[1], 0, publicFn[3], 0, "five"}

-
Nice one +1 Any way to allow our package function to return unevaluated (publicFn[..]) when it doesn't match the definitions we gave? –  Rojo Mar 11 '12 at 1:12
@Rojo Yes, see the new section Handling Partial Functions. –  WReach Mar 11 '12 at 1:55
but that last one works because privateFn fails with even arguments. If I remove the negative condition the user defined version never gets a chance to overload –  Rojo Mar 11 '12 at 1:59
@Rojo You are correct -- user definitions are ignored in the new version. The problem is that the condition in the new definition of publicFn causes Mathematica to regard the definition as specialized instead of broad. Consequently it is no longer applied after all other definitions. I'm withdrawing the section on partial functions until such time that I have an alternative solution. Assuming that time ever comes... :) –  WReach Mar 11 '12 at 6:01
Thanks WReach. With your and @MrWizard's ideas I think I came up with a way that doesn't require resorting DownValues or defining upvalues on set and posted it. –  Rojo Mar 13 '12 at 3:23
show 1 more comment

Here is method based on WReach's answer. It works by explicitly sorting the list of DownValues after any new assignment is made, placing the ___ last. It allows the return the public function unevaluated. By nature this only works for DownValues definitions.

ClearAll[func, privateFn]

privateFn[x_] /; x < 0 := 100 x
privateFn[x_?NumericQ] := 10 x

func[args___] := With[{eval = privateFn[args]}, eval /; ! MatchQ[eval, _privateFn]]

func /: (set : Set | SetDelayed)[lhs_func, rhs_] /; ! TrueQ[$deffunc] := Block[{$deffunc = True},
lhs ~set~ rhs;
DownValues[func] =
SortBy[
DownValues[func],
{MatchQ[ #[[1]], _[_[_[_, Verbatim[___]]]] ] &}
];
]


Testing:

func[n_Integer?EvenQ] := 0
func[n_Integer] := 1

func /@ Range[-5, 5]
func[0.7]
func["undefined"]

{1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1}

7.

func["undefined"]

-
_[_[_[_, Verbatim[___]]]] that's the scariest pattern I've ever seen. (note I said seen, not understood) –  Rojo Mar 13 '12 at 3:23
@Rojo is _~_~Verbatim@___//_//_ more cheerful? :o) It's just a pattern to match HoldPattern[func[args___]] :> something where args is arbitrary. –  Mr.Wizard Mar 13 '12 at 6:06
Haha, ok, I'm glad you used the first version –  Rojo Mar 14 '12 at 23:43

ClearAll[publicFn]
Module[{guard = True},
publicFn[args___] /; guard :=
Block[{guard = False}, Module[{res = publicFn[args]},
If[MatchQ[res, _publicFn],
res = privateFn[args]]; res /; ! MatchQ[res, _privateFn]]
]
]


This doesn't suffer problems of the automatic DownValue resorting, and without resorting to upvalues on Set functions... Shouldn't have problems with UpValues either. The public definition of the function is found perhaps before or perhaps after some user defined definitions. But if it is found, it makes sure there aren't any other good public definitions before transferring the job to the private definition. If there are none, it returns unevaluated.

privateFn[x_] /; x < 0 := 100 x
privateFn[x_?NumericQ] := 10 x
privateFn[3] = 99

publicFn /@ Range[-5 , 5]
publicFn["oij"]

{-500, 0, -300, 0, -100, 0, 10, 0, 30, 0, 50}
publicFn["oij"]


Now, making some definitions

publicFn[n_?EvenQ] := 0;
publicFn[3 | 33] = -100;
soh /: publicFn[soh] := "boo"


We see that

publicFn /@ Range[-5 , 5]
publicFn["oij"]
publicFn[soh]


returns

{-500, 0, -300, 0, -100, 0, 10, 0, -100, 0, 50}

publicFn["oij"]

"boo"

-