Ultimately, the culprit is not simply the lexical scoping of Module
, as comments have mentioned, but the scoping rules for Rule
and RuleDelayed
.
tl;dr: x_
appears to belong to the scope of the outermost ->
, not the innermost :>
, and so x
is not lexically scoped by :>
—even though it would be lexically scoped and protected from replacement in Module
in other cases, e.g. if we got rid of the top-level rule.
The following is a long discussion about the issue.
What's not the problem: how Module
behaves with rules
I want to contribute a few more examples, which I think illustrate how both the rule scoping and module scoping works:
In[1]:= x = 100; (x_ :> x + 1) -> 0
Out[1]:= (x_ :> x + 1) -> 0
In[2]:= x = 100; (5 /. x_ :> x + 1) -> 0
Out[2]:= 6 -> 0
In[3]:= Module[{x = 100}, x_ :> x + 1]
Out[3]:= x_ :> x + 1
In[4]:= Module[{x = 100}, (x_ :> x + 1) -> 0]
Out[4]:= (x_ :> x$90724 + 1) -> 0
In[5]:= Module[{x = 100}, (5 /. x_ :> x + 1) -> 0]
Out[5]:= 101 -> 0
In[6]:= Module[{x = 100}, Hold[x]]
Out[6]:= Hold[x$98978]
Note that all of the above behavior remains if we replace ->
with :>
.
These are a lot of examples all at once, but we'll use them.
Importantly, changing Module
to Block
produces the "expected behavior" in Out[1]
and Out[2]
. Module
uses lexical scoping, while Block
uses dynamic scoping.
To see how this works, consider In
s 4
, 5
, and 6
. They show that even held expressions get their variables renamed. :>
has the attribute HoldRest
, meaning that its latter part is held. In In[5]
, x$NNNNN
is released from its hold via the use of /.
, and evaluated. (This also shows us that variables are initialized only after renaming takes place; giving a symbol a definition is a dynamic operation, not a lexical one.)
But, even though Module
renames held symbols, it avoids renaming lexically bound symbols.
This means that for Module
the free symbols (ones that are not local to another construct inside it) are renamed before evaluation; for an example of a symbol that is local to a construct inside module, and therefore not renamed, consider Module[{x = 5}, Module[{x = 3}, x]]
, which returns 3
.
This is supposed to be the case for Rule
s too. From the documentation for Rule
:
Symbols that occur as pattern names in lhs are treated as local to the rule. This is true when the symbols appear on the right-hand side of /; conditions in lhs, and when the symbols appear anywhere in rhs, even inside other scoping constructs.
Further, consider the statement from the documentation for RuleDelayed
:
Module
and With
do not affect local variables of RuleDelayed
with the example
In[7] := Module[{x = 1}, a /. x_ :> x + 1]
Out[7]:= 1 + a
which is equivalent to our In[3]
.
So, what's happening here?
Let's describe what happens in the case of In[3]
. :>
looks at the symbols on its rhs, sees which appear as pattern names on its lhs, then binds them to be local to the rule, protecting them from Module
's lexical scoping.
In contrast, in the case of In[4]
, I believe that the x_
on the lhs of x_ :> x + 1
belongs to the scope of the outermost Rule
, not the innermost RuleDelayed
, and is therefore not a symbol on the rhs of :>
which is named by a pattern on its lhs, and so is not bound by :>
. Further, since the outermost Rule
does not consider symbols on its lhs which do not name patterns, such as x
, to be local, it releases x
to be bound and renamed by Module
.
(Note again that the distinction between Rule
and RuleDelayed
is only relevant insofar as RuleDelayed
has the attribute HoldRest
and holds x + 1
; replacing Rule
with RuleDelayed
produces the same behavior.)
This is unusual to me. I personally think the x_
there should belong to the scope of the inner :>
, not the outer ->
.
To bolster the notion that this is in fact what is going on, consider the following, which shows the latter-mentioned fact that symbols on the lhs of Rule
don't get lexically bound by named patterns on the lhs:
In[8] := Module[{x = 100}, Hold[{x_, x} -> x]
Out[8]:= Hold[{x_, x$100384} -> x]
This shows us that the x
named by a pattern on the lhs is only in the scope of Rule
when it's on its rhs. Otherwise, it's not local to Rule
, and Module
grabs it.
This is fine. The issue is, as you've pointed out, when there's an inner rule which seems like it ought to lexically scope the x
, and doesn't! So, take another look at In[4]
now; note that x
acts like any other symbol appearing on the lhs of Rule
, instead of being lexically bound by being on the rhs of :>
.
Your example corroborates the first notion: if x_
belonged to :>
, then x
would be lexically scoped in the :>
expression, and the replacement would proceed as expected.
Is this behavior good for users?
I can maybe understand why one might want the above behavior: then all patterns in the lhs of any rule are indeed patterns for that rule, which you could use in replacements. But this is still true even if you change the lexical scoping rules to make them more intuitive, since evaluation essentially implements the expected scoping behavior via attributes like HoldRest
. In contrast, as we've seen, Hold
s are transparent to lexical scoping.
The main thing wrong with it for me, really, is the unexpected failure of lexical scoping with nested rules, which is, as far as I can tell, not necessary for dynamic evaluation to behave as expected even in cases where one wants to use a whole rule as a pattern for a replacement—but I could be wrong on that.
Why should a user expect In[2]
to produce different output than In[5]
? The lexical scoping is "not compatible" with the dynamic scoping. Admittedly, in many cases this is exactly what we want. We want to transform held expressions in place before evaluating, as Module
and With
do.
The problem is that the evaluation rules create the appearance of lexical scoping while using a different mechanism, which makes the apparent scoping in In[2]
(x = 100; (5 /. x_ :> x + 1) -> 0
) a red herring. It works for a reason one wouldn't expect it to, given how Module
does respect lexical scoping by :>
: the x
here only appears well-scoped because it is held.
In either case, I think granting the lexical scope to the outermost rule instead of the innermost one (assuming that I'm correct in that being what's happening) is counterintuitive for the user, and relies on evaluation timing instead of the apparent scoping suggested by both the syntax highlighter and the behavior in Module
in non-nested cases. I'd be curious why the decision was made to grant the outermost rule lexical priority!
x
in the rules belongs to a pattern. If you consider this a bug, please report it to support@wolfram.com $\endgroup$ – Daniel Huber Jan 8 at 21:54x
scoped to theModule
is different fromx
in the outer scope. Check the documentation. UseBlock
(and be aware of the other differences) to avoid generated symbols. $\endgroup$ – Rohit Namjoshi Jan 8 at 21:54Module[{a,b},...]
, e.g.,a = 3;(x /. Replace[{x, t}, {a_, b_} -> a -> b])
returnsx
. $\endgroup$ – Roma Lee Jan 10 at 5:26