4
$\begingroup$

Why

(f[1, 0] /. {x : 0 | 1 :> 1 - x}) -> 0

returns (expectedly) f[0, 1] -> 0, while

Module[{x}, (f[1, 0] /. {x : 0 | 1 :> 1 - x}) -> 0]

returns f[1 - x$2749, 1 - x$2749] -> 0? I can not imagine a consistent explanation for this behavior. There since Mma 7 at least.

$\endgroup$
9
  • 1
    $\begingroup$ Your problem is that you declare a local variable with the same name as a pattern. Module replaces all local names by a unique name with the exception of pattern names. It does not realize that the x in the rules belongs to a pattern. If you consider this a bug, please report it to support@wolfram.com $\endgroup$ Jan 8 at 21:54
  • 3
    $\begingroup$ It is the way WL ensures that the x scoped to the Module is different from x in the outer scope. Check the documentation. Use Block (and be aware of the other differences) to avoid generated symbols. $\endgroup$ Jan 8 at 21:54
  • 1
    $\begingroup$ Note that while I mentioned this Q/A as a direct duplicate, I think that a somewhat more detailed explanation is given e.g. in this Q/A and references therein. $\endgroup$ Jan 8 at 23:06
  • $\begingroup$ @LeonidShifrin, related, but not exactly a duplicate as in your reference the inner rule appears in the right-hand side of the outer one, while in the present question it appears in the left-hand side. Even more important, in your reference the behavior does not depend on the outer Module[{a,b},...], e.g., a = 3;(x /. Replace[{x, t}, {a_, b_} -> a -> b]) returns x. $\endgroup$
    – Roma Lee
    Jan 10 at 5:26
  • $\begingroup$ @RomaLee My point is that the puzzling behavior in all these cases has the same origin. If you look into those other cases and read explanations in the related Q/As and references therein, that should give you a good idea about the reasons for this behavior, also in your particular case. $\endgroup$ Jan 10 at 9:34
4
$\begingroup$

Ultimately, the culprit is not simply the lexical scoping of Module, as comments have mentioned, but the scoping rules for Rule and RuleDelayed.

tl;dr: x_ appears to belong to the scope of the outermost ->, not the innermost :>, and so x is not lexically scoped by :>—even though it would be lexically scoped and protected from replacement in Module in other cases, e.g. if we got rid of the top-level rule.

The following is a long discussion about the issue.

What's not the problem: how Module behaves with rules

I want to contribute a few more examples, which I think illustrate how both the rule scoping and module scoping works:

 In[1]:=  x = 100; (x_ :> x + 1) -> 0

Out[1]:=  (x_ :> x + 1) -> 0

 In[2]:=  x = 100; (5 /. x_ :> x + 1) -> 0

Out[2]:=  6 -> 0

 In[3]:=  Module[{x = 100}, x_ :> x + 1]

Out[3]:=  x_ :> x + 1

 In[4]:=  Module[{x = 100}, (x_ :> x + 1) -> 0]

Out[4]:=  (x_ :> x$90724 + 1) -> 0

 In[5]:=  Module[{x = 100}, (5 /. x_ :> x + 1) -> 0]

Out[5]:=  101 -> 0

 In[6]:=  Module[{x = 100}, Hold[x]]

Out[6]:=  Hold[x$98978]

Note that all of the above behavior remains if we replace -> with :>.

These are a lot of examples all at once, but we'll use them.

Importantly, changing Module to Block produces the "expected behavior" in Out[1] and Out[2]. Module uses lexical scoping, while Block uses dynamic scoping.

To see how this works, consider Ins 4, 5, and 6. They show that even held expressions get their variables renamed. :> has the attribute HoldRest, meaning that its latter part is held. In In[5], x$NNNNN is released from its hold via the use of /., and evaluated. (This also shows us that variables are initialized only after renaming takes place; giving a symbol a definition is a dynamic operation, not a lexical one.)

But, even though Module renames held symbols, it avoids renaming lexically bound symbols.

This means that for Module the free symbols (ones that are not local to another construct inside it) are renamed before evaluation; for an example of a symbol that is local to a construct inside module, and therefore not renamed, consider Module[{x = 5}, Module[{x = 3}, x]], which returns 3.

This is supposed to be the case for Rules too. From the documentation for Rule:

Symbols that occur as pattern names in lhs are treated as local to the rule. This is true when the symbols appear on the right-hand side of /; conditions in lhs, and when the symbols appear anywhere in rhs, even inside other scoping constructs.

Further, consider the statement from the documentation for RuleDelayed:

Module and With do not affect local variables of RuleDelayed

with the example

In[7] :=  Module[{x = 1}, a /. x_ :> x + 1]

Out[7]:=  1 + a

which is equivalent to our In[3].

So, what's happening here?

Let's describe what happens in the case of In[3]. :> looks at the symbols on its rhs, sees which appear as pattern names on its lhs, then binds them to be local to the rule, protecting them from Module's lexical scoping.

In contrast, in the case of In[4], I believe that the x_ on the lhs of x_ :> x + 1 belongs to the scope of the outermost Rule, not the innermost RuleDelayed, and is therefore not a symbol on the rhs of :> which is named by a pattern on its lhs, and so is not bound by :>. Further, since the outermost Rule does not consider symbols on its lhs which do not name patterns, such as x, to be local, it releases x to be bound and renamed by Module.

(Note again that the distinction between Rule and RuleDelayed is only relevant insofar as RuleDelayed has the attribute HoldRest and holds x + 1; replacing Rule with RuleDelayed produces the same behavior.)

This is unusual to me. I personally think the x_ there should belong to the scope of the inner :>, not the outer ->.

To bolster the notion that this is in fact what is going on, consider the following, which shows the latter-mentioned fact that symbols on the lhs of Rule don't get lexically bound by named patterns on the lhs:

In[8] :=  Module[{x = 100}, Hold[{x_, x} -> x]

Out[8]:=  Hold[{x_, x$100384} -> x]

This shows us that the x named by a pattern on the lhs is only in the scope of Rule when it's on its rhs. Otherwise, it's not local to Rule, and Module grabs it.

This is fine. The issue is, as you've pointed out, when there's an inner rule which seems like it ought to lexically scope the x, and doesn't! So, take another look at In[4] now; note that x acts like any other symbol appearing on the lhs of Rule, instead of being lexically bound by being on the rhs of :>.

Your example corroborates the first notion: if x_ belonged to :>, then x would be lexically scoped in the :> expression, and the replacement would proceed as expected.

Is this behavior good for users?

I can maybe understand why one might want the above behavior: then all patterns in the lhs of any rule are indeed patterns for that rule, which you could use in replacements. But this is still true even if you change the lexical scoping rules to make them more intuitive, since evaluation essentially implements the expected scoping behavior via attributes like HoldRest. In contrast, as we've seen, Holds are transparent to lexical scoping.

The main thing wrong with it for me, really, is the unexpected failure of lexical scoping with nested rules, which is, as far as I can tell, not necessary for dynamic evaluation to behave as expected even in cases where one wants to use a whole rule as a pattern for a replacement—but I could be wrong on that.

Why should a user expect In[2] to produce different output than In[5]? The lexical scoping is "not compatible" with the dynamic scoping. Admittedly, in many cases this is exactly what we want. We want to transform held expressions in place before evaluating, as Module and With do.

The problem is that the evaluation rules create the appearance of lexical scoping while using a different mechanism, which makes the apparent scoping in In[2] (x = 100; (5 /. x_ :> x + 1) -> 0) a red herring. It works for a reason one wouldn't expect it to, given how Module does respect lexical scoping by :>: the x here only appears well-scoped because it is held.

In either case, I think granting the lexical scope to the outermost rule instead of the innermost one (assuming that I'm correct in that being what's happening) is counterintuitive for the user, and relies on evaluation timing instead of the apparent scoping suggested by both the syntax highlighter and the behavior in Module in non-nested cases. I'd be curious why the decision was made to grant the outermost rule lexical priority!

$\endgroup$
2
  • $\begingroup$ So, the outermost rule seems to not evaluate its left hand-side prior to checking for patterns? This is strange. $\endgroup$
    – Roma Lee
    Jan 10 at 5:32
  • $\begingroup$ So, because it's a matter of lexical scoping, not dynamic i.e. evaluation-based scoping, my understanding is that which symbols are local/in what lexical scope (i.e. checking for patterns) is in a sense determined before any evaluation takes place at all. it's only a matter of syntactic expression structure, not evaluation order. That's just my understanding, though, and ofc I'm still learning exactly how Mathematica works! $\endgroup$
    – thorimur
    Jan 11 at 4:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.