Why and how does RuleDelayed evaluate to itself?

It seems that RuleDelayed has some behaviour we cannot reproduce. It is not just a symbolic wrapper.

Consider this:

RuleDelayed[u, Unevaluated@RandomReal[]] // FullForm

is

RuleDelayed[u, RandomReal[]]

but I don't see how I could reproduce this:

ruleDelayed~SetAttributes~{HoldRest, SequenceHold}
ruleDelayed[u, Unevaluated@RandomReal[]] // FullForm


gives

ruleDelayed[u,Unevaluated[RandomReal[]]]

since no rule applies.

Setting ruleDelayed[x_, y_] := ruleDelayed[x, y] gives

Hold[ruleDelayed[u,0.2991801319146159]]

because we enter an infinite loop.

The closest I can get to what RuleDelayed does is

ruleDelayed2~SetAttributes~{HoldRest, SequenceHold}
ruleDelayed3~SetAttributes~{HoldRest, SequenceHold}
ruleDelayed2[x_, y_] := ruleDelayed3[x, y]

ruleDelayed2[u, Unevaluated@RandomReal[]] // FullForm


giving

ruleDelayed3[u,RandomReal[]]

but I would like the head to stay ruleDelayed2.

Is there any reason RuleDelayed evaluates instead of just being an inert token?

Am I right thinking that we cannot reproduce this behaviour?

• No matter how many times I read your question I still can't make any sense of your title. Aug 18 '16 at 20:52

RuleDelayed is unconventional in that it strips any number of occurrences of Unevaluated from the right-hand side:

a :> Unevaluated[1 + 1]

(* a :> 1 + 1 *)

a :> Unevaluated[Unevaluated[Unevaluated[1 + 1]]]

(* a :> 1 + 1 *)


This is different from the normal evaluation process which only strips one level:

ClearAll[f]
f[x_] := x

f[Unevaluated[Unevaluated[Unevaluated[1 + 1]]]]

(* Unevaluated[Unevaluated[1 + 1]] *)


I speculate that this non-standard behaviour is intended to simplify the semantics of rule application. Since Unevaluated has special meaning to the evaluator, it is hard to pattern-match (as we shall see below). I think that the brute-force strategy of stripping all levels of Unevaluated makes it easier to reason about the most common cases of rule application.

We might try to reproduce this behaviour naively as follows:

SetAttributes[ruleDelayed, {HoldRest, SequenceHold}]

ruleDelayed[u_, HoldPattern[Unevaluated[x_]]] := ruleDelayed[u, x]


But this always leaves one left-over Unevaluated:

a ~ruleDelayed~ Unevaluated[1 + 1]

(* ruleDelayed[a, Unevaluated[1 + 1]] *)

a ~ruleDelayed~ Unevaluated[Unevaluated[Unevaluated[1 + 1]]]

(* ruleDelayed[a, Unevaluated[1 + 1]] *)


The reason is that Unevaluated gets special treatment from the evaluator. It is stripped before the containing expression is evaluated, but is restored if there are no replacement rules applicable to that expression:

ClearAll[f]
f[x_] := Null /; (Print[Hold[x]]; False)

f[Unevaluated[1 + 1]]

(*
Hold[1 + 1]
f[Unevaluated[1 + 1]]
*)


Notice how the body of f saw the argument without Unevaluated. But the Unevaluated wrapper has been restored in the return value when the definition was rendered inapplicable by the False condition.

Since Unevaluated is unwrapped prior to the application of definitions, we can never pattern-match that outermost Unevaluated. (Never, that is, without resorting to HoldComplete -- but RuleDelayed does not use that.)

So what we need is to add a definition to our ruleDelayed that:

• is applicable to any input (to cause Unevaluated to be stripped but not restored),
• returns that input unchanged,
• and does not cause infinite recursion.

A tall order, but not impossible:

Module[{blocked = False}
, ruleDelayed[u_, x_] /; !blocked := Block[{blocked = True}, ruleDelayed[u, x]]
]


With this additional rule, we can reproduce the desired behaviour:

a ~ruleDelayed~ (1 + 1)

(* ruleDelayed[a, 1 + 1] *)

a ~ruleDelayed~ Unevaluated[1 + 1]

(* ruleDelayed[a, 1 + 1] *)

a ~ruleDelayed~ Unevaluated[Unevaluated[Unevaluated[1 + 1]]]

(* ruleDelayed[a, 1 + 1] *)


The extra definition might work, but it is somewhat brittle. The use of a dynamically-scoped variable to conditionally inhibit the operation of the definition can be defeated in pathological conditions (for example, up-value evaluations that sneak into the inner call). While this is very unlikely to happen in practice, a more bullet-proof solution would somehow trigger the stripping of Unevaluated more directly.

RuleDelayed appears to achieve this "more bullet-proof" solution through built-in definitions. Note that RuleDelayed has built-in "down code":

SystemPrivateHasDownCodeQ[RuleDelayed]

(* True *)


I suspect that this hard-coded definition directly tells the evaluator not to restore a stripped Unevaluated head, analogous to our magic Module[...] definition.

Furthermore, RuleDelayed has at least one built-in down-value:

DownValues[RuleDelayed]

(* False *)

SystemPrivateHasDownEvaluationsQ[RuleDelayed]

(* True *)


My guess is that this parallels our definition that strips all levels of Unevaluated but the first.

• I don't understand what SystemPrivateHasDownCodeQ and SystemPrivateHasDownEvaluationsQ do. It's not there in version 10.3.1, but it is there in 10.4. How do you know about them? Sep 4 '16 at 23:19
• @QuantumDot I stumbled across them during a debugging session. Naturally, they are unsupported. As far as I can tell, HasDownCodeQ and HasDownEvaluationsQ indicate that a symbol has definitions hard-coded into the kernel (written in C/C++ and WL respectively). Sep 7 '16 at 23:46

Consider the result of passing in RandomReal[] rather than Unevaluated @ RandomReal[] to RuleDelayed:

RuleDelayed[u, RandomReal[]] // FullForm


RuleDelayed[u, RandomReal[]]

which shows that passing in Unevaluated @ RandomReal[] is a distinction without a difference and there is no reason to do it.

Further, because

SetAttributes[ruleDelayed, {HoldRest, SequenceHold}]
ruleDelayed[u, RandomReal[]] // FullForm


gives

ruleDelayed[u, RandomReal[]]

it emulates all the essentially useful behavior of RuleDelayed` with respect to attributes. Consequently, your question reminds me of the old joke:

Patient: Doctor, it hurts when I do this.
Doctor: Well, then don't do that.