22
$\begingroup$

Why does the following pattern matching not succeed?

ClearAll[f];
SetAttributes[f,Orderless];
HoldForm[f[a,d[],b,c]]/.HoldPattern[f[a,d[],b,z_]]:>{z}
(* f[a,d[],b,c] *)

I expected that because f is Orderless, the pattern matcher should try all possible orders of arguments and find a match.

If the expression is not held, I find:

ClearAll[f];
SetAttributes[f,Orderless];
f[a,d[],b,c]/.HoldPattern[f[a,d[],b,z_]]:>{z}
(* {c} *)

I.e. the desired result. I know that because f is Orderless and the expression is evaluated, the arguments are sorted into canonical order a, b, c, d[]. It would appear in this case that the pattern matcher really does try all possible orders of arguments to find a match.

Why doesn't it work in the first example?

EDIT:

Here are some further examples:

ClearAll[f];
SetAttributes[f, Orderless];

Hold[f[a, b, c, d[]]] /. HoldPattern[f[a, b, c, z_]] :> {"Matched", {z}}
(* Hold[{"Matched", {d[]}}] *)

Hold[f[a, b, c[], d]] /. HoldPattern[f[a, b, c[], z_]] :> {"Matched", {z}}
(* Hold[{"Matched", {d}}] *)

Hold[f[a, b[], c, d]] /. HoldPattern[f[a, b[], c, z_]] :> {"Matched", {z}}
(* Hold[f[a, b[], c, d]] *)

Hold[f[a[], b, c, d]] /. HoldPattern[f[a[], b, c, z_]] :> {"Matched", {z}}
(* Hold[f[a[], b, c, d]] *)

This seems very inconsistent to me; how can I understand what is happening here?

EDIT:

An illuminating example:

Hold[f[a, c, b[], d]] /. HoldPattern[f[a, b[], c, z_]] :> {"Matched", {z}}
(* Hold[{"Matched", {d}}] *)

The Trace shows no argument reordering, but it appears that the pattern matcher must actually be reordering the pattern to (the canonical order) f[a, c, b[], z_] because f is Orderless, even though HoldPattern should prevent "regular" evaluation.

What remains confusing is what exactly is meant by "In matching patterns with Orderless functions, all possible orders of arguments are tried." in the documentation. On face value, the four examples shown above contradict this.

$\endgroup$
  • 2
    $\begingroup$ I am confused. HoldForm[f[a, d[], c]] /. HoldPattern[f[a, d[], c_]] :> {c} outputs {c}, and if you stop f from being Orderless then the we likewise get HoldForm[f[a, d[], b, c]] /. HoldPattern[f[a, d[], b, c_]] :> {c} outputting {c}. It seems inconsistent to me. $\endgroup$ – Patrick Stevens Sep 11 '15 at 8:05
  • $\begingroup$ Indeed, quite confusing! $\endgroup$ – bdforbes Sep 11 '15 at 8:05
  • $\begingroup$ But your Verbatim trick works! HoldForm[f[a, d[], b, c]] /. Verbatim[f][a, d[], b, z_] :> {z} yields {c}. Although if you lose the HoldForm, it doesn't work... $\endgroup$ – march Sep 11 '15 at 15:47
  • 1
    $\begingroup$ Verbatim works because f is no longer treated as a head for pattern matching purposes, and so its Orderless attribute has no effect. But why that attribute prevents the match in the first place is unclear :/ $\endgroup$ – bdforbes Sep 11 '15 at 22:34
  • $\begingroup$ And Verbatim doesn't work without the HoldForm on the expression, because the arguments of the expression will be sorted into canonical form but those of the pattern will not be able to shuffle, so there is no way for the pattern to match. $\endgroup$ – bdforbes Sep 11 '15 at 22:39
13
+50
$\begingroup$

I am betting that this is almost certainly an optimization short-cut. If Orderless had to try every ordering it would be extremely slow when there are a moderate number of arguments, but it is not.

Consider for example:

f @@@ Hold @@ {RandomSample[Range@12]}
Hold @@ {f @@ Range@12 /. {7 -> _}}
MatchQ[%%, %]
Hold[f[2, 6, 11, 7, 12, 10, 4, 1, 3, 5, 8, 9]]

Hold[f[1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, _]]

False

This test is surely not performing 479,001,600 comparisons, one for each permutation. Instead I suppose that pattern matching assumes that arguments of Orderless heads will first be placed in order and applies logic accordingly.

Since the first two of your "further examples" match I also suppose that this is because the sorted order of arguments on the RHS matches the verbatim order on the left; perhaps this case is tried first.

$\endgroup$
  • $\begingroup$ Best hypothesis so far... I suppose that the best practice would be to not Hold such patterns or simply deactivate Orderless heads during pattern matching? $\endgroup$ – bdforbes Sep 13 '15 at 10:37
6
$\begingroup$

The following is an expansion of the explanation given by Mr.Wizard.


The pattern-matcher works on the base of the assumption that Orderless attribute is already applied and the arguments of the Orderless function are already sorted in the canonical order:

ClearAll[o]
SetAttributes[o, Orderless]
MatchQ[Hold[o[y, x, a]], Hold[o[_, x, a]]] (* unsorted arguments: {x, a} is NOT ordered *)
MatchQ[Hold[Evaluate@o[y, x, a]], Hold[o[_, x, a]]] (* Orderless attribute is applied *)
False

True

Instead of Hold we can use Unevaluated with the same effect:

MatchQ[Unevaluated[o[y, x, a]], o[_, x, a]] (* {x, a} is NOT ordered *)
MatchQ[Unevaluated[o[y, a, x]], o[_, x, a]] (* {a, x} IS ordered *)
False

True

Hold and Unevaluated prevent applying the Orderless attribute to the arguments of the Orderless function. They only affect the evaluation of the expression, and do not affect the pattern-matcher.

The pattern-matcher does not reorder the arguments of the Orderless function but takes the whole expression verbatim. It assumes that the arguments are already sorted as it happens by default for evaluated function with Orderless attribute. At the same time the pattern-matcher does sort the arguments of the pattern (with the exception of Blanks) regardless of whether the pattern is evaluated or not. Then it compares the sorted pattern with the expression putting Blank(s) in each position possible in the pattern.

In the light of this the Documentation statement

In matching patterns with Orderless functions, all possible orders of arguments are tried.

is misleading: actually only all possible positions of the Blanks in the pattern are tried but other arguments in the pattern are preliminarily sorted in the fixed canonical order. This is evident from the following:

MatchQ[Hold[o[y, a, x]], Hold[o[_, x, a]]] (* {a, x} IS ordered *)
MatchQ[Hold[o[y, x, a]], Hold[o[_, x, a]]] (* {x, a} is NOT ordered  *)
MatchQ[Hold[o[y, a, x]], Hold[o[x, _, a]]] (* {a, x} IS ordered *)
MatchQ[Hold[o[y, a, x]], Hold[o[x, a, _]]] (* {a, x} IS ordered *)
True

False

True

True

In short, Hold prevents applying Orderless attribute in the sense that it prevents sorting of the arguments in the canonical order, but it does not prevent action of this attribute in the pattern (i.e. all possible positions of the Blanks are indeed tried).

$\endgroup$
  • $\begingroup$ This does not seem exactly correct. With g Orderless, I find MatchQ[Hold[g[y, a, x]], Hold[g[a, x, _]]] -> True but MatchQ[Hold[g[y, x, a]], Hold[g[x, a, _]]] -> False. What am I missing? $\endgroup$ – John McGee Apr 12 '16 at 10:52
  • $\begingroup$ @JohnMcGee Your second example is essentially identical to mine MatchQ[Hold[o[y, x, a]], Hold[o[_, x, a]]]. In the both cases the output is False because the sequence x, a in the expression we test against pattern is not in the canonical order. It immediately matches if we place arguments in the canonical order: MatchQ[Hold[g[y, a, x]], Hold[g[x, a, _]]] for your example and MatchQ[Hold[o[y, a, x]], Hold[o[_, x, a]]] for mine (both give True). This also explains your first example. $\endgroup$ – Alexey Popkov Apr 12 '16 at 11:32
  • 1
    $\begingroup$ It would be useful to more explicitly say that arguments of the expression are not reordered while those in the pattern are sorted, with the exception of blanks, put in each possible position. For a complete picture one can run Table[Print[{expr, patt} -> MatchQ[expr, patt]], {expr, Map[Hold[#] /. oo -> o &, Permutations[oo[a, x, y]]]}, {patt, Map[Hold[#] /. oo -> o &, Permutations[oo[a, x, _]]]}] and see that the order of the pattern patt does not matter. Matching happens exactly if a and x appear in the canonical order in the expression. $\endgroup$ – Bruno Le Floch Apr 12 '16 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.