Consider the following query:

Range[5] // Query[Select[EvenQ] /* Append[1], f]

(* {f[2], f[4], 1} *)


It selects the even numbers, applies f to them and finally appends 1 to the list. As expected, the application of f occurs between the descending operator Select and the ascending operator Append.

But watch what happens if we add a Sort operator in an attempt to sort the final result:

Range[5] // Query[Select[EvenQ] /* Append[1] /* Sort, f]

(* {1} *)


That's a surprise! Why didn't we get the sorted list {1, f[2], f[4]}?

This behaviour occurs in Mathematica version 11.0.1 and all previous versions back to 10.0.0.

• I don't sure I have understood your question.
– yode
Feb 14, 2017 at 17:00

This is a corner case that occurs when ascending operators are interleaved between descending operators. This case falls into an undocumented grey area.

In the Details and Options section of the Query documentation, we read the following special rule:

When one or more descending operators are composed with one or more ascending operators (e.g. desc /* asc), the descending part will be applied, then subsequent operators will be applied to deeper levels, and lastly the ascending part will be applied to the result.

The statement appears to apply to the case at hand. But appearances can be deceiving. Consider:

Dataset;

DatasetDescendingQ[Select[EvenQ]]    (* True  *)
DatasetDescendingQ[Append[1]]        (* False *)
DatasetDescendingQ[Sort]             (* True  *)


This means that the operator Select[EvenQ] /* Append[1] /* Sort has the form desc /* asc /* desc, with descending and ascending elements interleaved.

The documentation (weakly!) suggests that it only applies when all the descending operators precede all of the ascending operators. Since our ascending operator is sandwiched between two descending operators, the special rule is not applicable.

We can make the special rule apply by replacing the descending operator Sort with the ascending operator Query[Sort]:

DatasetDescendingQ[Query[Sort]]
(* False *)

Range[5] // Query[Select[EvenQ] /* Append[1] /* Query[Sort], f]
(* {1, f[2], f[4]} *)


This is the result we seek.

Commentary

Since the special rule is inapplicable, how are the interleaved operators interpreted? The documentation is silent, but apparently the resulting composition is treated as a single ascending operator. Thus, when a descending operator follows an ascending operator then all the descending operators lose their special status -- even the ones that precede the ascending operator.

This explains the result we see:

Range[5] // Query[Select[EvenQ] /* Append[1] /* Sort, f]

(* {1} *)


Since the level one operator is being treated as ascending in its entirety, it is applied after f. Therefore, Select[EvenQ] returns nothing since none of the elements {f[1], f[2], ...} are EvenQ. Append[1] acts upon an empty list, and the single element result is left unchanged by Sort.

Notwithstanding the documentation being silent on the matter, I am tempted to call this behaviour a bug. At the very least, I feel the operator composition rules would be simpler if leading descending operators always maintained their special status. It stands to reason that any descending operators that follow an ascending operator lose that status.

But until and unless the rules are ever changed like this, we must keep an eye out for this very subtle case.

Hackery

If we want to try out the revised rule to see how it feels, we can hack the V11.0.1 definition:

BeginPackage["DatasetQueryPackagePrivate", {"Dataset"}]

comp[ops:RightComposition[Longest[desc__ ? VectorDescendingQ], asc__]] :=
vectorDescendingOp[RightComposition[desc]] /* chainedOp[RightComposition[asc]];

comp[ops:RightComposition[desc___ ? VectorDescendingQ, descScalar_ ? ScalarDescendingQ, asc__]] :=
scalarDescendingOp[desc /* descScalar] /* chainedOp[RightComposition[asc]];

EndPackage[]


so then:

Range[5] // Query[Select[EvenQ] /* Append[1] /* Sort, f]

(* {1, f[2], f[4]} *)


Naturally, this hack is unsanctioned and brittle.

I don't know why the RightComposition don't work here,but the Composition will give a result as we expected.

Range[5] // Query[Select[EvenQ] /* Sort@*Append[1], f]


{1, f[2], f[4]}

• This form works because @* has higher precedence than /*. So RightComposition sees two terms: Select[EvenQ] and Sort@*Append[1]. Any operator involving @* is always treated as ascending, so these two terms conform to the pattern descending /* ascending which is documented to work. Feb 14, 2017 at 18:01