Struggling to understand Query

Question

The documentation for Query starts with:

data = {
   <|"a" -> 1, "b" -> "x", "c" -> {1}|>,
   <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
   <|"a" -> 3, "b" -> "z", "c" -> {3}|>,
   <|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
   <|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
   <|"a" -> 6, "b" -> "z", "c" -> {}|>};

They have the following example:

Query[Total, "a"] @ data
21

As I understand it Total is an ascending operator so it does the "a" selection, resulting in a list of the numbers 1 through 6. Then Query returns and does the Total operation. I think that I understand that each operator in turn operates on the next level, either ascending or descending.

How would I get Query to give me the total of the values of "a" in rows 1 through 5? That is, without preconditioning data. I realize that I don't have a good understanding of how Query works.

Answer 1

Short Answer

The query operator for a given level can perform both descending and ascending actions, separated by the /* operator. We can tack the descending filtering operator 1 ;; 5 onto the front of the ascending operator Total:

data // Query[(1 ;; 5) /* Total, "a"]
(* 15 *)

The parentheses are necessary due to the tightly binding precedence of /*.

Long Answer

Fasten your seatbelt...

The query Query[Total, "a"] contains two operators, Total and "a".

Total appears as the first operator of the query, so it is applied to the whole expression (i.e. level zero). It is documented to be an ascending operator. So in the descending phase of the operation, it does nothing and its action is essentially Identity. In the ascending phase, its action is to perform the function of the like-named built-in function Total.

"a" is documented to be a descending operator whose action is to invoke the operator form of the built-in function Key like this: Key["a"]. Since this operator appears as the second query operator, it will be applied to each element at level one in the expression. Notionally, this means that the operator further expands into Map[Key["a"]]. As a descending operator, "a" does nothing in the ascending phase. This is notionally equivalent to Map[Identity].

To summarize, we have the following descending and ascending options for each operator:

Operator         Descending Action        Ascending Action

TOTAL            Identity                 TOTAL
"a"              Map[Key["a"]]            Map[Identity]

Let's simulate the execution of the query. We will apply these actions to our data, with descending actions working down into the expression's levels and ascending actions working back up:

data

(* { <|"a" -> 1, "b" -> "x", "c" -> {1}|>
   , ...
   , <|"a" -> 6, "b" -> "z", "c" -> {}|>
   } *)

% // Identity         (* level 0 descending action *)

(* { <|"a" -> 1, "b" -> "x", "c" -> {1}|>
   , ...
   , <|"a" -> 6, "b" -> "z", "c" -> {}|>
   } *)

% // Map[Key["a"]]    (* level 1 descending action *)

(* {1, 2, 3, 4, 5, 6} *)

% // Map[Identity]    (* level 1 ascending action *)

(* {1, 2, 3, 4, 5, 6} *)

% // Total            (* level 0 ascending action *)

(* 21 *)

Now, to the question at hand. How do we restrict the range of rows? From the table above, we can see that we need to change the first descending operator from Identity to something that selects the first five rows. The Query documentation tells us that we can use the operator syntax i ;; j.

But... we already have a first operator: Total. How can we perform a descending operation as well? The documentation provides that answer as well:

When one or more descending operators are composed with one or more ascending operators (e.g. desc/*asc), the descending part will be applied, then subsequent operators will be applied to deeper levels, and lastly the ascending part will be applied to the result.

We can tack our descending operation onto the front of Total like this:

(1;;5) /* Total

The parentheses are necessary due to the tight binding of the /* operator.

The operator 1;;5 is notionally interpreted as the function #[[1;;5]]&.

The full query is:

Query[(1;;5) /* Total, "a"]

The operation table for this query looks like this:

Operator               Descending Action        Ascending Action

(1;;5) /* Total        #[[1;;5]]&               TOTAL
"a"                    Map[Key["a"]]            Map[Identity]

The execution proceeds exactly as detailed above, except that the new first descending action reduces the input list to its first five elements instead of the full list.

The input rows could be reduced by any descending filtering operator in place of (1;;5). Notably, the Select operator could be used for content-based filtering instead of part selection.

What's up with "Notionally"?

This response has used the word "notionally" several times when speaking of the interpretation of query operators (e.g. 1;;2 is notionally interpreted as #[[1;;2]]&). This qualification is necessary because in practice the query compiler uses a variety of internal functions to optimize or perform special processing during query execution. Furthermore, non-operations like Map[Identity] are optimized out.

The notional interpretations given above are roughly equivalent substitutes used to avoid distracting from the main points of discussion.

To see the actual interpretation of a query, use Normal:

Query[(1 ;; 5) /* Total, "a"] // Normal

(* GeneralUtilities`Slice[1 ;; 5] /* GeneralUtilities`Slice[All, "a"] /* Total *)

GeneralUtilities`Slice is a common query helper function that can perform all manner of slicing and mapping functions, applied simultaneously.

I recommend thinking in such notional terms, with query operators operating upon distinct expression levels with distinct descending and ascending phases. It helps that the documentation is written in such terms, and the optimized execution plans respect those semantics. Make it a habit to read the query:

Query[Total, "a"]

as if it were written as:

Query[   Identity  /*  Total
     ,   "a"       /*  Identity   ]

... mentally separating out the ascending and descending phases.

Query Operators vs. Normal Operators

Throughout this response, there has been an implicit distinction drawn between query operators and the functions that implement them. This is an important distinction to realize. It is just a coincidence that the Total query operator is implemented by the like-named Total function. This coincidence is not true in general. We already saw exceptions like "a" interpreted as Key["a"] and 1;;5 interpreted as #[[1;;5]]&.

GroupBy is an example of a query operator that behaves subtly differently from its function namesake. GroupBy["a"] is quietly rewritten as GroupBy[Key["a"]]. There are many other operators that have similar small changes in behaviour -- study the documentation for details.

Here is an interesting case: the /* query operator behaves very differently from the /* function. The /* function simply chains operations together, one right after the other. But the left and right operations of the /* query operator will not necessarily be executed one right after the other -- they might be separated by many intervening operators from lower query levels. Don't be fooled by the surface similarity of the syntax. Consider:

Query[Select[OddQ] /* f, Select[EvenQ] /* g] // Normal

(* Select[OddQ] /* Map[Select[EvenQ] /* g] /* f *)

Notice how the operands from Select[OddQ] /* f are not adjacent in the compiled query.

How do I know whether an operator is ascending or descending?

At the moment, the only real way is to read the documentation and memorize the descending operator list. If an operator is not descending, then it is ascending (duly noting the exception of composite operators). Any operator that is not listed in the documentation is ascending, meaning that the vast majority of expressions are ascending.

There are undocumented functions that will indicate whether an operator is ascending or descending:

Query; Needs["Dataset`"]

AscendingQ[Total]
(* True *)

DescendingQ[1;;5]
(* True *)