Query datasets with "variable" named slots

Question

I want to query a Dataset object, e.g. select some entries using

d = Dataset[{
   <|"a" -> 1, "b" -> "x", "c" -> {1}|>,
   <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
   <|"a" -> 3, "b" -> "z", "c" -> {3}|>,
   <|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
   <|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
   <|"a" -> 6, "b" -> "z", "c" -> {}|>}];
d[Select[Slot["a"] > 3 &]]

which works just fine, but now I want to select entries using a dynamic specification of the column key - like this:

colname = "a";
d[Select[Slot[colname] > 3 &]]

but this only gives the following error:

Function::slot: Slot[colname] (in Slot[colname]>3&) should contain a non-negative integer. >>

Am I doing it wrong, or is there a better way to do this?

Using V10.2...

Answer 1

These work:

d[Select[#[colname] > 3 &]]

In the Details section of Slot, it is said that "# is equivalent to Slot[1]"

so it is equivalent to write:

d[Select[Slot[1][colname] > 3 &]]

Answer 2

@SquareOne's answer shows the natural, idiomatic way to express the query. It exploits the fact that an association can be applied to a key to extract that key's value. A similar work-around would use part notation:

d[Select[#[[colname]] > 3 &]]

This response will show how to achieve the desired result using only Slot notation. The motivation is purely academic -- in normal code I would use @SquareOne's answer.

The argument to Slot is not evaluated when it appears within a Function. This is true even when using simple numeric slot references in earlier versions of Mathematica before the introduction of named slots:

n = 1;
Slot[n] &
(* Function::slot: Slot[n] (in Slot[n]&) should contain a non-negative integer. >> *)

When such evaluation is desired it must be done externally and then injected into the held expression. For example:

With[{c = colname}, Slot[c] > 3 &]
(* #a > 3 & *)

or

colname /. c_ :> (Slot[c] > 3 &)
(* #a > 3 & *)

In the original context, we could get the desired result by writing:

d[Select[With[{c = colname}, Slot[c] > 3 &]]]

This is notationally inferior to @SquareOne's solution, but might be useful in more complex scenarios that involve, for example, code generation.

Beware that this injection technique will only work if the Slot expression is written in full form. If we try to perform this replacement using the short form (#c), the replacement will fail:

With[{c = colname}, #c > 3 &]
(* #c > 3 & *)

Note carefully that we ended up with #c in the result, not the desired #a. The reason for this is subtle and is revealed by inspecting the full form of the slot reference:

#c // FullForm
(* Slot["c"] *)

Pay close attention to the fact that the full form involves the string "c" and not the symbol c. Therefore, the With statement does nothing to the function expression #c > 3 & because all it sees is Slot["c"] > 3 & with no mention of the symbol c.

---

Why does Slot[1][colname]& work when Slot[colname]& doesn't?

As noted in a comment, it is interesting to contrast the behaviours of Slot[1][colname]& and Slot[colname]&.

We will use simplified examples with the following definition:

a = <| "x" -> 123 |>;

Recall that keys can be extracted from associations by using function call syntax, or by using part syntax:

a["x"]
(* 123 *)

a[["x"]]
(* 123 *)

This works even if the key name is stored in a variable:

colname = "x";

a[colname]
(* 123 *)

a[[colname]]
(* 123 *)

This behaviour remains valid even if the association is passed as an argument to a pure function:

#[colname]&[a]
(* 123 *)

#[[colname]]&[a]
(* 123 *)

In these cases, colname remains unevaluated within the function body but is evaluated when the function is executed.

We can also write these expressions using the full form of #:

# // FullForm
(* Slot[1] *)

Slot[1][colname]&[a]
(* 123 *)

Slot[1][[colname]]&[a]
(* 123 *)

These expressions all work because both function call syntax and part syntax evaluate their keys as a matter of course.

All of this is in contrast to the use of Slot[colname] which, as noted in the first section, never evaluates the slot value and is thus invalid syntax. We are informed of this error even if simply try to define a function that uses it:

Slot[colname]&
(* Function::slot:Slot[colname](in Slot[colname]&) should contain a non-negative integer.
   Slot[colname] &
*)

If we go so far as to define and call the function, we get the error twice:

Slot[colname]&[a]
(* Function::slot:Slot[colname](in Slot[colname]&) should contain a non-negative integer.
   Function::slot:Slot[colname](in Slot[colname]&) should contain a non-negative integer.
   (Slot[colname]&)[<|x->123|>]
*)

The first message comes from defining the function, and the second from calling the function. colname remains unevaluated within the function body and also when the function is executed.