We must use Key[9] in place of 9:
dataset2[GroupBy[Key[9]]]
Note that dataset visualizer does not do a very good job with the result (non-string association keys often confuse it). It is easier to see that we have obtained the correct result by unwrapping the data:
dataset2[GroupBy[Key[9]]] // Normal
(* <| "x" -> {<|"a" -> 1, 9 -> "x"|>, <|"a" -> 4, 9 -> "x"|>}
, "y" -> {<|"a" -> 2, 9 -> "y"|>, <|"a" -> 5, 9 -> "y"|>}
, "z" -> {<|"a" -> 3, 9 -> "z"|>, <|"a" -> 6, 9 -> "z"|>}
|>
*)
Query Operator vs. Normal Function
This is a case where a query operator has different semantics from its liked-named function counterpart.
The normal function GroupBy takes a function argument for grouping purposes. Using the string "b" as a grouping function does not produce a particularly useful result:
dataset // Normal // GroupBy["b"]
(* <| "b"[<|"a" -> 1, "b" -> "x"|>] -> {<|"a" -> 1, "b" -> "x"|>}
, "b"[<|"a" -> 2, "b" -> "y"|>] -> {<|"a" -> 2, "b" -> "y"|>}
, "b"[<|"a" -> 3, "b" -> "z"|>] -> {<|"a" -> 3, "b" -> "z"|>}
, "b"[<|"a" -> 4, "b" -> "x"|>] -> {<|"a" -> 4, "b" -> "x"|>}
, "b"[<|"a" -> 5, "b" -> "y"|>] -> {<|"a" -> 5, "b" -> "y"|>}
, "b"[<|"a" -> 6, "b" -> "z"|>] -> {<|"a" -> 6, "b" -> "z"|>}
|>
*)
When using the GroupBy function, we must use Key["b"] instead of "b":
dataset // Normal // GroupBy[Key["b"]]
(* <| "x" -> {<|"a" -> 1, "b" -> "x"|>, <|"a" -> 4, "b" -> "x"|>}
, "y" -> {<|"a" -> 2, "b" -> "y"|>, <|"a" -> 5, "b" -> "y"|>}
, "z" -> {<|"a" -> 3, "b" -> "z"|>, <|"a" -> 6, "b" -> "z"|>}
|>
*)
By contrast, the GroupBy query operator receives special treatment. The Details section of the Dataset documentation says this:
Syntactic Sugar
Functions such as CountsBy, GroupBy, and TakeLargestBy normally take another function as one of their arguments. When working with associations in a Dataset, it is common to use this "by" function to look up the value of a column in a table.
To facilitate this, Dataset queries allow the syntax "string" to mean Key["string"] in such contexts. For example, the query operator GroupBy["string"] is automatically rewritten to GroupBy[Key["string"]] before being executed.
Similarly, the expression GroupBy[dataset,"string"] is rewritten as GroupBy[dataset,Key["string"]].
It is because of this query syntactic sugar that we can omit the Key[...] wrapper for strings -- and only strings. So in a query we can write GroupBy["b"] but not GroupBy[9].
The moral of this story is that we cannot safely assume that a query operator behaves in exactly the same fashion as a function with the same name.
