The problem is due to current limitations in the type system implemented by Dataset
(v10.2). Specifically, the type system:
- cannot determine the type that results from applying
Merge[Identity]
to a list of associations, and
- does not support
#a
syntax when applied to an unknown type.
These limitations could be considered bugs.
Work-around #1 - Turn Off the Type System
The simplest way to work around any type-related query limitations is to turn off the type system, i.e. do not keep the data within a typed Dataset
:
rawData = data // Normal;
rawData // Query[Merge[Identity] /* (Max[#a] + Total[#b] &), {"a", "b"}]
(* 2 + x + y *)
No querying flexibility is lost by keeping data outside of a dataset. A query result can always be wrapped into a Dataset
as the last step to benefit from the nice visualization.
Work-around #2: Dodge the First Limitation
Alternatively, we could use an initial operator that does not suffer from limitation #1, such as Transpose
as noted in the question:
data[Transpose/*(Max[#a]+Total[#b]&),{"a","b"}]
(* 2 + x + y *)
Work-around #3: Dodge the Second Limitation
Or again, we could use key access operators that do not suffer from limitation #2, e.g.
data[Merge[Identity]/*(Max[#[["a"]]]+Total[#[["b"]]]&),{"a","b"}]
(* 2 + x + y *)
data[Merge[Identity]/*(Max[#["a"]]+Total[#["b"]]&),{"a","b"}]
(* 2 + x + y *)
data[Merge[Identity]/*(Max[Lookup[#, "a"]]+Total[Lookup[#,"b"]]&),{"a","b"}]
(* 2 + x + y *)
Since it is hard to know without experimentation which operators are subject to which limitations, work-around #1 is usually preferable as it requires minimal effort.
Discussion (current as of version 10.2)
Here is a minimal example that demonstrates the issue:
Dataset[{<|"a" -> 1|>}][Merge[Identity] /* (#a &)]
(* Failure[...] *)
The first limitation can be seen by asking the type system to tell us the result of applying Merge[Identity]
to a list of associations:
Needs["TypeSystem`"]
TypeApply[Merge[Identity], {DeduceType[{<|"a" -> 1|>}]}]
(* UnknownType *)
UnknownType
means that system will not know anything about the type of the result. But "unknown" is not "failure", so the expression will still evaluate successfully:
data[Merge[Identity]]
(* Dataset[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>, ...] *)
If we use a recognized operator (e.g. First
), then the type system gives a more useful result:
TypeApply[First, {DeduceType[{<|"a"->1|>}]}]
(* Assoc[Atom[String], Atom[Integer], 1] *)
The second limitation can be seen if we try applying #a&
to an unknown type:
TypeApply[#a&, {UnknownType}]
(* FailureType[...] *)
However, other part-access syntax forms are acceptable:
TypeApply[Lookup["a"], {UnknownType}]
(* UnknownType *)
TypeApply[#["a"]&, {UnknownType}]
(* UnknownType *)
TypeApply[#[["a"]]&, {UnknownType}]
(* UnknownType *)
I expect future versions of Mathematica to remove these limitations.