Using Merge[Identity] and Transpose on a Dataset

Question

While experimenting with this answer, I ran across the following:

data = Dataset[{
    <|"a" -> 1, "b" -> "x"|>,
    <|"a" -> 2, "b" -> "y"|>}];

data[Transpose /* f, {"a", "b"}]

(* f[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>] *)

data[Merge[Identity] /* f, {"a", "b"}]

(* f[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>] *)

But when I fill in f with a function:

(*This works*)
data[Transpose /* (Max[#a] + Total[#b] &), {"a", "b"}]

(*This fails*)
data[Merge[Identity] /* (Max[#a] + Total[#b] &), {"a", "b"}]

Replacing #a and #b with #[[1]] and #[[2]] seems to work.

1. Why does using Merge[Identity] fail?
2. Is Transpose[] a descending operator in a Dataset, and is this related to 1.?

Answer 1

I believe what you experience should not happen because when you use the alternative notation for named slots, it works as expected:

data = Dataset[{<|"a" -> 1, "b" -> "x"|>, <|"a" -> 2, "b" -> "y"|>}];
data[Merge[Identity] /* (Max[#["a"]] + Total[#["b"]] &)]
(* 2 + "x" + "y" *)

Additionally, you could split your original function into two parts and it still works:

data[Merge[Identity]];
%[(Max[#a] + Total[#b] &)]
(* 2 + "x" + "y" *)

Therefore, without further inspection through Trace or friends, I would say this is a bug or a feature I don't understand. Nevertheless, I cannot answer your first question why it specifically fails.

To your second question: Transpose should be an ascending operator, when I understood the documentation and the meaning correctly. Let me illustrate this. Assume we have to following test Dataset:

test = Dataset[RandomInteger[9, {2, 3}]]

Now, we do some test with the descending operator All. Descending means, first all data is selected and the following min/max function is applied to each element on the next level, which are the vectors {5,7,9} and {8,3,7}

test[All, {Min[#], Max[#]} &] //Normal
(* {{5, 9}, {3, 8}} *)

The output are the expected min/max values of each vector.

Transpose works differently. First, the min/max function is calculated on all elements and then the result is transpose. And this, although Transpose comes first:

test[Transpose, {Min[#], Max[#]} &] // Normal
(* {{5, 3}, {9, 8}} *)

If Transpose would be a descending operator, the output should be

test[Transpose][All, {Min[#], Max[#]} &] // Normal
(* {{5, 8}, {3, 7}, {7, 9}} *)

but this is not the case.

Answer 2

The problem is due to current limitations in the type system implemented by Dataset (v10.2). Specifically, the type system:

1. cannot determine the type that results from applying Merge[Identity] to a list of associations, and
2. does not support #a syntax when applied to an unknown type.

These limitations could be considered bugs.

Work-around #1 - Turn Off the Type System

The simplest way to work around any type-related query limitations is to turn off the type system, i.e. do not keep the data within a typed Dataset:

rawData = data // Normal;
rawData // Query[Merge[Identity] /* (Max[#a] + Total[#b] &), {"a", "b"}]
(* 2 + x + y *)

No querying flexibility is lost by keeping data outside of a dataset. A query result can always be wrapped into a Dataset as the last step to benefit from the nice visualization.

Work-around #2: Dodge the First Limitation

Alternatively, we could use an initial operator that does not suffer from limitation #1, such as Transpose as noted in the question:

data[Transpose/*(Max[#a]+Total[#b]&),{"a","b"}]
(* 2 + x + y *)

Work-around #3: Dodge the Second Limitation

Or again, we could use key access operators that do not suffer from limitation #2, e.g.

data[Merge[Identity]/*(Max[#[["a"]]]+Total[#[["b"]]]&),{"a","b"}]
(* 2 + x + y *)

data[Merge[Identity]/*(Max[#["a"]]+Total[#["b"]]&),{"a","b"}]
(* 2 + x + y *)

data[Merge[Identity]/*(Max[Lookup[#, "a"]]+Total[Lookup[#,"b"]]&),{"a","b"}]
(* 2 + x + y *)

Since it is hard to know without experimentation which operators are subject to which limitations, work-around #1 is usually preferable as it requires minimal effort.

---

Discussion (current as of version 10.2)

Here is a minimal example that demonstrates the issue:

Dataset[{<|"a" -> 1|>}][Merge[Identity] /* (#a &)]
(* Failure[...] *)

The first limitation can be seen by asking the type system to tell us the result of applying Merge[Identity] to a list of associations:

Needs["TypeSystem`"]

TypeApply[Merge[Identity], {DeduceType[{<|"a" -> 1|>}]}]
(* UnknownType *)

UnknownType means that system will not know anything about the type of the result. But "unknown" is not "failure", so the expression will still evaluate successfully:

data[Merge[Identity]]
(* Dataset[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>, ...] *)

If we use a recognized operator (e.g. First), then the type system gives a more useful result:

TypeApply[First, {DeduceType[{<|"a"->1|>}]}]
(* Assoc[Atom[String], Atom[Integer], 1] *)

The second limitation can be seen if we try applying #a& to an unknown type:

TypeApply[#a&, {UnknownType}]
(* FailureType[...] *)

However, other part-access syntax forms are acceptable:

TypeApply[Lookup["a"], {UnknownType}]
(* UnknownType *)

TypeApply[#["a"]&, {UnknownType}]
(* UnknownType *)

TypeApply[#[["a"]]&, {UnknownType}]
(* UnknownType *)

I expect future versions of Mathematica to remove these limitations.