# Using Merge[Identity] and Transpose on a Dataset

While experimenting with this answer, I ran across the following:

data = Dataset[{
<|"a" -> 1, "b" -> "x"|>,
<|"a" -> 2, "b" -> "y"|>}];

data[Transpose /* f, {"a", "b"}]

(* f[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>] *)

data[Merge[Identity] /* f, {"a", "b"}]

(* f[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>] *)


But when I fill in f with a function:

(*This works*)
data[Transpose /* (Max[#a] + Total[#b] &), {"a", "b"}]

(*This fails*)
data[Merge[Identity] /* (Max[#a] + Total[#b] &), {"a", "b"}]


Replacing #a and #b with #[[1]] and #[[2]] seems to work.

1. Why does using Merge[Identity] fail?
2. Is Transpose[] a descending operator in a Dataset, and is this related to 1.?
• Welcome to Mathematica.SE! I suggest the following: 1) As you receive help, try to give it too, by answering questions in your area of expertise. 2) Read the faq! 3) When you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. Also, please remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign! Jul 22, 2015 at 1:32

I believe what you experience should not happen because when you use the alternative notation for named slots, it works as expected:

data = Dataset[{<|"a" -> 1, "b" -> "x"|>, <|"a" -> 2, "b" -> "y"|>}];
data[Merge[Identity] /* (Max[#["a"]] + Total[#["b"]] &)]
(* 2 + "x" + "y" *)


Additionally, you could split your original function into two parts and it still works:

data[Merge[Identity]];
%[(Max[#a] + Total[#b] &)]
(* 2 + "x" + "y" *)


Therefore, without further inspection through Trace or friends, I would say this is a bug or a feature I don't understand. Nevertheless, I cannot answer your first question why it specifically fails.

To your second question: Transpose should be an ascending operator, when I understood the documentation and the meaning correctly. Let me illustrate this. Assume we have to following test Dataset:

test = Dataset[RandomInteger[9, {2, 3}]]


Now, we do some test with the descending operator All. Descending means, first all data is selected and the following min/max function is applied to each element on the next level, which are the vectors {5,7,9} and {8,3,7}

test[All, {Min[#], Max[#]} &] //Normal
(* {{5, 9}, {3, 8}} *)


The output are the expected min/max values of each vector.

Transpose works differently. First, the min/max function is calculated on all elements and then the result is transpose. And this, although Transpose comes first:

test[Transpose, {Min[#], Max[#]} &] // Normal
(* {{5, 3}, {9, 8}} *)


If Transpose would be a descending operator, the output should be

test[Transpose][All, {Min[#], Max[#]} &] // Normal
(* {{5, 8}, {3, 7}, {7, 9}} *)


but this is not the case.

• Thanks for the response. Your example helped me see that Transpose is indeed ascending. Can you help explain why something like data[f,{"a","b"}]/.f->Transpose fails when data[Transpose, {"a","b"}] does not?
– user
Jul 22, 2015 at 2:41
• My best guess is that Transpose inside Dataset is syntactic sugar because people think of tabular data when they hear Dataset. Something like transposing tabular data seems very natural and this might be a reason why this was built in to work with Dataset. Transposing a list of Associations on the other hand will not work. An Association is an atom, something like 1 or a. It makes as much sense as doing Transpose[{1, 2}]. Jul 22, 2015 at 2:48
• As @halirutan says, Transpose within a query is actually transformed to another function, namely GeneralUtilitiesAssociationTranspose. We can see this by evaluating Query[Transpose] // Normal. Thus, we must write data[f] /. f -> GeneralUtilitiesAssociationTranspose to get something essentially equivalent to data[Transpose] // Normal. Another way would be to write Unevaluated[data[f]] /. f -> Transpose, which would leave the result as a dataset. Jul 22, 2015 at 18:06

The problem is due to current limitations in the type system implemented by Dataset (v10.2). Specifically, the type system:

1. cannot determine the type that results from applying Merge[Identity] to a list of associations, and
2. does not support #a syntax when applied to an unknown type.

These limitations could be considered bugs.

Work-around #1 - Turn Off the Type System

The simplest way to work around any type-related query limitations is to turn off the type system, i.e. do not keep the data within a typed Dataset:

rawData = data // Normal;
rawData // Query[Merge[Identity] /* (Max[#a] + Total[#b] &), {"a", "b"}]
(* 2 + x + y *)


No querying flexibility is lost by keeping data outside of a dataset. A query result can always be wrapped into a Dataset as the last step to benefit from the nice visualization.

Work-around #2: Dodge the First Limitation

Alternatively, we could use an initial operator that does not suffer from limitation #1, such as Transpose as noted in the question:

data[Transpose/*(Max[#a]+Total[#b]&),{"a","b"}]
(* 2 + x + y *)


Work-around #3: Dodge the Second Limitation

Or again, we could use key access operators that do not suffer from limitation #2, e.g.

data[Merge[Identity]/*(Max[#[["a"]]]+Total[#[["b"]]]&),{"a","b"}]
(* 2 + x + y *)

data[Merge[Identity]/*(Max[#["a"]]+Total[#["b"]]&),{"a","b"}]
(* 2 + x + y *)

data[Merge[Identity]/*(Max[Lookup[#, "a"]]+Total[Lookup[#,"b"]]&),{"a","b"}]
(* 2 + x + y *)


Since it is hard to know without experimentation which operators are subject to which limitations, work-around #1 is usually preferable as it requires minimal effort.

Discussion (current as of version 10.2)

Here is a minimal example that demonstrates the issue:

Dataset[{<|"a" -> 1|>}][Merge[Identity] /* (#a &)]
(* Failure[...] *)


The first limitation can be seen by asking the type system to tell us the result of applying Merge[Identity] to a list of associations:

Needs["TypeSystem"]

TypeApply[Merge[Identity], {DeduceType[{<|"a" -> 1|>}]}]
(* UnknownType *)


UnknownType means that system will not know anything about the type of the result. But "unknown" is not "failure", so the expression will still evaluate successfully:

data[Merge[Identity]]
(* Dataset[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>, ...] *)


If we use a recognized operator (e.g. First), then the type system gives a more useful result:

TypeApply[First, {DeduceType[{<|"a"->1|>}]}]
(* Assoc[Atom[String], Atom[Integer], 1] *)


The second limitation can be seen if we try applying #a& to an unknown type:

TypeApply[#a&, {UnknownType}]
(* FailureType[...] *)


However, other part-access syntax forms are acceptable:

TypeApply[Lookup["a"], {UnknownType}]
(* UnknownType *)

TypeApply[#["a"]&, {UnknownType}]
(* UnknownType *)

TypeApply[#[["a"]]&, {UnknownType}]
(* UnknownType *)


I expect future versions of Mathematica to remove these limitations.

• No querying flexibility is lost by keeping data outside of a dataset. - is the type system only used for optimization? Jan 4, 2016 at 19:50
• Up to v10.3, the data type information is used primarily for formatting the graphical representation of a Dataset` or for performing a small number of proactive sanity checks upon a query. I have speculated that future releases may use the type information for data and/or algorithm optimization. I discuss the use of type information more fully in (87479), (89081) and (102696). Jan 4, 2016 at 22:54