7
$\begingroup$

I think similar to unexpected-behavior-of-keytake there are other issues with v11.0 KeyTake and (more useful) KeyDrop that show up in Dataset only:

In Associations Transpose can precede manual listing of keys:

ds = <|"a" -> <|"x" -> 1, "y" -> 2|>, "b" -> <|"x" -> 3, "y" -> 4|>, 
  "c" ->  <|"x" -> 5, "y" -> 6|> |>

That is:

ds // Transpose // Query[{"x"}]

<|"x" -> <|"a" -> 1, "b" -> 3, "c" -> 5|>|>

And KeyTake

ds // Transpose // KeyTake["x"]

But in Dataset, this works:

Dataset[ds][Transpose, {"x"}]

Yet this fails (similarly for KeyDrop)

Dataset[ds][Transpose, KeyTake["x"]]

Message: The first two levels of Association[(String->)..] cannot be transposed.

Known bug?

$\endgroup$
  • $\begingroup$ In Dataset,you need Dataset[ds][Transpose /* "x"] or Dataset[ds][Transpose /* KeyTake["x"]] $\endgroup$ – yode Mar 6 '17 at 10:49
6
$\begingroup$

This operation used to work back in v10.0 when the dataset type-checking was less aggressive, but has been broken since v10.1. The root cause is that the type inferencer cannot deduce the result type of the KeyTake operation and thus errs on the side of caution to reject the final transposition.

Work-arounds

1) Query outside of Dataset (the #1 strategy when faced with dataset weirdness):

ds // Query[Transpose, KeyTake["x"]] // Dataset

2) Use KeySelect instead of KeyTake:

Dataset[ds][Transpose, KeySelect[# === "x" &]]

3) Hack like there's no tomorrow:

Needs["TypeSystem`"]

Block[{FailureType = AnyType&}
, Dataset[ds][Transpose, KeyTake["x"]]
]

Discussion

This analysis is current as of version 11.0.1.

The operation works just fine in a stand-alone query:

ds // Query[Transpose, KeyTake["x"]]
(* <|"x" -> <|"a" -> 1, "b" -> 3, "c" -> 5|>|> *)

but, as noted in the question, fails as a dataset query:

Dataset[ds][Transpose, KeyTake["x"]]
(* ... failure ... *)

This gives us a clue that it is probably a type-inferencing error. Using traceTypes from (89081) confirms that guess:

traceTypes[Dataset[ds2]][Transpose, KeyTake["x"]]

traceTypes output

Specifically, the type inferencer does not know what type results when KeyTake is applied to our association type:

Needs["TypeSystem`"]

TypeApply[KeyTake["x"], {DeduceType[<|"x" -> 1, "y" -> 2|>]}]
(* UnknownType *)

As a consequence, the inferencer cannot deduce whether the AssociationTranspose will succeed. It errs on the side of caution and rejects the operation.

Note that KeyTake in itself operates properly:

Dataset[ds][All, KeyTake["x"]]

Standalone KeyTake

But the lack of type information for the result will block the operation of subsequent operators for which the inferencer takes a defensive stance. Transpose is one such operator as we can see from its recognized type signatures:

Signatures[Transpose // Query // Normal]

Transpose Signatures

The second-last type signature rule kicks in when an unrecognized input type is encountered and is responsible for error message that we see.

We can check what type signatures are supported for KeyTake:

Signatures[KeyTake]
(* ... blank ... *)

Aha... there are none. Contrast this with, for example, KeySelect:

Signatures[KeySelect]

KeySelect signatures

The presence of the signature implies that KeySelect will work -- which it does:

Dataset[ds][Transpose, KeySelect[# == "x" &]]

KeySelect result

KeyDrop has no valid signatures either:

Signatures[KeyDrop]
(* ... blank ... *)

Our conclusion is that KeyTake and KeyDrop are not fully supported in dataset queries. This omission could be considered as either an incomplete implementation or as a bug (reader's choice).

Normally, setting FailureAction -> None will inhibit any aggressive error-checking in query operations. Unfortunately the type inferencer rarely (if ever?) respects this setting. And indeed it does not help us with the cases under discussion. Incomplete implementation and/or bug?

$\endgroup$
  • $\begingroup$ Thanks WReach, another detailed analysis. I think the type-system is a regression to C++/templates, is not that useful for its main intended application: formatting - in fact more often than not I Normalize it away and run ad-hoc formatting. $\endgroup$ – alancalvitti Nov 9 '16 at 21:12
  • $\begingroup$ Indeed. I tend to treat Dataset as a display wrapper much like MatrixForm, only adding it at the very end when it is time to visualize. I find the type deducer used for dataset creation to be much more reliable than the type inferencer used for dataset querying. On the other hand Query seems to be much more solid than it was in the early 10.x releases and I find myself using it quite a lot on general expressions (often displacing Map, Part, etc even for simple transformations). $\endgroup$ – WReach Nov 9 '16 at 23:40
  • $\begingroup$ is the type system and inference engine invoked in Query even when querying Associations (rather than Datasets)? $\endgroup$ – alancalvitti Nov 16 '16 at 19:08
  • $\begingroup$ Type-inferencing is not invoked for ... // Query[...] where the left-hand side is a non-dataset expression. We can observe this by establishing On[TypeSystem`TypeApply] before evaluating <|"a"->1|> // Query["a"] and Dataset[<|"a"->1|>]["a"]. Only in the latter case will we see invocation of the type-inferencing system (TypeApply). And, of course, both you and I need only count bruises from attempts to use Dataset[...][...]. :) It has been my experience that non-dataset queries are reliable (and as far as I can recall every rare exception has been fixed). $\endgroup$ – WReach Nov 16 '16 at 19:32
  • $\begingroup$ That's good news, gives hope for a ground-up re-implementation of Dataset's formatting based on the built-in expression manipulation (which in addition might allow pattern matching). For example, in v10 formatting workarounds using Framed, Column, Grid etc, within and outside Dataset allowed a lot of freedom, so it's feasible. Unfortunately v11 broke a lot of these. $\endgroup$ – alancalvitti Nov 16 '16 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.