8
$\begingroup$

The type system underlying Dataset seems a total kludge, as soon as one issue is fixed, here it recurs in a slightly different form - and difficult to isolate in minimal examples.

Here's a failure of Transpose in 11.1. Given this Dataset which has to be saved and imported as an .mx file otherwise copy/pasting its content into a new Dataset will not reproduce the error:

ds = Import["http://statigrafix.com/stackexchange/ds-transpose-issue.mx"]

enter image description here

This fails:

ds [All, Transpose]

Failure[[WarningSign] Message: The first two levels of _Association cannot be transposed. Tag: Transpose ]

But normalizing and or reforming a new dataset (after normal) shows that it works:

ds // Normal // Query[All, Transpose] // Dataset

and

ds // Normal // Dataset //  Query[All, Transpose] 

yield:

enter image description here

$\endgroup$
  • 3
    $\begingroup$ Somehow your Dataset is broken. Try (ds // Normal // Dataset)[All, Transpose] and see that it works. Your last line of code shows this as well as you do merely more that recreating the dataset. One hint is that the display of your dataset is not correct. You shouldn't see the raw Associations. $\endgroup$ – halirutan Apr 18 '17 at 23:29
  • $\begingroup$ @halirutan, it's not broken. the Dataset I saved is exactly what is imported - that's my point, once you cycle through // Normal // Dataset it works - the type system is reset. Also, that's how the dataset renders (Associations), - albeit I am only using a slice as it is proprietary data. $\endgroup$ – alancalvitti Apr 19 '17 at 1:54
  • $\begingroup$ If the dataset you saved looked exactly like your first screenshot it was already messed up. An Association of this structure should display differently, like your last screenshot but transposed. You question should rather be "what happened to my dataset and when?" because Transpose only fails because the dataset is internally incorrect. $\endgroup$ – halirutan Apr 19 '17 at 2:37
  • $\begingroup$ @halirutan, you are completely off on this issue - should is different than is. Why do you think I saved it as .mx instead of copy/pastying it as an Association (source code) and the wrapping Dataset around it? $\endgroup$ – alancalvitti Apr 19 '17 at 2:42
  • $\begingroup$ The workaround is ds[All, (Transpose@# &)].I have seen some strange Transpose behaviour as well. Think it has to do with the syntax sugar of Transpose in Datasets. Using a pure function hides Transpose from the syntax sugar and lets it just transpose the Association passed to it. Could be some sort of UpValue on Transpose in Dataset doing this. $\endgroup$ – Edmund Apr 19 '17 at 2:46
13
$\begingroup$

As observed in the question, the problem lies with the type information of the dataset. Specifically, the type information for the innermost association is missing (i.e. "any type"). This causes the operator Query[Transpose] to fail as the type system prohibits attempting to transpose an association with an arbitrary expression.

Work-arounds

As noted in the question, one work-around is to force the type information to be regenerated by type-deduction before querying. Here is a succinct way to do that:

ds[Dataset][All, Transpose]

Another work-around is to use the "regular" Transpose function instead of the Transpose query operator:

ds[All, Transpose[#]&]

The second work-around is brittle, but the first will cure most(?) type-inferencing errors.

Also see the definition of untypedDataset near the end of this response which bypasses dataset type-checking.

Analysis (current as of version 11.1.0)

Let us inspect the data type of the dataset:

Needs["Dataset`"]
Needs["TypeSystem`"]

ds // GetType
(* Assoc[Atom[String], Assoc[Atom[String], AnyType, AnyLength], AnyLength] *)

The type tells us that:

  • The top-level element is an association that maps strings to associations.
  • The level one element is an association that maps strings to AnyType.

It is this AnyType that causes the problem. The type-inferencer will reject the application of the Transpose query operator to this type:

TypeApply[Query[Transpose], {Assoc[Atom[String], AnyType, AnyLength]}]
(* FailureType[...] *)

Transpose Function Work-around

On the other hand, it will permit the regular Transpose function to be applied:

TypeApply[Transpose, {Assoc[Atom[String], AnyType, AnyLength]}]
(* UnknownType *)

It is not sure what to make of the operation, but it does not reject it. This difference is explained by the type signatures of the two operations:

Signatures[Query[Transpose]]

transpose operator signatures

Signatures[Transpose]
(* --- none --- *)

We can see that the Transpose operator will accept an association whose values are structures, tuples, vectors or associations... but any other value types will fail (see the FailureType entry for type t_ further down the list). The Transpose function has no type signatures at all, so anything goes.

This explains why the work-around using the Transpose function gives us a result.

Type Deduction Work-around

The first work-around is less mysterious. The type-deducer does a much better job of generating type information than the type-inferencer. So we can fix up the type information by forcing a type rescan:

ds[Dataset] // GetType

(* Assoc[Atom[String]
   , Struct[{"history", "future"}
     , { Assoc[Atom[Enumeration["Adjustment", "Balance", ...]], Atom[Real], 6]
       , Assoc[Atom[Enumeration["Adjustment", "Balance", ...]],  Atom[Real], 6]
       }
     ]
   , 2
   ] *)

We now have the full type information. Levels one and two are now a structure containing associations. If we look again at the signatures for the Transpose operator, we see that this is a permissible combination. Indeed, the query now works:

ds[Dataset][All, Transpose]

dataset screenshot

Missing Type Information?

So how did the type information get messed up in the first place? The internal structure of ds shows us that this dataset was generated by applying the following query to some other dataset:

Query[1, 1, 1, 60, (1 ;; 2) /* KeyMap[Function[expr, StringTake[expr, -4]]], "eventTotals"]

The source dataset is not included in the mx file. When the type inferencer applied this query to the original data, it was unable to compute the full type information. We got "watered down" information instead, where the inner associations were identified as AnyType.

General Strategy

To avoid these problems, I tend to avoid wrapping data in Dataset and use constructions like rawData // Query[...] instead. But, if I am using datasets, I often force the type information to be rebuilt by adding a trailing top-level Dataset operator, e.g.

myDataset[someOps /* Dataset, moreOps]

This way, the dataset has the best available type information and downstream queries do not have to worry about it.

Untyped Datasets?

Dataset provides three main services: type-checking, data visualization and an abbreviated syntax for chaining applications of Query. If we want to dispense with type-checking but retain the other features, we could write our own convenience wrapper. Here is a sketch implementation:

untypedDataset[ds_Dataset] := untypedDataset[ds // Normal]
untypedDataset[data_][queryOps___] := untypedDataset[data // Query[queryOps]]
Format[untypedDataset[data_]] := data // Dataset

So then:

utds = untypedDataset[ds]

dataset screenshot

utds[All, Transpose]

dataset screenshot

utds[All, Transpose]["b685"][All, "future"][{"Balance", "Charge"}]

dataset screenshot

$\endgroup$
  • 2
    $\begingroup$ Very informative (+1). I always thought it had something to do with special rules (a.k.a syntax sugar) for Transpose but had no clue how to investigate it. $\endgroup$ – Edmund Apr 19 '17 at 2:54
  • $\begingroup$ @WReach, " I tend to avoid wrapping data in Dataset" - in other words, you don't use Dataset? $\endgroup$ – alancalvitti Apr 19 '17 at 3:03
  • $\begingroup$ @WReach, by the way, same error without the KeyMap[Function...] - that's just to mask the account numbers. The rest is just routine projections of datasets. The real Q is why does the type system not understand it while inside Dataset, but gets it after // Normal // Dataset ? $\endgroup$ – alancalvitti Apr 19 '17 at 3:05
  • 2
    $\begingroup$ @alancalvitti I only use Dataset as a terminal visualization tool similar, say, to using MatrixForm. So I operate upon the raw data stored in variables and only at the very end visualize the result with Dataset. The answer to the "real Q" is that the type inferencer (invoked by query applications) gets different answers than the type deducer (invoked by dataset construction). $\endgroup$ – WReach Apr 19 '17 at 3:09
  • 1
    $\begingroup$ @WReach, thanks for the in-depth analysis as always. Your key phrase that it's the Query version that fails led me to use the non-operator form (Transpose[#] &) - where parentheses typically needed with RightComposition. $\endgroup$ – alancalvitti Jul 14 '17 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.