A subtlety in Query

Let us consider a simple Dataset:

ds=Dataset[{<|"a"->12,"b"->4|>,<|"a"->7,"b"->8|>,<|"b"->3|>}]


Column "b" is easy to find:

Query[All, "b"][ds]


The result is still a Dataset. I want to change the head of the result by using an ascending operator on level 0:

Query[Apply[anyhead], "b"][ds]



That works. But not when anyhead is List:

Query[Apply[List], "b"][ds]

(* returns a Dataset *)


What is the reason for this behaviour? Could I have expected the result being a Dataset?

• That's really awful, can't seem to see why it is, you've probably seen this. A workaround is to first apply a dummy head, then apply List. That is, List @@ ds[All, "b"] will return a Dataset, but List @@ anyhead @@ ds[All, "b"] returns a List. – Jason B. Dec 23 '15 at 10:15
• Also see this answer. A slightly shorter way to get the list you want is List @@ ds[anyhead,"b"] – Jason B. Dec 23 '15 at 10:52
• @Jason. Many thanks for your interest in this strange behaviour and your references. I know there are many workarounds (using Normal is another one), but I am wondering why it does not work as I expected. Usually this means that there is something I do not completely understand ... – Fred Simons Dec 23 '15 at 11:07
• One thing that I couldn't find: Where is documented, that Apply is an ascending operator? – halirutan Dec 23 '15 at 14:12
• @halirutan The Query documentation says: Unless an operator is specifically recognized to be descending, it is assumed to be ascending. The descending operators are explicitly listed in the documentation. Apply is not on that list. Some ascending operators are discussed explicitly in the documentation, but usually because they behave differently as query operators than they do when used as normal functions. – WReach Dec 23 '15 at 19:38

The behaviour is by design. When we query a Dataset, the system tries to infer whether we wish the result to be a dataset itself, or just a simple value. The decision is made using some heuristics, but it essentially boils down to this: if the result is "atomic data", then it is returned directly. Otherwise, the result is wrapped back up into a Dataset.

Dataset's Notion of Atomic Data

The notion of "atomic data" in this context is more elaborate than an AtomQ test. In particular, not only lists but also associations are treated as non-atomic for this purpose. Conversely, many non-atomic expressions (such as Failure[...] or Missing[...] or Graphics[...]) are treated as atomic data.

We can manually jam any value, atomic or otherwise, into a Dataset, e.g.

Dataset[1]


Dataset[{}]


Dataset[<|"a" -> 1|>]


Dataset[Graphics[Disk[]]]


But any query, even an apparent identity operation, will trigger the heuristic as to whether the query result should be unwrapped:

Dataset[1][All]
(* 1 *)

Dataset[{}][All]


Dataset[<|"a" -> 1|>][All]


Dataset[Graphics[Disk[]]][All]


Lists and associations are the principal expression types that are rewrapped into datasets. The most important kinds of expressions that are treated as atomic data needing unwrapping are true atoms (AtomQ, except associations), Missing[...], Failure[...] or expressions with any of the following heads:

TypeSystem\$AtomicHeads
(* Hold[Entity,Quantity,DateObject,TimeObject,TemporalData,Image,
GeoPosition,Graph,Graphics,GeoGraphics,Sound,CloudObject,
File,URL] *)


There are other exceptions as well, but they are fringe cases. Beware that due to the heuristic nature of the test, the fine details concerning such fringe cases may vary from release to release.

The Case At Hand

Now consider our example dataset query:

ds[Apply[List], "b"]


Since the result is a list, it is wrapped back into a Dataset.

Note that the use of Apply here is essentially a non-operation since the data value was a list already:

ds[All, "b"]


ds[Identity, "b"]


This is an important point: even a top-level ascending query operator acts upon the data value before the dataset-wrapping heuristic is applied. This means that even Normal cannot help us if it is used as a query operator. In fact, it is presently a type error to even attempt this on a list (v10.2.0, possible bug?):

ds[Normal, "b"]


We must use Normal outside of the query to force the unwrapping:

ds[All, "b"] // Normal
(* 4, 8, 3 *)


The use of Apply[anyhead] unwraps our data because expressions of the form anyhead[...] fall into one of the "fringe cases" alluded to above:

ds[Apply[anyhead], "b"]


Specifically, any non-atomic value whose type is unknown will presently be unwrapped. Using traceTypes from (89081), we see that this is the case:

It stands to reason that user-defined heads, especially inert ones, would be unwrapped after a query. But given the "fringe" nature of this case, this behaviour could change in future releases as the type system or heuristics develop.

• +1 clear informative explanation as always – Mike Honeychurch Dec 24 '15 at 0:48
• I completely agree with Mike. – Fred Simons Dec 24 '15 at 10:44
• Ugh, this behaviour seems horrible. One of Mathematica's core principles is consistency, and it's an absolutely necessary principle if you're to easily understand a language with five thousand primitives. If I meet a function I've never seen before, I can already know how it works in broad terms just by reading the first line of the docs, but Dataset is an exception because it works so differently from almost every other Mathematica function. I'm used to "failure-or-success" varying according to heuristics I can't see, but I'm not used to the actual output of the function varying like this. – Patrick Stevens Dec 24 '15 at 13:06

I'm not sure I'm missing something here, but as far as I can see, you should regard this as a feature. When you evaluate your Query, one of the later steps is to decide whether or not the final result will still be wrapped in Dataset.

Take your simple example and browse through the Trace. There, one thing caught my eye: the function DatasetReturnRawDataQ. Looking a bit closer it seems, that this is the obstacle

Block[{DatasetReturnRawDataQ},
DatasetReturnRawDataQ[___] := True;
Query[Apply[List], "b"][ds]
]


and you get