A subtlety in Query

Question

Let us consider a simple Dataset:

ds=Dataset[{<|"a"->12,"b"->4|>,<|"a"->7,"b"->8|>,<|"b"->3|>}]

Column "b" is easy to find:

Query[All, "b"][ds]

The result is still a Dataset. I want to change the head of the result by using an ascending operator on level 0:

Query[Apply[anyhead], "b"][ds]

(* anyhead[4,8,3] *)

That works. But not when anyhead is List:

Query[Apply[List], "b"][ds]

(* returns a Dataset *)

What is the reason for this behaviour? Could I have expected the result being a Dataset?

Answer 1

The behaviour is by design. When we query a Dataset, the system tries to infer whether we wish the result to be a dataset itself, or just a simple value. The decision is made using some heuristics, but it essentially boils down to this: if the result is "atomic data", then it is returned directly. Otherwise, the result is wrapped back up into a Dataset.

Dataset's Notion of Atomic Data

The notion of "atomic data" in this context is more elaborate than an AtomQ test. In particular, not only lists but also associations are treated as non-atomic for this purpose. Conversely, many non-atomic expressions (such as Failure[...] or Missing[...] or Graphics[...]) are treated as atomic data.

We can manually jam any value, atomic or otherwise, into a Dataset, e.g.

Dataset[1]

Dataset[{}]

Dataset[<|"a" -> 1|>]

Dataset[Graphics[Disk[]]]

But any query, even an apparent identity operation, will trigger the heuristic as to whether the query result should be unwrapped:

Dataset[1][All]
(* 1 *)

Dataset[{}][All]

Dataset[<|"a" -> 1|>][All]

Dataset[Graphics[Disk[]]][All]

Lists and associations are the principal expression types that are rewrapped into datasets. The most important kinds of expressions that are treated as atomic data needing unwrapping are true atoms (AtomQ, except associations), Missing[...], Failure[...] or expressions with any of the following heads:

TypeSystem`$AtomicHeads
(* Hold[Entity,Quantity,DateObject,TimeObject,TemporalData,Image,
        GeoPosition,Graph,Graphics,GeoGraphics,Sound,CloudObject,
        File,URL] *)

There are other exceptions as well, but they are fringe cases. Beware that due to the heuristic nature of the test, the fine details concerning such fringe cases may vary from release to release.

The Case At Hand

Now consider our example dataset query:

ds[Apply[List], "b"]

Since the result is a list, it is wrapped back into a Dataset.

Note that the use of Apply here is essentially a non-operation since the data value was a list already:

ds[All, "b"]

ds[Identity, "b"]

This is an important point: even a top-level ascending query operator acts upon the data value before the dataset-wrapping heuristic is applied. This means that even Normal cannot help us if it is used as a query operator. In fact, it is presently a type error to even attempt this on a list (v10.2.0, possible bug?):

ds[Normal, "b"]

We must use Normal outside of the query to force the unwrapping:

ds[All, "b"] // Normal
(* 4, 8, 3 *)

The use of Apply[anyhead] unwraps our data because expressions of the form anyhead[...] fall into one of the "fringe cases" alluded to above:

ds[Apply[anyhead], "b"]
(* anyhead[4,8,3] *)

Specifically, any non-atomic value whose type is unknown will presently be unwrapped. Using traceTypes from (89081), we see that this is the case:

It stands to reason that user-defined heads, especially inert ones, would be unwrapped after a query. But given the "fringe" nature of this case, this behaviour could change in future releases as the type system or heuristics develop.

Answer 2

I'm not sure I'm missing something here, but as far as I can see, you should regard this as a feature. When you evaluate your Query, one of the later steps is to decide whether or not the final result will still be wrapped in Dataset.

Take your simple example and browse through the Trace. There, one thing caught my eye: the function Dataset`ReturnRawDataQ. Looking a bit closer it seems, that this is the obstacle

Block[{Dataset`ReturnRawDataQ},
 Dataset`ReturnRawDataQ[___] := True;
 Query[Apply[List], "b"][ds]
]

and you get