The behaviour is by design. When we query a Dataset
, the system tries to infer whether we wish the result to be a dataset itself, or just a simple value. The decision is made using some heuristics, but it essentially boils down to this: if the result is "atomic data", then it is returned directly. Otherwise, the result is wrapped back up into a Dataset
.
Dataset's Notion of Atomic Data
The notion of "atomic data" in this context is more elaborate than an AtomQ
test. In particular, not only lists but also associations are treated as non-atomic for this purpose. Conversely, many non-atomic expressions (such as Failure[...]
or Missing[...]
or Graphics[...]
) are treated as atomic data.
We can manually jam any value, atomic or otherwise, into a Dataset
, e.g.
Dataset[1]

Dataset[{}]

Dataset[<|"a" -> 1|>]

Dataset[Graphics[Disk[]]]

But any query, even an apparent identity operation, will trigger the heuristic as to whether the query result should be unwrapped:
Dataset[1][All]
(* 1 *)
Dataset[{}][All]

Dataset[<|"a" -> 1|>][All]

Dataset[Graphics[Disk[]]][All]

Lists and associations are the principal expression types that are rewrapped into datasets. The most important kinds of expressions that are treated as atomic data needing unwrapping are true atoms (AtomQ
, except associations), Missing[...]
, Failure[...]
or expressions with any of the following heads:
TypeSystem`$AtomicHeads
(* Hold[Entity,Quantity,DateObject,TimeObject,TemporalData,Image,
GeoPosition,Graph,Graphics,GeoGraphics,Sound,CloudObject,
File,URL] *)
There are other exceptions as well, but they are fringe cases. Beware that due to the heuristic nature of the test, the fine details concerning such fringe cases may vary from release to release.
The Case At Hand
Now consider our example dataset query:
ds[Apply[List], "b"]

Since the result is a list, it is wrapped back into a Dataset
.
Note that the use of Apply
here is essentially a non-operation since the data value was a list already:
ds[All, "b"]

ds[Identity, "b"]

This is an important point: even a top-level ascending query operator acts upon the data value before the dataset-wrapping heuristic is applied. This means that even Normal
cannot help us if it is used as a query operator. In fact, it is presently a type error to even attempt this on a list (v10.2.0, possible bug?):
ds[Normal, "b"]

We must use Normal
outside of the query to force the unwrapping:
ds[All, "b"] // Normal
(* 4, 8, 3 *)
The use of Apply[anyhead]
unwraps our data because expressions of the form anyhead[...]
fall into one of the "fringe cases" alluded to above:
ds[Apply[anyhead], "b"]
(* anyhead[4,8,3] *)
Specifically, any non-atomic value whose type is unknown will presently be unwrapped. Using traceTypes
from (89081), we see that this is the case:

It stands to reason that user-defined heads, especially inert ones, would be unwrapped after a query. But given the "fringe" nature of this case, this behaviour could change in future releases as the type system or heuristics develop.
List
. That is,List @@ ds[All, "b"]
will return aDataset
, butList @@ anyhead @@ ds[All, "b"]
returns aList
. $\endgroup$List @@ ds[anyhead,"b"]
$\endgroup$Apply
is an ascending operator? $\endgroup$Query
documentation says: Unless an operator is specifically recognized to be descending, it is assumed to be ascending. The descending operators are explicitly listed in the documentation.Apply
is not on that list. Some ascending operators are discussed explicitly in the documentation, but usually because they behave differently as query operators than they do when used as normal functions. $\endgroup$