DeleteMissing doesn't match in Association tables

Question

DeleteMissing doesn't work in Association tables:

Dataset[{<|"a" -> 1, "b" -> Missing[]|>, <|"a" -> 2, 
     "b" -> 3|>}][DeleteMissing,"b"] // Normal

{Missing[], 3}

Whereas DeleteCases[_Missing] works, as does [All, "b"][DeleteMissing] and [{DeleteMissing, DeleteCases[_Missing]}, "b"] --> {{3},{3}}

Answer 1

The documentation for Dataset (and Query) indicates that DeleteMissing is a descending operator. This means that it is applied to the original list before the "b" suboperator is applied. Since none of the original list elements have the head Missing, the descending DeleteMissing is effectively a null operation in this case.

Original Case - DeleteMissing

What we need to do is convert the operator into an ascending operator. We start with the original case:

ds = Dataset[{<|"a"->1,"b"->Missing[]|>,<|"a"->2,"b"->3|>}];

ds[DeleteMissing, "b"] // Normal
(* {Missing[], 3} *)

Any descending operator can be converted into an ascending operator by wrapping it with Query[...]:

ds[Query[DeleteMissing], "b"] // Normal
(* {3} *)

Pure functions are also treated as ascending:

ds[DeleteMissing[#]&, "b"] // Normal
(* {3} *)

Composing a descending operator with a preceding ascending operator will also convert it -- #& is a terse choice:

ds[#& /* DeleteMissing, "b"] // Normal
(* {3} *)

As noted in the question, performing a second query will allow the descending operator to be applied in descending fashion, but after the operations in the first query are complete:

ds[All, "b"][DeleteMissing] // Normal
(* {3} *)

DeleteCases

DeleteCases is an ascending operator, which is why it gives the desired result:

ds[DeleteCases[_Missing], "b"] // Normal
(* {3} *)

{DeleteMissing}

The {...} operator is (usually) an ascending operator. Thus, any suboperators it contains are applied after the other query descending operators:

ds[{DeleteMissing}, "b"] // Normal
(* {{3}} *)

While tangential to the present question, note that in some circumstances the {...} operator can be considered a descending operator. This happens if and only if the contained suboperators are all simple part operators. For example:

{<|"a" -> 11|>, <|"a" -> 22|>, <|"a" -> 33|> } // Query[{2, 3}, "a"]
(* {22, 33} *)

This exception is made because even though part operators alter the descending structure, they do so in a well-defined fashion which is akin to filtering. The query machinery can still infer useful properties about the result of such part filtering. DeleteMissing is not so benign, and can change the shape of the data in ways that are more difficult to reason about up front. The difference is that a list of part operators generates a result of fixed cardinality, whereas {DeleteMissing} generates a result of varying cardinality. So {DeleteMissing} does not qualify for the exception and is interpreted as an ascending operator.