Query and Dataset Syntax Peculiarities

Question

Question

Can someone please explain to me the syntax of Query and why its behavior is so befuddling to me at the moment?

I am going to go through a few simple examples to demonstrate my confusion.

Example

Let's take the List of Associations from the Query documentation (and modify it slightly)

d1 = Dataset@{
   <|"a" -> 1, "b" -> "x", "c" -> Missing["KeyAbsent", ""]|>,
   <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
   <|"a" -> 3, "b" -> "z", "c" -> Missing["KeyAbsent", ""]|>,
   <|"a" -> 4, "b" -> "x", "c" -> "Bad Value"|>,
   <|"a" -> 5, "b" -> "y", "c" -> Missing|>,
   <|"a" -> 6, "b" -> "z", "c" -> {}|>}

For a sanity check, let's run the first example from the documentation:

Great. That works. What about DeleteDuplicates, which should delete anything with the Head Missing.

Ok that doesn't work. Well maybe these values do not have the Head Missing?

Huh... that worked! Now what if I just swap Head for DeleteMissing?

What just happened?

Maybe it is the #c?

Nope... Forcing the Notebook to rearrange this dataset (where a is the output from above)

Ok... let's try Select in a variety of forms:

Query[All, Select[Head@#c == List] &]@d1
d1[All, Select[#c == {2, 3}] &]
Select[d1, #c == {2, 3} &]

only the last one works...

What about the same thing with DeleteCases?

Query[All, DeleteCases[Head@#c == List] &]@d1
d1[All, DeleteCases[#c == {2, 3}] &]
DeleteCases[d1, #c == {2, 3} &]

Well now even the third form isn't working...

What is going on?

Documentation

Query Dataset DeleteMissing Select

Answer 1

Let's go through it blow-by-blow...

Query[DeleteMissing] @ d1

This will apply DeleteMissing to the list of associations. The documentation says:

DeleteMissing[list] drops elements with head Missing from a list.

All of the elements of the list have head Association, so this operation is a no-op.

d1[All, #c &, Head]

This works as advertised, showing the heads of the values associated with the key c in each association.

d1[All, #c &, DeleteMissing]

This asks to delete missing elements from the lists that appear as values associated with the key c in each association. Most of the c values are not lists. The error that appears reflects that fact.

Query[All, DeleteMissing] @ d1

This asks to delete from each association in the list any key/value pairs whose value has the head Missing. If we look at the Normal form of the result we find that this is what happened:

d1[All, DeleteMissing] // Normal

(* { <|a->1, b->x|>
   , <|a->2, b->y, c->{2,3}|>
   , <|a->3, b->z|>
   , <|a->4, b->x, c->Bad Value|>
   , <|a->5, b->y, c->Missing|>
   , <|a->6, b->z, c->{}|>
   }
*)

As noted in the question, the Dataset visualization of this result is broken (v11) due to incorrect inferred metadata, and it can be fixed by rededucing the metadata.

Note that the result still contains the association with the pair c->Missing. This is because DeleteMissing only considers values whose head is Missing, i.e. expressions of the form Missing[...]. The raw symbol Missing is left untouched. If we wish to delete pairs whose values are either form, we can write:

Query[All, DeleteCases[_Missing | Missing]] @ d1

This exhibits the same visualization bug as the original expression.

Query[All, Select[Head@#c == List] &]@d1

This one is operating upon the wrong level -- the Select should be applied to the association list, not to each association individually. Also, the & is misplaced:

Query[Select[Head@#c == List &]]@d1

As to the placement of &, note that Select requires a function as its argument. Since we want to use a pure function, that means the expression should be of the form Select[...&] rather than Select[...]&. Furthermore, since Select[...] is a function in its own right, Select[...]& defines a pure function that returns a function.

If we correct the & placement in the original expression, we can use Normal to see how the query is interpreted:

Query[All, Select[Head@#c == List] &] // Normal

(* Map[Select[Head[#c] == List] &] *)

From this expanded form, we can see that the query is attempting to map the Select across the associations.

For more information about levels and operators in queries, see (98181) or (89080).

d1[All, Select[#c == {2, 3} &]]

Same problem, should be d1[Select[#c == {2, 3} &]].

Select[d1, #c == {2, 3} &]

As noted, works.

Query[All, DeleteCases[Head@#c == List] &]@d1

The same error again but in addition DeleteCases requires a pattern -- not a function like Select:

Query[DeleteCases[_?(Head@#c == List &)]]@d1

d1[All, DeleteCases[#c == {2, 3}] &]

Same again, should be d1[DeleteCases[_?(#c == {2, 3} &)]].

DeleteCases[d1, #c == {2, 3} &]

Same again, should be DeleteCases[d1, _?(#c == {2, 3} &)].