Can someone please explain to me the syntax of Query and why its behavior is so befuddling to me at the moment?

I am going to go through a few simple examples to demonstrate my confusion.


Let's take the List of Associations from the Query documentation (and modify it slightly)

d1 = Dataset@{
   <|"a" -> 1, "b" -> "x", "c" -> Missing["KeyAbsent", ""]|>,
   <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
   <|"a" -> 3, "b" -> "z", "c" -> Missing["KeyAbsent", ""]|>,
   <|"a" -> 4, "b" -> "x", "c" -> "Bad Value"|>,
   <|"a" -> 5, "b" -> "y", "c" -> Missing|>,
   <|"a" -> 6, "b" -> "z", "c" -> {}|>}

enter image description here

For a sanity check, let's run the first example from the documentation:

enter image description here

Great. That works. What about DeleteDuplicates, which should delete anything with the Head Missing.

enter image description here

Ok that doesn't work. Well maybe these values do not have the Head Missing?

enter image description here

Huh... that worked! Now what if I just swap Head for DeleteMissing?

enter image description here

What just happened?

Maybe it is the #c?

enter image description here

Nope... Forcing the Notebook to rearrange this dataset (where a is the output from above)

enter image description here

Ok... let's try Select in a variety of forms:

Query[All, Select[Head@#c == List] &]@d1
d1[All, Select[#c == {2, 3}] &]
Select[d1, #c == {2, 3} &]

enter image description here

only the last one works...

What about the same thing with DeleteCases?

Query[All, DeleteCases[Head@#c == List] &]@d1
d1[All, DeleteCases[#c == {2, 3}] &]
DeleteCases[d1, #c == {2, 3} &]

enter image description here

Well now even the third form isn't working...

What is going on?


Query Dataset DeleteMissing Select


1 Answer 1


Let's go through it blow-by-blow...

Query[DeleteMissing] @ d1

This will apply DeleteMissing to the list of associations. The documentation says:

DeleteMissing[list] drops elements with head Missing from a list.

All of the elements of the list have head Association, so this operation is a no-op.

d1[All, #c &, Head]

This works as advertised, showing the heads of the values associated with the key c in each association.

d1[All, #c &, DeleteMissing]

This asks to delete missing elements from the lists that appear as values associated with the key c in each association. Most of the c values are not lists. The error that appears reflects that fact.

Query[All, DeleteMissing] @ d1

This asks to delete from each association in the list any key/value pairs whose value has the head Missing. If we look at the Normal form of the result we find that this is what happened:

d1[All, DeleteMissing] // Normal

(* { <|a->1, b->x|>
   , <|a->2, b->y, c->{2,3}|>
   , <|a->3, b->z|>
   , <|a->4, b->x, c->Bad Value|>
   , <|a->5, b->y, c->Missing|>
   , <|a->6, b->z, c->{}|>

As noted in the question, the Dataset visualization of this result is broken (v11) due to incorrect inferred metadata, and it can be fixed by rededucing the metadata.

Note that the result still contains the association with the pair c->Missing. This is because DeleteMissing only considers values whose head is Missing, i.e. expressions of the form Missing[...]. The raw symbol Missing is left untouched. If we wish to delete pairs whose values are either form, we can write:

Query[All, DeleteCases[_Missing | Missing]] @ d1

This exhibits the same visualization bug as the original expression.

Query[All, Select[Head@#c == List] &]@d1

This one is operating upon the wrong level -- the Select should be applied to the association list, not to each association individually. Also, the & is misplaced:

Query[Select[Head@#c == List &]]@d1

dataset screenshot

As to the placement of &, note that Select requires a function as its argument. Since we want to use a pure function, that means the expression should be of the form Select[...&] rather than Select[...]&. Furthermore, since Select[...] is a function in its own right, Select[...]& defines a pure function that returns a function.

If we correct the & placement in the original expression, we can use Normal to see how the query is interpreted:

Query[All, Select[Head@#c == List] &] // Normal

(* Map[Select[Head[#c] == List] &] *)

From this expanded form, we can see that the query is attempting to map the Select across the associations.

For more information about levels and operators in queries, see (98181) or (89080).

d1[All, Select[#c == {2, 3} &]]

Same problem, should be d1[Select[#c == {2, 3} &]].

Select[d1, #c == {2, 3} &]

As noted, works.

Query[All, DeleteCases[Head@#c == List] &]@d1

The same error again but in addition DeleteCases requires a pattern -- not a function like Select:

Query[DeleteCases[_?(Head@#c == List &)]]@d1

dataset screenshot

d1[All, DeleteCases[#c == {2, 3}] &]

Same again, should be d1[DeleteCases[_?(#c == {2, 3} &)]].

DeleteCases[d1, #c == {2, 3} &]

Same again, should be DeleteCases[d1, _?(#c == {2, 3} &)].

  • $\begingroup$ thank you for this answer. I have few follow up questions: Query[All, DeleteMissing] @ d1 still has one c->Missing why? Could you maybe elaborate on the depth for Query? and while this is not directly related, the where the end definition for a pure function needs to go, especially if there might be nested pure functions? $\endgroup$
    – SumNeuron
    Oct 21, 2016 at 5:49
  • $\begingroup$ See my edits. I'm not sure whether my additions concerning & fully address your question about pure functions... let me know. $\endgroup$
    – WReach
    Oct 21, 2016 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.