# DeleteMissing level spec confusion

Given:

titanic = ExampleData[{"Dataset", "Titanic"}]


Why does this correctly delete Missing[] elements:

 titanic[GroupBy@Key@"class", DeleteMissing[#, 1] &, "age"] // Normal


but not the default level spec? Seems to violate key transparency in Associations/Datasets.

 titanic[GroupBy@Key@"class", DeleteMissing, "age"] // Normal

(* <|"1st" -> {29, 1, 2, 30, 25, 48, 63, 39, 53, 71, 47, 18, 24, 26, 80,
Missing[], 24, 50, 32,... |>  *)


DeleteMissing also works on the bare list, as would any function applied at that Dataset slot. On the other hand:

<| "key" -> {29, 1, 2, 30, 25, 48, 63, 39, 53, 71, 47, 18, 24, 26, 80,
Missing[], 24, ... |> // DeleteMissing[#, 2] &


Level 2 is the minimum needed to delete missing element?

• I think your second form with a bare DeleteMissing should work. I am guessing this is just more Dataset weirdness. Personally I am waiting for 10.0.1 before continuing to use this functionality as there is too much inconsistency at this point IMO. Good question however, which will hopefully help to get the inconsistencies resolved. – Mr.Wizard Sep 13 '14 at 7:09
• I have taken to using Select[FreeQ[_Missing]] rather than DeleteMissing. It seems better behaved. For example, titanic[GroupBy@Key@"class", Select[FreeQ[_Missing]], "age"] // Normal works as expected. – m_goldberg Sep 13 '14 at 12:41
• @Mr.Wizard this is actually correct behavior... DeleteMissing is descending, basically. – Taliesin Beynon Sep 18 '14 at 6:06
• @Tali Well there's egg on my face. :o) I really don't understand Dataset yet but I'm trying. – Mr.Wizard Sep 18 '14 at 6:11
• @Mr.Wizard perhaps it would help if Query objects formatted in a way that helped you understand what they would actually do... – Taliesin Beynon Sep 18 '14 at 6:35

UPDATE The described behaviour is not a bug as DeleteMissing is explicitly listed as a descending operator in the documentation. See the response by @TaliesinBeynon.

I will delete this response after the "accept" has been transferred.

Original Response - WARNING: The following analysis is incorrect.

This is yet another manifestation of the WRI-confirmed bug encountered here and here.

Correct compilation, with DeleteMissing applied at the right time and level:

DatasetCompileQuery @ Query[GroupBy@Key@"class", DeleteMissing[#,1]&, "age"]

(* DatasetWithOverrides@*GeneralUtilitiesChecked[
GroupBy[Key[class]] /*
Map[Map[GeneralUtilitiesSlice[age]] /* (DeleteMissing[#1,1]&)]
, Identity
] *)


Incorrect compilation, with DeleteMissing applied too early:

DatasetCompileQuery @ Query[GroupBy@Key@"class", DeleteMissing, "age"]

(* DatasetWithOverrides@*GeneralUtilitiesChecked[
GroupBy[Key[class]] /*
Map[DeleteMissing /* Map[GeneralUtilitiesSlice[age]]]
, Identity
] *)

• WReach, I reopened this question because one of the linked bugs is now fixed but this problem remains. If you feel it is highly probable that this issue will be fixed if the one in the second linked question is then I shall (re)close this question as a duplicate of (only) that one. – Mr.Wizard Sep 17 '14 at 11:14
• @Mr.Wizard I have no strong opinion one way or another about whether this question should remain open. From the outside, all three questions appear to be a single bug as they all manifest in exactly the same fashion. From the inside (that is, from within the WRI implementation), it would seem that they are separate bugs. This is evidenced by the fact that at least one of them can be fixed independently of the others. Are the bugs independent, or was the fix incomplete? There is no way for us outsiders to discriminate these cases -- and I suspect the insiders find it difficult too :) – WReach Sep 17 '14 at 16:50
• @alancalvitti Please transfer the "accept" to Taliesin's answer. I will then delete this response as it presents an incorrect analysis. – WReach Sep 18 '14 at 21:47

There is no bug here. DeleteMissing is documented to be descending.

That means that it applies before the "age" operator extracts all the ages. And indeed at that level none of the rows themselves are Missing[], so nothing gets deleted.

You can easily get round this by writing:

 titanic[GroupBy@Key@"class", DeleteMissing[#]&, "age"]


Or if you find that ugly...

 titanic[GroupBy@Key@"class", Query[DeleteMissing], "age"]


or

 titanic[GroupBy@Key@"class", All, "age"][All, DeleteMissing]

• DOH! I completely missed that DeleteMissing was on the list of descending operators. Another way to convert a descending operator to ascending is like this: #& /* DeleteMissing. Note that Identity /* DeleteMissing will not do the job as the Identity operator is optimized out (it is effectively on the list of descending operators). In context: titanic[GroupBy@Key@"class", #& /* DeleteMissing, "age"]. – WReach Sep 18 '14 at 22:07