6
$\begingroup$

Given:

titanic = ExampleData[{"Dataset", "Titanic"}]

Why does this correctly delete Missing[] elements:

 titanic[GroupBy@Key@"class", DeleteMissing[#, 1] &, "age"] // Normal

but not the default level spec? Seems to violate key transparency in Associations/Datasets.

 titanic[GroupBy@Key@"class", DeleteMissing, "age"] // Normal

(* <|"1st" -> {29, 1, 2, 30, 25, 48, 63, 39, 53, 71, 47, 18, 24, 26, 80, 
   Missing[], 24, 50, 32,... |>  *)

DeleteMissing also works on the bare list, as would any function applied at that Dataset slot. On the other hand:

<| "key" -> {29, 1, 2, 30, 25, 48, 63, 39, 53, 71, 47, 18, 24, 26, 80, 
   Missing[], 24, ... |> // DeleteMissing[#, 2] &

Level 2 is the minimum needed to delete missing element?

Improve this question
$\endgroup$
5
  • $\begingroup$ I think your second form with a bare DeleteMissing should work. I am guessing this is just more Dataset weirdness. Personally I am waiting for 10.0.1 before continuing to use this functionality as there is too much inconsistency at this point IMO. Good question however, which will hopefully help to get the inconsistencies resolved. $\endgroup$
    – Mr.Wizard
    Commented Sep 13, 2014 at 7:09
  • $\begingroup$ I have taken to using Select[FreeQ[_Missing]] rather than DeleteMissing. It seems better behaved. For example, titanic[GroupBy@Key@"class", Select[FreeQ[_Missing]], "age"] // Normal works as expected. $\endgroup$
    – m_goldberg
    Commented Sep 13, 2014 at 12:41
  • $\begingroup$ @Mr.Wizard this is actually correct behavior... DeleteMissing is descending, basically. $\endgroup$
    – Taliesin Beynon
    Commented Sep 18, 2014 at 6:06
  • $\begingroup$ @Tali Well there's egg on my face. :o) I really don't understand Dataset yet but I'm trying. $\endgroup$
    – Mr.Wizard
    Commented Sep 18, 2014 at 6:11
  • $\begingroup$ @Mr.Wizard perhaps it would help if Query objects formatted in a way that helped you understand what they would actually do... $\endgroup$
    – Taliesin Beynon
    Commented Sep 18, 2014 at 6:35

2 Answers 2

Reset to default
6
$\begingroup$

UPDATE The described behaviour is not a bug as DeleteMissing is explicitly listed as a descending operator in the documentation. See the response by @TaliesinBeynon.

I will delete this response after the "accept" has been transferred.

Original Response - WARNING: The following analysis is incorrect.

This is yet another manifestation of the WRI-confirmed bug encountered here and here.

Correct compilation, with DeleteMissing applied at the right time and level:

Dataset`CompileQuery @ Query[GroupBy@Key@"class", DeleteMissing[#,1]&, "age"]

(* Dataset`WithOverrides@*GeneralUtilities`Checked[
     GroupBy[Key[class]] /*
     Map[Map[GeneralUtilities`Slice[age]] /* (DeleteMissing[#1,1]&)]
   , Identity
   ] *)

Incorrect compilation, with DeleteMissing applied too early:

Dataset`CompileQuery @ Query[GroupBy@Key@"class", DeleteMissing, "age"]

(* Dataset`WithOverrides@*GeneralUtilities`Checked[
     GroupBy[Key[class]] /* 
     Map[DeleteMissing /* Map[GeneralUtilities`Slice[age]]]
   , Identity
   ] *)
Improve this answer
$\endgroup$
3
  • $\begingroup$ WReach, I reopened this question because one of the linked bugs is now fixed but this problem remains. If you feel it is highly probable that this issue will be fixed if the one in the second linked question is then I shall (re)close this question as a duplicate of (only) that one. $\endgroup$
    – Mr.Wizard
    Commented Sep 17, 2014 at 11:14
  • $\begingroup$ @Mr.Wizard I have no strong opinion one way or another about whether this question should remain open. From the outside, all three questions appear to be a single bug as they all manifest in exactly the same fashion. From the inside (that is, from within the WRI implementation), it would seem that they are separate bugs. This is evidenced by the fact that at least one of them can be fixed independently of the others. Are the bugs independent, or was the fix incomplete? There is no way for us outsiders to discriminate these cases -- and I suspect the insiders find it difficult too :) $\endgroup$
    – WReach
    Commented Sep 17, 2014 at 16:50
  • $\begingroup$ @alancalvitti Please transfer the "accept" to Taliesin's answer. I will then delete this response as it presents an incorrect analysis. $\endgroup$
    – WReach
    Commented Sep 18, 2014 at 21:47
7
$\begingroup$

There is no bug here. DeleteMissing is documented to be descending.

That means that it applies before the "age" operator extracts all the ages. And indeed at that level none of the rows themselves are Missing[], so nothing gets deleted.

You can easily get round this by writing:

 titanic[GroupBy@Key@"class", DeleteMissing[#]&, "age"]

Or if you find that ugly...

 titanic[GroupBy@Key@"class", Query[DeleteMissing], "age"]

or

 titanic[GroupBy@Key@"class", All, "age"][All, DeleteMissing]
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ DOH! I completely missed that DeleteMissing was on the list of descending operators. Another way to convert a descending operator to ascending is like this: #& /* DeleteMissing. Note that Identity /* DeleteMissing will not do the job as the Identity operator is optimized out (it is effectively on the list of descending operators). In context: titanic[GroupBy@Key@"class", #& /* DeleteMissing, "age"]. $\endgroup$
    – WReach
    Commented Sep 18, 2014 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.