2
$\begingroup$

There’s an inconsistency in how Missing values are imputed in Query, with MissingBehavior->Automatic.

consider a couple of functions packaged in an association:

asF = <|"class1" -> Function[{x}, x + 10],    "class2" -> Function[{x}, Boole[x > 1]]|>

and a data association:

as = <| "a" -> 1, "b" -> Missing[]|>

Then

as // Query[All, asF]   (* same as adding option MissingBehavior -> Automatic to Query  *)
<|"a" -> <|"class1" -> 11, "class2" -> 0|>,  "b" -> <|"class1" -> 10, "class2" -> Boole[Missing[] > 1]|>|>

So for the addition function ("class1") Missing is imputed as 0 but not in the Boole function (“class2”) .

Using MissingBehavaior->None is more consistent but not necessarily useful.

as // Query[All, asF, MissingBehavior -> None]

<|"a" -> <|"class1" -> 11, "class2" -> 0|>, "b" -> <|"class1" -> 10 + Missing[], "class2" -> Boole[Missing[] > 1]|>|>

In some applications (eg when there's 100's of Boole's wrapped up in an EmpiricalDistribution, say) is it much more convenient to replace Missing whenever the as input argument is Missing[...], ie: desired output:

<|"a" -> <|"class1" -> 11, "class2" -> 0|>, "b" -> <|"class1" -> Missing[], "class2" -> Missing[] ]|>|>

Query has a FailureAction option “Replace” but not MissingBehavior. How can this functionality be emulated?

$\endgroup$
2
$\begingroup$

Within the Plus function Missing[...] only acts as if it were zero. Actually, it is taken to be Sequence[]. The result is:

Plus[10]
(* 10 *)

... which gives the same result as Plus[0, 10]. The functions which are subject to this "sequence" behaviour are listed in an internal variable (as of v11.3):

Dataset;    (* to trigger auto-loading *)
Dataset`Overrides`PackagePrivate`$MissingIsSequenceP
(* Plus|Min|Max|MinMax *)

There are other interesting lists as well:

Dataset`Overrides`PackagePrivate`$MissingIsIndeterminateP
(* Times | Minus | Power *)

Dataset`Overrides`PackagePrivate`$PreserveMissingP
(* Counts|CountsBy|GroupBy|Take|Drop|Select|...|TakeLargestBy|TakeSmallestBy *)

In the expression Boole[x > 1], it is the Greater function (rather than Boole) for which we desire special treatment. Alas, neither Greater nor Boole receive special treatment by MissingBehaviour.

For fun, we could add Greater to $MissingIsSequenceP so that it gets the same behaviour as Plus:

AppendTo[Dataset`Overrides`PackagePrivate`$MissingIsSequenceP, Greater]

So then:

as // Query[All, asF]
(* <| a -> <|class1 -> 11, class2 -> 0|>, b -> <|class1 -> 10, class2 -> 1|>|> *)

... but notice that the second "class2" is 1 instead of 0. This is because:

Greater[1]
(* True *)

If we always want an occurrence of Missing to be zero, we can take matters into our own hands:

Block[{Missing = 0&}
, as // Query[All, asF, MissingBehavior -> None]
]

(* <| a -> <|class1 -> 11, class2 -> 0|>, b -> <|class1 -> 10, class2 -> 0|>|> *)

or, somewhat more selectively:

Block[{Missing}
, Missing /: (f:Plus|Greater)[a___, m__Missing, b___] := f[a, 0, b]
; as // Query[All, asF, MissingBehavior -> None]
]

Note that it was necessary to turn off MissingBehaviour so that our overriding definitions of Missing would take effect. This is a standard and supported way to make Missing behave however we like.

Semi-unsanctioned Solution

If we want the standard MissingBehaviour -> Automatic functionality to be supplemented with special rules for Boole, we can get more aggressive:

Internal`InheritedBlock[{Boole}
, Unprotect[Boole]
; Boole[Except[True|False]] = 0
; as // Query[All, asF]
]

(* <| a -> <|class1 -> 11, class2 -> 0|>, b -> <|class1 -> 10, class2 -> 0|>|> *)

This is characterized as "semi-unsanctioned" because it uses Internal`InheritedBlock which is undocumented, but likely to remain around for a long time.

Fully unsanctioned solution

We could even go so far as to monkey-patch some internal definitions within the Query subsystem. As of version 11.3, this would entail adding definitions as desired to this function:

Dataset;
Needs["GeneralUtilities`"]

PrintDefinitions[Dataset`Overrides`PackagePrivate`attachMissingUpvalues]

Or for query operators (not relevant to the present case):

PrintDefinitions[Dataset`ToMissingSafe]

However, these solutions are very brittle and likely to break in future versions. I would recommend against using them. Use the Block[{Missing}, ...] pattern instead.

$\endgroup$
  • $\begingroup$ First of all, why does Greater[1] evaluate at all given just one parameter... $\endgroup$ – alancalvitti May 11 '18 at 20:51
  • $\begingroup$ Side-Q: Where should MissingBehavior-> None go here to result in the output b->Missing[]? <|"a" -> 2, "b" -> Missing[], "c" -> 10|> // Query[{Identity, MinMax} /* Replace[{data_, {min_, max_}} :> Query[All, (# - min)/(max - min) &, MissingBehavior -> None][ data]]] $\endgroup$ – alancalvitti May 11 '18 at 20:58
  • $\begingroup$ As to Greater[1], ask WRI :) As to the side-question, the closest we can get is to add MissingBehavior->None to the outer Query and replace MinMax with Query[MinMax, MissingBehavior->Automatic]. Why? Too large to fit into a comment :) Suffice to say, the presence of Replace hides the inner subquery from the compiler so that the innermost MissingBehavior->None is ignored. Bug? To get all the way, we would need to also change the innermost function to (# - min)/(max - min) & /* promoteMissing where promoteMissing[x_] := x /. m_ /; !FreeQ[m, _Missing] :> Missing[]. $\endgroup$ – WReach May 12 '18 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.