# Handling Missing values flexibly in Query (MissingBehavior problem)

There’s an inconsistency in how Missing values are imputed in Query, with MissingBehavior->Automatic.

consider a couple of functions packaged in an association:

asF = <|"class1" -> Function[{x}, x + 10],    "class2" -> Function[{x}, Boole[x > 1]]|>


and a data association:

as = <| "a" -> 1, "b" -> Missing[]|>


Then

as // Query[All, asF]   (* same as adding option MissingBehavior -> Automatic to Query  *)

<|"a" -> <|"class1" -> 11, "class2" -> 0|>,  "b" -> <|"class1" -> 10, "class2" -> Boole[Missing[] > 1]|>|>


So for the addition function ("class1") Missing is imputed as 0 but not in the Boole function (“class2”) .

Using MissingBehavaior->None is more consistent but not necessarily useful.

as // Query[All, asF, MissingBehavior -> None]


<|"a" -> <|"class1" -> 11, "class2" -> 0|>, "b" -> <|"class1" -> 10 + Missing[], "class2" -> Boole[Missing[] > 1]|>|>

In some applications (eg when there's 100's of Boole's wrapped up in an EmpiricalDistribution, say) is it much more convenient to replace Missing whenever the as input argument is Missing[...], ie: desired output:

<|"a" -> <|"class1" -> 11, "class2" -> 0|>, "b" -> <|"class1" -> Missing[], "class2" -> Missing[] ]|>|>

Query has a FailureAction option “Replace” but not MissingBehavior. How can this functionality be emulated?

Within the Plus function Missing[...] only acts as if it were zero. Actually, it is taken to be Sequence[]. The result is:

Plus[10]
(* 10 *)


... which gives the same result as Plus[0, 10]. The functions which are subject to this "sequence" behaviour are listed in an internal variable (as of v11.3):

Dataset;    (* to trigger auto-loading *)
DatasetOverridesPackagePrivate$MissingIsSequenceP (* Plus|Min|Max|MinMax *)  There are other interesting lists as well: DatasetOverridesPackagePrivate$MissingIsIndeterminateP
(* Times | Minus | Power *)

DatasetOverridesPackagePrivate$PreserveMissingP (* Counts|CountsBy|GroupBy|Take|Drop|Select|...|TakeLargestBy|TakeSmallestBy *)  In the expression Boole[x > 1], it is the Greater function (rather than Boole) for which we desire special treatment. Alas, neither Greater nor Boole receive special treatment by MissingBehaviour. For fun, we could add Greater to $MissingIsSequenceP so that it gets the same behaviour as Plus:

AppendTo[DatasetOverridesPackagePrivate\$MissingIsSequenceP, Greater]


So then:

as // Query[All, asF]
(* <| a -> <|class1 -> 11, class2 -> 0|>, b -> <|class1 -> 10, class2 -> 1|>|> *)


... but notice that the second "class2" is 1 instead of 0. This is because:

Greater[1]
(* True *)


If we always want an occurrence of Missing to be zero, we can take matters into our own hands:

Block[{Missing = 0&}
, as // Query[All, asF, MissingBehavior -> None]
]

(* <| a -> <|class1 -> 11, class2 -> 0|>, b -> <|class1 -> 10, class2 -> 0|>|> *)


or, somewhat more selectively:

Block[{Missing}
, Missing /: (f:Plus|Greater)[a___, m__Missing, b___] := f[a, 0, b]
; as // Query[All, asF, MissingBehavior -> None]
]


Note that it was necessary to turn off MissingBehaviour so that our overriding definitions of Missing would take effect. This is a standard and supported way to make Missing behave however we like.

Semi-unsanctioned Solution

If we want the standard MissingBehaviour -> Automatic functionality to be supplemented with special rules for Boole, we can get more aggressive:

InternalInheritedBlock[{Boole}
, Unprotect[Boole]
; Boole[Except[True|False]] = 0
; as // Query[All, asF]
]

(* <| a -> <|class1 -> 11, class2 -> 0|>, b -> <|class1 -> 10, class2 -> 0|>|> *)


This is characterized as "semi-unsanctioned" because it uses InternalInheritedBlock which is undocumented, but likely to remain around for a long time.

Fully unsanctioned solution

We could even go so far as to monkey-patch some internal definitions within the Query subsystem. As of version 11.3, this would entail adding definitions as desired to this function:

Dataset;
Needs["GeneralUtilities"]

PrintDefinitions[DatasetOverridesPackagePrivateattachMissingUpvalues]


Or for query operators (not relevant to the present case):

PrintDefinitions[DatasetToMissingSafe]


However, these solutions are very brittle and likely to break in future versions. I would recommend against using them. Use the Block[{Missing}, ...] pattern instead.

• First of all, why does Greater[1] evaluate at all given just one parameter... – alancalvitti May 11 '18 at 20:51
• Side-Q: Where should MissingBehavior-> None go here to result in the output b->Missing[]? <|"a" -> 2, "b" -> Missing[], "c" -> 10|> // Query[{Identity, MinMax} /* Replace[{data_, {min_, max_}} :> Query[All, (# - min)/(max - min) &, MissingBehavior -> None][ data]]] – alancalvitti May 11 '18 at 20:58
• As to Greater[1], ask WRI :) As to the side-question, the closest we can get is to add MissingBehavior->None to the outer Query and replace MinMax with Query[MinMax, MissingBehavior->Automatic]. Why? Too large to fit into a comment :) Suffice to say, the presence of Replace hides the inner subquery from the compiler so that the innermost MissingBehavior->None is ignored. Bug? To get all the way, we would need to also change the innermost function to (# - min)/(max - min) & /* promoteMissing where promoteMissing[x_] := x /. m_ /; !FreeQ[m, _Missing] :> Missing[]`. – WReach May 12 '18 at 4:06