# Upvalues for Missing do not work with Dataset

Consider the following Dataset:

d = Dataset[<| 1 -> Missing[], 2 -> 1.0, 3 -> 4.0 |>]


And the following Upvalue for Missing:

Unprotect[Missing];
Missing /:
f_Symbol[___, m_Missing,  ___] /; MemberQ[Attributes[f], NumericFunction ] := m;
Protect[Missing];


Now do the following:

d[All,  1 + # &]


It returns <|1 -> 1, 2 -> 2., 3 -> 5.|> rather than the expected <|1 -> Missing[], 2 -> 2., 3 -> 5.|>.

Why doesn't the upvalue for Missing[] work?

• I think you have probably discovered a bug in the implementation of Dataset. As it stands, your question is probably a bit long and it contains details irrelevant to the issue (i.e. the overloading of Missing). I'd suggest that you excise the "UPDATE" part of your question and repost it as a new one, since it has probably gone entirely unnoticed. For what it's worth, I am on MMA 10.4.0 / Win7-64, and I can reproduce the Dataset erroneous behavior you mentioned in the update. Would you also please report this to Wolfram Support so it can be identified as a bug? – MarcoB May 10 '16 at 16:07
• Thanks MarcoB. I agree there are two different issues. One, why doesn't overloading of Missing works. And two, why evaluating multiple times gives two different answers. – ecoxlinux May 12 '16 at 2:14
• @MarcoB, I changed this question. And created a new one here: mathematica.stackexchange.com/questions/114921/… – ecoxlinux May 12 '16 at 2:33

By default, any queries using Dataset or Query perform special processing of Missing values. The installed up-value will have no effect while that special processing is in place. We can use MissingBehavior -> None to disable the special treatment and to allow our up-value to take effect:

d[All, 1 + # &, MissingBehavior -> None]

(* <|1 -> Missing[], 2 -> 2., 3 -> 5.|> *)


In general, I would discourage adding an up-value to a protected system symbol like Missing. The present observed difficulty is an example of why -- many components make assumptions about the exact behaviour of built-in symbols. At the very least, I would recommend using Block (or InternalInheritedBlock) to keep the redefinition somewhat localized.

Incidentally, the dataset machinery that performs the special missing processing happens to ignore this advice. It temporarily redefines the behaviour of Missing (using Block). That is why Missing appears to behave erratically in this context.

• Thanks @WReach. Is this missingbehavior setting documented? – ecoxlinux May 13 '16 at 1:43
• I just saw it is documented – ecoxlinux May 13 '16 at 1:45
• starting from a fresh kernel and setting MissingBehavior to automatic does not make the evaluation of the first element of the expression d[All, 1 + # &] be Missing[]. It is still 1. – ecoxlinux May 13 '16 at 4:17
• The Automatic setting is the default. It must be set to None`. – WReach May 13 '16 at 4:43
• Thanks @WReach. But in this case, even with a fresh kernel and the Automatic setting, the evalution returns '1' in my computer, rather than 'Missing' (as expected according to the documentation). – ecoxlinux May 13 '16 at 13:48