Consider the following Dataset:

d = Dataset[<| 1 -> Missing[], 2 -> 1.0, 3 -> 4.0 |>]

And the following Upvalue for Missing:

Missing /:
f_Symbol[___, m_Missing,  ___] /; MemberQ[Attributes[f], NumericFunction ] := m;

Now do the following:

d[All,  1 + # &]

It returns <|1 -> 1, 2 -> 2., 3 -> 5.|> rather than the expected <|1 -> Missing[], 2 -> 2., 3 -> 5.|>.

Why doesn't the upvalue for Missing[] work?

  • $\begingroup$ I think you have probably discovered a bug in the implementation of Dataset. As it stands, your question is probably a bit long and it contains details irrelevant to the issue (i.e. the overloading of Missing). I'd suggest that you excise the "UPDATE" part of your question and repost it as a new one, since it has probably gone entirely unnoticed. For what it's worth, I am on MMA 10.4.0 / Win7-64, and I can reproduce the Dataset erroneous behavior you mentioned in the update. Would you also please report this to Wolfram Support so it can be identified as a bug? $\endgroup$
    – MarcoB
    May 10, 2016 at 16:07
  • $\begingroup$ Thanks MarcoB. I agree there are two different issues. One, why doesn't overloading of Missing works. And two, why evaluating multiple times gives two different answers. $\endgroup$
    – ecoxlinux
    May 12, 2016 at 2:14
  • $\begingroup$ @MarcoB, I changed this question. And created a new one here: mathematica.stackexchange.com/questions/114921/… $\endgroup$
    – ecoxlinux
    May 12, 2016 at 2:33

1 Answer 1


By default, any queries using Dataset or Query perform special processing of Missing values. The installed up-value will have no effect while that special processing is in place. We can use MissingBehavior -> None to disable the special treatment and to allow our up-value to take effect:

d[All, 1 + # &, MissingBehavior -> None]

(* <|1 -> Missing[], 2 -> 2., 3 -> 5.|> *)

In general, I would discourage adding an up-value to a protected system symbol like Missing. The present observed difficulty is an example of why -- many components make assumptions about the exact behaviour of built-in symbols. At the very least, I would recommend using Block (or Internal`InheritedBlock) to keep the redefinition somewhat localized.

Incidentally, the dataset machinery that performs the special missing processing happens to ignore this advice. It temporarily redefines the behaviour of Missing (using Block). That is why Missing appears to behave erratically in this context.

  • $\begingroup$ Thanks @WReach. Is this missingbehavior setting documented? $\endgroup$
    – ecoxlinux
    May 13, 2016 at 1:43
  • $\begingroup$ I just saw it is documented $\endgroup$
    – ecoxlinux
    May 13, 2016 at 1:45
  • $\begingroup$ starting from a fresh kernel and setting MissingBehavior to automatic does not make the evaluation of the first element of the expression d[All, 1 + # &] be Missing[]. It is still 1. $\endgroup$
    – ecoxlinux
    May 13, 2016 at 4:17
  • $\begingroup$ The Automatic setting is the default. It must be set to None. $\endgroup$
    – WReach
    May 13, 2016 at 4:43
  • $\begingroup$ Thanks @WReach. But in this case, even with a fresh kernel and the Automatic setting, the evalution returns '1' in my computer, rather than 'Missing' (as expected according to the documentation). $\endgroup$
    – ecoxlinux
    May 13, 2016 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.