Correct way to handle mysterious NaN result from MathLink function

I have a Mathematica expression that is mapped onto an external C function via MathLink. The external function passes a double array (using MLPutReal64List[]), which Mathematica interprets as a list of Real's. Sometimes the external function sends values for which the C math library function isnan(value) returns 1 (say from division by zero, or log(0)). Mathematica reads these values as NaN. For example,

In[1]:= result = externalFunction[<some input>]
Out[1]= {NaN, 0.18225}

Evaluating {NaN, 0.182255} gives a syntax error:

In[2]:= {NaN, 0.182255}
Syntax::tsntxi: "{NaN,0.182255}" is incomplete; more input is needed.
Syntax::sntxi: Incomplete expression; more input is needed.

which makes sense, since a variable name is expected to follow the , giving some variable in the NaN context.

But mathematica accepts result[[1]] as a number:

In[3]:= NumericQ[result[[1]]]
Out[3]= True
In[4]:= NumberQ[result[[1]]]
Out[4]:= True

!!

Yet I haven't been able to find a pattern that matches NaN and not all other numbers (for use in Position, Cases, etc.).

So what are ways of a) passing NaN's from c into Mathematica so that Mathematica interprets them as Indeterminate, or b) handling these NaNs inside mathematica (with a pattern match to replace them with Indeterminate, for example).

Is it possible to construct the first element of result using keyboard input? Note that InputForm[result[[1]]] gives NaN, and NumericQ[InputForm[result[[1]]]] gives False.

• @R.M Yes! As long as I load the ComputerArithmetic package before I call the external function, it works. Please write up your answer.
– JxB
Jan 20, 2012 at 22:49
• Does that NaN have a backtick at the end? That's unusual, as normally only numbers would have that, not symbols. I am curious what is result's FullForm. Jan 21, 2012 at 0:04
• I think it should be mentioned here that 1.030 indicates 1.0 to 30 digits of precision. 1 just indicated a machine precision number (same as 1.0). Jan 21, 2012 at 0:08
• @Szabolcs There is indeed a backtick suffix. The FullForm evaluates to NaN.
– JxB
Jan 21, 2012 at 2:03

NaN (or Not-a-Number is used in floating point arithmetic to represent values that are undefined or unrepresentable, such as $0/0,\ \infty/\infty$, etc. Mathematica typically returns Indeterminate for these, but several other languages return NaN.

To work with NaNs, you must load the ComputerArithmetic package as <<ComputerArithmetic prior to calling your external function. If you don't do so, then Mathematica will treat the NaNs like any other symbol (or perhaps with other, unknown consequences depending on the setup). Loading the package will give you the results as expected, and pattern matching is pretty straightforward too.

• Is it a bug that Mathematica correctly interprets the IEEE double encoding of a NaN as not-a-number, but stores it in the list as something that you can't even enter as input directly? I wonder if there are any cases when you would not want that nan to be automatically converted to Indeterminate.
– JxB
Jan 21, 2012 at 19:11
• @JxB I don't know if it's a bug or not, as I don't have anything to reproduce it, but since there are several WRI folks around here, one of them will probably comment if it is indeed one. Re: converting to Indeterminate, if you're using Mathematica as an intermediate step to process the data before feeding it into another software/language, you'd probably want to keep them as NaN so as to ensure that the other program uses it correctly.
– rm -rf
Jan 21, 2012 at 19:21
• Perhaps it's time to update this answer as it's a bit misleading: the NaN thing (supposedly a number---note the backtick) returned from MathLink is not the same thing as the NaN symbol from the ComputerArithmetic package. The backticked version is not a symbol, it's neither identical (SameQ) nor equal to the NaN symbol from ComputerArithmetic. Pattern matching is not reliable on it. You can obtain a NaN using MATLink. May 28, 2013 at 18:19
• So the advice "To work with NaNs, you must load the ComputerArithmetic package as <<ComputerArithmetic prior to calling your external function." is a bit misleading---loading ComputerArithmetic won't make it possible to work with the NaN . May 28, 2013 at 18:20
• I became aware of this, especially the pattern matching troubles, only after our previous discussions on this... I'm not sure why JxB said "As long as I load the ComputerArithmetic package before I call the external function, it works." in a comment (after I suggested this package). Unfortunately, the nitty-gritties of NaNs/IEEE specs and MathLink is out of my comfort zone, so I'm not sure what to include as an update other than to perhaps add a disclaimer about the NaN vs NaN . Instead, I'll make this answer a CW so that you (or anyone else) can update it as necessary. @Szabolcs
– rm -rf
May 31, 2013 at 1:08

This perhaps more of a comment than an answer, but I'm putting it in answer form so it will have more visibility.

The IEEE NaN number is not supposed to existing in the WL. That value is represented by the symbol Indeterminate. No WL computational function should ever produce an IEEE Nan, and I'm not aware of any bugs in which this is the case.

Now, when we allow external processes to directly write into our memory, e.g., MathLink or LibraryLink, there is some chance that an IEEE Nan will end up in the system. When that happens, you get this strange NaN thing. This is the text formatting of the numeric value followed by the machine precision mark, just like N[6/5] will give you 1.2 . The formatting is an accident of the implementation of something that shouldn't exist leaking into the system.

As noted in several comments, the expression NaN is not the same thing as the NaN symbol of the old ComputerArithmetic package, which is merely a symbolic representation of the IEEE value for the purposes of the package.

So why don't we prevent these from leaking in, and/or have better tools for dealing with them? Well, there are all sorts of complicated tradeoffs. For example, external packages often write large arrays directly into memory. If we stopped to check every single real number that came in, it would enormously slow down the system, defeating the whole purpose of allowing direct memory writes. OTOH, since the core system was written on the assumption that this value can't be created, it is not easy propagate it now.

With NumericArray making it easier for such things to enter the system, we've done some experiments and how to improve the situation. But we haven't yet reached a point where we have a clearly better story to tell, so in the mean time we're sticking with the status quo rather than randomly change things. If and when we have a better store to tell, I'll update this answer.

• Does a NumericArray of reals always preserve its contents bit-by-bit? If not (for example, because number representation may differ from platform to platform), does it support NaN and Infinity? Jun 21, 2021 at 20:33
• Re improving the situation: what I (and I think many other people) need is some way to be able to return infinities and indeterminates to Mathematica without too much difficulty. With MathLink, this wasn't possible last time I tried. LibraryLink behaved much better: originally, it returned a packed array which contained unusable "NaN" values. However, unpacking the array produced correct Indeterminates. Therefore, I wrote a helper function (in C) that checked if the array contained any unsafe values. If yes, I unpacked it. Jun 21, 2021 at 20:38
• In recent versions, things changed: now if an unsafe value is returned from LibraryLink, the system shows an error. Thus it seems Mathematica does spend the time to check for these values. There is a system option to turn this behaviour off, which I use, so I can work in the same way as previously. There are two improvements that I think would be useful: 1. since Mathematica checks for unsafe values anyway, it could do the necessary unpacking automatically. Jun 21, 2021 at 20:41
• 2. If at all possible, it would be great if indeterminates and infinities could be supported directly in packed arrays. I see the difficult with this, and I expect it won't be possible (a lot of arithmetic code would have to be updated). Jun 21, 2021 at 20:41

The best way to handle NaNs coming in from MathLink is to replace all NaNs with a symbol such as Indeterminate. One way this can be achieved is using the following piece of code:

NaNQ[x_] :=
Round[x] == -2147483648;

ReplaceNaNs[x_] := x /. y_?NaNQ :> Indeterminate;

You can then use the ReplaceNaNs function to replace any NaNs that you don't want:

In[1]:= result = externalFunction[<some input>]
Out[1]= {NaN, 0.18225}
In[2]:= ReplaceNaNs[result]
Out[2]= {Indeterminate, 0.18225}

Note when using 64 bit values the Round[] limit needs to be -9223372036854775808.

• This is unfortunately not reliable. Pattern matching does not work reliably on these special values, so replacing doesn't work either. Please see here. For example, MatchQ[NaN, -Inf] gives True. May 28, 2013 at 18:12
• I'm not sure if it's totally robust (for example, something different has to be done for 32-bit and 64-bit), but my suggestion does work as intended. I checked with Mathematica 9.0.1 on Mac OS X and ReplaceNaNs[NaN] returns Indeterminate while ReplaceNaNs[-Inf] returns -Inf. Aug 27, 2013 at 22:42
• Barry, my concern is that this is just a hack that accidentally works with a certain version of Mathematica on a certain platform. All the weirdness I experienced with these special values should be warning enough that they should not be used in Mathematica if you need any kind of reliability. Out of curiosity I tried Round[NaN] on two different Mac versions of Mathematica: 9.0.1 gives what you say, -9223372036854775808. 8.0.4 gives -2147483648. Who's to tell what 10 will give or what a Windows or Linux version will give? Aug 28, 2013 at 4:17
• I would strongly recommend to stay clear of these special values. I wish they were more reliable, it'd make it easier for us to handle them in MATLink, but in my experience they can generate some wildly surprising results ... Aug 28, 2013 at 4:19
• The difference you see between 9.0.1 and 8.0.4 is actually a result of a change from 64-bit to 32-bit. Note that the numbers are not totally arbitrary - they are the smallest possible two's complement signed 64-bit and 32-bit integers. I agree that this is very much a hack which relies on the machine-dependent internal representation of numbers. It shouldn't be used if it can be avoided in any way. However, if you can't modify the MathLink source code and are stuck with NaNs in Mathematica, it may still be useful as a way to get rid of them. Aug 29, 2013 at 13:39

Recently, I revisited this thread for dealing with NumericArrays produced via numpy where numpy.nan will be embedded as NAN, which can be simply generated using the following code:

cNaN = ExternalEvaluate["Python",
"import numpy as np; np.array([np.nan])"] // Normal // First

And I found a caveat in Barry's approach that a number close to -9223372036854775808 (on 64 bit system) will be misrecognised as NaN.

Meanwhile, I also noticed Threshold[{cNaN}] gives {0.}; so a more robust approach for testing NAN could be like this:

Clear[NaNQ]
NaNQ[nan_?NumericQ]:=Clip[Abs[Threshold[{nan}]-Round[{nan}]],{1,1},{0,Indeterminate}]==={Indeterminate}

Based on the above approach, it is also possible to have more convenient functions to deal with arrays/lists including intermittent NaNs:

(*replace NaN with Indeterminate*)
Clear[replaceNaN]
replaceNaN[x_List/;Length[x]>0]:=Clip[Abs[Threshold[x]-Round[x]],{1,1},{0,Indeterminate}]+Threshold[x]
replaceNaN[{}]:={}

and

(*fully remove NaN*)
Clear[removeNaN]
removeNaN[x_List]:=replaceNaN[x]/.Indeterminate:>Nothing