Correct way to handle mysterious NaN result from MathLink function

I have a Mathematica expression that is mapped onto an external C function via MathLink. The external function passes a double array (using MLPutReal64List[]), which Mathematica interprets as a list of Real's. Sometimes the external function sends values for which the C math library function isnan(value) returns 1 (say from division by zero, or log(0)). Mathematica reads these values as NaN. For example,

In[1]:= result = externalFunction[<some input>]
Out[1]= {NaN, 0.18225}


Evaluating {NaN, 0.182255} gives a syntax error:

In[2]:= {NaN, 0.182255}
Syntax::tsntxi: "{NaN,0.182255}" is incomplete; more input is needed.
Syntax::sntxi: Incomplete expression; more input is needed.


which makes sense, since a variable name is expected to follow the , giving some variable in the NaN context.

But mathematica accepts result[[1]] as a number:

In[3]:= NumericQ[result[[1]]]
Out[3]= True
In[4]:= NumberQ[result[[1]]]
Out[4]:= True


!!

Yet I haven't been able to find a pattern that matches NaN and not all other numbers (for use in Position, Cases, etc.).

So what are ways of a) passing NaN's from c into Mathematica so that Mathematica interprets them as Indeterminate, or b) handling these NaNs inside mathematica (with a pattern match to replace them with Indeterminate, for example).

Is it possible to construct the first element of result using keyboard input? Note that InputForm[result[[1]]] gives NaN, and NumericQ[InputForm[result[[1]]]] gives False.

-
@R.M Yes! As long as I load the ComputerArithmetic package before I call the external function, it works. Please write up your answer. –  JxB Jan 20 '12 at 22:49
Does that NaN have a backtick at the end? That's unusual, as normally only numbers would have that, not symbols. I am curious what is result's FullForm. –  Szabolcs Jan 21 '12 at 0:04
I think it should be mentioned here that 1.030 indicates 1.0 to 30 digits of precision. 1  just indicated a machine precision number (same as 1.0). –  Szabolcs Jan 21 '12 at 0:08
@Szabolcs There is indeed a backtick suffix. The FullForm evaluates to NaN. –  JxB Jan 21 '12 at 2:03

NaN (or Not-a-Number is used in floating point arithmetic to represent values that are undefined or unrepresentable, such as $0/0,\ \infty/\infty$, etc. Mathematica typically returns Indeterminate for these, but several other languages return NaN.

To work with NaNs, you must load the ComputerArithmetic package as <<ComputerArithmetic prior to calling your external function. If you don't do so, then Mathematica will treat the NaNs like any other symbol (or perhaps with other, unknown consequences depending on the setup). Loading the package will give you the results as expected, and pattern matching is pretty straightforward too.

-
Is it a bug that Mathematica correctly interprets the IEEE double encoding of a NaN as not-a-number, but stores it in the list as something that you can't even enter as input directly? I wonder if there are any cases when you would not want that nan to be automatically converted to Indeterminate. –  JxB Jan 21 '12 at 19:11
@JxB I don't know if it's a bug or not, as I don't have anything to reproduce it, but since there are several WRI folks around here, one of them will probably comment if it is indeed one. Re: converting to Indeterminate, if you're using Mathematica as an intermediate step to process the data before feeding it into another software/language, you'd probably want to keep them as NaN so as to ensure that the other program uses it correctly. –  rm -rf Jan 21 '12 at 19:21
Perhaps it's time to update this answer as it's a bit misleading: the NaN  thing (supposedly a number---note the backtick) returned from MathLink is not the same thing as the NaN symbol from the ComputerArithmetic package. The backticked version is not a symbol, it's neither identical (SameQ) nor equal to the NaN symbol from ComputerArithmetic. Pattern matching is not reliable on it. You can obtain a NaN  using MATLink. –  Szabolcs May 28 '13 at 18:19
So the advice "To work with NaNs, you must load the ComputerArithmetic package as <<ComputerArithmetic  prior to calling your external function." is a bit misleading---loading ComputerArithmetic won't make it possible to work with the NaN . –  Szabolcs May 28 '13 at 18:20
I became aware of this, especially the pattern matching troubles, only after our previous discussions on this... I'm not sure why JxB said "As long as I load the ComputerArithmetic package before I call the external function, it works." in a comment (after I suggested this package). Unfortunately, the nitty-gritties of NaNs/IEEE specs and MathLink is out of my comfort zone, so I'm not sure what to include as an update other than to perhaps add a disclaimer about the NaN vs NaN . Instead, I'll make this answer a CW so that you (or anyone else) can update it as necessary. @Szabolcs –  rm -rf May 31 '13 at 1:08

The best way to handle NaNs coming in from MathLink is to replace all NaNs with a symbol such as Indeterminate. One way this can be achieved is using the following piece of code:

NaNQ[x_] :=
Round[x] == -2147483648;

ReplaceNaNs[x_] := x /. y_?NaNQ :> Indeterminate;


You can then use the ReplaceNaNs function to replace any NaNs that you don't want:

In[1]:= result = externalFunction[<some input>]
Out[1]= {NaN, 0.18225}
In[2]:= ReplaceNaNs[result]
Out[2]= {Indeterminate, 0.18225}


Note when using 64 bit values the Round[] limit needs to be -9223372036854775808.

-
This is unfortunately not reliable. Pattern matching does not work reliably on these special values, so replacing doesn't work either. Please see here. For example, MatchQ[NaN, -Inf] gives True. –  Szabolcs May 28 '13 at 18:12
I'm not sure if it's totally robust (for example, something different has to be done for 32-bit and 64-bit), but my suggestion does work as intended. I checked with Mathematica 9.0.1 on Mac OS X and ReplaceNaNs[NaN] returns Indeterminate while ReplaceNaNs[-Inf] returns -Inf. –  barrywardell Aug 27 '13 at 22:42
Barry, my concern is that this is just a hack that accidentally works with a certain version of Mathematica on a certain platform. All the weirdness I experienced with these special values should be warning enough that they should not be used in Mathematica if you need any kind of reliability. Out of curiosity I tried Round[NaN] on two different Mac versions of Mathematica: 9.0.1 gives what you say, -9223372036854775808. 8.0.4 gives -2147483648. Who's to tell what 10 will give or what a Windows or Linux version will give? –  Szabolcs Aug 28 '13 at 4:17
I would strongly recommend to stay clear of these special values. I wish they were more reliable, it'd make it easier for us to handle them in MATLink, but in my experience they can generate some wildly surprising results ... –  Szabolcs Aug 28 '13 at 4:19
The difference you see between 9.0.1 and 8.0.4 is actually a result of a change from 64-bit to 32-bit. Note that the numbers are not totally arbitrary - they are the smallest possible two's complement signed 64-bit and 32-bit integers. I agree that this is very much a hack which relies on the machine-dependent internal representation of numbers. It shouldn't be used if it can be avoided in any way. However, if you can't modify the MathLink source code and are stuck with NaNs in Mathematica, it may still be useful as a way to get rid of them. –  barrywardell Aug 29 '13 at 13:39
show 1 more comment