3
$\begingroup$

I was trying to reset the imprecision record of a number like 0.0076022266122755632`1.025 by calling SetPrecision with second argument of Infinity. But then I was very surprised to see the following counterintuitive behavior:

SetPrecision[0.0076022266122755632`1.025, 30]
SetPrecision[0.0076022266122755632`1.025, Infinity]
N@%

0.00760222661227556321733231370352

1/128

0.0078125

I.e., when asked to set finite precision, SetPrecision just resets the precision marker, but when asked for infinite precision, it rounds the number to remove actual "extra" digits.

Is this inconsistency intended? How is it justified?

$\endgroup$
2
$\begingroup$

The first line just increases the mantissa size, so programmatically there is no need to do anything to the current mantissa bits.

The second line needs to read the mantissa and its more efficient to only read what is necessary according to the precision. The rest aren't rounded to zero, they are ignored.

Both are correct because they are elements of Interval[0.0076022266122755632´1.025], which is the only thing that makes sense to expect.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.