# Unpredictable behavior of numerical error

A comparison of float calculation and exact expression for

  NumberForm[0.3^2 - (3/10)^2, 18]


gives an exact zero for $0.3$. However,

R[n_Integer] = n 1./10.;
Rex[n_Integer] = n/10;
Table[R[n]^2 - Rex[n]^2, {n, 1, 5}]


gives

{1.73472*10^-18, 6.93889*10^-18, 2.77556*10^-17, 2.77556*10^-17, 0.}


Thus, here we get

0.3^2 - (3/10)^2 = 2.77556*10^-17


instead of zero. Why is the ouput different in the Table[] calculation?

• Is your question: why does 0.3^2 - (3/10)^2 return 0. while R[3]^2 - Rex[3]^2 returns 2.77556*10^-17? – anderstood Jul 20 '17 at 18:06
• Related, maybe a duplicate: mathematica.stackexchange.com/questions/135636/… – Michael E2 Jul 20 '17 at 18:30
• This could be simplified considerably. It amounts to having done some arithmetic just to get the ".3" in R[3]: In[17]:= {.3, (3*1./10)} // InputForm Out[17]//InputForm= {0.3, 0.30000000000000004} – Daniel Lichtblau Jul 20 '17 at 18:45