1
$\begingroup$

I have data sets with very close values and I am getting the error that points are being duplicated. Here is a small part of one data set.

 data = {{1.880167859175211`, 0.07717769628585`}, {1.9031673299071004`,
     0.07321050510180123`}, {1.9589651339133867`, 
    0.06358591161773906`}, {2.0039652605902973`, 
    0.05582381535409813`}, {2.0039652605902973`, 
    0.05582381535409813`}, {2.0039652605903098`, 
    0.05582381535409601`}, {2.0039652606025786`, 
    0.05582381535197977`}, {2.0101043764185507`, 
    0.054764875934334406`}, {2.0101043764222215`, 
    0.054764875933701225`}};
Interpolation[data, InterpolationOrder -> 1];

The solution is to try DeleteDuplicatesBy but when I try this I still get the error

Interpolation[DeleteDuplicatesBy[data, First], 
 InterpolationOrder -> 1]

Interpolation::inddp: The point 2.0039652605902973` in dimension 1 is duplicated.

When I look at differences I can see that there are some close points but they are not zero.

Differences@DeleteDuplicatesBy[data, First]


{{0.0229995, -0.00396719}, {0.0557978, -0.00962459}, {0.0450001, \
-0.0077621}, {1.24345*10^-14, -2.1233*10^-15}, {1.22689*10^-11, \
-2.11624*10^-12}, {0.00613912, -0.00105894}, {3.67084*10^-12, \
-6.33181*10^-13}}

Is there an obvious way forward? This is just a small part of what I am hitting repeatedly. Is there an easy way of deleting close points to some criterion that Interpolation would be happy with?

Thanks

$\endgroup$
7
  • $\begingroup$ You can use some threshold to decide when to remove adjacent points. One way is like this. In[595]:= Split[data, Abs[#1[[1]] - #2[[1]]] < .0001 &][[All, 1]] Out[595]= {{1.88017, 0.0771777}, {1.90317, 0.0732105}, {1.95897, 0.0635859}, {2.00397, 0.0558238}, {2.0101, 0.0547649}} $\endgroup$ Commented Apr 25, 2023 at 15:41
  • $\begingroup$ I don't know anything about the internal workings of Interpolation, but just based on the message, it seems clear that it doesn't want values in the first dimension that are that close together. So, one thing you could try is make your duplicate detector less precise, like maybe DeleteDuplicatesBy[data, Rationalize[First@#, 1/1000] &]. I don't kow if that'll scale with your data or whether it makes sense in your context. $\endgroup$
    – lericr
    Commented Apr 25, 2023 at 15:42
  • 2
    $\begingroup$ Consider points duplicates if their first values are the same. data2 = DeleteDuplicates[data, #1[[1]] == #2[[1]] &]; $\endgroup$
    – Bob Hanlon
    Commented Apr 25, 2023 at 15:48
  • $\begingroup$ Confirming @BobHanlon's solution. Interpolation[DeleteDuplicates[data, #1[[1]] == #2[[1]] &]] works. Note "==" is "approximately equal to within numerical precision" $\endgroup$ Commented Apr 25, 2023 at 18:28
  • $\begingroup$ Related: mathematica.stackexchange.com/questions/181424/… $\endgroup$
    – Michael E2
    Commented Apr 25, 2023 at 22:19

1 Answer 1

2
$\begingroup$

The shortest solution I am aware of for deduplicating a data list such as yours is:

data = {
    {1.880167859175211,  0.07717769628585},
    {1.9031673299071004, 0.07321050510180123},
    {1.9589651339133867, 0.06358591161773906},
    {2.0039652605902973, 0.05582381535409813},
    {2.0039652605902973, 0.05582381535409813},
    {2.0039652605903098, 0.05582381535409601},
    {2.0039652606025786, 0.05582381535197977},
    {2.0101043764185507, 0.054764875934334406},
    {2.0101043764222215, 0.054764875933701225}};

Interpolation[
  DeleteDuplicatesBy[data,
    Rationalize[#1[[1]], 2^(-46)] & ],
  InterpolationOrder -> 1]

After a close reading of DeleteDuplicates (note, no By), using Mathematica's Equal test instead of the Rationalize based test that I used previously:

Interpolation[
 DeleteDuplicates[data, #1[[1]] == #2[[1]] &],
  InterpolationOrder -> 1]

The above method can be used in $\mathbb{R}^n,\ n>1$ interpolation:

Interpolation[
 DeleteDuplicates[data, #1[[;;n]] == #2[[;;n]] &],
  InterpolationOrder -> 1]
{{1.8801679, 0.077177696}, {1.9031673, 0.073210505}, {1.9589651, 
  0.063585912}, {2.0039653, 0.055823815}, {2.0039653, 
  0.055823815}, {2.0101044, 0.054764876}, {2.0101044, 0.054764876}}

InterpolatingFunction[{{1.880167859175211, 2.0101043764222215`}}, {5, 
  7, 0, {7}, {2}, 0, 0, 0, 0, Automatic, {}, {}, 
  False}, {{1.880167859175211, 1.9031673299071004`, \
1.9589651339133867`, 2.0039652605902973`, 2.0039652606025786`, \
2.0101043764185507`, 2.0101043764222215`}}, \
{Developer`PackedArrayForm, {0, 1, 2, 3, 4, 5, 6, 7}, {
  0.07717769628585, 0.07321050510180123, 0.06358591161773906, 
  0.05582381535409813, 0.05582381535197977, 0.054764875934334406`, 
  0.054764875933701225`}}, {Automatic}]

No error messages were generated.

Equal states: "Approximate numbers with machine precision or higher are considered equal if they differ in at most their last seven binary digits (roughly their last two decimal digits)." *IEEE 754 Double Precision binary floating numbers generally have 53 binary in their mantissas. The use of 2^(-46) basically causes a truncation to 46 bits, $53 - 7 == 46$.

toHex = Function[n, StringJoin[
     (Characters["0123456789ABCDEF"][[#1 + 1]] & ) /@ 
      Flatten[Reverse[IntegerDigits[Normal[ExportByteArray[n, 
           "Real64"]], 16, 2]]]]];

The purpose of toHex is to produce a hexadecimal representation of the bits of IEEE-754 double precision binary floating point number.

Bwlow, I selected one of the numbers from the data list. I show the hexadecimal representation of that number and of some nearby decimal numbers to illustrate how much change is needed to change the hexadecimal representation and how much more change is needed to make the numbers different in the Wolfram Mathematica equality sense.

toHex[0.055823815354098130]

3FAC94F06A914D77

toHex[0.055823815354098128]

3FAC94F06A914D76

0.055823815354098926`18. == 0.055823815354098130

True

toHex[0.055823815354098136]

3FAC94F06A914D78

0.055823815354098130 == 0.055823815354098136

True

0.055823815354098926`18. == 0.055823815354098130

True

0.055823815354098927`18. == 0.055823815354098130

False

QED

I recommend reading material like Forman S. Acton's Real Computing Made Real: Preventing Errors in Scientific and Engineering Calculations. Another resource Testing equality of two floats: Realistic example.

ClearAll[toFloat]; 
toFloat[s_String /; StringLength[s] == 16 && 
     StringFreeQ[s, RegularExpression["[^0-9A-Fa-f]"]]] := 
  Block[{sign, exponent, mantissa}, 
   exponent = ToExpression[StringJoin["16^^", 
       StringTake[s, 3]]];
   sign = If[exponent >= 2048, 
      exponent -= 2048; -1., 1.]; exponent -= 1024; 
   mantissa = ToExpression[StringJoin["16^^1", 
       StringTake[s, -13]]];
   SetPrecision[
     sign*2^(exponent - 51)*mantissa, 55]]
toHex[1 + $MachineEpsilon]

(*"3FF0000000000001"*)

toHex[0.1]

(*"3FB999999999999A"*)

toFloat["3FB999999999999A"]

(*0.1000000000000000055511151231257827021181583404541015625`55.*)

N[toFloat["3FB999999999999A"]]

(*0.1`*)

toFloat["3FB9999999999999"]

(*0.09999999999999999167332731531132594682276248931884765625`55.*)

N[toFloat["3FB9999999999999"]]

(*0.1`*)

Another trap for the unaware is the sometimes vast differences in concepts of computer floating numbers (which are effectively limited range integers) and what is taught about real number. Computers do not have infinite precision!

$\endgroup$
3
  • $\begingroup$ This is a very helpful answer, thank you. I was unsure about what value to use for checking equality and you have sorted this out. The use of Rationalize deals with the issue for numbers of differing magnitude. As far as you are aware is Mathematica consistent for all instances where it is necessary to check equality? I guess WorkingPrecision is involved, e.g. for root finding, if equality is defined with fewer digits. $\endgroup$
    – Hugh
    Commented May 2, 2023 at 9:03
  • $\begingroup$ @Hugh As far as I am aware SetPrecision, WorkingPrecision, etc. have not effect on the real number equality test. $\endgroup$
    – anon
    Commented May 2, 2023 at 21:58
  • 1
    $\begingroup$ Tolerance of Equal is given by Internal`Internal`$EqualTolerance which is set to 2.10721 by default. Sginificantly increasing its value, for example Internal`$EqualTolerance = 16, will make 4. == 5. return True. $\endgroup$
    – Domen
    Commented May 3, 2023 at 5:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.