# DeleteDuplicates[] does not work as expected on floating point values

Here is my simple example, and in this case function DeleteDuplicates does not work as expected.

I want to FindRoot of my function $\chi[\nu]$, and since function $\chi$ is very sensitive to initial guess I decided to generate many initial conditions and leave only those solutions which are distinct. For this purpose I want to apply DeleteDuplicates on the resulting list of solution.

Here is the definition of my function:

χ[ν_] := 2*PolyGamma[1] - PolyGamma[1/2 + I*ν] - PolyGamma[1/2 - I*ν]


Here I generate many solution according to many random initial guesses

m = Table[v /. FindRoot[χ[v] == -1.2 - 0.2*I, {v, RandomComplex[]}], {i, 1, 10}]


And finally, I want to leave only distinct solutions:

DeleteDuplicates[m]


Unfortunately, the operation DeleteDuplicates[m] does not change the list m, although there are many identical values.

Namely:

DeleteDuplicates[m]

{1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I,
1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I, 0.0250407 + 1.00352 I,
1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I,
0.0250407 + 1.00352 I, 1.06423 + 0.0968739 I}


I'm puzzled.

Any help or suggestions are very welcome!

Thanks!

-
Try something like DeleteDuplicates[m, Abs[#1 - #2] < 0.01 &] . –  b.gatessucks Mar 20 '13 at 9:59
Thanks for editing Mr. Wizard! It looks much better :) @b.gatessucks: Great, works like a charm. Could you explain the reason? I'm quite new to this. Thanks a lot for help! –  A.K. Mar 20 '13 at 10:03
Andrew, please see the update to my answer for an important note about performance. –  Mr.Wizard Mar 21 '13 at 13:53
Andrew, what about changing "does not work properly" to "does not work as expected"? In fact it works properly, the cause is just seemingly identical numbers... –  Yves Klett Mar 21 '13 at 14:17

You would do well to understand the difference between tools that are intended for structural operations and those that are intended for mathematical operations. DeleteDuplicates is of the former, generally speaking. As such it is comparing the exact FullForm of the objects, or at least something close (caveat).

As b.gatessucks recommends in a comment you can use a mathematical comparison function for the equivalence test of DeleteDuplicates, e.g.:

 DeleteDuplicates[m, Abs[#1 - #2] < 10^-12 &]

{1.06423 + 0.0968739 I, 0.0250407 + 1.00352 I}


Indidentially you could also use Union, but the syntax is a bit different. Note the ( ).

 Union[m, SameTest -> (Abs[#1 - #2] < 10^-12 &)]

{0.0250407 + 1.00352 I, 1.06423 + 0.0968739 I}


Using InputForm to show all of the digits of your expression you can see that they are not structurally identical in the (approximate) way that Mathematica "sees" them:

m // InputForm

{1.0642275928442373 + 0.09687392021742822*I, 1.0642275928442366 + 0.09687392021742817*I,
1.0642275928442366 + 0.09687392021742797*I, 1.064227592844237 + 0.09687392021742822*I,
1.0642275928442373 + 0.09687392021742852*I, 1.0642275928442366 + 0.09687392021742793*I,
1.0642275928442368 + 0.09687392021742801*I, 0.025040728196256346 + 1.0035162552538588*I,
1.0642275928442377 + 0.0968739202174282*I, 1.0642275928442375 + 0.0968739202174283*I}


### Performance

Yves reminded me to mention something about the performance of using a custom comparison function in DeleteDuplicates or Union as I did above. For long lists this is always considerably slower than using the default method. I gave an example with timings in How to represent a list as a cycle.

To apply that method here we could Round the numbers beforehand:

Round[m, 10^-12] // DeleteDuplicates // N

{1.06423 + 0.0968739 I, 0.0250407 + 1.00352 I}


I added // N to convert back to machine precision, but the values will not be precisely the same. This probably doesn't matter if you consider numbers this close to be duplicates, but should you want the unchanged numbers you could use GatherBy and get performance not far distant.

First /@ GatherBy[m, Round[#, 10^-6] &]

-
Thanks for the answer, I see. It's like a comparison with epsilon of two floating point numbers. I assumed, that MATHEMATICA already does it for me. –  A.K. Mar 20 '13 at 10:08
Great, I missed InputForm. Without it it looks the same. –  A.K. Mar 20 '13 at 10:09
@Andrew On some mathematical operations it may, but as I stated DeleteDuplicates is generally a structural tool. You will run into a similar problem in many cases. For example, replacing elements in a list, because Replace and ReplaceAll are also structural tools: {0, 0., 0.00000} /. 0 -> 1 -- you need a mathematical tool like Equal: {0, 0., 0.00000} /. x_ /; x == 0 -> 1 –  Mr.Wizard Mar 20 '13 at 10:12
@Andrew FullForm was a bit needlessly verbose here, but in general you should look at the FullForm when trying to understand how Mathematica will treat an expression using structural tools. There are many cases where not using it will leave you quite befuddled as it can be very different from what is shown in standard output notation. –  Mr.Wizard Mar 20 '13 at 10:13
Admonish - lovely acoustics but far too harsh. I thought of it as "gently encouraging" at most... –  Yves Klett Mar 21 '13 at 14:13