FindClusters for 3D Pancake/Platelet Shapes

Question

I'm looking to properly cluster a dataset of roughly 3 million data points in 3-space. The shapes form closely spaced "clusters" that resemble pancakes.

Here is a downsampled dataset:

data = Import["https://dl.dropbox.com/u/28603777/data.wdx"];

I have tried the standard distance methods (Canberra, Euclidean, Manhattan and BrayCurtis), yet each leaves some points from one cluster improperly sorted into another.

out = FindClusters[data, 6];

The navy platelet "leaks" into the gold platelet due to the fact that the distance methods are isotropic. Also the gold platelet "leaks" into the red platelet.

Is there a different way to cluster such that the distance method can be optimized in this way, or is there a fast way to cluster these platelets that does not rely on this function.

Answer 1

Based on the data you provide, it seems that hierarchical clustering (see wiki here) with type "agglomerate" (bottom up) solves your problem, i.e.:

out = FindClusters[data, 6, Method -> "Agglomerate"];
ListPointPlot3D[out]

and get:

Based on how your full dataset looks like (e.g. if you know how many clusters there are etc.), you might need to adapt the code a bit (possibly also with distance function or matrix) - but for the sample you are providing, it seems to work nicely. Also, performance could be an issue, as noted in the wiki article.