# ClusteringComponent and FindClusters are returning the wrong number of clusters and are inconsistent

I am trying to find the optimum number of clusters in a correlation matrix.

To do this I run a clustering algorithm that loops through 3 variables ( number of clusters, Method and RandomSeeding) and then select the best clustering from there.

However, while running these loops I run in to errors every now and then when FindClusters and ClusteringComponents return

1. Inconsistent number of clusters, i.e.: FindClusters returns 4 clusters and ClusteringComponents returns 5 clusters

OR

1. Wrong number of clusters, i.e.: both return 4 clusters when 5 clusters was requested

An example of when this happens is in snippet 2 when number of clusters = 5, RandomSeeding=3 and Method->"KMeans"

(* ===DEFINE REQUIRED FUNCTION=== *)
properDistanceMatrix[corrMat_] := Sqrt[0.5*Chop[1. - corrMat]];

(* ===IMPORT DATA=== *)
retMat = ToPackedArray[Import[importPath]];
used = retMat[[1 ;; 1008, 226 ;; 275]];
corMat = Correlation[used];
distMat = properDistanceMatrix[corMat];
allDistance = DistanceMatrix[distMat];

(* ===RUN CODE=== *)

(* ===snippet 1=== *)
cc = ClusteringComponents[distMat, 5, 1, Method -> "KMedoids",
RandomSeeding -> 3, DistanceFunction -> EuclideanDistance,
PerformanceGoal -> "Quality"];
fc = FindClusters[distMat, 5, Method -> "KMedoids",
RandomSeeding -> 3, DistanceFunction -> EuclideanDistance,
PerformanceGoal -> "Quality"];
Print[{First@Dimensions[fc], Max[cc]}];
(* correctly returns 5 clusters on both as specified above *)

(* ===snippet 2=== *)
ccBug = ClusteringComponents[distMat, 5, 1, Method -> "KMeans",
RandomSeeding -> 3, DistanceFunction -> EuclideanDistance,
PerformanceGoal -> "Quality"];
fcBug = FindClusters[distMat, 5, Method -> "KMeans",
RandomSeeding -> 3, DistanceFunction -> EuclideanDistance,
PerformanceGoal -> "Quality"];
Print[{First@Dimensions[fcBug],Max[ccBug]}];
(* returns {4,5} when 5 clusters were requested.
returns {4,4} if snippet 1 is not run.
problem disappears when seed is changed (to 4 for example) and correctly returns {5,5} *)


Based on this thread FindClusters versus ClusteringComponents? I was under the impression that FindCluster and ClusteringComponents had been rejigged to return the same number of clusters given the same inputs.

Is this a bug or am I using the functions wrongly?

Thanks

PS: I am running this on Windows 7 with Mathematica 11.2.0.0

EDIT : I realise that pointing to 3rd party sites for file hosting is frowned upon so I have added the following code that should replicate the same error

properDistanceMatrix[corrMat_] := Sqrt[0.5*Chop[1. - corrMat]];
retMatR = RandomReal[{-1, 1}, {10, 10}];

(*{{-0.420038, 0.477, 0.462921, -0.650315, -0.391649,
0.294729, -0.226934, -0.735427, 0.331658, -0.693274}, {-0.494563,
0.021645, 0.0178725, 0.266314, -0.986802, -0.0496106, -0.732457,
0.137578, 0.0482319, -0.616772}, {-0.073485, -0.655964,
0.00530954, -0.823253, 0.303467, 0.540479,
0.731417, -0.608943, -0.953788, 0.614624}, {0.403796,
0.895508, -0.354059, 0.0572363, 0.664937, -0.470154, -0.856749,
0.794929, 0.945532, 0.413173}, {-0.509987, 0.709611, 0.402642,
0.274048, -0.0804114, 0.261213, 0.205204, 0.754889,
0.406956, -0.701399}, {0.669529, -0.972304, -0.377802, 0.207726,
0.740803, -0.32163, 0.733927, 0.066348,
0.465975, -0.938974}, {0.485361, -0.987685, -0.525496,
0.965126, -0.802624, 0.232629, -0.660341, -0.0636777,
0.443938, -0.603204}, {-0.243766, -0.629372, 0.690897,
0.0910828, -0.192148, 0.272657, -0.124762, 0.591744, 0.0664948,
0.807557}, {-0.820118, -0.978278, -0.689691,
0.776383, -0.389317, -0.987641, 0.712486, -0.338271,
0.449278, -0.258858}, {-0.375498, -0.0127232, -0.148995, 0.754593,
0.720646, -0.0362857, 0.614544, 0.583437, 0.842111, 0.505644}}*)

Table[
cc = ClusteringComponents[distMat, numClusters, 1,
Method -> "KMeans", RandomSeeding -> seed,
DistanceFunction -> EuclideanDistance,
PerformanceGoal -> "Quality"];
fc = FindClusters[distMat, numClusters, Method -> "KMeans",
RandomSeeding -> seed, DistanceFunction -> EuclideanDistance,
PerformanceGoal -> "Quality"];
If[First@Dimensions[fc] != Max[cc],
Print[{{First@Dimensions[fc], Max[cc]}, {seed, numClusters}}]],
{seed, 1, 25},
{numClusters, 2, 9}
];

(*{{8,9},{25,9}}*)


CSV file (retMat.zip) containing the 'retMat' matrix is available on this link.

• I had this same issue with FindClusters not returning the requested number of components when using the second argument, for which I submitted [CASE:4092039]. Unfortunately, WRI replied that 'the second argument n in FindClusters[] instructs the function to determine at most n clusters and not exactly n clusters.' So this is a 'feature' not a 'bug' - but obviously it is an entirely detrimental feature, and I requested that they improve the design to something more useful. If you also make a complaint/request along these lines it might actually get fixed... Commented Oct 6, 2018 at 14:36
• Also, IIRC, the KMeans and the KMediods 'Methods' were both unstable, but the Agglomerate was (more?) consistent. Commented Oct 6, 2018 at 14:42
• Thanks for pointing out that it returns "at most" n clusters which addresses point 2 above. Now we still have the problem that the ClustersComponents returned are inconsistent with FindClusters. I have contacted Wolfram about this too and will highlight your point above and post any updates here. Commented Oct 7, 2018 at 3:08
• This might be helpful for consistency instead of using ClusteringComponents fc=FindClusters[data]; alternativeClusterComponents=DeleteCases[Flatten[Table[If[MemberQ[fc[[#]],x],#]&/@Range[Length[fc]],{x,data}]],Null] Commented Oct 7, 2018 at 9:24

Wolfram has reponded confirming that "the number n form FindClusters[data, n] or ClusteringComponents[data,n] is the actual maximum number of clusters, the data might be placed in less than n clusters" and also "that changing RandomSeeding indeed produces different results."
The clustering family (FindClusters, ClusterClassify, ClusteringComponents) now treats the syntax f[data,n] to signify that data has to be partitioned in exactly n clusters. To get back the old behavior that treated n as an upper limit, one can use f[data,UpTo[n]].