# Cluster analysis returns questionable results

A few weeks ago I ran a cluster analysis on some economic data and (unfortunately only) now I found out that some results do not make sense. For instance I analyzed the development of unemployment rates form end-2005 to mid-2011 for 32 countries. One cluster contained among others Germany (Figure 1) and Spain (Figure 2) and another cluster among others Estonia (Figure 3). I couldn't understand why Mathematica would put Germany and Spain together in one cluster, instead of Estonia and Spain. I just calculated the EuclideanDistance between Germany and Spain (17.3) as well as Spain and Estonia (9.1). These results underline my doubts on my cluster-analysis-results.

Here is the code with which I worked:

First I generate a random sample (n=1200) of the original data, to minimize the effect of the input-order in FindClusters (Note from Mathematica-Documentation: "The order of elements can have an effect on the clusters found")

data;  (*see below*)
Table[DeleteDuplicates[
Table[RandomSample[DeleteCases[data[[j]], {}]], {i, 1200}]], {j,Length[data]}];


Then I run FindCluster for the first time to determine the commonest length of each cluster.

ClustersUnSorted[list_]:=Table[Map[FindClusters,list[[i]]],{i, Length[list]}]


In my next step I say, that if the previous step found less than four clusters, I would like Mathematica instead to set the minimum number of clusters to 4.

ClusterLength[list_]:=Table[If[
Commonest[Map[Length,list[[i]]]][[1]]<4,
4, Commonest[Map[Length, list[[i]]]][[1]]],
{i, Length[list]}]


Then I run FindClusters again on the basis of the previously determined number of clusters.

ClustersUnSortedFixLength[list_,Flatten[ClusterLength_]]:=
Table[Map[FindClusters[#,ClusterLength[[i]]] &, list[[i]]], {i, Length[list]}]


The last two steps are there to get rid of duplicates as for each indicator I generated a random sample (see step 1).

ClustersSorted[list_] := Table[Table[
Sort[Map[Sort, list[[j]][[i]]]], {i, Length[list[[j]]]}],
{j, Length[list]}]

CommonestCluster[list_] := Map[Commonest, list]


The last two steps can be easily comprehended by the following example (instead of RandomSample I could use Permutations[{1, 5, 3, 10, 100}] for a much more reliable result but since Permutations is limited to input with length less than 11 it is not applicable to my data.):

Table[RandomSample[{1, 5, 3, 10, 100}], {i, 1200}];
Map[FindClusters, %];
Table[Map[Sort, %[[i]]], {i, Length[%]}];
Map[Sort, %];
Commonest[%]


exemplary data (Note: the time series are not always of the same length.):

Country1GDP = Join[{{"Country1", "GDP"}}, RandomReal[{-1, 1}, {6}]];
Country2GDP = Join[{{"Country2", "GDP"}}, RandomReal[{-1, 1}, {6}]];
Country3GDP = Join[{{"Country3", "GDP"}}, RandomReal[{-1, 1}, {6}]];
Country4GDP = Join[{{"Country4", "GDP"}}, RandomReal[{-1, 1}, {5}]];
Country5GDP = Join[{{"Country5", "GDP"}}, RandomReal[{-1, 1}, {6}]];

Country1Imports =
Join[{{"Country1", "Imports"}}, RandomReal[{-1, 1}, {6}]];
Country2Imports =
Join[{{"Country2", "Imports"}}, RandomReal[{-1, 1}, {5}]];
Country3Imports =
Join[{{"Country3", "Imports"}}, RandomReal[{-1, 1}, {6}]];
Country4Imports =
Join[{{"Country4", "Imports"}}, RandomReal[{-1, 1}, {6}]];
Country5Imports =
Join[{{"Country5", "Imports"}}, RandomReal[{-1, 1}, {6}]];

CompleteGDP = {Country1GDP, Country2GDP, Country3GDP, Country4GDP,
Country5GDP};
CompleteImports = {Country1Imports, Country2Imports, Country3Imports,
Country4Imports, Country5Imports};

data = {CompleteGDP, CompleteImports};

-
You know, you can supply multiple iterators to a single Table call... i.e., you can write Table[..., {i,...}, {j,...}] instead of Table[Table[...,{i,...}], {j,...}] –  rm -rf Mar 20 '12 at 0:50
@Mr.Wizard We must've edited at the same time :) It's ok, mine was more thorough =) Besides, the poster is probably not American, hence the use of s instead of z, which I think we should respect... –  rm -rf Mar 20 '12 at 0:51
Also, it would be helpful if you could give the same dataset you are using, namely what is data to start out with? (Sorry if I missed it) –  tkott Mar 20 '12 at 1:34
I'll try to generate some exemplary data, as the original one is far too large. –  John Mar 20 '12 at 8:43
@rm -rf: Actually, the equivalent to Table[Table[...,{i,...}],{j,...}] is Table[...,{j,...},{i,...}]. This constantly throws me off, as it does not simply allow to delete brackets, but one has to change the order of iterators as well. –  Thomas Oct 1 '12 at 18:08