Here's the set-up: I'm generating data based on a process, but the important thing is that eventually the data diverges and starts forming branches. This is a close up of some of the behavior:
The data consists of a list of data points, where the x-value is the time step (1,2,...,N) and the y-position is the value that that point has at that time step. Now I am curious about the "clusters" for the y-values at each time step (the x-axis), so what I first did was use
clusterData = Table[ClusteringComponents[data[[i,All,2]]],{i,Length@data}]
to find the clusters of the y-data at each individual time step, and then I combined the information into a Dataset object to plot, like so:
dataset = Dataset@Flatten@Table[Table[<|"x"->data[[j,i,1]],"y"->data[[j,i,2]],"cluster"->clusterData[[j,i]]|>,{i,Length@data[[j]]}],{j,Length@clusterData}]
dataset = dataset[All,{"cluster"->ToString}]; (* otherwise it won't plot *)
ListPlot[dataset[GroupBy[Key["cluster"]], All, {"x", "y"}]]
What I'm essentially trying to look at is the evolution of these clusters over time:
which is why I want to look at the clusters at each time step, rather than just giving the clustering algorithm all of the data at one shot. This is also why I'd ideally like to have some "consistency" in the clusters, and keep, say, cluster 1 at the top and cluster 2 on the bottom, rather than them flip-flopping continually like this.
Here's my problem: obviously the clustering mechanic is unaware of the previous time step, so Mathematica's ClusteringComponents[] just clusters the data as it sees fit. This is why in the plot we see that clusters 1 and 2 (red and blue) switch back so often. This is a problem because I'm trying to keep track of the number of data points assigned to each cluster as each time step. So if, for example, the top branch at time step 30 was assigned to cluster 1, I'd prefer if I could force the algorithm to keep assigning that data to cluster 1, rather than switch to assigning it to cluster 2.
This struck me that this would be some sort of time-space clustering problem, and when I googled I came across this question, which isn't quite what I'm trying to do, I don't think - their clusters are rather clear. This question seemed to be getting at a similar question, but the comment about checking Segmentation Analysis doesn't currently seem the most useful in my case. I tried googling "clusters over time" and similar combinations, but no cigar.
I acknowledge however that this sounds like a pretty hard problem, so I'm really just trying to see if anyone wouldn't have any ideas they could spit ball my way. Maybe there's a different way to approach this than what I'm doing and I just can't currently see it. Different methods to approach this, Wolfram Alpha pages to look at, etc, would be appreciated.
Edit: Some of the data:
data = {{{0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0,
0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0,
0}, {0, 0}, {0, 0}, {0, 0}, {0,
0}}, {{1, -0.513217549931869`}, {1, -0.09365557645450462`}, {1,
0}, {1, 0}, {1, 0}, {1, -0.026320223366777196`}, {1,
0.7984677253527943`}, {1, 0.8503933934966845`}, {1,
0.022991013603006483`}, {1, 0.4068139206905119`}, {1,
0}, {1, -0.0928930335040925`}, {1, 0.6073459057143387`}, {1,
0}, {1, -0.10417227689265118`}, {1, -0.14658163557138826`}, {1,
-0.4491265654775356`}, {1, -0.596466550193052`}, {1,
0}, {1, -1.0612645079488878`}}, {{2,
1.3985650183744234`}, {2, -0.09365557645450462`}, {2,
-2.027216835726654`}, {2, -0.988752647380104`}, {2,
0.9126264845196519`}, {2, 1.808045050702348`}, {2,
1.413184562566951`}, {2, -0.883559376209254`}, {2,
-1.1331046017892703`}, {2, -0.00348663337270283`}, {2, 0}, {2,
0.584699773594958`}, {2, 0.6073459057143387`}, {2,
0.9251140978694967`}, {2,
0.07196133626826118`}, {2, -0.5354548069038403`}, {2,
1.2148526506168342`}, {2, 1.0462407939068803`}, {2,
0}, {2, -1.0612645079488878`}}, {{3,
1.3985650183744234`}, {3, -0.09365557645450462`}, {3,
-2.027216835726654`}, {3, -0.988752647380104`}, {3,
-2.5197630981831387`}, {3, -1.5512956637821786`}, {3,
1.413184562566951`}, {3, -1.900679156719825`}, {3,
-1.1331046017892703`}, {3, -2.0474709749300977`}, {3, 0}, {3,
0.584699773594958`}, {3,
2.0190621081573896`}, {3, -1.015089370369215`}, {3,
1.0026103774048463`}, {3, -1.784903428952083`}, {3,
-0.16221484809586872`}, {3, -0.5282018418789214`}, {3,
0}, {3, -1.2741192443035625`}}, {{4, -2.631946907662266`}, {4,
1.756524979321504`}, {4,
2.441503475699692`}, {4, -0.988752647380104`}, {4,
-2.5197630981831387`}, {4, -1.5512956637821786`}, {4,
1.413184562566951`}, {4, 2.9356173416597957`}, {4,
0.9244100524189189`}, {4, -2.0474709749300977`}, {4,
0.9334647485862009`}, {4,
0.24745778524139173`}, {4, -2.583165664117323`}, {4,
-1.015089370369215`}, {4,
0.8985767114257333`}, {4, -1.784903428952083`}, {4,
0.6453853494667907`}, {4, -0.5282018418789214`}, {4,
0}, {4, -1.2741192443035625`}}, {{5, -2.631946907662266`}, {5,
1.756524979321504`}, {5, 1.6305366409991486`}, {5,
1.932828807232926`}, {5, -2.5197630981831387`}, {5,
1.0543944935161025`}, {5,
1.7291981045725167`}, {5, -0.7778133302474788`}, {5,
0.8266925844824518`}, {5, -2.0474709749300977`}, {5,
0.9334647485862009`}, {5,
1.38044633989962`}, {5, -2.583165664117323`}, {5,
-1.015089370369215`}, {5,
0.8985767114257333`}, {5, -1.9708082514350593`}, {5,
0.6453853494667907`}, {5, -3.0908950511953535`}, {5,
-1.0549277088681484`}, {5, -1.2332813650077705`}}, {{6,
-2.631946907662266`}, {6, -1.4268066402071566`}, {6,
-1.546177931907455`}, {6, -1.825586428301003`}, {6,
2.8886952362534006`}, {6, 1.0543944935161025`}, {6,
2.5764316031983348`}, {6, 3.4994923719503026`}, {6,
1.7544443848175204`}, {6, -4.530610424408232`}, {6,
0.9334647485862009`}, {6,
1.38044633989962`}, {6, -2.583165664117323`}, {6,
1.2245294723645972`}, {6,
0.8985767114257333`}, {6, -1.9708082514350593`}, {6,
-2.8254012847210324`}, {6, -3.5637568371858372`}, {6,
-2.5773691618497794`}, {6, -1.656590739869932`}}, {{7,
-2.631946907662266`}, {7,
1.9333620138695253`}, {7, -3.3897417400543794`}, {7,
-1.825586428301003`}, {7, -2.303751772372871`}, {7,
1.4008767015955246`}, {7, 2.5764316031983348`}, {7,
3.4994923719503026`}, {7, -2.247446164713139`}, {7,
-1.77663051350171`}, {7, -1.353876282731421`}, {7,
-3.062747022737813`}, {7,
1.3985697477854817`}, {7, -0.7795201756305139`}, {7,
-2.3103917181887574`}, {7, -1.9708082514350593`}, {7,
-2.8254012847210324`}, {7, -2.008564038646559`}, {7,
-2.5051053471821416`}, {7, -1.656590739869932`}}, {{8,
-1.6090782870989337`}, {8, 0.20070654088218776`}, {8,
1.0953134681786576`}, {8,
1.3606485003810067`}, {8, -0.8333106346146395`}, {8,
-1.6711509474142892`}, {8, 1.6307347240903876`}, {8,
2.3450948041242654`}, {8, -2.247446164713139`}, {8,
-1.327629351772806`}, {8,
1.3445972924735146`}, {8, -2.8601080409051445`}, {8,
1.3985697477854817`}, {8, -2.1486422317416447`}, {8,
-2.3103917181887574`}, {8, -1.4522239787119098`}, {8,
-2.8254012847210324`}, {8, -2.008564038646559`}, {8,
-2.5051053471821416`}, {8,
2.010117503566065`}}, {{9, -1.6090782870989337`}, {9,
-2.186407779834681`}, {9, 3.4492818460577315`}, {9,
1.3606485003810067`}, {9, -0.626970014244637`}, {9,
0.9835372388196459`}, {9, -2.4343468720401615`}, {9,
2.3450948041242654`}, {9, 1.9275074190927186`}, {9,
2.2615200022853092`}, {9,
1.3445972924735146`}, {9, -2.8601080409051445`}, {9,
2.6528116484325`}, {9, -2.070306914057001`}, {9,
2.257770844418676`}, {9, -1.4522239787119098`}, {9,
-2.8254012847210324`}, {9, 2.3477751597403578`}, {9,
2.4522488628033208`}, {9,
2.010117503566065`}}, {{10, -2.7427139685194737`}, {10,
-2.186407779834681`}, {10,
3.4492818460577315`}, {10, -2.2184066831063123`}, {10,
-0.626970014244637`}, {10,
0.8627635574950969`}, {10, -2.4343468720401615`}, {10,
-1.7699746901775564`}, {10,
2.544128937134569`}, {10, -0.6976504925593403`}, {10,
-2.4769420188888844`}, {10, -2.4169701969396455`}, {10,
2.462039340912788`}, {10, -2.070306914057001`}, {10,
2.967533420351023`}, {10, -1.5785023546994714`}, {10,
-1.9153869508059653`}, {10, 2.3477751597403578`}, {10,
2.2843895094454636`}, {10,
2.010117503566065`}}, {{11, -2.243774745177397`}, {11,
-2.186407779834681`}, {11, -2.6116134381230856`}, {11,
-2.2184066831063123`}, {11, 3.1999184313706963`}, {11,
3.1833138319767205`}, {11, -2.4343468720401615`}, {11,
-1.0160611554482348`}, {11, 0.0699075061149661`}, {11,
2.105814609932241`}, {11, 3.503155577334455`}, {11,
3.46883450424943`}, {11, -3.086172257731213`}, {11,
3.041858709356843`}, {11,
2.967533420351023`}, {11, -1.5785023546994714`}, {11,
2.6289317407571167`}, {11, -3.2275847831378868`}, {11,
3.809127502466673`}, {11, 2.626399232058327`}}, {{12,
3.570073954889942`}, {12,
2.0451575429306588`}, {12, -2.6116134381230856`}, {12,
4.321843024363826`}, {12, -0.33638207422522504`}, {12,
2.7293006320336812`}, {12, -0.7032639687766782`}, {12,
-1.0160611554482348`}, {12, 2.4445677108866337`}, {12,
2.105814609932241`}, {12,
3.503155577334455`}, {12, -1.3373271419360089`}, {12,
-3.086172257731213`}, {12, -2.4553685842695274`}, {12,
2.967533420351023`}, {12, -1.5785023546994714`}, {12,
-1.8410492782677599`}, {12, -1.512115953146461`}, {12,
3.809127502466673`}, {12,
1.083447430804604`}}, {{13, -2.689819771603731`}, {13,
2.0451575429306588`}, {13, -2.6116134381230856`}, {13,
4.321843024363826`}, {13, 3.3059143041695673`}, {13,
3.244590972055542`}, {13, 2.0051554526468904`}, {13,
4.324014831334397`}, {13, 3.8618737162157695`}, {13,
2.105814609932241`}, {13, 3.503155577334455`}, {13,
3.6129228725365126`}, {13, -1.9952338031778551`}, {13,
-2.9220279774640816`}, {13,
2.967533420351023`}, {13, -1.5785023546994714`}, {13,
2.054386639181586`}, {13, -3.3267172419576783`}, {13,
3.6526309320145995`}, {13,
1.083447430804604`}}, {{14, -2.689819771603731`}, {14,
3.53279137731269`}, {14, 2.9854607055686144`}, {14,
4.321843024363826`}, {14, -1.9650609340102805`}, {14,
-2.9166543621389445`}, {14, 2.0051554526468904`}, {14,
4.818374761324`}, {14,
1.5433122680009346`}, {14, -3.2731478151496214`}, {14,
-2.6945843454021334`}, {14, -2.8636015199502536`}, {14,
-1.9952338031778551`}, {14, 4.376016805575464`}, {14,
0.58473667437242`}, {14, -2.6101165472097434`}, {14,
2.054386639181586`}, {14, -3.3267172419576783`}, {14,
1.6584676076367435`}, {14,
1.083447430804604`}}, {{15, -2.689819771603731`}, {15,
3.53279137731269`}, {15, -2.2870625975701757`}, {15,
-4.135995224575007`}, {15, -1.9650609340102805`}, {15,
1.3217534073659332`}, {15, 2.0051554526468904`}, {15,
4.101564011944627`}, {15,
4.093717188241731`}, {15, -3.2731478151496214`}, {15,
4.847640930316921`}, {15,
1.2874161783320233`}, {15, -1.9952338031778551`}, {15,
2.9188024605949905`}, {15, 2.6040087298447294`}, {15,
5.339176136113665`}, {15, -3.3103228034511547`}, {15,
-4.301808830812817`}, {15, -2.746951587052287`}, {15,
1.083447430804604`}}, {{16, -2.037800118270176`}, {16,
-4.751304697154737`}, {16, -2.2870625975701757`}, {16,
-4.135995224575007`}, {16, 3.931661341549311`}, {16,
1.3217534073659332`}, {16, -1.2029502884368948`}, {16,
-3.3104335259499846`}, {16, 2.9445811081697375`}, {16,
1.6615135519154092`}, {16,
4.847640930316921`}, {16, -4.321941722394576`}, {16,
1.2182270883920108`}, {16, -4.291752184788049`}, {16,
4.7637032876555905`}, {16,
0.30175211922907064`}, {16, -3.0785524566345988`}, {16,
-5.252385219193551`}, {16, -4.569154370815168`}, {16,
-3.118539549776655`}}, {{17, -2.037800118270176`}, {17,
4.33619721417101`}, {17, -2.4656831140767244`}, {17,
-2.2591563156547902`}, {17, 3.931661341549311`}, {17,
4.914434688154813`}, {17, -3.0981837631411415`}, {17,
-1.2697115367525944`}, {17, -2.373162078788219`}, {17,
-3.1248528123371404`}, {17,
4.847640930316921`}, {17, -4.39707153759843`}, {17,
2.0587372281651035`}, {17, -4.291752184788049`}, {17,
4.61273317261773`}, {17,
4.244308961002415`}, {17, -3.0785524566345988`}, {17,
-5.252385219193551`}, {17, -3.2591716726389803`}, {17,
4.145561030957269`}}, {{18, -2.037800118270176`}, {18,
4.33619721417101`}, {18, 3.978450682164329`}, {18,
3.9076485751972734`}, {18, 3.931661341549311`}, {18,
4.914434688154813`}, {18, 4.795251574929358`}, {18,
5.253588611748619`}, {18, 3.922462678282567`}, {18,
4.9564257158773035`}, {18, -3.620123925372994`}, {18,
-2.7362096812830066`}, {18, -1.5377787297626027`}, {18,
-3.8367031218917154`}, {18,
4.61273317261773`}, {18, -1.8930669483966014`}, {18,
3.3304550080763398`}, {18, -0.45980901774225336`}, {18,
-3.2591716726389803`}, {18,
4.83477986545003`}}, {{19, -0.780709357322956`}, {19,
4.034849546951172`}, {19,
3.978450682164329`}, {19, -4.787529629078456`}, {19,
4.105457331485535`}, {19,
3.1276500533269465`}, {19, -2.385634657940179`}, {19,
5.248322249519411`}, {19, 3.922462678282567`}, {19,
4.9564257158773035`}, {19, -3.620123925372994`}, {19,
-2.8758698931403233`}, {19, -3.1713176003674555`}, {19,
-3.8367031218917154`}, {19,
4.61273317261773`}, {19, -1.8930669483966014`}, {19,
3.0273337781158753`}, {19,
6.251699456818568`}, {19, -1.606852464277316`}, {19,
3.745469927768456`}}};
dataand make sure the posted code evaluates. $\endgroup$