# Generate clusters based off clusters in previous time step?

Here's the set-up: I'm generating data based on a process, but the important thing is that eventually the data diverges and starts forming branches. This is a close up of some of the behavior:

The data consists of a list of data points, where the x-value is the time step (1,2,...,N) and the y-position is the value that that point has at that time step. Now I am curious about the "clusters" for the y-values at each time step (the x-axis), so what I first did was use

clusterData = Table[ClusteringComponents[data[[i,All,2]]],{i,Length@data}]


to find the clusters of the y-data at each individual time step, and then I combined the information into a Dataset object to plot, like so:

dataset = Dataset@Flatten@Table[Table[<|"x"->data[[j,i,1]],"y"->data[[j,i,2]],"cluster"->clusterData[[j,i]]|>,{i,Length@data[[j]]}],{j,Length@clusterData}]
dataset = dataset[All,{"cluster"->ToString}]; (* otherwise it won't plot *)
ListPlot[dataset[GroupBy[Key["cluster"]], All, {"x", "y"}]]


What I'm essentially trying to look at is the evolution of these clusters over time:

which is why I want to look at the clusters at each time step, rather than just giving the clustering algorithm all of the data at one shot. This is also why I'd ideally like to have some "consistency" in the clusters, and keep, say, cluster 1 at the top and cluster 2 on the bottom, rather than them flip-flopping continually like this.

Here's my problem: obviously the clustering mechanic is unaware of the previous time step, so Mathematica's ClusteringComponents[] just clusters the data as it sees fit. This is why in the plot we see that clusters 1 and 2 (red and blue) switch back so often. This is a problem because I'm trying to keep track of the number of data points assigned to each cluster as each time step. So if, for example, the top branch at time step 30 was assigned to cluster 1, I'd prefer if I could force the algorithm to keep assigning that data to cluster 1, rather than switch to assigning it to cluster 2.

This struck me that this would be some sort of time-space clustering problem, and when I googled I came across this question, which isn't quite what I'm trying to do, I don't think - their clusters are rather clear. This question seemed to be getting at a similar question, but the comment about checking Segmentation Analysis doesn't currently seem the most useful in my case. I tried googling "clusters over time" and similar combinations, but no cigar.

I acknowledge however that this sounds like a pretty hard problem, so I'm really just trying to see if anyone wouldn't have any ideas they could spit ball my way. Maybe there's a different way to approach this than what I'm doing and I just can't currently see it. Different methods to approach this, Wolfram Alpha pages to look at, etc, would be appreciated.

Edit: Some of the data:

data = {{{0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0,
0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0, 0}, {0,
0}, {0, 0}, {0, 0}, {0, 0}, {0,
0}}, {{1, -0.513217549931869}, {1, -0.09365557645450462}, {1,
0}, {1, 0}, {1, 0}, {1, -0.026320223366777196}, {1,
0.7984677253527943}, {1, 0.8503933934966845}, {1,
0.022991013603006483}, {1, 0.4068139206905119}, {1,
0}, {1, -0.0928930335040925}, {1, 0.6073459057143387}, {1,
0}, {1, -0.10417227689265118}, {1, -0.14658163557138826}, {1,
-0.4491265654775356}, {1, -0.596466550193052}, {1,
0}, {1, -1.0612645079488878}}, {{2,
1.3985650183744234}, {2, -0.09365557645450462}, {2,
-2.027216835726654}, {2, -0.988752647380104}, {2,
0.9126264845196519}, {2, 1.808045050702348}, {2,
1.413184562566951}, {2, -0.883559376209254}, {2,
-1.1331046017892703}, {2, -0.00348663337270283}, {2, 0}, {2,
0.584699773594958}, {2, 0.6073459057143387}, {2,
0.9251140978694967}, {2,
0.07196133626826118}, {2, -0.5354548069038403}, {2,
1.2148526506168342}, {2, 1.0462407939068803}, {2,
0}, {2, -1.0612645079488878}}, {{3,
1.3985650183744234}, {3, -0.09365557645450462}, {3,
-2.027216835726654}, {3, -0.988752647380104}, {3,
-2.5197630981831387}, {3, -1.5512956637821786}, {3,
1.413184562566951}, {3, -1.900679156719825}, {3,
-1.1331046017892703}, {3, -2.0474709749300977}, {3, 0}, {3,
0.584699773594958}, {3,
2.0190621081573896}, {3, -1.015089370369215}, {3,
1.0026103774048463}, {3, -1.784903428952083}, {3,
-0.16221484809586872}, {3, -0.5282018418789214}, {3,
0}, {3, -1.2741192443035625}}, {{4, -2.631946907662266}, {4,
1.756524979321504}, {4,
2.441503475699692}, {4, -0.988752647380104}, {4,
-2.5197630981831387}, {4, -1.5512956637821786}, {4,
1.413184562566951}, {4, 2.9356173416597957}, {4,
0.9244100524189189}, {4, -2.0474709749300977}, {4,
0.9334647485862009}, {4,
0.24745778524139173}, {4, -2.583165664117323}, {4,
-1.015089370369215}, {4,
0.8985767114257333}, {4, -1.784903428952083}, {4,
0.6453853494667907}, {4, -0.5282018418789214}, {4,
0}, {4, -1.2741192443035625}}, {{5, -2.631946907662266}, {5,
1.756524979321504}, {5, 1.6305366409991486}, {5,
1.932828807232926}, {5, -2.5197630981831387}, {5,
1.0543944935161025}, {5,
1.7291981045725167}, {5, -0.7778133302474788}, {5,
0.8266925844824518}, {5, -2.0474709749300977}, {5,
0.9334647485862009}, {5,
1.38044633989962}, {5, -2.583165664117323}, {5,
-1.015089370369215}, {5,
0.8985767114257333}, {5, -1.9708082514350593}, {5,
0.6453853494667907}, {5, -3.0908950511953535}, {5,
-1.0549277088681484}, {5, -1.2332813650077705}}, {{6,
-2.631946907662266}, {6, -1.4268066402071566}, {6,
-1.546177931907455}, {6, -1.825586428301003}, {6,
2.8886952362534006}, {6, 1.0543944935161025}, {6,
2.5764316031983348}, {6, 3.4994923719503026}, {6,
1.7544443848175204}, {6, -4.530610424408232}, {6,
0.9334647485862009}, {6,
1.38044633989962}, {6, -2.583165664117323}, {6,
1.2245294723645972}, {6,
0.8985767114257333}, {6, -1.9708082514350593}, {6,
-2.8254012847210324}, {6, -3.5637568371858372}, {6,
-2.5773691618497794}, {6, -1.656590739869932}}, {{7,
-2.631946907662266}, {7,
1.9333620138695253}, {7, -3.3897417400543794}, {7,
-1.825586428301003}, {7, -2.303751772372871}, {7,
1.4008767015955246}, {7, 2.5764316031983348}, {7,
3.4994923719503026}, {7, -2.247446164713139}, {7,
-1.77663051350171}, {7, -1.353876282731421}, {7,
-3.062747022737813}, {7,
1.3985697477854817}, {7, -0.7795201756305139}, {7,
-2.3103917181887574}, {7, -1.9708082514350593}, {7,
-2.8254012847210324}, {7, -2.008564038646559}, {7,
-2.5051053471821416}, {7, -1.656590739869932}}, {{8,
-1.6090782870989337}, {8, 0.20070654088218776}, {8,
1.0953134681786576}, {8,
1.3606485003810067}, {8, -0.8333106346146395}, {8,
-1.6711509474142892}, {8, 1.6307347240903876}, {8,
2.3450948041242654}, {8, -2.247446164713139}, {8,
-1.327629351772806}, {8,
1.3445972924735146}, {8, -2.8601080409051445}, {8,
1.3985697477854817}, {8, -2.1486422317416447}, {8,
-2.3103917181887574}, {8, -1.4522239787119098}, {8,
-2.8254012847210324}, {8, -2.008564038646559}, {8,
-2.5051053471821416}, {8,
2.010117503566065}}, {{9, -1.6090782870989337}, {9,
-2.186407779834681}, {9, 3.4492818460577315}, {9,
1.3606485003810067}, {9, -0.626970014244637}, {9,
0.9835372388196459}, {9, -2.4343468720401615}, {9,
2.3450948041242654}, {9, 1.9275074190927186}, {9,
2.2615200022853092}, {9,
1.3445972924735146}, {9, -2.8601080409051445}, {9,
2.6528116484325}, {9, -2.070306914057001}, {9,
2.257770844418676}, {9, -1.4522239787119098}, {9,
-2.8254012847210324}, {9, 2.3477751597403578}, {9,
2.4522488628033208}, {9,
2.010117503566065}}, {{10, -2.7427139685194737}, {10,
-2.186407779834681}, {10,
3.4492818460577315}, {10, -2.2184066831063123}, {10,
-0.626970014244637}, {10,
0.8627635574950969}, {10, -2.4343468720401615}, {10,
-1.7699746901775564}, {10,
2.544128937134569}, {10, -0.6976504925593403}, {10,
-2.4769420188888844}, {10, -2.4169701969396455}, {10,
2.462039340912788}, {10, -2.070306914057001}, {10,
2.967533420351023}, {10, -1.5785023546994714}, {10,
-1.9153869508059653}, {10, 2.3477751597403578}, {10,
2.2843895094454636}, {10,
2.010117503566065}}, {{11, -2.243774745177397}, {11,
-2.186407779834681}, {11, -2.6116134381230856}, {11,
-2.2184066831063123}, {11, 3.1999184313706963}, {11,
3.1833138319767205}, {11, -2.4343468720401615}, {11,
-1.0160611554482348}, {11, 0.0699075061149661}, {11,
2.105814609932241}, {11, 3.503155577334455}, {11,
3.46883450424943}, {11, -3.086172257731213}, {11,
3.041858709356843}, {11,
2.967533420351023}, {11, -1.5785023546994714}, {11,
2.6289317407571167}, {11, -3.2275847831378868}, {11,
3.809127502466673}, {11, 2.626399232058327}}, {{12,
3.570073954889942}, {12,
2.0451575429306588}, {12, -2.6116134381230856}, {12,
4.321843024363826}, {12, -0.33638207422522504}, {12,
2.7293006320336812}, {12, -0.7032639687766782}, {12,
-1.0160611554482348}, {12, 2.4445677108866337}, {12,
2.105814609932241}, {12,
3.503155577334455}, {12, -1.3373271419360089}, {12,
-3.086172257731213}, {12, -2.4553685842695274}, {12,
2.967533420351023}, {12, -1.5785023546994714}, {12,
-1.8410492782677599}, {12, -1.512115953146461}, {12,
3.809127502466673}, {12,
1.083447430804604}}, {{13, -2.689819771603731}, {13,
2.0451575429306588}, {13, -2.6116134381230856}, {13,
4.321843024363826}, {13, 3.3059143041695673}, {13,
3.244590972055542}, {13, 2.0051554526468904}, {13,
4.324014831334397}, {13, 3.8618737162157695}, {13,
2.105814609932241}, {13, 3.503155577334455}, {13,
3.6129228725365126}, {13, -1.9952338031778551}, {13,
-2.9220279774640816}, {13,
2.967533420351023}, {13, -1.5785023546994714}, {13,
2.054386639181586}, {13, -3.3267172419576783}, {13,
3.6526309320145995}, {13,
1.083447430804604}}, {{14, -2.689819771603731}, {14,
3.53279137731269}, {14, 2.9854607055686144}, {14,
4.321843024363826}, {14, -1.9650609340102805}, {14,
-2.9166543621389445}, {14, 2.0051554526468904}, {14,
4.818374761324}, {14,
1.5433122680009346}, {14, -3.2731478151496214}, {14,
-2.6945843454021334}, {14, -2.8636015199502536}, {14,
-1.9952338031778551}, {14, 4.376016805575464}, {14,
0.58473667437242}, {14, -2.6101165472097434}, {14,
2.054386639181586}, {14, -3.3267172419576783}, {14,
1.6584676076367435}, {14,
1.083447430804604}}, {{15, -2.689819771603731}, {15,
3.53279137731269}, {15, -2.2870625975701757}, {15,
-4.135995224575007}, {15, -1.9650609340102805}, {15,
1.3217534073659332}, {15, 2.0051554526468904}, {15,
4.101564011944627}, {15,
4.093717188241731}, {15, -3.2731478151496214}, {15,
4.847640930316921}, {15,
1.2874161783320233}, {15, -1.9952338031778551}, {15,
2.9188024605949905}, {15, 2.6040087298447294}, {15,
5.339176136113665}, {15, -3.3103228034511547}, {15,
-4.301808830812817}, {15, -2.746951587052287}, {15,
1.083447430804604}}, {{16, -2.037800118270176}, {16,
-4.751304697154737}, {16, -2.2870625975701757}, {16,
-4.135995224575007}, {16, 3.931661341549311}, {16,
1.3217534073659332}, {16, -1.2029502884368948}, {16,
-3.3104335259499846}, {16, 2.9445811081697375}, {16,
1.6615135519154092}, {16,
4.847640930316921}, {16, -4.321941722394576}, {16,
1.2182270883920108}, {16, -4.291752184788049}, {16,
4.7637032876555905}, {16,
0.30175211922907064}, {16, -3.0785524566345988}, {16,
-5.252385219193551}, {16, -4.569154370815168}, {16,
-3.118539549776655}}, {{17, -2.037800118270176}, {17,
4.33619721417101}, {17, -2.4656831140767244}, {17,
-2.2591563156547902}, {17, 3.931661341549311}, {17,
4.914434688154813}, {17, -3.0981837631411415}, {17,
-1.2697115367525944}, {17, -2.373162078788219}, {17,
-3.1248528123371404}, {17,
4.847640930316921}, {17, -4.39707153759843}, {17,
2.0587372281651035}, {17, -4.291752184788049}, {17,
4.61273317261773}, {17,
4.244308961002415}, {17, -3.0785524566345988}, {17,
-5.252385219193551}, {17, -3.2591716726389803}, {17,
4.145561030957269}}, {{18, -2.037800118270176}, {18,
4.33619721417101}, {18, 3.978450682164329}, {18,
3.9076485751972734}, {18, 3.931661341549311}, {18,
4.914434688154813}, {18, 4.795251574929358}, {18,
5.253588611748619}, {18, 3.922462678282567}, {18,
4.9564257158773035}, {18, -3.620123925372994}, {18,
-2.7362096812830066}, {18, -1.5377787297626027}, {18,
-3.8367031218917154}, {18,
4.61273317261773}, {18, -1.8930669483966014}, {18,
3.3304550080763398}, {18, -0.45980901774225336}, {18,
-3.2591716726389803}, {18,
4.83477986545003}}, {{19, -0.780709357322956}, {19,
4.034849546951172}, {19,
3.978450682164329}, {19, -4.787529629078456}, {19,
4.105457331485535}, {19,
3.1276500533269465}, {19, -2.385634657940179}, {19,
5.248322249519411}, {19, 3.922462678282567}, {19,
4.9564257158773035}, {19, -3.620123925372994}, {19,
-2.8758698931403233}, {19, -3.1713176003674555}, {19,
-3.8367031218917154}, {19,
4.61273317261773}, {19, -1.8930669483966014}, {19,
3.0273337781158753}, {19,
6.251699456818568}, {19, -1.606852464277316}, {19,
3.745469927768456}}};

• Please provide data` and make sure the posted code evaluates. Commented May 14, 2020 at 13:08
• It seems to me that what you are looking for can be (approximately) solved using Quantile Regression and / or Independent Component Analysis. Commented May 14, 2020 at 13:10
• @AntonAntonov I added in a comment with some of the data, and can confirm that the code in the post works as written. The data provided will only go up to time 20, however. Interesting suggestion re: Quantile Regressions, but I'm trying to consider how a continuous model might cope with a branch randomly dying? Commented May 14, 2020 at 17:45
• I was thinking more about it and what I might try is defining some sort of max distance between points to determine if they are in the same cluster and if that distance is exceeded, then it is a new "cluster", for each time step, and if you iterate top-down in the list for a time step, that should force the sort of "space coherence" that I'm looking for in the "clusters". Of course I was hoping to use ClusteringComponent[] so I wouldn't have to manually define this max distance myself. Commented May 14, 2020 at 17:49