# Does mathematica have support for mixture models?

Does Mathematica have support mixture models? Similar to the R language libraries mclust or mixtools. The documentation for mixtools is quite good.

The closest support I can find in Mathematica is the LearnDistribution[] function as documented here. I can get a PDF out of this, but I cant figure out how to get parameter estimates out, or to define the number of distributions it should determine, or define any start conditions.

• Nov 13 '20 at 15:37
• This is all in the documention. Use Method->"GaussianMixture" and the sub-options here. For example: iris = ExampleData[{"MachineLearning", "FisherIris"}, "Data"][[All, 1, {1, 3}]]; ld = LearnDistribution[iris, Method -> {"GaussianMixture", "ComponentsNumber" -> 3}]. All the information about the parameters is in ld[] if you need it - that's poking into the internals of Mathematica so this part is not documented. Nov 13 '20 at 15:38
• Well, goodness. How embarrassing. Thanks @flinty
– Q.P.
Nov 13 '20 at 15:40
• @Q.P. it is a little more complicated than that in fact - I will write up an answer. Nov 13 '20 at 15:49
• @flinty Yes I can see, I just ran some code. It's also quite different to some of the methods I've seen in the R libraries mclust and mixtools.
– Q.P.
Nov 13 '20 at 16:16

Mathematica often pre-Standardize's data before learning. For example, see my answer here about learning SVM hyperplanes where I also encountered this problem. Therefore things are a bit more involved than my comment about the documentation if you want to extract the right parameters. I hope you can follow along:

(* some sample data, learn a 3 component mixture *)
iris = ExampleData[{"MachineLearning", "FisherIris"}, "Data"][[All,1,{1,3}]];
ld = LearnDistribution[iris,
Method -> {"GaussianMixture", "ComponentsNumber" -> 3,
"CovarianceType" -> "Full"}];

(* extract the parameters from the internals *)
model = ld[]["Model"];
{weights, means, choleskys} =
model /@ {"MixingCoefficients", "Means", "CholeskyCovariances"};

(* turn the cholesky covariances into +ve def symm matrices with U.U^T*)
covariances = #.ConjugateTranspose[#] & /@ choleskys;

(* create the mixture *)
mixture = MixtureDistribution[weights, distributions];

(* we first plot against the standardized points. Mathematica standardizes
all data before learning the parameters *)
pdf = PDF[mixture, {x, y}];
DensityPlot[pdf, {x, -2, 3}, {y, -2, 3},
Epilog -> {Red, Point[Standardize[iris]]}, PlotPoints -> 25] In the above, we extracted the weights, means, and cholesky covariances, and we formed the symmetric covariances using $$U.U^\top$$ on each. Unfortunately these parameters are have been fit to Standardize[iris], not the original points iris. Therefore we need to apply the appropriate linear transformations to get parameters that fit the original data.

(* we need to unstandardize the parameters *)
sd = DiagonalMatrix@StandardDeviation[iris];
mu = Mean[iris];
finalmeans = (sd.# + mu) & /@ means;
finalcholeskys = (sd.#) & /@ choleskys;
finalcovariances = #.ConjugateTranspose[#] & /@ finalcholeskys;

(* create the un-standardized mixture and finally we can plot against
the original points *)
finalmixture = MixtureDistribution[weights, finaldistributions];
finalpdf = PDF[finalmixture, {x, y}];

(* Plot it. The ^0.125 is for better scaling and visualization only *)
DensityPlot[finalpdf^0.125, {x, 0, 12}, {y, 0, 12},
Epilog -> {Red, Point[iris]}, PlotPoints -> 50] It looks mostly correct by eye. I'm not sure if my transformations of the Cholesky covariances are correct. Somebody with more stats knowledge might want to verify this is the right way to un-standardize a covariance matrix.

• I'm going to accept your answer as it answers the question. But I don't think this is the solution I am looking for, the R libraries Implement maximum-likelihood approaches, and I must confess to having no idea how LearnDistribution works!
– Q.P.
Nov 13 '20 at 17:34
• @Q.P. thanks. Please leave the question up though as this standardization stuff is useful for the community to know. The max likelihood approach is best done in Mathematica with FindDistributionParameters[..., ParameterEstimator -> "MaximumLikelihood"] (the "MaximumLikelihood" is the default anyway). See my other answer: mathematica.stackexchange.com/a/234210/72682 Nov 13 '20 at 18:07
• @Q.P. by the way, from looking at the ld[] I can see it produces this key: "LossName" -> "MeanCrossEntropy". See here and here . Also this appears: "EMIterations" -> 1 so it's doing expectation maximization. Nov 13 '20 at 18:13
• That's a very good spot!! Can you see how to get a likelihood number out as a measure of how good the model numbers are?
– Q.P.
Nov 13 '20 at 18:35