Mathematica has undocumented functionality of combination classifiers / predictors in ensemble. How to get access to this functionality?

X = Table[
    ToString /@ RandomReal[{-1, 1}, 1],
    RandomReal[{-1, 1}, 4],
    RandomSample[{"A", "B", "C"}, 2]
Y = RandomInteger[{0, 1}, 1000];
c = Classify[X -> Y]

enter image description here


enter image description here

ClassifierMeasurements[c, X -> Y, "ConfusionMatrixPlot"]

enter image description here

  • $\begingroup$ How is this not a duplicate of the question I posted? $\endgroup$
    – Abel Brown
    Dec 11, 2016 at 18:13

2 Answers 2

c1 = Classify[X -> Y, Method -> "LogisticRegression"]
c2 = Classify[X -> Y, Method -> "NearestNeighbors"]

c = MachineLearning`PackageScope`CombinePredictors[{c1, c2}]


enter image description here

p1 = Predict[X -> Y, Method -> "LinearRegression"]
p2 = Predict[X -> Y, Method -> "NearestNeighbors"]

p = MachineLearning`PackageScope`CombinePredictors[{p1, p2}]


enter image description here

  • 1
    $\begingroup$ Great (+1) -- do you think you can find how to do a weighted combination of the classifiers? $\endgroup$ Jan 12, 2017 at 15:05
  • $\begingroup$ @AntonAntonov I am not sure. Right now Combiner can be only 'Geometric". GeneralUtilities`PrintDefinitions@MachineLearning`PackageScope`CombinePredictors $\endgroup$ Jan 12, 2017 at 15:11
  • $\begingroup$ What about 11.3? It doesn't seem to work anymore. $\endgroup$
    – swish
    Mar 22, 2018 at 20:37
  • 1
    $\begingroup$ @swish Yes, it stopped work in V11.2. Function was renamed to Combiner. But it doesn't work the old way. During exploration I found that Classify has the new method Combination. But I don't know what is wrong: Classify[X -> Y, Method -> {"Combination", "Methods" -> {"LogisticRegression", "NearestNeighbors"}, "CombinerMethod" -> "Geometric"}, "DirectTraining" -> True] See GeneralUtilities`PrintDefinitions@MachineLearning`PackageScope`TrainPredictor $\endgroup$ Mar 23, 2018 at 4:12
  • $\begingroup$ How to combine two trained models of Classify in 13.1 or 13.2? $\endgroup$ Dec 20, 2022 at 9:35

The question formulation asks for access to the internals of Classify in order to get the ensembles, but there is way to make ensembles of classifiers (i.e. ClassifierFunction[___] functions) through the argument "Probabilities" -- see the (short) package ClassifierEnsembles.m.

Very detailed explanations for using and evaluating classifier ensembles made with that package are given in:

Here is an image of ROC curves for comparing the performance of a classifier ensemble with its individual classifiers:

  • 1
    $\begingroup$ Very nice, that. $\endgroup$ Jan 12, 2017 at 0:08
  • $\begingroup$ @DanielLichtblau Thank you, good to hear. $\endgroup$ Jan 12, 2017 at 3:36
  • $\begingroup$ Anton, good answer! +1 here and in the Wolfram Community. But I can not accept your answer because, as you have noted, my question is about the internals of Classify. With Spelunking I have read Classify code but didn't understand how to call Combiner-function directly. $\endgroup$ Jan 12, 2017 at 6:27
  • $\begingroup$ @AlexeyGolyshev Thank you for your note! I understand your perspective. I posted this answer because with that package I am using the internals of Classify indirectly through "Probabilities". $\endgroup$ Jan 12, 2017 at 11:26
  • 1
    $\begingroup$ I have found solution! $\endgroup$ Jan 12, 2017 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.