12
$\begingroup$

Mathematica has undocumented functionality of combination classifiers / predictors in ensemble. How to get access to this functionality?

SeedRandom[1];
X = Table[
   Join[
    ToString /@ RandomReal[{-1, 1}, 1],
    RandomReal[{-1, 1}, 4],
    RandomSample[{"A", "B", "C"}, 2]
    ],
   {1000}
   ];
Y = RandomInteger[{0, 1}, 1000];
c = Classify[X -> Y]

enter image description here

ClassifierInformation[c]

enter image description here

ClassifierMeasurements[c, X -> Y, "ConfusionMatrixPlot"]

enter image description here

$\endgroup$
  • $\begingroup$ How is this not a duplicate of the question I posted? $\endgroup$ – Abel Brown Dec 11 '16 at 18:13
8
$\begingroup$

The question formulation asks for access to the internals of Classify in order to get the ensembles, but there is way to make ensembles of classifiers (i.e. ClassifierFunction[___] functions) through the argument "Probabilities" -- see the (short) package ClassifierEnsembles.m.

Very detailed explanations for using and evaluating classifier ensembles made with that package are given in:

Here is an image of ROC curves for comparing the performance of a classifier ensemble with its individual classifiers:

$\endgroup$
  • 1
    $\begingroup$ Very nice, that. $\endgroup$ – Daniel Lichtblau Jan 12 '17 at 0:08
  • $\begingroup$ @DanielLichtblau Thank you, good to hear. $\endgroup$ – Anton Antonov Jan 12 '17 at 3:36
  • $\begingroup$ Anton, good answer! +1 here and in the Wolfram Community. But I can not accept your answer because, as you have noted, my question is about the internals of Classify. With Spelunking I have read Classify code but didn't understand how to call Combiner-function directly. $\endgroup$ – Alexey Golyshev Jan 12 '17 at 6:27
  • $\begingroup$ @AlexeyGolyshev Thank you for your note! I understand your perspective. I posted this answer because with that package I am using the internals of Classify indirectly through "Probabilities". $\endgroup$ – Anton Antonov Jan 12 '17 at 11:26
  • 1
    $\begingroup$ I have found solution! $\endgroup$ – Alexey Golyshev Jan 12 '17 at 15:03
8
$\begingroup$
c1 = Classify[X -> Y, Method -> "LogisticRegression"]
c2 = Classify[X -> Y, Method -> "NearestNeighbors"]

c = MachineLearning`PackageScope`CombinePredictors[{c1, c2}]

ClassifierInformation[c]

enter image description here

p1 = Predict[X -> Y, Method -> "LinearRegression"]
p2 = Predict[X -> Y, Method -> "NearestNeighbors"]

p = MachineLearning`PackageScope`CombinePredictors[{p1, p2}]

PredictorInformation[p]

enter image description here

$\endgroup$
  • $\begingroup$ Great (+1) -- do you think you can find how to do a weighted combination of the classifiers? $\endgroup$ – Anton Antonov Jan 12 '17 at 15:05
  • $\begingroup$ @AntonAntonov I am not sure. Right now Combiner can be only 'Geometric". GeneralUtilities`PrintDefinitions@MachineLearning`PackageScope`CombinePredictors $\endgroup$ – Alexey Golyshev Jan 12 '17 at 15:11
  • $\begingroup$ What about 11.3? It doesn't seem to work anymore. $\endgroup$ – swish Mar 22 '18 at 20:37
  • 1
    $\begingroup$ @swish Yes, it stopped work in V11.2. Function was renamed to Combiner. But it doesn't work the old way. During exploration I found that Classify has the new method Combination. But I don't know what is wrong: Classify[X -> Y, Method -> {"Combination", "Methods" -> {"LogisticRegression", "NearestNeighbors"}, "CombinerMethod" -> "Geometric"}, "DirectTraining" -> True] See GeneralUtilities`PrintDefinitions@MachineLearning`PackageScope`TrainPredictor $\endgroup$ – Alexey Golyshev Mar 23 '18 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.