# Classify across ten methods

Mathematica supports ten different methods for Classify.

The ten methods are: "ClassDistributions", "DecisionTree", "GradientBoostedTrees", "LogisticRegression", "Markov", "NaiveBayes", "NearestNeighbors", "NeuralNetwork", "RandomForest", "SupportVectorMachine".

Here is an example using the first method.

  resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];

trainingData // Length

c1 = Classify[trainingData, Method -> "ClassDistributions"]

cm1 = ClassifierMeasurements[c1, testData]

cm1["AccuracyRejectionPlot"]


My question

What is the best approach to repeat this for each of the ten methods and take the classifier results and accuracy rejection plots and put them in a grid for each method?

I was doing the same three steps method by method, but that seems clunky.

Next steps are to play around with various other approaches like convolutional neural networks and LeNet variants.

Update Add example of other ClassifierMeasurementsObject properties.

The code will work for any property that is a Graphics e.g. for "ConfusionMatrixPlot"

confusionMatrixPlots = #["ConfusionMatrixPlot"] & /@ classifierMeasurements;

KeyValueMap[Show[#2, PlotLabel -> #1, ImageSize -> 300] &, confusionMatrixPlots] //
Partition[#, UpTo@3] & //
Grid[#, Alignment -> Bottom, Frame -> All] & Map over the available methods

methods = {"ClassDistributions", "DecisionTree", "GradientBoostedTrees",
"LogisticRegression", "Markov", "NaiveBayes", "NearestNeighbors",
"NeuralNetwork", "RandomForest", "SupportVectorMachine"};

(* Only 100 examples and TimeGoal = 5s for testing *)
classifiers =
AssociationMap[
Classify[RandomSample[trainingData, 100], TimeGoal -> 5, Method -> #] &, methods];

classifierMeasurements =
ClassifierMeasurements[#, RandomSample[testData, 100]] & /@ classifiers;

accuracyRejectionPlots = #["AccuracyRejectionPlot"] & /@ classifierMeasurements;

KeyValueMap[Show[#2, PlotLabel -> #1, ImageSize -> 300] &, accuracyRejectionPlots] //
Partition[#, UpTo@3] & //
Grid[#, Alignment -> Bottom, Frame -> All] & • Can you provide the second variant that uses the entire datasets as a second variant, or better yet, let the user choose based on some flag?
– Moo
Jul 13 at 1:25
• I don't have a GPU so training on 60000 images would take a long time for each method so I chose 100. To use a different number change RandomSample[trainingData, 100] to RandomSample[trainingData, n] where n <= Length@trainingData and RandomSample[testData, 100]] to RandomSample[testData, m]] where m <= Length@testData. Also remove TimeGoal -> 5. Jul 13 at 12:59
• n / m should be ~ 6 (ratio between training and testing sample lengths) Jul 13 at 13:09