Mathematica supports ten different methods for Classify.

The ten methods are: "ClassDistributions", "DecisionTree", "GradientBoostedTrees", "LogisticRegression", "Markov", "NaiveBayes", "NearestNeighbors", "NeuralNetwork", "RandomForest", "SupportVectorMachine".

Here is an example using the first method.

  resource = ResourceObject["MNIST"];
  trainingData = ResourceData[resource, "TrainingData"];
  testData = ResourceData[resource, "TestData"];

  trainingData // Length

  c1 = Classify[trainingData, Method -> "ClassDistributions"]

  cm1 = ClassifierMeasurements[c1, testData]


My question

What is the best approach to repeat this for each of the ten methods and take the classifier results and accuracy rejection plots and put them in a grid for each method?

I was doing the same three steps method by method, but that seems clunky.

Next steps are to play around with various other approaches like convolutional neural networks and LeNet variants.


Update Add example of other ClassifierMeasurementsObject properties.

The code will work for any property that is a Graphics e.g. for "ConfusionMatrixPlot"

confusionMatrixPlots = #["ConfusionMatrixPlot"] & /@ classifierMeasurements;

KeyValueMap[Show[#2, PlotLabel -> #1, ImageSize -> 300] &, confusionMatrixPlots] // 
Partition[#, UpTo@3] & // 
Grid[#, Alignment -> Bottom, Frame -> All] &

enter image description here

Map over the available methods

methods = {"ClassDistributions", "DecisionTree", "GradientBoostedTrees",
           "LogisticRegression", "Markov", "NaiveBayes", "NearestNeighbors",
           "NeuralNetwork", "RandomForest", "SupportVectorMachine"};

(* Only 100 examples and TimeGoal = 5s for testing *)
classifiers = 
   Classify[RandomSample[trainingData, 100], TimeGoal -> 5, Method -> #] &, methods];

classifierMeasurements = 
 ClassifierMeasurements[#, RandomSample[testData, 100]] & /@ classifiers;

accuracyRejectionPlots = #["AccuracyRejectionPlot"] & /@ classifierMeasurements;

KeyValueMap[Show[#2, PlotLabel -> #1, ImageSize -> 300] &, accuracyRejectionPlots] // 
Partition[#, UpTo@3] & // 
Grid[#, Alignment -> Bottom, Frame -> All] &

enter image description here

  • $\begingroup$ Can you provide the second variant that uses the entire datasets as a second variant, or better yet, let the user choose based on some flag? $\endgroup$
    – Moo
    Jul 13 at 1:25
  • 1
    $\begingroup$ I don't have a GPU so training on 60000 images would take a long time for each method so I chose 100. To use a different number change RandomSample[trainingData, 100] to RandomSample[trainingData, n] where n <= Length@trainingData and RandomSample[testData, 100]] to RandomSample[testData, m]] where m <= Length@testData. Also remove TimeGoal -> 5. $\endgroup$ Jul 13 at 12:59
  • 1
    $\begingroup$ n / m should be ~ 6 (ratio between training and testing sample lengths) $\endgroup$ Jul 13 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.