9
$\begingroup$

Does anyone have knowledge of the O(n) time & space complexities for various model types supported by Classify[] and Predict[] during training and evaluation?

Here are a list of supported model types:

models = {"LogisticRegression", "Markov", "RandomForest", 
   "SupportVectorMachine", "NearestNeighbors", "NeuralNetwork", 
   "NaiveBayes"};

I tried using BenchmarkPlot[] but it is currently broken:

trainingData = ExampleData[{"MachineLearning", "MNIST"}, "TrainingData"];

Needs["GeneralUtilities`"]; Clear[g, f];
g[n_] := Classify[RandomSample[trainingData, n], Method -> "RandomForest"]
f[g_] := ClassifierInformation[g, "TrainingTime"]

BenchmarkPlot[f, g, {10, 100, 200, 400, 800, 1600, 3200, 6400}, "IncludeFits" -> True]

enter image description here

Update:

Even though BenchmarkPlot doesn't always work, it can on some inputs:

<<GeneralUtilities`
trainingData = ExampleData[{"MachineLearning", "MNIST"}, "TrainingData"];
plotComplexity[m_, trainingData_] := Module[{n,c,time,space,tdata,sdata},
n=Table[100*n,{n,1,50,2}];
c=Classify[RandomSample[trainingData,#],Method->m]&/@n;
time=QuantityMagnitude[ClassifierInformation[#,"TrainingTime"]]&/@c;
space=ByteCount/@c;
tdata=Thread[{n,time}]; sdata=Thread[{n,space}];
Return @ TextGrid @ {{BenchmarkPlot[sdata,"IncludeFits"->True,PlotLabel->m<>" Training Space"],
BenchmarkPlot[tdata,"IncludeFits"->True,PlotLabel->m<>" Training Time"]}}
]

Here's what the above code gives for numeric features:

trainingData = ExampleData[{"MachineLearning", "UCILetter"}, "TrainingData"];
models = {"LogisticRegression", "Markov", "RandomForest", 
   "SupportVectorMachine", "NearestNeighbors", "NeuralNetwork", "NaiveBayes"};
Column[plotComplexity[#, trainingData] & /@ models];

enter image description here

$\endgroup$
  • 1
    $\begingroup$ What is holding you back finding out this yourself? BenchmarkPlot in the GeneralUtilities package may be helpful here. $\endgroup$ – Sjoerd C. de Vries Feb 1 '16 at 18:50
  • $\begingroup$ Good idea, will try tonight... Ive always found complexity hard to get a handle on in mma because there's a lot going on under the hood. I thought maybe a dev would know offhand which libraries mma uses for each type of classifier $\endgroup$ – M.R. Feb 2 '16 at 1:32
  • $\begingroup$ @SjoerdC.deVries seems that BenchmarkPlot doesn't like me, can you see what's going wrong here? $\endgroup$ – M.R. Feb 2 '16 at 4:17
  • $\begingroup$ BenchmarkPlot is broken. $\endgroup$ – Karsten 7. Feb 2 '16 at 6:13
  • $\begingroup$ Yeah, I forgot about that. I actually tested it myself. But of course, what BenchmarkPlot does (timing executions with various amount of data) is not that hard to do yourself. Just a Table and AbsoluteTiming should do the trick. $\endgroup$ – Sjoerd C. de Vries Feb 2 '16 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.