# Is the Method -> Automatic option for Classify not deterministic?

I just prepared an (well known) example in machine learning, the weather dependent determination of whether to play golf or not, as shown in this picture (of a corresponding Dataset)

Now I set up this data to be classified as follows (one can copy and paste the data for easy verifying my example):

data =
{
{"outlook", "temperature", "humidity", "wind", "play"},
{"sunny", "hot", "high", "false", "no"},
{"sunny", "hot", "high", "true", "no"},
{"overcast", "hot", "high", "false", "yes"},
{"rainy", "mild", "high", "false", "yes"},
{"rainy", "cold", "normal", "false", "yes"},
{"rainy", "cold", "normal", "true", "no"},
{"overcast", "cold", "normal", "true", "yes"},
{"sunny", "mild", "high", "false", "no"},
{"sunny", "cold", "normal", "false", "yes"},
{"rainy", "mild", "normal", "false", "yes"},
{"sunny", "mild", "normal", "true", "yes"},
{"overcast", "mild", "high", "true", "yes"},
{"overcast", "hot", "normal", "false", "yes"},
{"rainy", "mild", "high", "true", "no"}
};


Now drop the header:

data = Rest@data;


Get the target

target = Last /@ data;


get the training data

training = Most /@ data


Now do the classification 100 times with automatic choosing of Method:

tab = Table[Classify[training -> target], {100}]


then....

tab = ClassifierInformation[#, Method] & /@ tab; Tally @ tab


And I get the following result (which I do not understand, because I thought the choosing of the Method in Classify is deterministic):

Can anyone give me a hint or explanation?

• I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results. I would be curious if anyone knows more about how Automatic works, though. Mar 2, 2016 at 20:42
• nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself. Mar 2, 2016 at 20:53
• @rcollyer "The only way to get a consistent result is to use SeedRandom " - does this extend to the "RandomForest" method? e.g. {#, SameQ @@ Table[SeedRandom@10; Predict[ExampleData[{"MachineLearning", "BostonHomes"}, "Data"], Method -> #] // Query[1, "Models", 1, 2] // Hash, 2]} & /@ {"LinearRegression", "NearestNeighbors", "NeuralNetwork", "RandomForest", "GaussianProcess"} Jun 17, 2016 at 10:11
• @RonaldMonson presumably, it should. I'll pass along that it doesn't appear to. Jun 17, 2016 at 12:40
• @rcollyer My impression too. Thanks. Jun 17, 2016 at 19:50

## 1 Answer

It would seem an answer can be given by quoting the two comments.

$\qquad$I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results.

$\qquad$nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself.

• Thank your for commenting / answering this problem. One comment: choosing by random means in fact that not necessaryly the best classifier is chosen. In the Titanic example in the documentation the logarithmic regression is shown. When one uses all (possible) classifiers it is shown that according to error/accuracy the best one is the NearestNeighbors. others are close but show a clearly wrong classification where the survival rate tends to 100% for increasing age ;-) Mar 3, 2016 at 6:02
• @mgamer but knowing which method will be the best is virtually impossible without trying all of them which would take forever to do. So, it is a trade-off. Based on your data, there are several methods that might be reasonable, so one of them is chosen. If you don't like it, for some criteria of "don't like", pick another one. Mar 3, 2016 at 14:37
• @rcollyer: Right! This is exactly my strategy :-) Mar 3, 2016 at 17:22