# Is the Method -> Automatic option for Classify not deterministic?

I just prepared an (well known) example in machine learning, the weather dependent determination of whether to play golf or not, as shown in this picture (of a corresponding Dataset)

Now I set up this data to be classified as follows (one can copy and paste the data for easy verifying my example):

data =
{
{"outlook", "temperature", "humidity", "wind", "play"},
{"sunny", "hot", "high", "false", "no"},
{"sunny", "hot", "high", "true", "no"},
{"overcast", "hot", "high", "false", "yes"},
{"rainy", "mild", "high", "false", "yes"},
{"rainy", "cold", "normal", "false", "yes"},
{"rainy", "cold", "normal", "true", "no"},
{"overcast", "cold", "normal", "true", "yes"},
{"sunny", "mild", "high", "false", "no"},
{"sunny", "cold", "normal", "false", "yes"},
{"rainy", "mild", "normal", "false", "yes"},
{"sunny", "mild", "normal", "true", "yes"},
{"overcast", "mild", "high", "true", "yes"},
{"overcast", "hot", "normal", "false", "yes"},
{"rainy", "mild", "high", "true", "no"}
};


data = Rest@data;


Get the target

target = Last /@ data;


get the training data

training = Most /@ data


Now do the classification 100 times with automatic choosing of Method:

tab = Table[Classify[training -> target], {100}]


then....

tab = ClassifierInformation[#, Method] & /@ tab; Tally @ tab


And I get the following result (which I do not understand, because I thought the choosing of the Method in Classify is deterministic):

Can anyone give me a hint or explanation?

• I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results. I would be curious if anyone knows more about how Automatic works, though. – Niki Estner Mar 2 '16 at 20:42
• nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself. – rcollyer Mar 2 '16 at 20:53
• @rcollyer "The only way to get a consistent result is to use SeedRandom " - does this extend to the "RandomForest" method? e.g. {#, SameQ @@ Table[SeedRandom@10; Predict[ExampleData[{"MachineLearning", "BostonHomes"}, "Data"], Method -> #] // Query[1, "Models", 1, 2] // Hash, 2]} & /@ {"LinearRegression", "NearestNeighbors", "NeuralNetwork", "RandomForest", "GaussianProcess"} – Ronald Monson Jun 17 '16 at 10:11
• @RonaldMonson presumably, it should. I'll pass along that it doesn't appear to. – rcollyer Jun 17 '16 at 12:40
• @rcollyer My impression too. Thanks. – Ronald Monson Jun 17 '16 at 19:50

$\qquad$I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results.
$\qquad$nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself.