# Is the Method -> Automatic option for Classify not deterministic?

I just prepared an (well known) example in machine learning, the weather dependent determination of whether to play golf or not, as shown in this picture (of a corresponding Dataset)

Now I set up this data to be classified as follows (one can copy and paste the data for easy verifying my example):

data =
{
{"outlook", "temperature", "humidity", "wind", "play"},
{"sunny", "hot", "high", "false", "no"},
{"sunny", "hot", "high", "true", "no"},
{"overcast", "hot", "high", "false", "yes"},
{"rainy", "mild", "high", "false", "yes"},
{"rainy", "cold", "normal", "false", "yes"},
{"rainy", "cold", "normal", "true", "no"},
{"overcast", "cold", "normal", "true", "yes"},
{"sunny", "mild", "high", "false", "no"},
{"sunny", "cold", "normal", "false", "yes"},
{"rainy", "mild", "normal", "false", "yes"},
{"sunny", "mild", "normal", "true", "yes"},
{"overcast", "mild", "high", "true", "yes"},
{"overcast", "hot", "normal", "false", "yes"},
{"rainy", "mild", "high", "true", "no"}
};


data = Rest@data;


Get the target

target = Last /@ data;


get the training data

training = Most /@ data


Now do the classification 100 times with automatic choosing of Method:

tab = Table[Classify[training -> target], {100}]


then....

tab = ClassifierInformation[#, Method] & /@ tab; Tally @ tab


And I get the following result (which I do not understand, because I thought the choosing of the Method in Classify is deterministic):

Can anyone give me a hint or explanation?

• I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results. I would be curious if anyone knows more about how Automatic works, though. Mar 2, 2016 at 20:42
• nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself. Mar 2, 2016 at 20:53
• @rcollyer "The only way to get a consistent result is to use SeedRandom " - does this extend to the "RandomForest" method? e.g. {#, SameQ @@ Table[SeedRandom@10; Predict[ExampleData[{"MachineLearning", "BostonHomes"}, "Data"], Method -> #] // Query[1, "Models", 1, 2] // Hash, 2]} & /@ {"LinearRegression", "NearestNeighbors", "NeuralNetwork", "RandomForest", "GaussianProcess"} Jun 17, 2016 at 10:11
• @RonaldMonson presumably, it should. I'll pass along that it doesn't appear to. Jun 17, 2016 at 12:40
• @rcollyer My impression too. Thanks. Jun 17, 2016 at 19:50

$\qquad$I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results.
$\qquad$nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself.