8
$\begingroup$

I just prepared an (well known) example in machine learning, the weather dependent determination of whether to play golf or not, as shown in this picture (of a corresponding Dataset)

golf example

Now I set up this data to be classified as follows (one can copy and paste the data for easy verifying my example):

data = 
  {
    {"outlook", "temperature", "humidity", "wind", "play"}, 
    {"sunny", "hot", "high", "false", "no"}, 
    {"sunny", "hot", "high", "true", "no"}, 
    {"overcast", "hot", "high", "false", "yes"}, 
    {"rainy", "mild", "high", "false", "yes"}, 
    {"rainy", "cold", "normal", "false", "yes"}, 
    {"rainy", "cold", "normal", "true", "no"}, 
    {"overcast", "cold", "normal", "true", "yes"}, 
    {"sunny", "mild", "high", "false", "no"}, 
    {"sunny", "cold", "normal", "false", "yes"}, 
    {"rainy", "mild", "normal", "false", "yes"}, 
    {"sunny", "mild", "normal", "true", "yes"}, 
    {"overcast", "mild", "high", "true", "yes"}, 
    {"overcast", "hot", "normal", "false", "yes"}, 
    {"rainy", "mild", "high", "true", "no"}
  };

Now drop the header:

data = Rest@data;

Get the target

target = Last /@ data;

get the training data

training = Most /@ data

Now do the classification 100 times with automatic choosing of Method:

tab = Table[Classify[training -> target], {100}]

then....

tab = ClassifierInformation[#, Method] & /@ tab; Tally @ tab

And I get the following result (which I do not understand, because I thought the choosing of the Method in Classify is deterministic):

Results of classification

Can anyone give me a hint or explanation?

|improve this question
$\endgroup$
  • 2
    $\begingroup$ I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results. I would be curious if anyone knows more about how Automatic works, though. $\endgroup$ – Niki Estner Mar 2 '16 at 20:42
  • 6
    $\begingroup$ nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself. $\endgroup$ – rcollyer Mar 2 '16 at 20:53
  • $\begingroup$ @rcollyer "The only way to get a consistent result is to use SeedRandom " - does this extend to the "RandomForest" method? e.g. {#, SameQ @@ Table[SeedRandom@10; Predict[ExampleData[{"MachineLearning", "BostonHomes"}, "Data"], Method -> #] // Query[1, "Models", 1, 2] // Hash, 2]} & /@ {"LinearRegression", "NearestNeighbors", "NeuralNetwork", "RandomForest", "GaussianProcess"} $\endgroup$ – Ronald Monson Jun 17 '16 at 10:11
  • $\begingroup$ @RonaldMonson presumably, it should. I'll pass along that it doesn't appear to. $\endgroup$ – rcollyer Jun 17 '16 at 12:40
  • $\begingroup$ @rcollyer My impression too. Thanks. $\endgroup$ – Ronald Monson Jun 17 '16 at 19:50
4
$\begingroup$

It would seem an answer can be given by quoting the two comments.

nikie:

$\qquad $I'm guessing that Automatic actually tries multiple classifiers and chooses the best one, according to some criteria. And since some learners like random forests and neural networks aren't deterministic, that could lead to different results.

rcollyer:

$\qquad $nikie's correct. Classify and Predict will randomly choose a method from a weighted list that is determined by your data. So, if your data changes in some significant way (size, type, etc.), the possible methods change. The only way to get a consistent result is to use SeedRandom which is true of any random process, or to specify the method yourself.

|improve this answer
$\endgroup$
  • $\begingroup$ Thank your for commenting / answering this problem. One comment: choosing by random means in fact that not necessaryly the best classifier is chosen. In the Titanic example in the documentation the logarithmic regression is shown. When one uses all (possible) classifiers it is shown that according to error/accuracy the best one is the NearestNeighbors. others are close but show a clearly wrong classification where the survival rate tends to 100% for increasing age ;-) $\endgroup$ – mgamer Mar 3 '16 at 6:02
  • 1
    $\begingroup$ @mgamer but knowing which method will be the best is virtually impossible without trying all of them which would take forever to do. So, it is a trade-off. Based on your data, there are several methods that might be reasonable, so one of them is chosen. If you don't like it, for some criteria of "don't like", pick another one. $\endgroup$ – rcollyer Mar 3 '16 at 14:37
  • $\begingroup$ @rcollyer: Right! This is exactly my strategy :-) $\endgroup$ – mgamer Mar 3 '16 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.