7
$\begingroup$

When using the classic tennis data (reference here) with Classify

sunny = "Sunny";
overcast = "Overcast";
rainy = "Rainy";
high = "High";
low = "Low";
tennis = {
 {sunny, low, True} -> True,
 {sunny, high, True} -> False,
 {sunny, high, False} -> False,
 {overcast, low, True} -> True,
 {overcast, high, False} -> True,
 {overcast, low, False} -> True,
 {rainy, low, True} -> False,
 {rainy, low, False} -> True
};
cl = Classify[tennis]

(resulting in a Breiman-Cutler ensemble classification) I get the unexpected result

cl[Keys[tennis]]

{True, True, True, True, True, True, True, True}

I would expect that at least the given input is being classified correctly. What is wrong here?

Wolfram Desktop Option

$\endgroup$
  • 2
    $\begingroup$ Perhaps your assumption is incorrect. Should the given input be classified "correct"? In a data fit, should the fitted line go through all original data points? $\endgroup$ – Sjoerd C. de Vries Jul 6 '14 at 21:44
  • $\begingroup$ Decision trees and machine learning is not data fitting. If you feed the same data to other (machine learning) frameworks it will return the correct classification; in Python (Numl), SASS, SQL Server Analysis Services and so on. $\endgroup$ – Francois Vanderseypen Jul 11 '14 at 5:44
  • $\begingroup$ Definitions may vary but I know for a fact that regression (data fitting) is taught in machine learning courses. Take for example Andrew Ng's machine learning course on coursera. "LogisticRegression" is one of the methods that Mathematica's Classify can choose. Perhaps it helps if you choose a specific classifier. $\endgroup$ – Sjoerd C. de Vries Jul 11 '14 at 6:30
  • $\begingroup$ By the way, I am not able to reproduce your result. In my freshly downloaded trial of v10 I get {True, False, False, True, True, True, False, True}. $\endgroup$ – Sjoerd C. de Vries Jul 11 '14 at 6:50
  • 1
    $\begingroup$ I just performed this on the Programming Cloud (don't yet have v10 where I am now). The result is the same as what I got before. But I do see a difference between the output you have now added and mine and that is the method chosen by Mathematica. In my case I see NearestNeighbors, and in your case it is RandomForest. Perhaps you have a different default setting or so? What is your $Version? $\endgroup$ – Sjoerd C. de Vries Jul 11 '14 at 12:38
4
$\begingroup$

There are two distinct issues here:

  1. I would expect that at least the given input is being classified correctly.

As mentioned, machine learning algorithms are not designed to fit the training data, rather they seek to fit unseen test data well. So trained classifiers will not necessarily classify your test data correctly.

  1. The classifier outputs only True.

The fundamental reason for this are class priors: when predictions are made by Classify, the prior probabilities of each class occurring are taken into account. By default, these probabilities are estimated from the frequencies in which the classes appear in the training data. In the tennis example, there are 5 instances of True, and 3 of False, so predictions are weighted in favour of True. As a concrete example (actual class False):

In:= cl[{sunny, high, True}, "Probabilities"]
Out= <|False -> 0.486438, True -> 0.513562|>

True is predicted. Now specify that both classes are a priori equally likely using ClassPriors:

In:= cl[{sunny, high, True}, "Probabilities", ClassPriors -> <|True -> 0.5, False -> 0.5|>]
Out= <|False -> 0.586909, True -> 0.413091|>

False is now predicted!

The lesson: if the Classifier is not sure of which class to assign, class priors can have a large impact on the predicted class.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.