3
$\begingroup$

I am using Classify[] function with Support vector machine model. I need to analyze the quality of results by analyzing the training error and the validation error to avoid the over-fitting and under-fitting problem. I tried the ClassifierInformation[] function to these errors but It seems not exist. Does anyone tell me how to get these error.

$\endgroup$
  • $\begingroup$ Look at ClassifierMeasurements and see if it fits your needs. $\endgroup$ – rcollyer Feb 9 '16 at 17:01
  • $\begingroup$ @rcollyer, Normally ClassifierMeasurments[] is designed for testing the model. When we test the model on training set, it is normal to get Accuracy 1. $\endgroup$ – BetterEnglish Feb 9 '16 at 17:05
  • $\begingroup$ Do you have a validation set? If so, why don't you use that with ClassifierMeasurements[] instead of the training set? Have a look at the documentation for ClassifierMeasurements. The first example shows how to do this. $\endgroup$ – Sebastian Feb 9 '16 at 19:39
  • $\begingroup$ @Sebastian, The validation set should be used in the Classify[] function as a parameter to avoid the overfitting problem. $\endgroup$ – BetterEnglish Feb 9 '16 at 20:11
  • $\begingroup$ @Developer2000 Your statement "When we test the model on training set, it is normal to get Accuracy 1" is not actually true. It's the same as saying that a linear fit through noisy data will go through every data point. Try the first example in the Application section of the Classify doc page and you'll see that testing the 3rd training example will get you an incorrect answer. $\endgroup$ – Sjoerd C. de Vries Feb 9 '16 at 20:59
-1
$\begingroup$

More observations about Sjoerd C. de Vries' answer but too long for comments.

My own re-runs produce similar accuracy - at least to the second decimal place (with variations due to the initial sample randomisation).

c = Classify[trainingSet];

ClassifierMeasurements[c, trainingSet, "Accuracy"]
ClassifierMeasurements[c, validationSet, "Accuracy"]
ClassifierMeasurements[c, testSet, "Accuracy"]

(* 0.9666 *)
(* 0.961467 *)
(* 0.9604 *)

Note that no ValidationSet option is specified since it apparently has no effect on this data source (i.e. with the same training set - bug? or yet to implement as of V10.3.1?)

Note also that the method automatically chosen is "NearestNeighbors" and not "SupportVectorMachine" as requested by the OP.

Query[1, "Models", 1, "Method"]@c  (* ClassifierInformation[c,"Method"] is buggy  *)

(* "NearestNeighbors" *)

Specifying this gives a marginal improvement in accuracy on testSet

svm = Classify[trainingSet, Method -> "SupportVectorMachine"]

ClassifierMeasurements[svm, trainingSet, "Accuracy"]
ClassifierMeasurements[svm, validationSet, "Accuracy"]
ClassifierMeasurements[svm, testSet, "Accuracy"]

(* 0.999675 *)
(* 0.977467 *)
(* 0.9766 *)

Given that ValidationSet seems to have no effect, might as well use it as part of the trainingSet together with selecting "SupportVectorMachine" as the method.

biggerTraining = Join[trainingSet, validationSet];
svmBigger = Classify[biggerTraining, Method -> "SupportVectorMachine"];
ClassifierMeasurements[svmBigger, testSet, "Accuracy"]

(* 0.979467 *)
$\endgroup$
  • $\begingroup$ The machine learning package of mathematica is full of bugs and errors. I am using the Sklearn on python which is a very popular lib in machine learning and the results are more reasonable. $\endgroup$ – BetterEnglish Feb 16 '16 at 15:32
  • $\begingroup$ @Developer2000 I agree that there the ML is still rapidly maturing but my sense is that opportunity to integrate this capability with curated data and Mathematica's programming environment will eventually differentiate this implementation from others. For this to happen though and IMO a bigger issue is the current "black box" nature of the whole ML set-up; while useful for standard situations, more "glass box" insights would allow you to do so much more ... $\endgroup$ – Ronald Monson Feb 17 '16 at 2:34
  • $\begingroup$ I have used Mathematica since 2012 and I am agree with you that is very interesting software.However, it has is a lot of problems as other new softwares. For example, the errors message is misunderstood, documentation is not well written. For the package of machine learning, it is very closed and i think it is still not ready to be public. $\endgroup$ – BetterEnglish Feb 17 '16 at 2:45
  • $\begingroup$ The reason that ValidationSet hasn't got a noticeable effect here is that its default setting is Automatic which causes the part of the training set to be used for validation. Since the training set is huge this doesn't impact the final performance, but usually (with less data) if you have a separate training set and validation set this should improve out-of-sample testing compared to training without validation set or with a very small validation set. So, what you see is quite probably not a bug. $\endgroup$ – Sjoerd C. de Vries Feb 17 '16 at 8:53
  • $\begingroup$ @SjoerdC.deVries, I have received the confirmation from the technical support that is a bug. $\endgroup$ – BetterEnglish Feb 17 '16 at 11:41
4
$\begingroup$

Get the MNIST digit recognition data set (70,000 hand-drawn digits with classifications):

totalSet = ExampleData[{"MachineLearning", "MNIST"}, "Data"];

Divide it into training set, validation set (used to find optimum values for hyperparameters, such as regularization constants) and test set (which is not used in building the classifier at all, but which will be used for an independent test):

{trainingSet, validationSet, testSet} = 
    {#[[;; 40000]], #[[40001 ;; 55000]], #[[55001 ;; 70000]]} &[RandomSample[totalSet]];

Do the training:

c = Classify[trainingSet, ValidationSet -> validationSet];

and examine the results for each of the three sets:

ClassifierMeasurements[c, trainingSet, "Accuracy"]
(* 0.9667 *)

ClassifierMeasurements[c, validationSet, "Accuracy"]
(* 0.963867 *)

ClassifierMeasurements[c, testSet, "Accuracy"]
(* 0.962267 *)

As the results of the test set show, the classifier is certainly not overfitted as it reaches almost the same performance as the training set. The validation set has been used to good effect.

$\endgroup$
  • $\begingroup$ "The validation set has been used to good effect" - is that the case here? I find it has no effect $\endgroup$ – Ronald Monson Feb 17 '16 at 2:52
  • 1
    $\begingroup$ @ronald You often see that a trained classifier has a lower performance on an out-of-sample test. This is often due to overfitting. In this case performance of the out-of-sample fraction is equal to the training sample, which is quite good and which I feel is due to the presence of the validation sample. $\endgroup$ – Sjoerd C. de Vries Feb 17 '16 at 8:39
  • $\begingroup$ I agree the improved performance is due to some validation but whether or not it is due to the validation specified in the option or else due to the inbuilt cross-validation still seems questionable to me - see, for example, these runs. $\endgroup$ – Ronald Monson Jun 9 '16 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.