I am using Classify[] function with Support vector machine model. I need to analyze the quality of results by analyzing the training error and the validation error to avoid the over-fitting and under-fitting problem. I tried the ClassifierInformation[] function to these errors but It seems not exist. Does anyone tell me how to get these error.
$\begingroup$
$\endgroup$
8
2 Answers
$\begingroup$
$\endgroup$
9
More observations about Sjoerd C. de Vries' answer but too long for comments.
My own re-runs produce similar accuracy - at least to the second decimal place (with variations due to the initial sample randomisation).
c = Classify[trainingSet];
ClassifierMeasurements[c, trainingSet, "Accuracy"]
ClassifierMeasurements[c, validationSet, "Accuracy"]
ClassifierMeasurements[c, testSet, "Accuracy"]
(* 0.9666 *)
(* 0.961467 *)
(* 0.9604 *)
Note that no ValidationSet option is specified since it apparently has no effect on this data source (i.e. with the same training set - bug? or yet to implement as of V10.3.1?)
Note also that the method automatically chosen is "NearestNeighbors" and not "SupportVectorMachine" as requested by the OP.
Query[1, "Models", 1, "Method"]@c (* ClassifierInformation[c,"Method"] is buggy *)
(* "NearestNeighbors" *)
Specifying this gives a marginal improvement in accuracy on testSet
svm = Classify[trainingSet, Method -> "SupportVectorMachine"]
ClassifierMeasurements[svm, trainingSet, "Accuracy"]
ClassifierMeasurements[svm, validationSet, "Accuracy"]
ClassifierMeasurements[svm, testSet, "Accuracy"]
(* 0.999675 *)
(* 0.977467 *)
(* 0.9766 *)
Given that ValidationSet seems to have no effect, might as well use it as part of the trainingSet together with selecting "SupportVectorMachine" as the method.
biggerTraining = Join[trainingSet, validationSet];
svmBigger = Classify[biggerTraining, Method -> "SupportVectorMachine"];
ClassifierMeasurements[svmBigger, testSet, "Accuracy"]
(* 0.979467 *)
-
$\begingroup$ The machine learning package of mathematica is full of bugs and errors. I am using the Sklearn on python which is a very popular lib in machine learning and the results are more reasonable. $\endgroup$ Commented Feb 16, 2016 at 15:32
-
$\begingroup$ @Developer2000 I agree that there the ML is still rapidly maturing but my sense is that opportunity to integrate this capability with curated data and Mathematica's programming environment will eventually differentiate this implementation from others. For this to happen though and IMO a bigger issue is the current "black box" nature of the whole ML set-up; while useful for standard situations, more "glass box" insights would allow you to do so much more ... $\endgroup$ Commented Feb 17, 2016 at 2:34
-
$\begingroup$ I have used Mathematica since 2012 and I am agree with you that is very interesting software.However, it has is a lot of problems as other new softwares. For example, the errors message is misunderstood, documentation is not well written. For the package of machine learning, it is very closed and i think it is still not ready to be public. $\endgroup$ Commented Feb 17, 2016 at 2:45
-
$\begingroup$ The reason that ValidationSet hasn't got a noticeable effect here is that its default setting is Automatic which causes the part of the training set to be used for validation. Since the training set is huge this doesn't impact the final performance, but usually (with less data) if you have a separate training set and validation set this should improve out-of-sample testing compared to training without validation set or with a very small validation set. So, what you see is quite probably not a bug. $\endgroup$ Commented Feb 17, 2016 at 8:53
-
$\begingroup$ @SjoerdC.deVries, I have received the confirmation from the technical support that is a bug. $\endgroup$ Commented Feb 17, 2016 at 11:41
$\begingroup$
$\endgroup$
3
Get the MNIST digit recognition data set (70,000 hand-drawn digits with classifications):
totalSet = ExampleData[{"MachineLearning", "MNIST"}, "Data"];
Divide it into training set, validation set (used to find optimum values for hyperparameters, such as regularization constants) and test set (which is not used in building the classifier at all, but which will be used for an independent test):
{trainingSet, validationSet, testSet} =
{#[[;; 40000]], #[[40001 ;; 55000]], #[[55001 ;; 70000]]} &[RandomSample[totalSet]];
Do the training:
c = Classify[trainingSet, ValidationSet -> validationSet];
and examine the results for each of the three sets:
ClassifierMeasurements[c, trainingSet, "Accuracy"]
(* 0.9667 *)
ClassifierMeasurements[c, validationSet, "Accuracy"]
(* 0.963867 *)
ClassifierMeasurements[c, testSet, "Accuracy"]
(* 0.962267 *)
As the results of the test set show, the classifier is certainly not overfitted as it reaches almost the same performance as the training set. The validation set has been used to good effect.
answered Feb 9, 2016 at 21:34
-
$\begingroup$ "The validation set has been used to good effect" - is that the case here? I find it has no effect $\endgroup$ Commented Feb 17, 2016 at 2:52
-
1$\begingroup$ @ronald You often see that a trained classifier has a lower performance on an out-of-sample test. This is often due to overfitting. In this case performance of the out-of-sample fraction is equal to the training sample, which is quite good and which I feel is due to the presence of the validation sample. $\endgroup$ Commented Feb 17, 2016 at 8:39
-
$\begingroup$ I agree the improved performance is due to some validation but whether or not it is due to the validation specified in the option or else due to the inbuilt cross-validation still seems questionable to me - see, for example, these runs. $\endgroup$ Commented Jun 9, 2016 at 10:10
ClassifierMeasurementsand see if it fits your needs. $\endgroup$Classifydoc page and you'll see that testing the 3rd training example will get you an incorrect answer. $\endgroup$