Get the MNIST digit recognition data set (70,000 hand-drawn digits with classifications):
totalSet = ExampleData[{"MachineLearning", "MNIST"}, "Data"];
Divide it into training set, validation set (used to find optimum values for hyperparameters, such as regularization constants) and test set (which is not used in building the classifier at all, but which will be used for an independent test):
{trainingSet, validationSet, testSet} =
{#[[;; 40000]], #[[40001 ;; 55000]], #[[55001 ;; 70000]]} &[RandomSample[totalSet]];
Do the training:
c = Classify[trainingSet, ValidationSet -> validationSet];
and examine the results for each of the three sets:
ClassifierMeasurements[c, trainingSet, "Accuracy"]
(* 0.9667 *)
ClassifierMeasurements[c, validationSet, "Accuracy"]
(* 0.963867 *)
ClassifierMeasurements[c, testSet, "Accuracy"]
(* 0.962267 *)
As the results of the test set show, the classifier is certainly not overfitted as it reaches almost the same performance as the training set. The validation set has been used to good effect.