An example showcasing ValidationSet's utility

Question

Validation sets form a fundamental part of machine learning by minimising overfitting but is there a nice example showcasing this effect when using Predict/Classify?

Answer 1

This is a negative answer in the sense of suggesting where not to look - the built-in datasets. The following evaluation is readily described: (definitions follow)

ValidationEffect[{UCILetterMLData, 1}, {0.5, 0.25, 0.25}, 10]

<|"Frac" -> {1, 20000}, "noVS" -> {0.9624, 0.919216, 0.928214},
"yesVS" -> {0.9624, 0.919216, 0.928214}, "diffs" -> {0., 0}|>

This shows that if you take all 20 000 classifications in the UCILetter dataset and partition into training/validation/test sets according to the proportions {0.5, 0.25, 0.25} then (with random seed 10) the resulting model, noVS, without the validation set (i.e. trained only on the training set) has corresponding accuracies {0.9624, 0.919216, 0.928214} on the training/validation/test sets whereas the model, yesVS, trained with the validation set has corresponding accuracies {0.9624, 0.919216, 0.928214}. These accuracies are identical suggesting the same model is being generated as confirmed by diffs of (0,0) - the proportion and raw number of different outputs when the respective models act on elements of the test set.

Now, changing the partition proportions does seem to result in a setting of ValidationSet having an effect

ValidationEffect[{UCILetterMLData, 1}, {0.8, 0.1, 0.1}, 10]

<|"Frac" -> {1, 20000}, "noVS" -> {0.958563, 0.936532, 0.936532}, "yesVS" -> {0.971688, 0.937531, 0.942029}, "diffs" -> {0.0414793, 83}|>

but not a significant one with both models disagreeing only 0.04 of the time with both having an accuracy of ~0.94 on the test set.

This effect seems to be linked to the dataset's size. Taking 0.1 of the largest built-in dataset, MNISTMLData shows ValidationSet having no impact (even with 7000 datapoints)

ValidationEffect[{MNISTMLData, .1}, {40000/70000, 15000/70000, 15000/
  70000}, 21]

<|"Frac" -> {0.1, 7000}, "noVS" -> {0.9265, 0.897402, 0.897402},
"yesVS" -> {0.9265, 0.897402, 0.897402}, "diffs" -> {0., 0}|>

but even when a ValidationSet does have an effect, the new model is not significantly more accurate (warning - this takes a while, >20 min, to evaluate)

ValidationEffect[{MNISTMLData, 1}, {40000/70000, 15000/70000, 15000/
  70000}, 21]

<|"Frac" -> {1, 70000}, "noVS" -> {0.961025, 0.954403, 0.949337}, "yesVS" -> {0.952075, 0.950137, 0.94367}, "diffs" -> {0.0198653, 298}|>

Note the fraction proportions are from another answer in partly explaining its exhibited anomalies.

This behaviour seems unchanged for other random seeds (unaccountably though, often even with the same random seed negligibly different models are produced?) nor does it seem to change for other inbuilt datasets as run on {BostonHomesMLData,FisherIrisMLData, MNISTMLData,MovieReviewMLData,MushroomMLData,SatelliteMLData,TitanicMLData,UCILetterMLData,WineQualityMLData}.

The upshot seems to be that, at least for the inbuilt datasets, setting ValidationSet either has no effect either in terms of generating new models or else in terms of new models that surpass those generated by default - via the built-in (cross) validation.

Definitions:

oClassifierMeasurements[model_] := 
  Function[set, ClassifierMeasurements[model, set, "Accuracy"]];

oDrop[specs__] := Function[ls, Drop[ls, specs]];

oPartition[specs__] := Function[ls, Partition[ls, specs]];

ProportionalSplit[ls_, split_List /; Total@split == 1] := Let[
   n = Length@ls,
   parts = (split // oDrop@-1 // Accumulate) // (Map[Round[n*#] &]) //
        Prepend@0 // Append@n // oPartition[2, 1],
   ls // Query[Span @@@ parts]];

ProportionalSplit[split_] := 
  Function[ls, ProportionalSplit[ls, split]];

ValidationEffect[{dataSet_, frac_}, split_List, seed_: 10] := Let[
   n = (frac*Length@dataSet // Ceiling),
   {trainingSet, validationSet, testSet} =
    RandomSample[dataSet, n] // ProportionalSplit@split,
   noVS = (RandomSeed@seed; Classify[trainingSet]),
   yesVS = (RandomSeed@seed; 
     Classify[trainingSet, ValidationSet -> validationSet]),
   testKeys = Keys@testSet,
   diffCount = 
    Count[UnsameQ @@@ (Thread[{noVS@testKeys, yesVS@testKeys}]), True],
    <|
    "Frac" -> {frac, n},
    "noVS_Acc" -> ({trainingSet, validationSet, testSet} // 
       Map@oClassifierMeasurements@noVS),
    "yesVS_Acc" -> ({trainingSet, validationSet, testSet} // 
       Map@oClassifierMeasurements@yesVS),
    "diffs" -> {N@diffCount/(Length@testSet), diffCount}
    |>
   ];

ExampleData["MachineLearning"] // Scan[
   Let[symb = ToExpression[#[[2]] <> "MLData"],
     dummy; symb = ExampleData[#, "Data"]] &];