How exactly does NetTrain use a ValidationSet (can we reuse it as TestSet)?

Question

Typically when you train a model you have three sets of data: training, validation, and testing - and they can’t be mixed, i.e. you can’t use images from one in the other.

I want to know if the samples that you give NetTrain through the option ValidationSet are used to effect the parameters of the network? Or is it considered a holdout that I can test on later, or is that data burnt after NetTrain completes?

For example, given that testData is already used in NetTrain:

trainingData = ResourceData["MNIST", "TrainingData"];
testData = ResourceData["MNIST", "TestData"];
lenetModel = NetModel["LeNet Trained on MNIST Data"]
n = NetTrain[NetModel["LeNet"], trainingData, ValidationSet -> testData]

Is this an ill-informed thing to do:

ClassifierMeasurements[n, testData]

Answer 1

Yes, you can do ClassifierMeasurements on the testData.

http://reference.wolfram.com/language/tutorial/NeuralNetworksRegularization.html