Options for ValidationSet accuracy from NetTrain?

Question

With NetTrain[] if you use the option ValidationSet->Scale[.1] it will run a validation test on 10% of the training data after each epoch. For example,

data = RandomSample@Table[x -> Exp[-x^2] + RandomVariate[NormalDistribution[0, .15]], {x, -3, 3, .2}];
net = NetChain[{150, Tanh, 150, Tanh, 1}, "Input" -> "Scalar", "Output" -> "Scalar"];
trained = NetTrain[net, data, ValidationSet -> Scaled[.1]];

My question is how to get more information on the validation tests performed, i.e. what was the set (indices of examples) and what were the metrics (precision, recall, ...). And are there any (undocumented) suboptions that can control the splits/shuffling choices for each validation test performed?

Follow-up: is there any mechanism for breaking out of the training loop if the validation results start to plateau?

Answer 1

I'll add to our ToDo list the idea of having the validation indices be available as a property of NetTrain.

Answer 2

I think for the ValidationSet -> Scaled[.1] the validation will be sampled with randomly from the training data. If we want to control the sample ourselves, we can split them manually and supply it as ValidationSet -> testData.

We can use the TrainingProgressFunction function to get more information about the validation accuracy, and abort the training after a certain goal is achieved.

For example, this will stop the training process once the validation loss is smaller than 0.1:

resource = ResourceObject["MNIST"];
trainingData = 
  RandomSample[ResourceData[resource, "TrainingData"], 6000];
testData = RandomSample[ResourceData[resource, "TestData"], 1000];
lenet = NetChain[{
    ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
    ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
    FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
   "Output" -> NetDecoder[{"Class", Range[0, 9]}],
   "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]
   ];

loss = {};
trained = 
 NetTrain[lenet, trainingData, ValidationSet -> testData, 
  MaxTrainingRounds -> 20, 
  TrainingProgressFunction -> {AppendTo[loss, #ValidationLoss]; 
     If[#ValidationLoss < 0.1, Abort[]]; &, 
    "Interval" -> Quantity[1, "Rounds"]}]

loss
(* {None, 0.474543, 0.290683, 0.174869, 0.146322, 0.110496, 0.109694, 0.0923467} *)