Accuracy during training?

Question

Is there any way to see a plot of the accuracy over the Validation set for each round instead of the loss graph, during a NetTrain session?

There probably is and I'm just missing it...

Answer 1

The loss is a more general quantity than accuracy since accuracy is only defined for classification problems. However, you can easily make a custom report function that plots the accuracy as it trains.

resource = ResourceObject["MNIST"];

trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];

lenet = NetChain[{
    ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
    ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
    FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
   "Output" -> NetDecoder[{"Class", Range[0, 9]}],
   "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]
   ];

calacc[net_] := Module[{prediction},
  prediction = net[testData[[;; , 1]], TargetDevice -> "GPU"];
  N@Count[prediction - testData[[;; , 2]], 0]/Length[prediction]
  ]

plotSolution[net_] := 
 ListPlot[AppendTo[acc, calacc[net]], Joined -> True, Axes -> False, 
  Frame -> True, Mesh -> All, FrameLabel -> {"Round", "Accuracy"}]

acc = {};
trained = 
 NetTrain[lenet, trainingData, ValidationSet -> testData, 
  MaxTrainingRounds -> 20, TargetDevice -> "GPU", 
  TrainingProgressReporting -> {plotSolution[#Net] &, 
    "Interval" -> Quantity[1, "Rounds"]}]