Extract training and test set performance from a neural net

Question

I would like to extract a graph of the history of a neural net's performance, as it is trained. This is so I can compare it with the equivalent from Tensorflow (for some reason an equivalent network in Mathematica is training much faster, and more accurately).

As an example, take the case study on MNIST digit classification from here: https://www.wolfram.com/language/11/neural-networks/digit-classification.html

Download training and testing data,

resource = ResourceObject["MNIST"]; 
trainingData = ResourceData[resource, "TrainingData"]; 
testData = ResourceData[resource, "TestData"];

Build a convolutional neural network,

lenet = NetChain[ {ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2], 
         ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[], 500, 
         Ramp, 10, SoftmaxLayer[]}, 
         "Output" -> NetDecoder[{"Class", Range[0, 9]}], 
         "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}] ]

And train it,

lenet = NetTrain[lenet, trainingData, ValidationSet -> testData,
         MaxTrainingRounds -> 3];

You'll see a little box appear when this is run (on Mathematica 11; don't think it'll run on 10) like the following,

I would like to extract the data used to plot both the orange line (performance on the training set), and the blue line when it appears (yet to appear above but illustrates performance on the test set).

Does anyone know how to do this?

Best,

Ben

Answer 1

You can either use the TrainingProgressFunction as suggested by Edmund, or simply use the property argument in NetTrain. For example, this will return the trained net and the batch loss and validation lost history:

{lenet, bloss, vloss} = 
  NetTrain[lenet, trainingData, 
   Automatic, {"TrainedNet", "BatchLossList", "ValidationLossList"}, 
   ValidationSet -> testData, MaxTrainingRounds -> 3];

batchsize=24;
round=10;

Show[
 ListLogPlot[bloss, DataRange -> {1, batchsize*round}, Joined -> True,
   PlotStyle -> Orange, Axes -> False, Frame -> True, 
  FrameLabel -> {"batch", "loss"}],
 ListLogPlot[vloss, Joined -> True, Mesh -> All]]