# Loss evolution plot computed by NetTrain looks wrong

I'm using V11.1. I trained a net for the MNIST data using some fraction of the data as a validation set. Everything works fine, but the validation ioss list does not agree with the loss evolution plot.

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];
lenet = NetChain[{ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
ConvolutionLayer[50, 5], Ramp, PoolingLayer[2,2],
FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
"Output" -> NetDecoder[{"Class", Range[0, 9]}],
"Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]]

{trainedModel, trainLossL, valLossL, lossPlot} =
NetTrain[lenet,
trainingData,
Automatic,
{"TrainedNet", "RoundLossList", "ValidationLossList", "LossEvolutionPlot"},
ValidationSet -> Scaled[0.2],
BatchSize -> 128,
Method -> "SGD",
MaxTrainingRounds -> 10,
TrainingProgressReporting -> {"Print", "Interval" -> Quantity[1, "Rounds"]}];


After this call the returned validation loss list (valLossL) is

{{376, 0.32971}, {751, 0.24226}, {1126, 0.219627}, {1501, 0.193021},
{1876, 0.160327}, {2251, 0.148028}, {2626, 0.12826}, {3001, 0.128028},
{3376, 0.118934}, {3750, 0.106219}}


but the loss evolution plot (lossPlot) looks like the following.

It seems that the curve for the validation data is much too high. Any idea what's going on?

Ps: Using CPU on win10, @partida have the same problem as the OP

My environment is 11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)

This is training process:

{trainedModel, trainLossL, valLossL, lossPlot} = Uncompress@Import@"https://wolfr.am/mzIGew77";
valLossL(*right*)
lossPlot (*so weird*)
ClassifierMeasurements[trainedModel, testData]["Accuracy"]
(*0.9905 right*)


If I use panel,LossEvolutionPlot works well.

Only on CPU it has this problem.Is it a bug of LossEvolutionPlot? PS:11.1.1 fix this bugs

I think the reason is the batch size. reference:What is batch size in neural network?

Because training on CPU spend much time,I run this on GeForce GTX 1080.

And only I use GPU can get the right LossEvolutionPlot

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];
lenet = NetChain[{
ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
"Output" -> NetDecoder[{"Class", Range[0, 9]}],
"Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]]


batch size:128

{trainedModel, trainLossL, valLossL, lossPlot} =
NetTrain[lenet, trainingData, Automatic, {"TrainedNet", "RoundLossList", "ValidationLossList", "LossEvolutionPlot"},
BatchSize -> 128, Method -> "SGD",
MaxTrainingRounds -> 10,
ValidationSet -> Scaled[0.2],
TargetDevice -> "GPU",
TrainingProgressReporting -> "Print"
(*important when you want to trace all the process and print it*)];


You can also see when the input is Image.Then the batch size is the number of images per round.

And batch number is Floor[Length[trainingData]*(1-0.2)/128]==375(*True*)

If you print trainLossL and valLossL,you will find it has the print exactly the same number as TrainingProgressReporting -> "Print" dose.

trainLossL -> round loss (orange)

valLossL -> test loss(dark green)

batch size:256

batch size:512

batch size:1024

You could find the result is better when batch size become larger.

But if the Batch Size is very very large,then the memory maybe can't hold on.

• Hi partida, thanks for your answer, but maybe I was not clear enough. My problem is that the values of the returned ValidationLossList do not agree with the returned LossEvolutionPlot. In my evolution plot all validation values are above 10^0, i.e. above 1, while the numerical values are not . Interestingly in your plots everything looks okay. Is it possible to upload a whole notebook, then you could check if something is wrong with my code !? – Alex Jun 19 '17 at 17:18
• @Alex you can copy my code and run it again – partida Jun 20 '17 at 0:41
• @Alex Ｉ add more detailed info – partida Jun 20 '17 at 1:32
• You could cancel the acceptance of my answer because I think it's a bug.And it has not be resolved – partida Jun 21 '17 at 5:54
• hi, do you try GTX1080 in Windows or Linux? I've failed in Ubuntu. – HyperGroups Aug 21 '17 at 5:03

I rerun your code on my machine and the output of NetTrain looks similar (only much slower with my CPU) :

But then I have the same problem as before, the validation values in lossPlot don't agree with the numbers in valLossL :-( I'm using Mma 11.1 under Windows 7. What are you using ?

• I am using linux mma11.1.1 – partida Jun 21 '17 at 0:19
• To check,I use Mma11.1.0 on win10 wow..same problem as you! – partida Jun 21 '17 at 5:26
• Okay! I upgraded to 11.1.1 and the problem is gone. Many thanks for your help ! – Alex Jun 21 '17 at 13:30
• No problem you’re welcome :) – partida Jun 21 '17 at 13:47