4
$\begingroup$

I'm using V11.1. I trained a net for the MNIST data using some fraction of the data as a validation set. Everything works fine, but the validation ioss list does not agree with the loss evolution plot.

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];
lenet = NetChain[{ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
                  ConvolutionLayer[50, 5], Ramp, PoolingLayer[2,2],
                  FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
                  "Output" -> NetDecoder[{"Class", Range[0, 9]}],
                  "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]]

{trainedModel, trainLossL, valLossL, lossPlot} = 
   NetTrain[lenet, 
     trainingData, 
     Automatic, 
     {"TrainedNet", "RoundLossList", "ValidationLossList", "LossEvolutionPlot"}, 
     ValidationSet -> Scaled[0.2], 
     BatchSize -> 128, 
     Method -> "SGD", 
     MaxTrainingRounds -> 10,
     TrainingProgressReporting -> {"Print", "Interval" -> Quantity[1, "Rounds"]}];

After this call the returned validation loss list (valLossL) is

{{376, 0.32971}, {751, 0.24226}, {1126, 0.219627}, {1501, 0.193021}, 
 {1876, 0.160327}, {2251, 0.148028}, {2626, 0.12826}, {3001, 0.128028}, 
 {3376, 0.118934}, {3750, 0.106219}}

but the loss evolution plot (lossPlot) looks like the following.

enter image description here

It seems that the curve for the validation data is much too high. Any idea what's going on?

Ps: Using CPU on win10, @partida have the same problem as the OP

My environment is 11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)

I upload the result. here

This is training process:

enter image description here

{trainedModel, trainLossL, valLossL, lossPlot} = Uncompress@Import@"https://wolfr.am/mzIGew77";
valLossL(*right*)
lossPlot (*so weird*)
ClassifierMeasurements[trainedModel, testData]["Accuracy"]
(*0.9905 right*)

enter image description here

If I use panel,LossEvolutionPlot works well.

enter image description here

Only on CPU it has this problem.Is it a bug of LossEvolutionPlot? PS:11.1.1 fix this bugs

$\endgroup$
1
$\begingroup$

I think the reason is the batch size. reference:What is batch size in neural network?

Because training on CPU spend much time,I run this on GeForce GTX 1080.

And only I use GPU can get the right LossEvolutionPlot

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];
lenet = NetChain[{
ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
 "Output" -> NetDecoder[{"Class", Range[0, 9]}],
 "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]]

batch size:128

{trainedModel, trainLossL, valLossL, lossPlot} = 
  NetTrain[lenet, trainingData, Automatic, {"TrainedNet", "RoundLossList", "ValidationLossList", "LossEvolutionPlot"}, 
    BatchSize -> 128, Method -> "SGD", 
    MaxTrainingRounds -> 10,
    ValidationSet -> Scaled[0.2], 
    TargetDevice -> "GPU",
    TrainingProgressReporting -> "Print"
    (*important when you want to trace all the process and print it*)];

You can also see when the input is Image.Then the batch size is the number of images per round.

And batch number is Floor[Length[trainingData]*(1-0.2)/128]==375(*True*)

enter image description here

If you print trainLossL and valLossL,you will find it has the print exactly the same number as TrainingProgressReporting -> "Print" dose.

trainLossL -> round loss (orange)

valLossL -> test loss(dark green)

enter image description here

enter image description here

batch size:256

enter image description here

batch size:512

enter image description here

batch size:1024

enter image description here

You could find the result is better when batch size become larger.

But if the Batch Size is very very large,then the memory maybe can't hold on.

$\endgroup$
  • $\begingroup$ Hi partida, thanks for your answer, but maybe I was not clear enough. My problem is that the values of the returned ValidationLossList do not agree with the returned LossEvolutionPlot. In my evolution plot all validation values are above 10^0, i.e. above 1, while the numerical values are not . Interestingly in your plots everything looks okay. Is it possible to upload a whole notebook, then you could check if something is wrong with my code !? $\endgroup$ – Alex Jun 19 '17 at 17:18
  • $\begingroup$ @Alex you can copy my code and run it again $\endgroup$ – partida Jun 20 '17 at 0:41
  • $\begingroup$ @Alex I add more detailed info $\endgroup$ – partida Jun 20 '17 at 1:32
  • $\begingroup$ You could cancel the acceptance of my answer because I think it's a bug.And it has not be resolved $\endgroup$ – partida Jun 21 '17 at 5:54
  • $\begingroup$ hi, do you try GTX1080 in Windows or Linux? I've failed in Ubuntu. $\endgroup$ – HyperGroups Aug 21 '17 at 5:03
0
$\begingroup$

I rerun your code on my machine and the output of NetTrain looks similar (only much slower with my CPU) :

enter image description here

But then I have the same problem as before, the validation values in lossPlot don't agree with the numbers in valLossL :-( I'm using Mma 11.1 under Windows 7. What are you using ?

enter image description here

$\endgroup$
  • $\begingroup$ I am using linux mma11.1.1 $\endgroup$ – partida Jun 21 '17 at 0:19
  • $\begingroup$ To check,I use Mma11.1.0 on win10 wow..same problem as you! $\endgroup$ – partida Jun 21 '17 at 5:26
  • 1
    $\begingroup$ Okay! I upgraded to 11.1.1 and the problem is gone. Many thanks for your help ! $\endgroup$ – Alex Jun 21 '17 at 13:30
  • $\begingroup$ No problem you’re welcome :) $\endgroup$ – partida Jun 21 '17 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.