4
$\begingroup$

I'm using V11.1. I trained a net for the MNIST data using some fraction of the data as a validation set. Everything works fine, but the validation ioss list does not agree with the loss evolution plot.

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];
lenet = NetChain[{ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
                  ConvolutionLayer[50, 5], Ramp, PoolingLayer[2,2],
                  FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
                  "Output" -> NetDecoder[{"Class", Range[0, 9]}],
                  "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]]

{trainedModel, trainLossL, valLossL, lossPlot} = 
   NetTrain[lenet, 
     trainingData, 
     Automatic, 
     {"TrainedNet", "RoundLossList", "ValidationLossList", "LossEvolutionPlot"}, 
     ValidationSet -> Scaled[0.2], 
     BatchSize -> 128, 
     Method -> "SGD", 
     MaxTrainingRounds -> 10,
     TrainingProgressReporting -> {"Print", "Interval" -> Quantity[1, "Rounds"]}];

After this call the returned validation loss list (valLossL) is

{{376, 0.32971}, {751, 0.24226}, {1126, 0.219627}, {1501, 0.193021}, 
 {1876, 0.160327}, {2251, 0.148028}, {2626, 0.12826}, {3001, 0.128028}, 
 {3376, 0.118934}, {3750, 0.106219}}

but the loss evolution plot (lossPlot) looks like the following.

enter image description here

It seems that the curve for the validation data is much too high. Any idea what's going on?

Ps: Using CPU on win10, @partida have the same problem as the OP

My environment is 11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)

I upload the result. here

This is training process:

enter image description here

{trainedModel, trainLossL, valLossL, lossPlot} = Uncompress@Import@"https://wolfr.am/mzIGew77";
valLossL(*right*)
lossPlot (*so weird*)
ClassifierMeasurements[trainedModel, testData]["Accuracy"]
(*0.9905 right*)

enter image description here

If I use panel,LossEvolutionPlot works well.

enter image description here

Only on CPU it has this problem.Is it a bug of LossEvolutionPlot? PS:11.1.1 fix this bugs

$\endgroup$
0

2 Answers 2

1
$\begingroup$

I think the reason is the batch size. reference:What is batch size in neural network?

Because training on CPU spend much time,I run this on GeForce GTX 1080.

And only I use GPU can get the right LossEvolutionPlot

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];
lenet = NetChain[{
ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
 "Output" -> NetDecoder[{"Class", Range[0, 9]}],
 "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]]

batch size:128

{trainedModel, trainLossL, valLossL, lossPlot} = 
  NetTrain[lenet, trainingData, Automatic, {"TrainedNet", "RoundLossList", "ValidationLossList", "LossEvolutionPlot"}, 
    BatchSize -> 128, Method -> "SGD", 
    MaxTrainingRounds -> 10,
    ValidationSet -> Scaled[0.2], 
    TargetDevice -> "GPU",
    TrainingProgressReporting -> "Print"
    (*important when you want to trace all the process and print it*)];

You can also see when the input is Image.Then the batch size is the number of images per round.

And batch number is Floor[Length[trainingData]*(1-0.2)/128]==375(*True*)

enter image description here

If you print trainLossL and valLossL,you will find it has the print exactly the same number as TrainingProgressReporting -> "Print" dose.

trainLossL -> round loss (orange)

valLossL -> test loss(dark green)

enter image description here

enter image description here

batch size:256

enter image description here

batch size:512

enter image description here

batch size:1024

enter image description here

You could find the result is better when batch size become larger.

But if the Batch Size is very very large,then the memory maybe can't hold on.

$\endgroup$
8
  • $\begingroup$ Hi partida, thanks for your answer, but maybe I was not clear enough. My problem is that the values of the returned ValidationLossList do not agree with the returned LossEvolutionPlot. In my evolution plot all validation values are above 10^0, i.e. above 1, while the numerical values are not . Interestingly in your plots everything looks okay. Is it possible to upload a whole notebook, then you could check if something is wrong with my code !? $\endgroup$
    – Alex
    Commented Jun 19, 2017 at 17:18
  • $\begingroup$ @Alex you can copy my code and run it again $\endgroup$
    – partida
    Commented Jun 20, 2017 at 0:41
  • $\begingroup$ @Alex I add more detailed info $\endgroup$
    – partida
    Commented Jun 20, 2017 at 1:32
  • $\begingroup$ You could cancel the acceptance of my answer because I think it's a bug.And it has not be resolved $\endgroup$
    – partida
    Commented Jun 21, 2017 at 5:54
  • $\begingroup$ hi, do you try GTX1080 in Windows or Linux? I've failed in Ubuntu. $\endgroup$ Commented Aug 21, 2017 at 5:03
0
$\begingroup$

I rerun your code on my machine and the output of NetTrain looks similar (only much slower with my CPU) :

enter image description here

But then I have the same problem as before, the validation values in lossPlot don't agree with the numbers in valLossL :-( I'm using Mma 11.1 under Windows 7. What are you using ?

enter image description here

$\endgroup$
4
  • $\begingroup$ I am using linux mma11.1.1 $\endgroup$
    – partida
    Commented Jun 21, 2017 at 0:19
  • $\begingroup$ To check,I use Mma11.1.0 on win10 wow..same problem as you! $\endgroup$
    – partida
    Commented Jun 21, 2017 at 5:26
  • 1
    $\begingroup$ Okay! I upgraded to 11.1.1 and the problem is gone. Many thanks for your help ! $\endgroup$
    – Alex
    Commented Jun 21, 2017 at 13:30
  • $\begingroup$ No problem you’re welcome :) $\endgroup$
    – partida
    Commented Jun 21, 2017 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.