1
$\begingroup$

This is cross-posted from the forum.

I am trying to just reproduce the loss computed during training a very simple network. In this case the loss is just the standard L2 loss function.

Start with some data to learn a sin function:

trainingData = Table[x -> Sin[x], {x, 0, 2 Pi, 2 Pi/100.0}];

Next, create a very simple network:

chain = NetChain[{10, Tanh, 1}];

Normally, I would train it just like this:

result = NetTrain[chain, trainingData, All]

But to reproduce the loss, I need to know what the batch data was used at each iteration. So let's also save that:

lastBatchIn = {};
lastBatchOut = {};
lastBatchLossList = None;

appendBatch = Block[{},
    (* Save the last two batch data *)

    AppendTo[lastBatchIn, Normal[#BatchData["Input"]]];
    AppendTo[lastBatchOut, Normal[#BatchData["Output"]]];
    If[Length[lastBatchIn] > 2,
     lastBatchIn = lastBatchIn[[-2 ;;]];
     lastBatchOut = lastBatchOut[[-2 ;;]];
     ];

    (* Save the last two losses for these batches *)

    lastBatchLossList = #BatchLossList[[-2 ;;]];
    ] &;

result = NetTrain[chain, trainingData, All, 
  TrainingProgressFunction -> appendBatch]

Here I have create a function to save the #BatchData and the #BatchLossList for the last two examples. The factor 2 comes from the fact that I see it is using 2 batches for each round of training.

Now to the question:

The loss of the last round is reported as this:

Print["Last round loss reported: ", result["RoundLoss"]]
(* Last round loss reported: 6.05732*10^-6 *)

I can reproduce this from the stored loss list for each batch. Since every round contains two batches, I average the two:

Print["Mean round loss recalculated: ", Mean[lastBatchLossList], 
  " from loss of the last 2 batches: ", lastBatchLossList];
(* Mean round loss recalculated: 6.05732*10^-6 from loss of the last 2 batches: {4.61266*10^-6,7.50198*10^-6} *)

Great! Now I want to recompute it by feeding the actual batch data into the network:

trained = result["TrainedNet"];
mb1 = Mean[(trained[lastBatchIn[[1]]] - lastBatchOut[[1]])^2];
mb2 = Mean[(trained[lastBatchIn[[2]]] - lastBatchOut[[2]])^2];
Print["Mean round loss recomputed: ", 0.5*(mb1 + mb2), 
  " from last 2 batches: ", mb1, " ", mb2];
(* Mean round loss recomputed: 5.77591*10^-6 from last 2 batches: 5.35384*10^-6 6.19799*10^-6 *)

It's definitely not the same. It may be close, but I want to figure out how to get the exact same result. How can I reproduce the loss exactly with the trained network?

Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

Interesting question. Looks like a bug: incorrect reported BatchIndices and BatchData.

SeedRandom[0];
X = RandomReal[{-1, 1}, {16, 3}];
Y = RandomReal[{0, 1}, {16, 1}];

net = NetChain[{LinearLayer[10], LinearLayer[1]}, "Input" -> 3] // NetInitialize;

saveInfo[assoc_] := Module[
  {},
  AppendTo[batchIndices, assoc["BatchIndices"]];
  AppendTo[batchLoss, assoc["BatchLoss"]];
  AppendTo[trainedNet, assoc["Net"]];
  ]

report[batchSize_] := Module[
  {},
  
  batchIndices = {};
  batchLoss = {};
  trainedNet = {};
  
  NetTrain[net, X -> Y, MaxTrainingRounds -> 2, 
   BatchSize -> batchSize, LossFunction -> MeanSquaredLossLayer[], 
   TrainingProgressFunction -> {saveInfo, 
     "Interval" -> Quantity[1, "Round"]}];
  
  Print@batchLoss;
  Print@MeanSquaredLossLayer[][<|
     "Input" -> net@X[[batchIndices[[1]]]], 
     "Target" -> Y[[batchIndices[[1]]]]|>];
  Print@MeanSquaredLossLayer[][<|
     "Input" -> trainedNet[[1]]@X[[batchIndices[[2]]]], 
     "Target" -> Y[[batchIndices[[2]]]]|>];
  ]

Size of the data = 16 and batch = 16:

report[16]

{0.484094,0.476693}

0.484094

0.476693

Incorrect here:

report[8]

{0.633787,0.312369}

0.642032

0.316417

$\endgroup$
1
  • $\begingroup$ Thanks for checking it out. Hoping someone from Wolfram will clarify if it indeed is a bug.... $\endgroup$
    – smörkex
    Jan 16, 2021 at 1:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.