# How is the loss computed from the batch and round?

This is cross-posted from the forum.

I am trying to just reproduce the loss computed during training a very simple network. In this case the loss is just the standard L2 loss function.

trainingData = Table[x -> Sin[x], {x, 0, 2 Pi, 2 Pi/100.0}];


Next, create a very simple network:

chain = NetChain[{10, Tanh, 1}];


Normally, I would train it just like this:

result = NetTrain[chain, trainingData, All]


But to reproduce the loss, I need to know what the batch data was used at each iteration. So let's also save that:

lastBatchIn = {};
lastBatchOut = {};
lastBatchLossList = None;

appendBatch = Block[{},
(* Save the last two batch data *)

AppendTo[lastBatchIn, Normal[#BatchData["Input"]]];
AppendTo[lastBatchOut, Normal[#BatchData["Output"]]];
If[Length[lastBatchIn] > 2,
lastBatchIn = lastBatchIn[[-2 ;;]];
lastBatchOut = lastBatchOut[[-2 ;;]];
];

(* Save the last two losses for these batches *)

lastBatchLossList = #BatchLossList[[-2 ;;]];
] &;

result = NetTrain[chain, trainingData, All,
TrainingProgressFunction -> appendBatch]


Here I have create a function to save the #BatchData and the #BatchLossList for the last two examples. The factor 2 comes from the fact that I see it is using 2 batches for each round of training.

Now to the question:

The loss of the last round is reported as this:

Print["Last round loss reported: ", result["RoundLoss"]]
(* Last round loss reported: 6.05732*10^-6 *)


I can reproduce this from the stored loss list for each batch. Since every round contains two batches, I average the two:

Print["Mean round loss recalculated: ", Mean[lastBatchLossList],
" from loss of the last 2 batches: ", lastBatchLossList];
(* Mean round loss recalculated: 6.05732*10^-6 from loss of the last 2 batches: {4.61266*10^-6,7.50198*10^-6} *)


Great! Now I want to recompute it by feeding the actual batch data into the network:

trained = result["TrainedNet"];
mb1 = Mean[(trained[lastBatchIn[]] - lastBatchOut[])^2];
mb2 = Mean[(trained[lastBatchIn[]] - lastBatchOut[])^2];
Print["Mean round loss recomputed: ", 0.5*(mb1 + mb2),
" from last 2 batches: ", mb1, " ", mb2];
(* Mean round loss recomputed: 5.77591*10^-6 from last 2 batches: 5.35384*10^-6 6.19799*10^-6 *)


It's definitely not the same. It may be close, but I want to figure out how to get the exact same result. How can I reproduce the loss exactly with the trained network?

Thanks!

Interesting question. Looks like a bug: incorrect reported BatchIndices and BatchData.

SeedRandom;
X = RandomReal[{-1, 1}, {16, 3}];
Y = RandomReal[{0, 1}, {16, 1}];

net = NetChain[{LinearLayer, LinearLayer}, "Input" -> 3] // NetInitialize;

saveInfo[assoc_] := Module[
{},
AppendTo[batchIndices, assoc["BatchIndices"]];
AppendTo[batchLoss, assoc["BatchLoss"]];
AppendTo[trainedNet, assoc["Net"]];
]

report[batchSize_] := Module[
{},

batchIndices = {};
batchLoss = {};
trainedNet = {};

NetTrain[net, X -> Y, MaxTrainingRounds -> 2,
BatchSize -> batchSize, LossFunction -> MeanSquaredLossLayer[],
TrainingProgressFunction -> {saveInfo,
"Interval" -> Quantity[1, "Round"]}];

Print@batchLoss;
Print@MeanSquaredLossLayer[][<|
"Input" -> net@X[[batchIndices[]]],
"Target" -> Y[[batchIndices[]]]|>];
Print@MeanSquaredLossLayer[][<|
"Input" -> trainedNet[]@X[[batchIndices[]]],
"Target" -> Y[[batchIndices[]]]|>];
]


Size of the data = 16 and batch = 16:

report


{0.484094,0.476693}

0.484094

0.476693

Incorrect here:

report


{0.633787,0.312369}

0.642032

0.316417

• Thanks for checking it out. Hoping someone from Wolfram will clarify if it indeed is a bug.... Jan 16, 2021 at 1:32