# What's the difference between train and test evaluation modes of a BatchNormalizationLayer?

After creating a BatchNormalizationLayer:

batchnorm = NetInitialize@BatchNormalizationLayer["Input" -> 3]


We can evaluate it in "Test" or "Train" mode:

x = RandomReal[1, 3];
batchnorm[x, NetEvaluationMode -> "Test"]
batchnorm[x, NetEvaluationMode -> "Train"]

(* {0.935889, 0.580286, 0.424091} *)
(* {0., 0., 0.} *)


What is the difference? I.e. what is the function being evaluated?

https://en.wikipedia.org/wiki/Batch_normalization

You will see the difference if the batch size is 2 or more. Because if batch = 1, mean = x.

batchnorm = NetInitialize@BatchNormalizationLayer["Input" -> 3];

x = RandomReal[1, {2, 3}]


{{0.90044,0.177369,0.356773},{0.784026,0.446095,0.664769}}

batchnorm[x, NetEvaluationMode -> "Train"]


{{0.878696,-0.973404,-0.979561},{-0.878696,0.973404,0.979561}}

batchnorm[x, NetEvaluationMode -> "Test"]


{{0.89999, 0.177281, 0.356595}, {0.783635, 0.445872, 0.664437}}

In "Train"-mode mean is calculated.

The "Test"-mode uses learned weights.

NetExtract[batchnorm, "MovingMean"] // Normal


{0., 0., 0.}

NetExtract[batchnorm, "MovingVariance"] // Normal


{1., 1., 1.}

NetExtract[batchnorm, "Biases"] // Normal


{0., 0., 0.}

NetExtract[batchnorm, "Scaling"] // Normal


{1., 1., 1.}