NetTrain ClassifierFunction/NetChain Entropy

Question

Obtain a trained network from NetTrain as

net = NetTrain[net, dat];

which i believe returns a ClassifierFunction (although Head[net] returns NetChain).

In the example docs for NetTrain it says:

From a random sample, select the images for which the net produces highest and lowest entropy predictions. High entropy inputs can be interpreted as those for which the net is most uncertain about the correct class:

images = RandomSample[Keys[dat], 5000];
entropies = net[images, "Entropy"];

My question is: what is "Entropy" here and how is it calculated? Is the entropy returned similar to entropy from ImageMeasurments or is it actually based on some information produced by the network etc?

Answer 1

Look at the source code:

C:\Program Files\Wolfram Research\Mathematica\11.0\SystemFiles\Components\NeuralNetworks\Decoders\Class.m

DEntropy = Compile[{{probs, _Real, 1}}, Module[{e = 0.},
    Do[
        If[2*$MachineEpsilon <= p <= 1., e -= p * Log[p]],
        {p, probs}
    ];
    e
], RuntimeAttributes -> {Listable}];