ContrastiveLossLayer: How do I convert a scalar to a number? (And what's the difference?)

Question

I'm trying to use hinge loss for training a neural net. I think ContrastiveLossLayer does what I want, but I don't know how to make my net output compatible with ContrastiveLossLayer's input. Minimal example:

testNet = NetChain[{1}, "Input" -> {10}, "Output" -> "Scalar"];
NetTrain[testNet, {Range[10] -> True}, ContrastiveLossLayer[]]

NetTrain::invploss2: Provided loss layer ContrastiveLossLayer[0.5,[Ellipsis]], which expects a number, is incompatible with "Output" port, which produces a scalar.

(emphasis mine)

So: What's the difference between a number and a scalar?

Answer 1

For seeing what is going on it is (in my opinion) best to use NetGraph instead of NetChain

Compare

NetGraph[{LinearLayer[1]}, {NetPort["Input"] -> 1 -> NetPort["Output"]},
"Input" -> 10 , "Output" -> "Scalar"]

with

NetGraph[{LinearLayer[1], PartLayer[1]}, 
{NetPort["Input"] -> 1 -> 2 -> NetPort["Output"]}, "Input" -> 10 ]

and notice the dimensions of the layer outputs. You can also click on the pink dot indicating the PartLayer and check that it converts a length one vector (think {1.}) into a real number (think 1.).

In this case you could also use SummationLayer instead of PartLayer.

---

Addendum

The way you reverse this operation (taking a real number and turning it into a length one vector again) is ReplicateLayer[1] as in

NetGraph[{LinearLayer[1], PartLayer[1], ReplicateLayer[1]},
{NetPort["Input"] -> 1 -> 2 -> 3 -> NetPort["Output"]}, "Input" -> 10]

Answer 2

A number and a scalar are of course the same thing. That message is buggy (I've reported it). It should say something more like provided loss layer ContrastiveLossLayer[0.5,...], which expects a number, is incompatible with "Output" port, which produces a 1-vector decoded to a number.

So the net you're trying to train is producing a 1-vector, but ContrastiveLossLayer wants a scalar. Decoding happens 'outside' the net, so when the loss layer and the net are joined to make the training net, any encoders you had on the output of the original net go away and so the mismatch becomes evident.

You can fix this by producing a scalar in the first place, rather than producing a 1-vector and then having a decoder. Here's the simplest way, which is to take advantage of the fact that LinearLayer can produce any shape of output, including a number/scalar (which is a 0-tensor, in some sense, hence the dimensions of {}):

testNet = LinearLayer[{}, "Input" -> 10];
NetTrain[testNet, {Range[10] -> True}, ContrastiveLossLayer[]]

You can also write that linear layer as LinearLayer["Input" -> 10, "Output" -> "Real"] if you prefer.

Inside a NetChain, there is no special syntax for LinearLayer[{}], whereas an integer n neans LinearLayer[{n}], so you'd have to write out LinearLayer[{}] directly, or if you really want to save some keystrokes, take advantage of inference to infer the output size and just write LinearLayer[]. Writing a 'blank' linear layer can also be quite useful as well for when you don't want to hardcode the number of classes, and to leave that to be inferred from the training data itself.