0
$\begingroup$

I'm working on a class of machine learning problems which fit functions from scalars to vectors. Let's suppose the output of this function is just a set of two-component vectors. I have found that my neural network is doing a good job of fitting this function, except that the outputs are consistently off by a multiplicative factor. To fix this, I would like to insert a layer into my neural network which rescales the outputs by a learnable multiplicative factor.

Essentially, such an operation would look like LinearLayer[] without biases and with the constraint that every weight is identical. Is there a built-in layer for this? Other seemingly related functions like NormalizationLayer[] seem much more involved than the desired operation of rescaling the final output by a learnable factor.

$\endgroup$
1
$\begingroup$

What you need is something like a ConstantTimesLayer, but with only one parameter that is applied to all values in the input.

Since Mathematica 12.2, ConstantTimesLayer[] translates to a specific NetGraph object. We construct a similar NetGraph:

scaleLayer[opt : OptionsPattern[]] := 
 NetGraph[{NetArrayLayer["Output" -> {}], 
   ThreadingLayer[Times, 1]}, {{1, NetPort["Input"]} -> 2}, opt]

Which then you can use in your neural network.

For example:

net = NetChain[{
   5, Tanh, 5, Tanh, 5, scaleLayer[]}]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.