3
$\begingroup$

In the popular neural network framework Keras one can constrain the learned weights in a LinearLayer to have a unit norm. For example:

model.add(Dense(500, input_dim=2, activation='relu', kernel_constraint=unit_norm()))

One can think of various motivations for doing this in practice, so it is not just a hypothetical question. Also note that we are talking about constraining the learned weights, not the output (the latter was discussed in another post.

How would one implement this in Mathematica?

  • The NetTrain Method specification allows for "WeightClipping", which lets you set a maximum value (presumably absolute value) for the weights, but this does not enforce an equality constraint. (Aside: I suspect this is implemented with the MXNet clip_global_norm...and perhaps also relevant, is that the MXNet documentation doesn't seem to suggest a capability for applying the desired type of layer constraint.)

  • One path would be to implement a custom network component using NetArrayLayer to store the learned weights and biases:

unitNormWeightLinearLayer[nIn_Integer, nOut_Integer] := NetGraph[
  <|"weight" -> NetArrayLayer["Output" -> {nOut, nIn}],
   "bias" -> NetArrayLayer["Output" -> {nOut}],
   "normalize" -> FunctionLayer[Map[Normalize]],
   "linear" -> FunctionLayer[ Apply[#1 . #2 + #3 &]]|>,
  {"weight" -> "normalize",
   {"normalize", NetPort["Input"], "bias"} -> 
    "linear" -> NetPort["Output"]},
  "Input" -> nIn]

unitNormWeightLinearLayer[5, 10]

NetGraph output

  • Although this only provides normalized values of the weights, it is perhaps suboptimal because it does not guarantee that the weights won't pathologically get too large during training (although perhaps that is doable by setting a "WeightClipping" to this layer to enforce the maximum value or attaching a loss function to the norm of the weight layer to keep it small) ... both of which seem a bit like hacky.

Is there a better way to implement this?

$\endgroup$
1
  • 1
    $\begingroup$ If you disable biases by setting "Biases"->None then you could also pass a constant array layer containing only 1s via NetArrayLayer["Output" -> ConstantArray[1, n], {LearningRateMultipliers -> All -> None}] into the LinearLayer. The output of the LinearLayer is exactly the weights. We can then do the norm with this later. If we do all this in a separate LossFunction then we may be able to feed the normal data into it as usual, while using only our 1-vector in the special loss function. $\endgroup$
    – flinty
    Feb 2 at 12:41

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.