6
$\begingroup$

I'm trying to use LearningRateMultipliers in a recurrent neural network, but I haven't found the right way yet.

An example of the code is as follows:

fixed = LinearLayer[2, "Weights" -> {{1., 0.5}, {0., 1.}}, "Biases" -> None]
core = NetGraph[<|"l1" -> TotalLayer[], "l2" -> LinearLayer[2, "Input" -> 2], "l3" -> fixed|>, {{NetPort["Input"], NetPort["State"]} -> "l1" -> "l2" -> "l3"}]
fold = NetFoldOperator[core, "Input" -> {"Varying", 2}]

net = NetTrain[fold, io, LearningRateMultipliers -> {_ -> 0.1, 3 -> 0}]
net[["Parameters"]]["Net"][["l3"]][["Weights"]]
(*The network training ends but the value of the l3 weights are changed*)

net = NetTrain[fold, io, LearningRateMultipliers -> {3 -> 0, _ -> 0.1}]
(*The command gives an error*)

net = NetTrain[fold, io, LearningRateMultipliers -> {_ -> 0.1, fold[["Parameters"]]["Net"][["l3"]] -> 0}]
(*The network training ends but the value of the l3 weights are changed*)

net = NetTrain[fold, io, LearningRateMultipliers -> {fold[["Parameters"]]["Net"][["l3"]] -> 0, _ -> 0.1}]
(*The command gives an error*)

What is the right way to change the learning rate in only one layer of a recurrent neural network?

Moreover, while looking for a way to solve the problem, I found a strange behaviour in a NetGraph. Please check the code.

fixed = LinearLayer[2, "Weights" -> {{1., 0.5}, {0., 1.}},"Biases" -> None]
core = NetInitialize@NetGraph[{LinearLayer[2, "Input" -> 2], fixed}, {NetPort["Input"] -> 1 -> 2}]
io = {{1, 2} -> {1, 2}};
net = NetTrain[core, io, LearningRateMultipliers -> {2 -> 0, _ -> 0.1}]
net[[2]][["Weights"]] (*Works*)
net = NetTrain[core, io, LearningRateMultipliers -> {_ -> 0.1, 2 -> 0}]
net[[2]][["Weights"]] (*Not works without warrning*)

The fact that the order of this list is important is reported nowhere.

$\endgroup$
  • 1
    $\begingroup$ From the documentation on LearningRateMultipliers: "For each trainable array, the rate used is given by the first matching rule, or 1 if no rule matches." That's why the order of the list is important. $\endgroup$ – Sjoerd Smit Sep 12 '18 at 15:24
7
$\begingroup$

You can use NetInformation to see how you should refer to the layers inside of the network:

In[89]:= fold = NetInitialize[fold];
NetInformation[fold, "Arrays"]

Out[90]= <|{"Net", "l2", "Biases"} -> {0., 0.}, {"Net", "l2", "Weights"} -> {{0.517432, -0.660375},
  {-0.524418, 0.517747}}, {"Net", "l3", "Weights"} -> {{1., 0.5}, {0., 1.}}|>

So if you want to freeze the 3rd layer in core, use

LearningRateMultipliers -> {{"Net", "l3"} -> 0, _ -> 1}

or

LearningRateMultipliers -> {{"Net", "l3", "Weights"} -> 0, _ -> 1}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.