# Using LearningRateMultipliers in a recurrent neural network

I'm trying to use LearningRateMultipliers in a recurrent neural network, but I haven't found the right way yet.

An example of the code is as follows:

fixed = LinearLayer[2, "Weights" -> {{1., 0.5}, {0., 1.}}, "Biases" -> None]
core = NetGraph[<|"l1" -> TotalLayer[], "l2" -> LinearLayer[2, "Input" -> 2], "l3" -> fixed|>, {{NetPort["Input"], NetPort["State"]} -> "l1" -> "l2" -> "l3"}]
fold = NetFoldOperator[core, "Input" -> {"Varying", 2}]

net = NetTrain[fold, io, LearningRateMultipliers -> {_ -> 0.1, 3 -> 0}]
net[["Parameters"]]["Net"][["l3"]][["Weights"]]
(*The network training ends but the value of the l3 weights are changed*)

net = NetTrain[fold, io, LearningRateMultipliers -> {3 -> 0, _ -> 0.1}]
(*The command gives an error*)

net = NetTrain[fold, io, LearningRateMultipliers -> {_ -> 0.1, fold[["Parameters"]]["Net"][["l3"]] -> 0}]
(*The network training ends but the value of the l3 weights are changed*)

net = NetTrain[fold, io, LearningRateMultipliers -> {fold[["Parameters"]]["Net"][["l3"]] -> 0, _ -> 0.1}]
(*The command gives an error*)


What is the right way to change the learning rate in only one layer of a recurrent neural network?

Moreover, while looking for a way to solve the problem, I found a strange behaviour in a NetGraph. Please check the code.

fixed = LinearLayer[2, "Weights" -> {{1., 0.5}, {0., 1.}},"Biases" -> None]
core = NetInitialize@NetGraph[{LinearLayer[2, "Input" -> 2], fixed}, {NetPort["Input"] -> 1 -> 2}]
io = {{1, 2} -> {1, 2}};
net = NetTrain[core, io, LearningRateMultipliers -> {2 -> 0, _ -> 0.1}]
net[][["Weights"]] (*Works*)
net = NetTrain[core, io, LearningRateMultipliers -> {_ -> 0.1, 2 -> 0}]
net[][["Weights"]] (*Not works without warrning*)


The fact that the order of this list is important is reported nowhere.

• From the documentation on LearningRateMultipliers: "For each trainable array, the rate used is given by the first matching rule, or 1 if no rule matches." That's why the order of the list is important. – Sjoerd Smit Sep 12 '18 at 15:24

You can use NetInformation to see how you should refer to the layers inside of the network:

In:= fold = NetInitialize[fold];
NetInformation[fold, "Arrays"]

Out= <|{"Net", "l2", "Biases"} -> {0., 0.}, {"Net", "l2", "Weights"} -> {{0.517432, -0.660375},
{-0.524418, 0.517747}}, {"Net", "l3", "Weights"} -> {{1., 0.5}, {0., 1.}}|>


So if you want to freeze the 3rd layer in core, use

LearningRateMultipliers -> {{"Net", "l3"} -> 0, _ -> 1}


or

LearningRateMultipliers -> {{"Net", "l3", "Weights"} -> 0, _ -> 1}