As suggested by the title, is it possible to fix some part of the neural network while training?

Since Mathematica provides a way to extract part of a neural network and combine it with some layers to make a new one:

newNet = NetChain[{Take[oldNet, 3], 10, Ramp, 10}]

It would be very helpful to fix the layers taken from the old network. In this way one can reuse the neural network and investigate how transferrable are features in the neural network (c.f. https://arxiv.org/abs/1411.1792).

  • 1
    $\begingroup$ Is LearningRateMultipliers what you are looking for? And here are examples of transfer learning. $\endgroup$ Commented Aug 17, 2017 at 2:57

1 Answer 1


The LearningRateMultipliers supports using different learning rates with different layers. When setting the learning rate of a particular layer to None, the weight of that layer will be fixed during training.

You can have a look at the example in the documentation of LearningRateMultipliers.

In that example, net is constructed from part of the network trained for MNIST

net = NetChain[Join[Drop[Normal[lenet], -2],
   {LinearLayer[], SoftmaxLayer[]}],
  "Input" -> enc, "Output" -> dec]

Then at the training time, only the newly added linear layer is allowed to change. All the other layers are fixed by setting the learning rate to None.

trained = 
 NetTrain[net, trainingData, 
  MaxTrainingRounds -> Quantity[10, "Seconds"], 
  LearningRateMultipliers -> {-2 -> 1, _ -> None}, 
  ValidationSet -> Scaled[0.1]]
  • $\begingroup$ Thanks so much for the helpful answer! $\endgroup$
    – Tom
    Commented Aug 17, 2017 at 21:11
  • $\begingroup$ @xslittlegrass, Why LearningRateMultipliers -> {-2 -> 1, _ -> None} and not LearningRateMultipliers -> { _ -> None, -2->1}? Aren't you undoing the first LearningRateMultipliers assigment? How can one transfer some layers without their trained values? $\endgroup$
    – Miladiouss
    Commented Aug 24, 2017 at 5:09
  • 1
    $\begingroup$ @mpourrah We need to use {-2 -> 1, _ -> None} because the rules are matched with the order. If you use { _ -> None, -2->1} then the first rule is always matched so that the learning rate is 0 for all layers. $\endgroup$ Commented Aug 24, 2017 at 14:27
  • 1
    $\begingroup$ @mpourrah You can use NeuralNetworks`NetDeinitialize to remove the weights. See more here. $\endgroup$ Commented Aug 24, 2017 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.