As suggested by the title, is it possible to fix some part of the neural network while training?

Since Mathematica provides a way to extract part of a neural network and combine it with some layers to make a new one:

newNet = NetChain[{Take[oldNet, 3], 10, Ramp, 10}]

It would be very helpful to fix the layers taken from the old network. In this way one can reuse the neural network and investigate how transferrable are features in the neural network (c.f. https://arxiv.org/abs/1411.1792).

  • 1
    $\begingroup$ Is LearningRateMultipliers what you are looking for? And here are examples of transfer learning. $\endgroup$ Aug 17 '17 at 2:57

The LearningRateMultipliers supports using different learning rates with different layers. When setting the learning rate of a particular layer to None, the weight of that layer will be fixed during training.

You can have a look at the example in the documentation of LearningRateMultipliers.

In that example, net is constructed from part of the network trained for MNIST

net = NetChain[Join[Drop[Normal[lenet], -2],
   {LinearLayer[], SoftmaxLayer[]}],
  "Input" -> enc, "Output" -> dec]

Then at the training time, only the newly added linear layer is allowed to change. All the other layers are fixed by setting the learning rate to None.

trained = 
 NetTrain[net, trainingData, 
  MaxTrainingRounds -> Quantity[10, "Seconds"], 
  LearningRateMultipliers -> {-2 -> 1, _ -> None}, 
  ValidationSet -> Scaled[0.1]]
  • $\begingroup$ Thanks so much for the helpful answer! $\endgroup$
    – Andy
    Aug 17 '17 at 21:11
  • $\begingroup$ @xslittlegrass, Why LearningRateMultipliers -> {-2 -> 1, _ -> None} and not LearningRateMultipliers -> { _ -> None, -2->1}? Aren't you undoing the first LearningRateMultipliers assigment? How can one transfer some layers without their trained values? $\endgroup$
    – Miladiouss
    Aug 24 '17 at 5:09
  • 1
    $\begingroup$ @mpourrah We need to use {-2 -> 1, _ -> None} because the rules are matched with the order. If you use { _ -> None, -2->1} then the first rule is always matched so that the learning rate is 0 for all layers. $\endgroup$ Aug 24 '17 at 14:27
  • 1
    $\begingroup$ @mpourrah You can use NeuralNetworks`NetDeinitialize to remove the weights. See more here. $\endgroup$ Aug 24 '17 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.