10
$\begingroup$

Is there a way to change the weights of a layer in a neural network, without reconstructing the whole network?

For example,

lenet = NetInitialize@NetChain[{
    ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
    ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
    FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
   "Output" -> NetDecoder[{"Class", Range[0, 9]}],
   "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]
   ]

The weights of the second convolution layer can be extracted using NetExtract

NetExtract[lenet, {4, "Weights"}]

But this is extracted convolution layer is now independent of the neural network. How can I change the weights of this layer in the neural network in place? I'm looking something like

lenet[[4]]["Weights"] = myWeights

Keras has a similar functionality that allows we to modify the weights directly like

layer.set_weights([ker,bia])

Being able to change the weights in place is very useful, for instance, in loading pre-trained weights file.

$\endgroup$
  • $\begingroup$ I don't know if there is a way to change the weights of an existing layer, but you can specify the initial weights of a new layer using ConvolutionLayer[... "Weights"->w]. So you could create a new network and initialize all layers to the weights you want $\endgroup$ – Niki Estner Mar 29 '17 at 19:27
  • $\begingroup$ Also, NetReplacePart looks like a promising way to replace part of an existing network. Haven't tried it, though. $\endgroup$ – Niki Estner Mar 29 '17 at 19:29
  • $\begingroup$ @nikie NetReplacePart seems to work by return a new network with the updated weights. I somehow missed that function in the documentation. Thanks! $\endgroup$ – xslittlegrass Mar 29 '17 at 20:17
6
$\begingroup$

As nikie suggests, the function NetReplacePart can be used to replace weights of layers in a neural network. It returns a new neural network with the updated weights. For example

myWeights1 = ConstantArray[0., {20, 1, 5, 5}];
myWeights2 = ConstantArray[0., {50, 20, 5, 5}];

lenet = NetReplacePart[lenet, {{1, "Weights"} -> myWeights1, {4, "Weights"} -> myWeights2}]

And we can check that the weights are updated

NetExtract[lenet, {1, "Weights"}] == myWeights1
NetExtract[lenet, {4, "Weights"}] == myWeights2
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.