2
$\begingroup$

I would like to implement the PPO algorithm - Reinforcement Learning - in Mathematica. And for that, I need to be able to copy network weights from say subnetwork NN1 to subnetwork NN2 as an update and as a part of the training process.

Is doing this possible in Mathematica? maybe my question can be split into two parts:

  1. how can I copy weights?

  2. how can I do that as a part of the training process, i.e. using NetTrain?

$\endgroup$
1
$\begingroup$

RNNs have ports "State" (all) and "CellState" (LSTM only).

NetGraph[
 {
  BasicRecurrentLayer[8],
  BasicRecurrentLayer[8]
  },
 {
  NetPort[1, "State"] -> NetPort[2, "State"]
  }
 ]

enter image description here

With ConvolutionLayer and LinearLayer it's more complicated. They have no ports other than "Input" and "Output". We can copy and paste weights. But I do not know how to do this in NetTrain without breaking the training loop.

conv = ConvolutionLayer[1, {3, 3}, "Input" -> {1, Automatic, Automatic}] // NetInitialize;
w = NetExtract[conv, "Weights"];
b = NetExtract[conv, "Biases"];
convNew = ConvolutionLayer[1, {3, 3}, "Weights" -> w, "Biases" -> b]
$\endgroup$
1
  • $\begingroup$ Thanks for taking the time to answer this question. I have done a lot more research since posting this question, and it seems that there is no way to copy and paste weight from one net to another during the training. Unfortunately NetTake, NetReplacePart, NetExtract, and all other Mathematica net commands all give you a new copy of the net. Like you mentioned, convNew is a new net, not an updated one... $\endgroup$ – Terrell N. Jan 12 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.