Replacing parts of a neural net during training using TrainingProgressFunction

NetReplacePart is interesting because it allows to change, for instance, the weights of a layer (in a net), however it does return a new net and not mutate the existing net.

TrainingProgressFunction on the other hand allows us to call arbitrary functions during training.

Is is possible to combine NetReplacePart and TrainingProgressFunction so that we can change the net we are currently training?

An useful application could a soft tying of weights in an auto encoder net. Usually tying of weights means that the decoder weight matrix is just the transposed encoder weight matrix. As far as I am aware there is currently no method to do so directly, so I though one could call a function that takes both encoder and transposed decoder matrix and replaces both by the average of the two every few batches. My naive approach of just using Set on the #Net parameter supplied to TrainingProgressFunction prints out the usual warnings of misusing Set.

1 Answer

No that's not possible at the moment. We plan on exposing a mutable API so you can write your own training loop, optimizers, etc. Currently the "\$MXTrainer" property is pretty close to this, but this will be higher-level: there will be a way of getting a mutable network that contains handles to mutable arrays, changing those arrays makes the network behave differently.