8
$\begingroup$

From what the documentation says about NetTrain and NetInitialize and what the progress indicator shows when reevaluating examples from the Documentation it looks like NetTrain always calls NetInitialize.

The Properties & Relations section of NetInitialize states

NetTrain will automatically call NetInitialize before training begins. The weights and biases of a simple layer are initialized before training

Can someone confirm this behavior and if so offer a workaround? It would be really useful to be able to train networks in stages for instance for pre-training a network on some readily available (similar) data or for playing around with Complementary Reinforcement Backpropagation.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
11
$\begingroup$

So actually this is already implied by the documentation for NetInitialize, which says "gives a net in which all uninitialized learnable parameters in net have been given initial values". So when you retrain, you're starting with weights, and the internal NetInitialize does nothing (you can use the second argument of 'All' to force it to overwrite existing weights).

But in any case I've added a note to NetTrain to mention that it doesn't re-initialize pre-existing weights. Thanks for drawing my attention to this possible confusion.

Improve this answer
$\endgroup$
0
$\begingroup$

I played around more with this and found my initial observation that NetTrain starts anew each time to be wrong!

Take for instance the MNIST example from the NetTrain>>Applications>>Image Classification- section. There replace the line 

lenet = NetTrain[lenet, trainingData, ValidationSet -> testData, 
MaxTrainingRounds -> 3]

with 

(lenet = NetTrain[lenet, RandomSample[trainingData, 100], 
 BatchSize -> 10, MaxTrainingRounds -> 1]) //Do[#, 10^3]&

and observe that indeed the network learns when increasing the number of rounds via Do or repeatedly evaluating this line. The following lines, also found in that example, can be used to check classification accuracy against the test set.

cm = ClassifierMeasurements[lenet, testData]
cm["Accuracy"]
Improve this answer
$\endgroup$
3
  • 4
    $\begingroup$ you should be aware that by doing this you'll be losing the internal state used by the optimizers, so training will be less stable and efficient. Luckily, for 11.1 they'll be a nice internal API to do your own staged training with zero overhead using a "Trainer" object. You could use it for quite complex stuff that isn't possible to parameterize under NetTrain. $\endgroup$
    – Taliesin Beynon
    Sep 26, 2016 at 21:21
  • $\begingroup$ @TaliesinBeynon Thank you for your answer and comment! Can you tell me wether this "Trainer"-object stuff will also support some flavor of Reinforcement learning (in the future)? I just recently tried the algorithm from the paper I linked in my question and it seems to work well for easy problems (MNIST with only 0's and 1's) but implementing it is not as comfortable as one might wish. To be able and assign a critic/feedback system to a trainer would be really convenient. $\endgroup$
    – Sascha
    Sep 27, 2016 at 8:49
  • 2
    $\begingroup$ we intend to support reinforcement learning at some point. The state of the art with experience replay etc in is quite complicated and would probably require a new function (not NetTrain). But trainers will be quite general, you define which ports provide the loss (or a custom function to impose the final loss gradient to be backpropogated), feed them batches of training data, and then they do the forward and backward pass and update the weights via the optimizers. NetTrain will just do the 'feeding' in a Do loop, basically. $\endgroup$
    – Taliesin Beynon
    Sep 27, 2016 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.