3
$\begingroup$

I am using Mathematica's NetTrain[] function for training neural nets. There is a way to set the learning rate (LearningRate and LearningRateMultiplier options) but I want a decreasing learning rate or changes based on development of the loss.

Current call for transfer learning with fixed learning rate:

NetTrain[
  preTrainedNet, trainData, All, 
  ValidationSet -> valData, MaxTrainingRounds -> epochs, TargetDevice -> "GPU", 
  LearningRateMultipliers -> {
      "classifier" -> lr, 
      {"base", 1, "conv_conv2d"} -> lr, 
      {"base", 1, "conv_relu"} -> lr, 
       _ -> 0},
  BatchSize -> 8,
  TrainingProgressCheckpointing -> 
    {"Directory", "C:\\DataSets\\RZ-DL-Aug-Pre", "Interval" -> Quantity[10, "Rounds"]}
];

How can I implement what I need?

$\endgroup$
1
$\begingroup$

Maybe the suboption "LearningRateSchedule" of the option Method helps? See the doc page of NetTrain > Options > Method. But I have to admit that the documentation is quite vague about the meaning of the function that can be fed to "LearningRateSchedule".

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.