7
$\begingroup$

For example:

INPUTNOTE = 1000;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data = Transpose[{Range[INPUTNOTE], Labels}];
generator = Function[Map[#[[1]] -> #[[2]] &, RandomSample[data, #BatchSize]]];
net = NetChain[{EmbeddingLayer[128, "Input" -> NetEncoder[{"Class", Range[1, INPUTNOTE]}]], 256, Ramp,256, Ramp, 43}]; 
{net, LossEvolutionPlot} = NetTrain[net, generator, MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, BatchSize -> 32, MaxTrainingRounds -> 2000];
LossEvolutionPlot

enter image description here

And I want to save network at the lowest point (red point). How to do it?

PS:In TensorFlow this task can do like this:

min_loss_value=float("inf")
saver = tf.train.Saver()
for i in range(2000):
    ...train the network...
    loss_value = get_loss(sess,x,y)
    if loss_value<min_loss_value:
        min_loss_value=loss_value
        save_path = saver.save(sess,SAVE_NET_TRAIN,global_step=global_step)
        print("Training Set got a smaller value,network save to path: "+ save_path)
$\endgroup$
0

1 Answer 1

6
$\begingroup$

One way to do this is to check the loss while training using the TrainingProgressFunction.

This will terminate the training and give approximately the best net if the loss doesn't improve in certain batches:

loss = {};
window = 100;
INPUTNOTE = 1000;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data = Transpose[{Range[INPUTNOTE], Labels}];
generator = 
  Function[Map[#[[1]] -> #[[2]] &, RandomSample[data, #BatchSize]]];
net = NetChain[{EmbeddingLayer[128, 
     "Input" -> NetEncoder[{"Class", Range[1, INPUTNOTE]}]], 256, 
    Ramp, 256, Ramp, 43}];
{net, LossEvolutionPlot} = 
  NetTrain[net, generator, 
   MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, 
   BatchSize -> 32, MaxTrainingRounds -> 2000, 
   TrainingProgressFunction -> {If[AppendTo[loss, #BatchLoss]; 
       Length[loss] - First@Flatten@Position[loss, Min[loss]] >= 
        window, "StopTraining"] &, 
     "Interval" -> Quantity[1, "Rounds"]}];
LossEvolutionPlot

enter image description here

You can also change the window size and smooth the loss history a little bit to get a better location of the best net.

Another way is to save the net as we train the network

stepsize = 100;
{net, LossEvolutionPlot} = 
  NetTrain[net, generator, 
   MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, 
   BatchSize -> 32, MaxTrainingRounds -> 2000, 
   TrainingProgressFunction -> {If[#BatchLoss < minloss, 
       minloss = #BatchLoss; Print[#BatchLoss, "   ", #Round]; 
       Export["best_net.wlnet", #Net]] &, 
     "Interval" -> Quantity[stepsize, "Batches"]}];
$\endgroup$
3
  • 2
    $\begingroup$ Oh I didn't know about the "StopTraining" trigger, that's helpful! $\endgroup$
    – M.R.
    May 5, 2017 at 16:14
  • $\begingroup$ @M.R. That one comes from Taliesin :) $\endgroup$ May 5, 2017 at 16:20
  • $\begingroup$ if want to save net at lowest Validation Loss.TrainingProgressFunction -> {If[#LowestValidationLoss < minloss, Print[#LowestValidationLoss, " ", #Round]; Export["Target_model.wlnet", #Net]] &, "Interval" -> Quantity[1, "Rounds"]} will help $\endgroup$
    – partida
    May 19, 2017 at 2:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.