7
$\begingroup$

For example:

INPUTNOTE = 1000;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data = Transpose[{Range[INPUTNOTE], Labels}];
generator = Function[Map[#[[1]] -> #[[2]] &, RandomSample[data, #BatchSize]]];
net = NetChain[{EmbeddingLayer[128, "Input" -> NetEncoder[{"Class", Range[1, INPUTNOTE]}]], 256, Ramp,256, Ramp, 43}]; 
{net, LossEvolutionPlot} = NetTrain[net, generator, MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, BatchSize -> 32, MaxTrainingRounds -> 2000];
LossEvolutionPlot

enter image description here

And I want to save network at the lowest point (red point). How to do it?

PS:In TensorFlow this task can do like this:

min_loss_value=float("inf")
saver = tf.train.Saver()
for i in range(2000):
    ...train the network...
    loss_value = get_loss(sess,x,y)
    if loss_value<min_loss_value:
        min_loss_value=loss_value
        save_path = saver.save(sess,SAVE_NET_TRAIN,global_step=global_step)
        print("Training Set got a smaller value,network save to path: "+ save_path)
$\endgroup$
6
$\begingroup$

One way to do this is to check the loss while training using the TrainingProgressFunction.

This will terminate the training and give approximately the best net if the loss doesn't improve in certain batches:

loss = {};
window = 100;
INPUTNOTE = 1000;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data = Transpose[{Range[INPUTNOTE], Labels}];
generator = 
  Function[Map[#[[1]] -> #[[2]] &, RandomSample[data, #BatchSize]]];
net = NetChain[{EmbeddingLayer[128, 
     "Input" -> NetEncoder[{"Class", Range[1, INPUTNOTE]}]], 256, 
    Ramp, 256, Ramp, 43}];
{net, LossEvolutionPlot} = 
  NetTrain[net, generator, 
   MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, 
   BatchSize -> 32, MaxTrainingRounds -> 2000, 
   TrainingProgressFunction -> {If[AppendTo[loss, #BatchLoss]; 
       Length[loss] - First@Flatten@Position[loss, Min[loss]] >= 
        window, "StopTraining"] &, 
     "Interval" -> Quantity[1, "Rounds"]}];
LossEvolutionPlot

enter image description here

You can also change the window size and smooth the loss history a little bit to get a better location of the best net.

Another way is to save the net as we train the network

stepsize = 100;
{net, LossEvolutionPlot} = 
  NetTrain[net, generator, 
   MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, 
   BatchSize -> 32, MaxTrainingRounds -> 2000, 
   TrainingProgressFunction -> {If[#BatchLoss < minloss, 
       minloss = #BatchLoss; Print[#BatchLoss, "   ", #Round]; 
       Export["best_net.wlnet", #Net]] &, 
     "Interval" -> Quantity[stepsize, "Batches"]}];
|improve this answer|||||
$\endgroup$
  • 2
    $\begingroup$ Oh I didn't know about the "StopTraining" trigger, that's helpful! $\endgroup$ – M.R. May 5 '17 at 16:14
  • $\begingroup$ @M.R. That one comes from Taliesin :) $\endgroup$ – xslittlegrass May 5 '17 at 16:20
  • $\begingroup$ if want to save net at lowest Validation Loss.TrainingProgressFunction -> {If[#LowestValidationLoss < minloss, Print[#LowestValidationLoss, " ", #Round]; Export["Target_model.wlnet", #Net]] &, "Interval" -> Quantity[1, "Rounds"]} will help $\endgroup$ – partida May 19 '17 at 2:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.