# How to save the network at lowest loss?

For example:

INPUTNOTE = 1000;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data = Transpose[{Range[INPUTNOTE], Labels}];
generator = Function[Map[#[] -> #[] &, RandomSample[data, #BatchSize]]];
net = NetChain[{EmbeddingLayer[128, "Input" -> NetEncoder[{"Class", Range[1, INPUTNOTE]}]], 256, Ramp,256, Ramp, 43}];
{net, LossEvolutionPlot} = NetTrain[net, generator, MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, BatchSize -> 32, MaxTrainingRounds -> 2000];
LossEvolutionPlot And I want to save network at the lowest point (red point). How to do it?

PS：In TensorFlow this task can do like this:

min_loss_value=float("inf")
saver = tf.train.Saver()
for i in range(2000):
...train the network...
loss_value = get_loss(sess,x,y)
if loss_value<min_loss_value:
min_loss_value=loss_value
save_path = saver.save(sess,SAVE_NET_TRAIN,global_step=global_step)
print("Training Set got a smaller value,network save to path: "+ save_path)


One way to do this is to check the loss while training using the TrainingProgressFunction.

This will terminate the training and give approximately the best net if the loss doesn't improve in certain batches:

loss = {};
window = 100;
INPUTNOTE = 1000;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data = Transpose[{Range[INPUTNOTE], Labels}];
generator =
Function[Map[#[] -> #[] &, RandomSample[data, #BatchSize]]];
net = NetChain[{EmbeddingLayer[128,
"Input" -> NetEncoder[{"Class", Range[1, INPUTNOTE]}]], 256,
Ramp, 256, Ramp, 43}];
{net, LossEvolutionPlot} =
NetTrain[net, generator,
MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"},
BatchSize -> 32, MaxTrainingRounds -> 2000,
TrainingProgressFunction -> {If[AppendTo[loss, #BatchLoss];
Length[loss] - First@Flatten@Position[loss, Min[loss]] >=
window, "StopTraining"] &,
"Interval" -> Quantity[1, "Rounds"]}];
LossEvolutionPlot You can also change the window size and smooth the loss history a little bit to get a better location of the best net.

Another way is to save the net as we train the network

stepsize = 100;
{net, LossEvolutionPlot} =
NetTrain[net, generator,
MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"},
BatchSize -> 32, MaxTrainingRounds -> 2000,
TrainingProgressFunction -> {If[#BatchLoss < minloss,
minloss = #BatchLoss; Print[#BatchLoss, "   ", #Round];
Export["best_net.wlnet", #Net]] &,
"Interval" -> Quantity[stepsize, "Batches"]}];

• Oh I didn't know about the "StopTraining" trigger, that's helpful! – M.R. May 5 '17 at 16:14
• @M.R. That one comes from Taliesin :) – xslittlegrass May 5 '17 at 16:20
• if want to save net at lowest Validation Loss.TrainingProgressFunction -> {If[#LowestValidationLoss < minloss, Print[#LowestValidationLoss, " ", #Round]; Export["Target_model.wlnet", #Net]] &, "Interval" -> Quantity[1, "Rounds"]} will help – partida May 19 '17 at 2:01