# Main Problem

I'm exploring the neural network in v11 and I would like to monitor the training progress in my own way instead of Mathematica's default way. So my question is, how to do this conveniently?

# Example to try

So now I want to train a classifier on a set of points:

trainset = #1 -> {#2} & @@@
Rest[AnglePath[# Pi, Table[1./t^(2/3), {t, 1, 200}]]] ->
2 # - 1] & /@ {0, 1}, 1];


and I would like to get a result like：

Where the small points are data points and color represents classification.

It's easy to write the code implementing basic form of neural network training to accomplish this：

net = NetInitialize@
NetChain[{DotPlusLayer[20, 2], Tanh, 20, Tanh, 20, Tanh, 20, Tanh,
1, Tanh}];

net = NetTrain[net, trainset, MaxTrainingRounds -> 2500];

Show[ContourPlot[net[{x, y}][[1]], {x, -40, 40}, {y, -40, 40},
ColorFunction -> (ColorData["RedGreenSplit"][(#/2 + 1)/2] &),
ColorFunctionScaling -> False],
ListPlot[Style[#[[1]],
ColorData["RedGreenSplit"][(#[[2, 1]] + 1)/2]] & /@ trainset,
PlotStyle -> PointSize[Large]]]


However, I would like to monitor the whole learning process, so this is what I've tried: splitting the whole learning process into pieces, then use Dynamic to track them.

Dynamic[Show[
ContourPlot[net[{x, y}][[1]], {x, -40, 40}, {y, -40, 40},
ColorFunction -> (ColorData["RedGreenSplit"][(#/2 + 1)/2] &),
ColorFunctionScaling -> False],
ListPlot[Style[#[[1]],
ColorData["RedGreenSplit"][(#[[2, 1]] + 1)/2]] & /@ trainset]], TrackedSymbols :> {net}]

Do[net = NetTrain[net, trainset, MaxTrainingRounds -> 100], {30}];


This piece of code will produce a pretty good dynamic result, but there's two main problems inside it. Solving any of them will be quite helpful:

1. The default tracker keeps bumping out and is quite annoying. How to close the default tracker and stop it from showing up? are there any options that can do this job?

2. As you can see, keep re-initializing the NetTrain process can slow down the whole calculation a lot. So are there any way to do this job without NetTrain for multiple times?(Note that it will be a much better answer if anyone solved this!)

Thanks!

• 1. Could you define: "default tracker", "bumping out and influence", "notebooks order"?
– Kuba
Aug 11, 2016 at 14:34
• Default tracker means the small window showing you the progress, loss rate and loss-step curve. As for "Bumping out", I edit it a bit, I just don't want to see the default tracker bumping out.
– Wjx
Aug 11, 2016 at 14:47
• To turn off the training progress panel, pass "ShowTrainingProgress" -> False as an option to NetTrain Aug 11, 2016 at 18:01
• @Wjx Sorry for confusion, I have Print To Console by default and I haven't seen that default tracker at all. Messages window wasn't opened automatically.
– Kuba
Aug 11, 2016 at 18:37

You can pass the undocumented option "ShowTrainingProgress" -> False to turn off the training progress blob thing.

As for monitoring the progress yourself, there is a feature coming in 11.1 called Checkpointing that lets you specify a callback to every so often that gets access to the partially trained net. You WOULD use it like so:

NetTrain[
...,
Checkpointing -> <|
"Interval" -> Quantity[5, "Rounds"],
"Function" -> f
|>
]


where f will be called on the partial networks every 5 training rounds. There's other units like "Seconds", "Minutes", "Hours", and "Batches". And you can also save the partially trained networks to a directory using a different suboption (they get named uniquely).

This is actually all implemented already, and lets you do interesting visualizations, like the following of the Lena "Neat Example" you can find in the NetTrain documentation:

animation of lena training

• May I ask how you guys find out those undocumented options? Well well, you are seemingly working in Wolfram, so that's not a problem~ Thank you very much!
– Wjx
Aug 12, 2016 at 2:55
• @Wjx: Options@NetTrain Aug 12, 2016 at 14:36
• Oh, yeah, This is of couse a way. I usually simply go over the documentation and forget this. Thanks! But Even with this we won't be able to know about Checkpointing ~ :P
– Wjx
Aug 12, 2016 at 15:12
• @lgmendes no, it'll come in 11.1. But there is an internal way to get out the loss history as a simple list after you have trained, one loss per batch: NeuralNetworks$LastLossHistory. If you specified a ValidationSet you can get those losses out too with NeuralNetworks$LastValidationLossHistory. Aug 20, 2016 at 0:38
• It's months, but not 6 months. The new progress monitoring we've got running on 11.1 builds is already much, much better, and gives you access to a whole goody-bag of functionality. Oct 3, 2016 at 20:32