How to monitor the process of Neural Network Learning

Question

Main Problem

I'm exploring the neural network in v11 and I would like to monitor the training progress in my own way instead of Mathematica's default way. So my question is, how to do this conveniently?

---

Example to try

So now I want to train a classifier on a set of points:

trainset = #1 -> {#2} & @@@ 
   Flatten[Thread[
       Rest[AnglePath[# Pi, Table[1./t^(2/3), {t, 1, 200}]]] -> 
        2 # - 1] & /@ {0, 1}, 1];

and I would like to get a result like：

Where the small points are data points and color represents classification.

It's easy to write the code implementing basic form of neural network training to accomplish this：

net = NetInitialize@
   NetChain[{DotPlusLayer[20, 2], Tanh, 20, Tanh, 20, Tanh, 20, Tanh, 
     1, Tanh}];

net = NetTrain[net, trainset, MaxTrainingRounds -> 2500];

Show[ContourPlot[net[{x, y}][[1]], {x, -40, 40}, {y, -40, 40}, 
  ColorFunction -> (ColorData["RedGreenSplit"][(#/2 + 1)/2] &), 
  ColorFunctionScaling -> False], 
 ListPlot[Style[#[[1]], 
     ColorData["RedGreenSplit"][(#[[2, 1]] + 1)/2]] & /@ trainset, 
  PlotStyle -> PointSize[Large]]]

However, I would like to monitor the whole learning process, so this is what I've tried: splitting the whole learning process into pieces, then use Dynamic to track them.

Dynamic[Show[
  ContourPlot[net[{x, y}][[1]], {x, -40, 40}, {y, -40, 40}, 
   ColorFunction -> (ColorData["RedGreenSplit"][(#/2 + 1)/2] &), 
   ColorFunctionScaling -> False], 
  ListPlot[Style[#[[1]], 
      ColorData["RedGreenSplit"][(#[[2, 1]] + 1)/2]] & /@ trainset]], TrackedSymbols :> {net}]

Do[net = NetTrain[net, trainset, MaxTrainingRounds -> 100], {30}];

This piece of code will produce a pretty good dynamic result, but there's two main problems inside it. Solving any of them will be quite helpful:

1. The default tracker keeps bumping out and is quite annoying. How to close the default tracker and stop it from showing up? are there any options that can do this job?

2. As you can see, keep re-initializing the NetTrain process can slow down the whole calculation a lot. So are there any way to do this job without NetTrain for multiple times?(Note that it will be a much better answer if anyone solved this!)

Thanks!

Answer 1

You can pass the undocumented option "ShowTrainingProgress" -> False to turn off the training progress blob thing.

As for monitoring the progress yourself, there is a feature coming in 11.1 called Checkpointing that lets you specify a callback to every so often that gets access to the partially trained net. You WOULD use it like so:

NetTrain[
   ..., 
   Checkpointing -> <|
      "Interval" -> Quantity[5, "Rounds"], 
       "Function" -> f
   |>
]

where f will be called on the partial networks every 5 training rounds. There's other units like "Seconds", "Minutes", "Hours", and "Batches". And you can also save the partially trained networks to a directory using a different suboption (they get named uniquely).

This is actually all implemented already, and lets you do interesting visualizations, like the following of the Lena "Neat Example" you can find in the NetTrain documentation:

animation of lena training