25
$\begingroup$

Main Problem

I'm exploring the neural network in v11 and I would like to monitor the training progress in my own way instead of Mathematica's default way. So my question is, how to do this conveniently?


Example to try

So now I want to train a classifier on a set of points:

trainset = #1 -> {#2} & @@@ 
   Flatten[Thread[
       Rest[AnglePath[# Pi, Table[1./t^(2/3), {t, 1, 200}]]] -> 
        2 # - 1] & /@ {0, 1}, 1];

and I would like to get a result like:

img1

Where the small points are data points and color represents classification.

It's easy to write the code implementing basic form of neural network training to accomplish this:

net = NetInitialize@
   NetChain[{DotPlusLayer[20, 2], Tanh, 20, Tanh, 20, Tanh, 20, Tanh, 
     1, Tanh}];

net = NetTrain[net, trainset, MaxTrainingRounds -> 2500];

Show[ContourPlot[net[{x, y}][[1]], {x, -40, 40}, {y, -40, 40}, 
  ColorFunction -> (ColorData["RedGreenSplit"][(#/2 + 1)/2] &), 
  ColorFunctionScaling -> False], 
 ListPlot[Style[#[[1]], 
     ColorData["RedGreenSplit"][(#[[2, 1]] + 1)/2]] & /@ trainset, 
  PlotStyle -> PointSize[Large]]]

However, I would like to monitor the whole learning process, so this is what I've tried: splitting the whole learning process into pieces, then use Dynamic to track them.

Dynamic[Show[
  ContourPlot[net[{x, y}][[1]], {x, -40, 40}, {y, -40, 40}, 
   ColorFunction -> (ColorData["RedGreenSplit"][(#/2 + 1)/2] &), 
   ColorFunctionScaling -> False], 
  ListPlot[Style[#[[1]], 
      ColorData["RedGreenSplit"][(#[[2, 1]] + 1)/2]] & /@ trainset]], TrackedSymbols :> {net}]

Do[net = NetTrain[net, trainset, MaxTrainingRounds -> 100], {30}];

This piece of code will produce a pretty good dynamic result, but there's two main problems inside it. Solving any of them will be quite helpful:

  1. The default tracker keeps bumping out and is quite annoying. How to close the default tracker and stop it from showing up? are there any options that can do this job?

  2. As you can see, keep re-initializing the NetTrain process can slow down the whole calculation a lot. So are there any way to do this job without NetTrain for multiple times?(Note that it will be a much better answer if anyone solved this!)

Thanks!

$\endgroup$
  • $\begingroup$ 1. Could you define: "default tracker", "bumping out and influence", "notebooks order"? $\endgroup$ – Kuba Aug 11 '16 at 14:34
  • $\begingroup$ Default tracker means the small window showing you the progress, loss rate and loss-step curve. As for "Bumping out", I edit it a bit, I just don't want to see the default tracker bumping out. $\endgroup$ – Wjx Aug 11 '16 at 14:47
  • 3
    $\begingroup$ To turn off the training progress panel, pass "ShowTrainingProgress" -> False as an option to NetTrain $\endgroup$ – Arnoud Buzing Aug 11 '16 at 18:01
  • $\begingroup$ @Wjx Sorry for confusion, I have Print To Console by default and I haven't seen that default tracker at all. Messages window wasn't opened automatically. $\endgroup$ – Kuba Aug 11 '16 at 18:37
21
$\begingroup$

You can pass the undocumented option "ShowTrainingProgress" -> False to turn off the training progress blob thing.

As for monitoring the progress yourself, there is a feature coming in 11.1 called Checkpointing that lets you specify a callback to every so often that gets access to the partially trained net. You WOULD use it like so:

NetTrain[
   ..., 
   Checkpointing -> <|
      "Interval" -> Quantity[5, "Rounds"], 
       "Function" -> f
   |>
]

where f will be called on the partial networks every 5 training rounds. There's other units like "Seconds", "Minutes", "Hours", and "Batches". And you can also save the partially trained networks to a directory using a different suboption (they get named uniquely).

This is actually all implemented already, and lets you do interesting visualizations, like the following of the Lena "Neat Example" you can find in the NetTrain documentation:

animation of lena training

$\endgroup$
  • $\begingroup$ May I ask how you guys find out those undocumented options? Well well, you are seemingly working in Wolfram, so that's not a problem~ Thank you very much! $\endgroup$ – Wjx Aug 12 '16 at 2:55
  • $\begingroup$ @Wjx: Options@NetTrain $\endgroup$ – Sebastian Aug 12 '16 at 14:36
  • $\begingroup$ Oh, yeah, This is of couse a way. I usually simply go over the documentation and forget this. Thanks! But Even with this we won't be able to know about Checkpointing ~ :P $\endgroup$ – Wjx Aug 12 '16 at 15:12
  • 2
    $\begingroup$ @lgmendes no, it'll come in 11.1. But there is an internal way to get out the loss history as a simple list after you have trained, one loss per batch: NeuralNetworks`$LastLossHistory. If you specified a ValidationSet you can get those losses out too with NeuralNetworks`$LastValidationLossHistory. $\endgroup$ – Taliesin Beynon Aug 20 '16 at 0:38
  • 1
    $\begingroup$ It's months, but not 6 months. The new progress monitoring we've got running on 11.1 builds is already much, much better, and gives you access to a whole goody-bag of functionality. $\endgroup$ – Taliesin Beynon Oct 3 '16 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.