Optimize a network topology?

Question

I'd like to know if one can use optimization function like Maximize[] or BayesianMaximize[] to optimize the hyper-parameters of a NetGraph?

Specifically, I want to optimize the kernel size, the number of hidden units for each of the convolutional layers, and the number of hidden units in the fully connected layer (for mnist task for instance).

Answer 1

In principle you can wrap a neural network inside a function and then optimize the hyper-parameters using the optimization functions. Here is an example of using BayesianMaximize to optimize the kernel size of one of the convolution layer.

Loading the MNIST data

trainingData = ResourceData["MNIST", "TrainingData"];
testData = ResourceData["MNIST", "TestData"];

Construct network with one of the kernel size as parameter

net[kerSize_] := NetChain[{
   ConvolutionLayer[20, {kerSize, kerSize}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   ConvolutionLayer[50, {5, 5}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   FlattenLayer[],
   LinearLayer[500],
   ElementwiseLayer[Ramp],
   LinearLayer[10],
   SoftmaxLayer[]
   },
  "Output" -> NetDecoder[{"Class", Range[0, 9]}],
  "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]
  ]

Define a function that trains the network and compute the loss

f[kerSize_] := Module[{trained, loss},
  loss = 
   NetTrain[net[kerSize], trainingData, "ValidationLoss", 
    ValidationSet -> testData, BatchSize -> 2048, 
    MaxTrainingRounds -> 5, TargetDevice -> "GPU"];
  -loss
  ]

Optimize the kernel size using BayesianMaximization.

bo = BayesianMaximization[f, {3, 5, 7}]; // AbsoluteTiming 
(*{51.5012, Null}*)

The best kernel size is 5 according to the optimization result

bo["MaximumConfiguration"]
(*5*)