8
$\begingroup$

I'd like to know if one can use optimization function like Maximize[] or BayesianMaximize[] to optimize the hyper-parameters of a NetGraph?

Specifically, I want to optimize the kernel size, the number of hidden units for each of the convolutional layers, and the number of hidden units in the fully connected layer (for mnist task for instance).

$\endgroup$

1 Answer 1

4
$\begingroup$

In principle you can wrap a neural network inside a function and then optimize the hyper-parameters using the optimization functions. Here is an example of using BayesianMaximize to optimize the kernel size of one of the convolution layer.

Loading the MNIST data

trainingData = ResourceData["MNIST", "TrainingData"];
testData = ResourceData["MNIST", "TestData"];

Construct network with one of the kernel size as parameter

net[kerSize_] := NetChain[{
   ConvolutionLayer[20, {kerSize, kerSize}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   ConvolutionLayer[50, {5, 5}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   FlattenLayer[],
   LinearLayer[500],
   ElementwiseLayer[Ramp],
   LinearLayer[10],
   SoftmaxLayer[]
   },
  "Output" -> NetDecoder[{"Class", Range[0, 9]}],
  "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]
  ]

Define a function that trains the network and compute the loss

f[kerSize_] := Module[{trained, loss},
  loss = 
   NetTrain[net[kerSize], trainingData, "ValidationLoss", 
    ValidationSet -> testData, BatchSize -> 2048, 
    MaxTrainingRounds -> 5, TargetDevice -> "GPU"];
  -loss
  ]

Optimize the kernel size using BayesianMaximization.

bo = BayesianMaximization[f, {3, 5, 7}]; // AbsoluteTiming 
(*{51.5012, Null}*)

The best kernel size is 5 according to the optimization result

bo["MaximumConfiguration"]
(*5*)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.