There’s an example in the docs for BayesianMaximization for optimizing two suboptions of a classifier with a fixed method. I would like to optimize over multiple options for Method and their various sub-options simultaneously. But it seems there’s no way to give BayesianMaximization configurations that are interdependent, of mixed types, and/or hierarchical.

In the documentation there is an example of maximizing the log-likelihood function over a domain defined by a list of different methods for Predict:

trainingset = Table[n -> Sin[n], {n, 1, 15}];
testset = {3.9 -> Sin[3.9], 7.3 -> Sin[7.3]};
func[method_] := 
   Predict[trainingset, Method -> {method (* methodSubOptions here! *) }], testset, 
methods = {"LinearRegression", "NearestNeighbors", "NeuralNetwork", 
bo = BayesianMaximization[func, methods]

In the above, how would I add optimization over methods and their specific sub-options together? For example, for Method -> "LogisticRegression" I would like to optimize "L1Regularization" and "L1Regularization" parameters, but for Method -> "RandomForest" I want to optimize "TreeNumber" and "LeafSize".


You can pass a function that will be called to provide samples, and define it hierarchically so that it randomly chooses configurations with the desired parameters. Small example:

trainingset = Table[n -> Sin[n], {n, 1, 15}];

testset = {3.9 -> Sin[3.9], 7.3 -> Sin[7.3]};

randomForestSample[] := {"RandomForest", 
  "TreeNumber" -> RandomInteger[5], "LeafSize" -> RandomInteger[5]}
linearRegressionSample[] := {"LinearRegression", 
  "L1Regularization" -> RandomReal[1], 
  "L2Regularization" -> RandomReal[1]}
sampleConfiguration[] := 
 RandomChoice[{linearRegressionSample, randomForestSample}][]

maximizer = 
  PredictorMeasurements[Predict[trainingset, Method -> #1], testset, 
    "LogLikelihood"] &, sampleConfiguration]


I also played around with explicitly defining a region that corresponds to the parameter space you want to optimize over, but it proved too complicated. Maybe someone with a better understanding of the system for geometric regions could explain how to do this. It might work better -- I believe the Bayesian optimizer has some notion of "nearness" which might be lost by just providing random samples.


Here is an example of sampling from a region instead. The region consists of two orthogonal spaces for the four parameters in the example.

linearRegion = 
 RegionProduct[Point[{{0, 0}}], Point[List /@ Range[0, 1, .2]], 
  Point[List /@ Range[0, 1, .2]]]
treeRegion = 
 RegionProduct[Point[List /@ Range[5]], Point[List /@ Range[5]], 
  Point[{0, 0}]]
combinedRegion = RegionUnion[linearRegion, treeRegion]

convertToLinearParams[config_List] := {"LinearRegression", 
  "L1Regularization" -> config[[3]], "L2Regularization" -> config[[4]]}
convertToTreeParams[config_List] :=
  "TreeNumber" -> config[[1]], "LeafSize" -> config[[2]]}
configurationToPredictor[config_List] := 
 Module[{resultConfig = 
    If[config \[Element] linearRegion, convertToLinearParams[config], 
  PredictorMeasurements[Predict[trainingset, Method -> resultConfig], 
   testset, "LogLikelihood"]]

regionMaximizer = 
 BayesianMaximization[configurationToPredictor, combinedRegion]

Note that the linear regression parameters, which should be continuous, are instead discrete. You can easily make it a continuous rectangle but then, because there are infinitely many points in a continuous rectangle, the probability of sampling a random forest point (with discrete params so finitely many points) drops to zero. I can't figure out a way around this.

linearRegion = RegionProduct[Point[{{0, 0}}], Rectangle[{0, 0}]]
treeRegion = 
 RegionProduct[Point[List /@ Range[5]], Point[List /@ Range[5]], 
  Point[{0, 0}]]
combinedRegion = RegionUnion[linearRegion, treeRegion]
(* will only ever try linear regression *)
regionMaximizer = 
 BayesianMaximization[configurationToPredictor, combinedRegion]
  • $\begingroup$ Yes I think without being able to specify regions there's much less benefit. $\endgroup$ – M.R. Nov 16 '16 at 4:51
  • $\begingroup$ Thanks for your answer. I have to agree with @M.R. The strength of the search is that it will choose the next point in some subregion. Can you adapt your code to work with intervals or rectangles etc..? $\endgroup$ – user5601 Nov 16 '16 at 6:28
  • $\begingroup$ I experimented with it. $\endgroup$ – Michael Curry Nov 16 '16 at 13:19
  • $\begingroup$ BayesianMaximization does seem to call the sampling function many times and only evaluate the main function for a few points. So it is still taking advantage of "nearness" to some extent. $\endgroup$ – Michael Curry Nov 16 '16 at 13:20
  • $\begingroup$ I tried defining a geometric region as a RegionProduct of rectangles (for continuous parameters) and discrete point sets. I wasn't clever enough to come up with a good mapping from hyperparameters -> nice well-defined region that would also work with RandomPoint. $\endgroup$ – Michael Curry Nov 16 '16 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.