There’s an example in the docs for BayesianMaximization for optimizing two suboptions of a classifier with a fixed method. I would like to optimize over multiple options for Method and their various sub-options simultaneously. But it seems there’s no way to give BayesianMaximization configurations that are interdependent, of mixed types, and/or hierarchical.

In the documentation there is an example of maximizing the log-likelihood function over a domain defined by a list of different methods for Predict:

trainingset = Table[n -> Sin[n], {n, 1, 15}];
testset = {3.9 -> Sin[3.9], 7.3 -> Sin[7.3]};
func[method_] :=
PredictorMeasurements[
Predict[trainingset, Method -> {method (* methodSubOptions here! *) }], testset,
"LogLikelihood"];
methods = {"LinearRegression", "NearestNeighbors", "NeuralNetwork",
"GaussianProcess"};
bo = BayesianMaximization[func, methods]


In the above, how would I add optimization over methods and their specific sub-options together? For example, for Method -> "LogisticRegression" I would like to optimize "L1Regularization" and "L1Regularization" parameters, but for Method -> "RandomForest" I want to optimize "TreeNumber" and "LeafSize".

You can pass a function that will be called to provide samples, and define it hierarchically so that it randomly chooses configurations with the desired parameters. Small example:

trainingset = Table[n -> Sin[n], {n, 1, 15}];

testset = {3.9 -> Sin[3.9], 7.3 -> Sin[7.3]};

ClearAll[randomForestSample];
randomForestSample[] := {"RandomForest",
"TreeNumber" -> RandomInteger[5], "LeafSize" -> RandomInteger[5]}
ClearAll[linearRegressionSample];
linearRegressionSample[] := {"LinearRegression",
"L1Regularization" -> RandomReal[1],
"L2Regularization" -> RandomReal[1]}
ClearAll[sampleConfiguration];
sampleConfiguration[] :=
RandomChoice[{linearRegressionSample, randomForestSample}][]

maximizer =
BayesianMaximization[
PredictorMeasurements[Predict[trainingset, Method -> #1], testset,
"LogLikelihood"] &, sampleConfiguration]

maximizer["EvaluationHistory"]


I also played around with explicitly defining a region that corresponds to the parameter space you want to optimize over, but it proved too complicated. Maybe someone with a better understanding of the system for geometric regions could explain how to do this. It might work better -- I believe the Bayesian optimizer has some notion of "nearness" which might be lost by just providing random samples.

Update:

Here is an example of sampling from a region instead. The region consists of two orthogonal spaces for the four parameters in the example.

linearRegion =
RegionProduct[Point[{{0, 0}}], Point[List /@ Range[0, 1, .2]],
Point[List /@ Range[0, 1, .2]]]
treeRegion =
RegionProduct[Point[List /@ Range[5]], Point[List /@ Range[5]],
Point[{0, 0}]]
combinedRegion = RegionUnion[linearRegion, treeRegion]

ClearAll[convertToLinearParams];
convertToLinearParams[config_List] := {"LinearRegression",
"L1Regularization" -> config[[3]], "L2Regularization" -> config[[4]]}
ClearAll[convertToTreeParams];
convertToTreeParams[config_List] :=
{"RandomForest",
"TreeNumber" -> config[[1]], "LeafSize" -> config[[2]]}
ClearAll[configurationToPredictor];
configurationToPredictor[config_List] :=
Module[{resultConfig =
If[config \[Element] linearRegion, convertToLinearParams[config],
convertToTreeParams[config]]},
PredictorMeasurements[Predict[trainingset, Method -> resultConfig],
testset, "LogLikelihood"]]

regionMaximizer =
BayesianMaximization[configurationToPredictor, combinedRegion]


Note that the linear regression parameters, which should be continuous, are instead discrete. You can easily make it a continuous rectangle but then, because there are infinitely many points in a continuous rectangle, the probability of sampling a random forest point (with discrete params so finitely many points) drops to zero. I can't figure out a way around this.

linearRegion = RegionProduct[Point[{{0, 0}}], Rectangle[{0, 0}]]
treeRegion =
RegionProduct[Point[List /@ Range[5]], Point[List /@ Range[5]],
Point[{0, 0}]]
combinedRegion = RegionUnion[linearRegion, treeRegion]
(* will only ever try linear regression *)
regionMaximizer =
BayesianMaximization[configurationToPredictor, combinedRegion]

• Yes I think without being able to specify regions there's much less benefit. – M.R. Nov 16 '16 at 4:51
• Thanks for your answer. I have to agree with @M.R. The strength of the search is that it will choose the next point in some subregion. Can you adapt your code to work with intervals or rectangles etc..? – user5601 Nov 16 '16 at 6:28
• I experimented with it. – Michael Curry Nov 16 '16 at 13:19
• BayesianMaximization does seem to call the sampling function many times and only evaluate the main function for a few points. So it is still taking advantage of "nearness" to some extent. – Michael Curry Nov 16 '16 at 13:20
• I tried defining a geometric region as a RegionProduct of rectangles (for continuous parameters) and discrete point sets. I wasn't clever enough to come up with a good mapping from hyperparameters -> nice well-defined region that would also work with RandomPoint. – Michael Curry Nov 16 '16 at 13:25