What does the value of Predict's "L2Regularization" correspond to?

Question

What does the value supplied to the Predict Method option "L2Regularization" correspond to?

It does not seem to correspond to the conventional definition of $\lambda$ in the context of linear regression:

$$J(\mathbf{w})= \text{MSE}(\mathbf{w})+\lambda \mathbf{w}^{T}\cdot\mathbf{w}$$

For example

Predict[dmat -> y, Method->{"LinearRegression", "L2Regularization"->lambda}, PerformanceGoal->"Quality"]

does not produce the same predictions as a function generated from

Minimize[Mean[(dmat.w-y)^2] + lambda*w.w, w]

unless there's no regularization (lambda = 0).

Moreover, I can't see how to use Predict with "L2Regularization" to regularize only weights, but not bias (as is common on NN applications), as in

p=Prepend[w, b];
Minimize[Mean[(dmat.p-y)^2] + lambda*w.w, p]

where dmat is of the form DesignMatrix[...,IncludeConstantBasis->True].

What value should I use for "L2Regularization" to match the effect of lambda? What options or other changes do I need to make to have Predict only regularize weights?

---

(* 
Generate things for an MWE 
*)
x = Partition[ RandomReal[{0,10},200], 2];  
f = #1-4#2+2*#1^3-50*#1 Sin[#1]Cos[#2]&;
y = MapThread[f ][Transpose@x]+RandomReal[{-400,400},Length@x];
basisFuncs = {1, x1, x2, x1^2, x2^2,  x1^4, x2^4, x1^5, x2^5,x1 Sin[x1]Cos[x2]};
weights = Symbol/@(ToString[StringForm["w``",#]]&/@Range[Length[basisFuncs]-1]);
bias = Symbol@"b";
params = Prepend[weights,bias];
dmat = DesignMatrix[MapThread[Append,{x, y}],basisFuncs, {x1, x2},IncludeConstantBasis->False];


(* 
With no regularization, the fit matches, though the Predict soluton has an extra, near-zero term that appears to be a side effect of the biases column in dmat.
*)

lambda=0;

Minimize[Mean[(dmat.params-y)^2] + lambda*weights.weights,params][[2]]

PredictorInformation[Predict[dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

PredictorInformation[Predict[Rest/@dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

(* 
With regularization, the fits don't match, even if all params are regularized.
*)

lambda=50; 

Minimize[Mean[(dmat.params-y)^2] + lambda*weights.weights,params][[2]]

PredictorInformation[Predict[dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

PredictorInformation[Predict[Rest/@dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

Minimize[Mean[(dmat.params-y)^2] + lambda*params.params,params][[2]]

Answer 1

I think this is a partly answer.

There are some questions in my understanding.

1. How can I reproduce the result of Predict by Minimize? If your question is that, another title is better.

2. Why lambda in Minimize doesn't gives the same result in Predict?

3. Does L2Regularization in Predict work like that in lambda in Minimize?

---

Answer for question1: I cann't reproduce now, is it possible?

Answer for question3: My answer is Yes now, L2Regularization is corresponding to lambda.

By tune parameters of L2, the weights decay and converge, though it's not like the case in Ridge, most values decay to nearly zero. L2Regularization is so-called weight decay, though in some context, we can say L2Regularization is Ridge[from context of PRML], but that will confuse someone and think L2Regularization should give the same result like Ridge Regression in SKLearn or other implementation like Minimize.

Answer for question2:

1. Predict can use many methods to solve or train, like SVD, or Cholesky, SAG, and so on. L2 form is so called Ridge Regression, I've tested with SKLearn's Ridge, you may try it, and change the solver with sag and svd with alpha[Lambda/L2Reg] setting. And coefficients of model is not fixed and uniq, even the dataset is fixed. Bad thing is Mathematica maybe use ElasticNet or some other object functions or processing tricks with L2 norm.

2. There is no methods options, and detail method description in Minimize, so the results to compare is not rigid. When lambda is 0, they gives the same result, because they use the simplest OLSE least square methods.

3. Predict is going on a complex workflow, like encoding and data standardize and DimensionReduce and other manipulations. one example In this my example, I've spent some time to control some conditions, and find out this fact. Some methods like PCA will also need Standardize, sometimes ghost knows where and when Mathematica does it.

4. In different implements, the parameters scope is different and varying even with the same method. I've played with many ML packages, this is very normal in the ML packages. Maybe the L2Regularization works like this : L2Regularizeation=f(alpha)?, alpha is an lambda parameter in other implement.

5. Even in different object functions, we can get the same value when lambda=0.

I can reproduce SKLearn's Ridge in Mathematica by setting L2Regularizeation and normalize in some datasets. And I can reproduce Ridge in SKLearn by Minimize also.

Since OP edit the question, and add some Datasets and Minimize method to compare, so I also edit something.

In different datasets, result will be not the same, that's why I say: What's your dataset？ it's very important in doing experiments, becuase Predict treat different data with different workflows.

• In my first dataset, Mathematica's Predict doing some Normalize manipulation, and I can reproduce the result by SKLearn's Ridge with parameters: normalize=True and alpha. In OP's dataset, I can only reproduce Minimize's result by SKLearn's Ridge with normalize=False and alpha, since OP use Mean, In Ridge we can use Total.

So, I'm sensitive to OP's thought: But in MMA, "L2Regularization" clearly does not have this meaning.How clearly?

---

code show the reason 3

    trainingset={1,2,3,4,5,6}->{2,3,5,8,9,7};
    linear=Predict[trainingset, Method->"LinearRegression"];
    Options[linear][[1]]["Models"][[1]]

---

I have some problem reproduce the OP's result of Minimize Minimize[Mean[(dmat.w-y)^2]+lambda*w.w,w] General::stop: Further output of NMinimize::nnum will be suppressed during this calculation.

---

Reproduce Ridge result of SKLearn in Mathematica.

from sklearn import linear_model
reg=linear_model.Ridge(alpha=0.01, copy_X=True, fit_intercept=True, max_iter=None,
normalize=True, random_state=None, solver='svd', tol=0.001)

reg.fit(
[[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]], [0.0783993, 1.06691, 2.02407, 3.65314, 4.14089, 5.51513, 6.75203, 7.06544, 8.17167, 9.13125]
)
print(reg.coef_)
print(reg.intercept_)
[ 1.00262068]
-0.754520809274

model=Predict[dmat->y,Method->{"LinearRegression","L2Regularization"->.1},PerformanceGoal->"Quality"];
PredictorInformation[model,"Function"]
-0.754521 + 1.00262 #1 &

So here, as you see, alpha 0.01 in SKLearn's Ridge gives the same result with L2Regularization 0.1 in Mathematica.