# What does the value of Predict's “L2Regularization” correspond to?

What does the value supplied to the Predict Method option "L2Regularization" correspond to?

It does not seem to correspond to the conventional definition of $\lambda$ in the context of linear regression:

$$J(\mathbf{w})= \text{MSE}(\mathbf{w})+\lambda \mathbf{w}^{T}\cdot\mathbf{w}$$

For example

Predict[dmat -> y, Method->{"LinearRegression", "L2Regularization"->lambda}, PerformanceGoal->"Quality"]


does not produce the same predictions as a function generated from

Minimize[Mean[(dmat.w-y)^2] + lambda*w.w, w]


unless there's no regularization (lambda = 0).

Moreover, I can't see how to use Predict with "L2Regularization" to regularize only weights, but not bias (as is common on NN applications), as in

p=Prepend[w, b];
Minimize[Mean[(dmat.p-y)^2] + lambda*w.w, p]


where dmat is of the form DesignMatrix[...,IncludeConstantBasis->True].

What value should I use for "L2Regularization" to match the effect of lambda? What options or other changes do I need to make to have Predict only regularize weights?

(*
Generate things for an MWE
*)
x = Partition[ RandomReal[{0,10},200], 2];
f = #1-4#2+2*#1^3-50*#1 Sin[#1]Cos[#2]&;
basisFuncs = {1, x1, x2, x1^2, x2^2,  x1^4, x2^4, x1^5, x2^5,x1 Sin[x1]Cos[x2]};
weights = Symbol/@(ToString[StringForm["w",#]]&/@Range[Length[basisFuncs]-1]);
bias = Symbol@"b";
params = Prepend[weights,bias];
dmat = DesignMatrix[MapThread[Append,{x, y}],basisFuncs, {x1, x2},IncludeConstantBasis->False];

(*
With no regularization, the fit matches, though the Predict soluton has an extra, near-zero term that appears to be a side effect of the biases column in dmat.
*)

lambda=0;

Minimize[Mean[(dmat.params-y)^2] + lambda*weights.weights,params][[2]]

PredictorInformation[Predict[dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

PredictorInformation[Predict[Rest/@dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

(*
With regularization, the fits don't match, even if all params are regularized.
*)

lambda=50;

Minimize[Mean[(dmat.params-y)^2] + lambda*weights.weights,params][[2]]

PredictorInformation[Predict[dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

PredictorInformation[Predict[Rest/@dmat->y,Method->{"LinearRegression","L2Regularization"-> lambda}, PerformanceGoal->"Quality"],"Function"]

Minimize[Mean[(dmat.params-y)^2] + lambda*params.params,params][[2]]

• What's your dataset？ it's very important. – HyperGroups Aug 28 '17 at 16:21
• I see this with any dataset. With lambda = 0, Predict and Minimize produce exactly the same results (as expected). With lambda ≠ 0 Predict and Minimize give completely different results, suggesting that MMA's "L2Regularization" parameter is something distinct from $\lambda$. The question is: what is it? – orome Aug 28 '17 at 16:26
• lambda is a step in solve the solution, and if use different method, the result may be not the same. – HyperGroups Aug 28 '17 at 16:28
• @HyperGroups: Let me spell this out a bit: "L2 Regularization" has a meaning: generally when provided as an argument for an API it means something like "treat the provided value as $\lambda$ and incorporate L2-regularization into the prediction/learning/fit" or more precisely, add the second term in the equation above to the cost function, using the provided value as $\lambda$. But in MMA, "L2Regularization" clearly does not have this meaning. What does it mean? – orome Aug 28 '17 at 16:31
• But in MMA, "L2Regularization" clearly does not have this meaning. How clearly?  Maybe the L2Regularization works like Ridge or L2Regularizeation=f(lambda)?, but just the parameters scope is different from another implement? This is very normal in the ML packages. I even cann't produce your result of Minimize[Mean[(dmat.w-y)^2]+lambda*w.w,w] General::stop: Further output of NMinimize::nnum will be suppressed during this calculation. – HyperGroups Aug 28 '17 at 17:16

I think this is a partly answer.

There are some questions in my understanding.

1. How can I reproduce the result of Predict by Minimize? If your question is that, another title is better.
2. Why lambda in Minimize doesn't gives the same result in Predict?

3. Does L2Regularization in Predict work like that in lambda in Minimize?

Answer for question1: I cann't reproduce now, is it possible?

Answer for question3: My answer is Yes now, L2Regularization is corresponding to lambda.

By tune parameters of L2, the weights decay and converge, though it's not like the case in Ridge, most values decay to nearly zero. L2Regularization is so-called weight decay, though in some context, we can say L2Regularization is Ridge[from context of PRML], but that will confuse someone and think L2Regularization should give the same result like Ridge Regression in SKLearn or other implementation like Minimize.

1. Predict can use many methods to solve or train, like SVD, or Cholesky, SAG, and so on. L2 form is so called Ridge Regression, I've tested with SKLearn's Ridge, you may try it, and change the solver with sag and svd with alpha[Lambda/L2Reg] setting. And coefficients of model is not fixed and uniq, even the dataset is fixed. Bad thing is Mathematica maybe use ElasticNet or some other object functions or processing tricks with L2 norm.

2. There is no methods options, and detail method description in Minimize, so the results to compare is not rigid. When lambda is 0, they gives the same result, because they use the simplest OLSE least square methods.

3. Predict is going on a complex workflow, like encoding and data standardize and DimensionReduce and other manipulations. one example In this my example, I've spent some time to control some conditions, and find out this fact. Some methods like PCA will also need Standardize, sometimes ghost knows where and when Mathematica does it.

4. In different implements, the parameters scope is different and varying even with the same method. I've played with many ML packages, this is very normal in the ML packages. Maybe the L2Regularization works like this : L2Regularizeation=f(alpha)?, alpha is an lambda parameter in other implement.

5. Even in different object functions, we can get the same value when lambda=0.

I can reproduce SKLearn's Ridge in Mathematica by setting L2Regularizeation and normalize in some datasets. And I can reproduce Ridge in SKLearn by Minimize also.

Since OP edit the question, and add some Datasets and Minimize method to compare, so I also edit something.

In different datasets, result will be not the same, that's why I say: What's your dataset？ it's very important in doing experiments, becuase Predict treat different data with different workflows.

• In my first dataset, Mathematica's Predict doing some Normalize manipulation, and I can reproduce the result by SKLearn's Ridge with parameters: normalize=True and alpha. In OP's dataset, I can only reproduce Minimize's result by SKLearn's Ridge with normalize=False and alpha, since OP use Mean, In Ridge we can use Total.

So, I'm sensitive to OP's thought: But in MMA, "L2Regularization" clearly does not have this meaning.How clearly?

code show the reason 3

    trainingset={1,2,3,4,5,6}->{2,3,5,8,9,7};
linear=Predict[trainingset, Method->"LinearRegression"];
Options[linear][[1]]["Models"][[1]]


I have some problem reproduce the OP's result of Minimize Minimize[Mean[(dmat.w-y)^2]+lambda*w.w,w] General::stop: Further output of NMinimize::nnum will be suppressed during this calculation.

Reproduce Ridge result of SKLearn in Mathematica.

from sklearn import linear_model
reg=linear_model.Ridge(alpha=0.01, copy_X=True, fit_intercept=True, max_iter=None,
normalize=True, random_state=None, solver='svd', tol=0.001)

reg.fit(
[[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]], [0.0783993, 1.06691, 2.02407, 3.65314, 4.14089, 5.51513, 6.75203, 7.06544, 8.17167, 9.13125]
)
print(reg.coef_)
print(reg.intercept_)
[ 1.00262068]
-0.754520809274

model=Predict[dmat->y,Method->{"LinearRegression","L2Regularization"->.1},PerformanceGoal->"Quality"];
PredictorInformation[model,"Function"]
-0.754521 + 1.00262 #1 &


So here, as you see, alpha 0.01 in SKLearn's Ridge gives the same result with L2Regularization 0.1 in Mathematica.

• I'm not sure what you're saying here. Since the argument has a value, it must have some meaning. If you're just saying that sometimes Predict can produce results that don't match (the corresponding) Minimize results, that's not the problem here. With lambda = 0 the results are exactly the same. – orome Aug 28 '17 at 16:23
• You got a down vote, I think, because you seem to be answering an eatery different question. – orome Aug 28 '17 at 18:11
• @raxacoricofallapatorius OK, see edit, With lambda!=0, result can also be exactly the same. – HyperGroups Aug 29 '17 at 6:21
• Interesting. That's progress. So for sklearn.linear_model.Ridge I can get things to match by setting alpha to len(dmat) times $\lambda$ (and intercept=True and normalize=False and omitting the constant column from dmat, since, following deep learning practice, I'm not regularizing the bias). But I still can't get anything recognizable out of Predict. What's the source of the difference in the values. It's not merely 10x or multiplied by the sample size. – orome Aug 29 '17 at 17:54
• I wish the computations and model parameters weren't so obscured – M.R. Sep 7 '17 at 16:01