# How to Classify with LogisticRegression with a non linear border?

I am currently taking the Coursera Machine Learning course and want to reproduce some of the coding exercises in Mathematica. In Exercise 2 the problem is to Classify with Logistic Regression. So I tried to use Classify with Method -> "LogisticRegression".

Getting the data:

dataraw =
Import["https://raw.githubusercontent.com/anirudhjayaraman/Machine-\
Learning/master/Andrew%20Ng%20Stanford%20Coursera/Week%2003/ex2/\
ex2data2.txt", "CSV"];
X = dataraw[[All, 1 ;; 2]];
y = dataraw[[All, 3]];


Using Classify with LogisticRegression and Automatic settings:

cflogistic = Classify[data, Method -> {"LogisticRegression"}];
cfauto = Classify[data];


Plotting:

onecases = Cases[data, HoldPattern[__ -> 1]];
zeorcases = Cases[data, HoldPattern[__ -> 0]];
features =
ListPlot[{onecases, zeorcases},
PlotMarkers -> \
{"\!$$\*StyleBox[\"\[HappySmiley]\",FontSize->18]$$",
"\!$$\*StyleBox[\"\[SadSmiley]\",FontSize->18]$$"}];
decisionboundarylogistic =
ContourPlot[
cflogistic[{x1, x2}, "Probability" -> 1], {x1, -1, 1}, {x2, -1, 1},
Contours -> {0, 0.5, 1}, ContourShading -> False,
PlotLegends -> {"LogisticRegression"}];
decisionboundaryauto =
ContourPlot[
cfauto[{x1, x2}, "Probability" -> 1], {x1, -1, 1}, {x2, -1, 1},
Contours -> {0, 0.5, 1}, ContourShading -> False,
ContourStyle -> Blue, PlotLegends -> {"Auto"}];
Show[decisionboundarylogistic, decisionboundaryauto, features]


So the automatic one is overfitted and the Logistic Regression is under fitted. Now I would like to adapt logistic regression to use higher order terms. In order to have a boundary like this:

But the documentation of LogisticRegression says: "Models class probabilities with logistic functions of linear combinations of features." Also if I look closer into the classifier function it uses a LinearLayer.

Isn't the goal of Wolfram to have high level functions but if you need you can adjust it do your needs. Now how would I adjust Classify to use LogisticRegression with higher order polynomials and get a border similar to the second image?

• One can certainly use higher order polynomials. A logistic regression and a linear regression are "linear" in the coefficients and not necessarily linear in the predictors. As such one can introduce higher order polynomials. Here is an example: predictors = {#[[1]], #[[1]]^2, #[[2]], #[[2]]^2, #[[1]] #[[2]]} & /@ X; data2 = Thread[predictors -> y]; cflogistic = Classify[data2, Method -> {"LogisticRegression"}]. – JimB Sep 28 '18 at 16:57
• @JimB Ok thats good to know, but this seems a bit hacky to alter the data, but it is a start. – gogoolplex Sep 28 '18 at 17:16
• Yes, it would be nice if a formula would be allowed (as in NonlinearModelFit - which would allow nonlinear models) but that's exactly what one does for LinearModelFit, LogisticModelFit, and GeneralizedLinearModelFit. So it is at least consistent with most other fitting functions. – JimB Sep 28 '18 at 17:24
• Ok, I need to look into that, and also how python libraries handle that. – gogoolplex Sep 28 '18 at 17:28
• Big corrections: It is LogitModelFit rather than LogisticModelFit and one doesn't need to augment the data for LinearModelFit, LogitModelFit, and GeneralizedLinearModelFit. Not sure why that wrong idea was in my head. Sorry about that. – JimB Sep 28 '18 at 18:27

Here's how to do it using Classify:

X = dataraw[[All, 1 ;; 2]];
y = dataraw[[All, 3]];
data = Flatten[#] & /@ Transpose[{X, y}];
data1 = Select[data, #[[3]] == 1 &];
data0 = Select[data, #[[3]] == 0 &];

XX = {#[[1]], Exp[#[[1]]^2], #[[2]],
Exp[#[[2]]^2], #[[1]] #[[2]]} & /@ X;
cflogistic = Classify[data2, Method -> {"LogisticRegression"}];
decisionboundarylogistic =
ContourPlot[
cflogistic[{x1, Exp[x1^2], x2, Exp[x2^2], x1 x2},
"Probability" -> 1], {x1, -1, 1}, {x2, -1, 1},
Contours -> {0, 0.5, 1}, ContourShading -> False,
PlotLegends -> {"LogisticRegression"}];
Show[decisionboundarylogistic,
ListPlot[{data1[[All, {1, 2}]], data0[[All, {1, 2}]]},


It appears that one must eponentiate each even-numbered predictor value to get the expected logistic regression. (If I have some time in the near future, I'll see if I can pin that down. Don't know if that might be a bug or an undocumented feature.)

• I accept this answer because it uses Classify, which was my original question. I like your answer but, not how Wolfram has implemented this in Mathematica, and also the Exp is strange. This also shows, that automatic choosing the "best" algorithm is quite limited in Classify when logistic regression is only considered in the most simple case where the boundary is a line. – gogoolplex Oct 1 '18 at 11:12

Here is one approach.

dataraw =
Import["https://raw.githubusercontent.com/anirudhjayaraman/Machine-Learning/master/Andrew%20Ng%20Stanford%20Coursera/Week%2003/ex2/ex2data2.txt", "CSV"];
X = dataraw[[All, 1 ;; 2]];
y = dataraw[[All, 3]];
data = Flatten[#] & /@ Transpose[{X, y}];
lr = LogitModelFit[data, {x1, x1^2, x2, x2^2, x1 x2}, {x1, x2}];

data1 = Select[data, #[[3]] == 1 &];
data0 = Select[data, #[[3]] == 0 &];
Show[ContourPlot[lr[x1, x2], {x1, -1, 1.3}, {x2, -1, 1.3},
Contours -> {Length[data1]/(Length[data1] + Length[data0])},
ContourShading -> None, ContourStyle -> Thick],
ListPlot[{data1[[All, {1, 2}]], data0[[All, {1, 2}]]},
PlotStyle -> {Green, Red}, PlotLegends -> {"1", "0"}]]