# FindMinimum produces error message

I'm trying to solve a logistic regression problem using Mathematica -- more to improve my Mathematica skills than to solve the problem as I have already solved it using Octave.

The problem I am experiencing is with the FindMinimum function which always returns an error no matter what I do. I have tried it without passing the gradient function and it produces the error:

Encountered a gradient that is effectively zero.

If I pass it the gradient function, I get the error:

The gradient is not a vector of real numbers at {θ} = {{{0.}, {0.}, {0.}}}.

I've read through other posts with similar problems but none of recommended solutions work for me.

Below is the code from my Notebook (I hope you can copy and paste it into your own notebook).

data = {{34.62365962451697, 78.0246928153624, 0}, {30.28671076822607,
43.89499752400101, 0}, {35.84740876993872, 72.90219802708364,
0}, {60.18259938620976, 86.30855209546826, 1}, {79.0327360507101,
75.3443764369103, 1}, {45.08327747668339, 56.3163717815305,
0}, {61.10666453684766, 96.51142588489624, 1}, {75.02474556738889,
46.55401354116538, 1}, {76.09878670226257, 87.42056971926803,
1}, {84.43281996120035, 43.53339331072109, 1}, {95.86155507093572,
38.22527805795094, 0}, {75.01365838958247, 30.60326323428011,
0}, {82.30705337399482, 76.48196330235604, 1}, {69.36458875970939,
97.71869196188608, 1}, {39.53833914367223, 76.03681085115882,
0}, {53.9710521485623, 89.20735013750205, 1}, {69.07014406283025,
52.74046973016765, 1}, {67.94685547711617, 46.67857410673128,
0}, {70.66150955499435, 92.92713789364831, 1}, {76.97878372747498,
47.57596364975532, 1}, {67.37202754570876, 42.83843832029179,
0}, {89.6767757507208, 65.79936592745237, 1}, {50.534788289883,
48.85581152764205, 0}, {34.21206097786789, 44.20952859866288,
0}, {77.9240914545704, 68.9723599933059, 1}, {62.27101367004632,
69.95445795447587, 1}, {80.1901807509566, 44.82162893218353,
1}, {93.114388797442, 38.80067033713209, 0}, {61.83020602312595,
50.25610789244621, 0}, {38.78580379679423, 64.99568095539578, 0}};


Break data into correct matricies

myX = Take[data, All, 2];
myy = Take[data, All, -1];


Categorize rows as 0 or 1

posRows = Flatten[Position[myy, {1}]];
negRows = Flatten[Position[myy, {0}]];


Plot the dataset

resultsPlot =
ListPlot[{Partition[Riffle[myX[[posRows, 1]], myX[[posRows, 2]]], 2],
Partition[Riffle[myX[[negRows, 1]], myX[[negRows, 2]]], 2]},
PlotMarkers -> {"X", "O"}, PlotLegends -> {"Positive", "Negative"},
Frame -> True]


Fill out the X matrix by prepending a column of 1's

myX = PadLeft[myX, {Length[myX], 3}, 1];


Create the working functions

Sigmoid function - used to ensure we have a convex function with no local minima

sigmoid[mat_] := 1 /(1 + E^-mat);


Cost function

cost[θ_, X_, y_] := Module[{m, hThetaX},
m = Length[y];
hThetaX = sigmoid[X.θ];
Flatten[1/
m*(-y\[Transpose].Log[hThetaX] - (1 - y)\[Transpose].Log[1 - hThetaX])]
]


grad[θ_, X_, y_] := Module[{m, hThetaX},
m = Length[y];
hThetaX = sigmoid[X.θ];
Flatten[1/m*(hThetaX - y)\[Transpose].X]
]


Test the functions

thetaInitial = {{0}, {0}, {0}}

cost[thetaInitial, myX, myy][[1]]

Out[105]= 0.693147

Out[106]= {-0.0333333, -6.63738, -6.82055}


These are the expected results.

Find the values of theta that minimise the cost

FindMinimum[cost[θ, myX, myy][[1]], {θ, thetaInitial}, Gradient -> grad[θ, myX, myy]]


During evaluation of In[108]:= FindMinimum::nrgnum: The gradient is not a vector of real numbers at {θ} = {{{0.},{0.},{0.}}}. >>

Out[108]= {0.0333333, {θ -> {{0.}, {0.}, {0.}}}}


----- EDIT -----

As per requests, I've added the full notebook below in one block to make it easier to copy and paste for testing.

data = {{34.62365962451697, 78.0246928153624, 0}, {30.28671076822607,
43.89499752400101, 0}, {35.84740876993872, 72.90219802708364,
0}, {60.18259938620976, 86.30855209546826, 1}, {79.0327360507101,
75.3443764369103, 1}, {45.08327747668339, 56.3163717815305,
0}, {61.10666453684766, 96.51142588489624, 1}, {75.02474556738889,
46.55401354116538, 1}, {76.09878670226257, 87.42056971926803,
1}, {84.43281996120035, 43.53339331072109, 1}, {95.86155507093572,
38.22527805795094, 0}, {75.01365838958247, 30.60326323428011,
0}, {82.30705337399482, 76.48196330235604, 1}, {69.36458875970939,
97.71869196188608, 1}, {39.53833914367223, 76.03681085115882,
0}, {53.9710521485623, 89.20735013750205, 1}, {69.07014406283025,
52.74046973016765, 1}, {67.94685547711617, 46.67857410673128,
0}, {70.66150955499435, 92.92713789364831, 1}, {76.97878372747498,
47.57596364975532, 1}, {67.37202754570876, 42.83843832029179,
0}, {89.6767757507208, 65.79936592745237, 1}, {50.534788289883,
48.85581152764205, 0}, {34.21206097786789, 44.20952859866288,
0}, {77.9240914545704, 68.9723599933059, 1}, {62.27101367004632,
69.95445795447587, 1}, {80.1901807509566, 44.82162893218353,
1}, {93.114388797442, 38.80067033713209, 0}, {61.83020602312595,
50.25610789244621, 0}, {38.78580379679423, 64.99568095539578, 0}};

myX = Take[data, All, 2];
myy = Take[data, All, -1];

posRows = Flatten[Position[myy, {1}]];
negRows = Flatten[Position[myy, {0}]];

resultsPlot =
ListPlot[{Partition[Riffle[myX[[posRows, 1]], myX[[posRows, 2]]], 2],
Partition[Riffle[myX[[negRows, 1]], myX[[negRows, 2]]], 2]},
PlotMarkers -> {"X", "O"}, PlotLegends -> {"Positive", "Negative"},
Frame -> True]

myX = PadLeft[myX, {Length[myX], 3}, 1];

sigmoid[mat_] := 1 /(1 + E^-mat);

cost[\[Theta]_, X_, y_] := Module[{m, hThetaX},
m = Length[y];
hThetaX = sigmoid[X.\[Theta]];
Flatten[1/
m*(-y\[Transpose].Log[hThetaX] - (1 - y)\[Transpose].Log[1 - hThetaX])]
]

grad[\[Theta]_, X_, y_] := Module[{m, hThetaX},
m = Length[y];
hThetaX = sigmoid[X.\[Theta]];
Flatten[1/m*(hThetaX - y)\[Transpose].X]
]

thetaInitial = {0, 0, 0}

cost[thetaInitial, myX, myy][[1]]

FindMinimum[cost[\[Theta], myX, myy][[1]], {\[Theta], thetaInitial},

• Testing your code requires a lot of copy/pasting. Perhaps it would be a good idea to repeat the full code in just one code box at the end of your question Nov 14, 2013 at 14:24
• Why do you use thetaInitial = {{0}, {0}, {0}} rather than thetaInitial = {0, 0, 0}? Nov 14, 2013 at 14:35
• It is not necessary to differentiate between row vectors and column vectors in Mathematica. Octave and MATLAB do not have real vectors, only matrices, so with those systems it is a necessaity to be aware whether something is a row vector or a column vector. Mathematica can handle arbitrary dimensional tensors, so this is not necessary. Treating vectors as simple vectors will save you some transpositions and flattenings here, and will improve readability. Nov 14, 2013 at 16:17
• Theta is a vector? I dont think FindMinimum will work with a vector unknown, you need to give it the components explicitly eg {{theta1,0},{theta2,0},{theta3,0}} .. Nov 14, 2013 at 22:48
• Ok, correct myself. You CAN minimize over a vector..however you need to make sure your function doesn't evaluate for a scalar (which yours does i think). Put theta_List in your cost function definition (first Clear[cost]). Nov 14, 2013 at 22:59

## Vector-valued variable's input syntax for FindMinimum

The Documentation states (emphasis mine):

<...> since the value of the function would be meaningless unless x had the correct structure, the definition is restricted to arguments with that structure. For example, if you defined the function for any pattern x_, then evaluating with an undefined symbol x (which is what FindMinimum does) gives meaningless unintended results. It is often the case that when working with functions for vector-valued variables, you will have to restrict the definitions.

The main problem with your code is that your cost function and its gradient are defined for any pattern ϴ_ and give unintended results when evaluated with undefined ϴ. You should restrict them to be evaluated only for the values of the vector variable ϴ with correct structure.

## Solution for your particular problem

As george2079 correctly stated in the comments you should put ϴ_List in the cost function and in the gradient function and make the cost function returning a scalar (note First added to the definition):

Clear[cost, grad]
cost[θ_List, X_, y_] := First@Module[{m, hThetaX}, m = Length[y];
hThetaX = sigmoid[X.θ];
Flatten[1/m*(-y\[Transpose].Log[hThetaX] - (1 - y)\[Transpose].Log[1 - hThetaX])]]
grad[θ_List, X_, y_] := Module[{m, hThetaX}, m = Length[y];
hThetaX = sigmoid[X.θ];
Flatten[1/m*(hThetaX - y)\[Transpose].X]]


Now using the "Newton" (as well as Gradient-free "PrincipalAxis") method everything work nicely:

FindMinimum[cost[θ, myX, myy], {θ, thetaInitial},