2
$\begingroup$

I'm trying to solve a logistic regression problem using Mathematica -- more to improve my Mathematica skills than to solve the problem as I have already solved it using Octave.

The problem I am experiencing is with the FindMinimum function which always returns an error no matter what I do. I have tried it without passing the gradient function and it produces the error:

Encountered a gradient that is effectively zero.

If I pass it the gradient function, I get the error:

The gradient is not a vector of real numbers at {θ} = {{{0.}, {0.}, {0.}}}.

I've read through other posts with similar problems but none of recommended solutions work for me.

Below is the code from my Notebook (I hope you can copy and paste it into your own notebook).

data = {{34.62365962451697`, 78.0246928153624`, 0}, {30.28671076822607`, 
    43.89499752400101`, 0}, {35.84740876993872`, 72.90219802708364`, 
    0}, {60.18259938620976`, 86.30855209546826`, 1}, {79.0327360507101`, 
    75.3443764369103`, 1}, {45.08327747668339`, 56.3163717815305`, 
    0}, {61.10666453684766`, 96.51142588489624`, 1}, {75.02474556738889`, 
    46.55401354116538`, 1}, {76.09878670226257`, 87.42056971926803`, 
    1}, {84.43281996120035`, 43.53339331072109`, 1}, {95.86155507093572`, 
    38.22527805795094`, 0}, {75.01365838958247`, 30.60326323428011`, 
    0}, {82.30705337399482`, 76.48196330235604`, 1}, {69.36458875970939`, 
    97.71869196188608`, 1}, {39.53833914367223`, 76.03681085115882`, 
    0}, {53.9710521485623`, 89.20735013750205`, 1}, {69.07014406283025`, 
    52.74046973016765`, 1}, {67.94685547711617`, 46.67857410673128`, 
    0}, {70.66150955499435`, 92.92713789364831`, 1}, {76.97878372747498`, 
    47.57596364975532`, 1}, {67.37202754570876`, 42.83843832029179`, 
    0}, {89.6767757507208`, 65.79936592745237`, 1}, {50.534788289883`, 
    48.85581152764205`, 0}, {34.21206097786789`, 44.20952859866288`, 
    0}, {77.9240914545704`, 68.9723599933059`, 1}, {62.27101367004632`, 
    69.95445795447587`, 1}, {80.1901807509566`, 44.82162893218353`, 
    1}, {93.114388797442`, 38.80067033713209`, 0}, {61.83020602312595`, 
    50.25610789244621`, 0}, {38.78580379679423`, 64.99568095539578`, 0}};

Break data into correct matricies

myX = Take[data, All, 2];
myy = Take[data, All, -1];

Categorize rows as 0 or 1

posRows = Flatten[Position[myy, {1}]];
negRows = Flatten[Position[myy, {0}]];

Plot the dataset

resultsPlot =
 ListPlot[{Partition[Riffle[myX[[posRows, 1]], myX[[posRows, 2]]], 2], 
   Partition[Riffle[myX[[negRows, 1]], myX[[negRows, 2]]], 2]}, 
  PlotMarkers -> {"X", "O"}, PlotLegends -> {"Positive", "Negative"}, 
  Frame -> True]

Fill out the X matrix by prepending a column of 1's

myX = PadLeft[myX, {Length[myX], 3}, 1];

Create the working functions

Sigmoid function - used to ensure we have a convex function with no local minima

sigmoid[mat_] := 1 /(1 + E^-mat);

Cost function

cost[θ_, X_, y_] := Module[{m, hThetaX},
  m = Length[y];
  hThetaX = sigmoid[X.θ];
  Flatten[1/
     m*(-y\[Transpose].Log[hThetaX] - (1 - y)\[Transpose].Log[1 - hThetaX])]
  ]

Gradient function

grad[θ_, X_, y_] := Module[{m, hThetaX},
  m = Length[y];
  hThetaX = sigmoid[X.θ];
  Flatten[1/m*(hThetaX - y)\[Transpose].X]
  ]

Test the functions

thetaInitial = {{0}, {0}, {0}}

cost[thetaInitial, myX, myy][[1]]
grad[thetaInitial, myX, myy]
Out[105]= 0.693147

Out[106]= {-0.0333333, -6.63738, -6.82055}

These are the expected results.

Find the values of theta that minimise the cost

FindMinimum[cost[θ, myX, myy][[1]], {θ, thetaInitial}, Gradient -> grad[θ, myX, myy]]

During evaluation of In[108]:= FindMinimum::nrgnum: The gradient is not a vector of real numbers at {θ} = {{{0.},{0.},{0.}}}. >>

Out[108]= {0.0333333, {θ -> {{0.}, {0.}, {0.}}}}

----- EDIT -----

As per requests, I've added the full notebook below in one block to make it easier to copy and paste for testing.

data = {{34.62365962451697`, 78.0246928153624`, 0}, {30.28671076822607`, 
    43.89499752400101`, 0}, {35.84740876993872`, 72.90219802708364`, 
    0}, {60.18259938620976`, 86.30855209546826`, 1}, {79.0327360507101`, 
    75.3443764369103`, 1}, {45.08327747668339`, 56.3163717815305`, 
    0}, {61.10666453684766`, 96.51142588489624`, 1}, {75.02474556738889`, 
    46.55401354116538`, 1}, {76.09878670226257`, 87.42056971926803`, 
    1}, {84.43281996120035`, 43.53339331072109`, 1}, {95.86155507093572`, 
    38.22527805795094`, 0}, {75.01365838958247`, 30.60326323428011`, 
    0}, {82.30705337399482`, 76.48196330235604`, 1}, {69.36458875970939`, 
    97.71869196188608`, 1}, {39.53833914367223`, 76.03681085115882`, 
    0}, {53.9710521485623`, 89.20735013750205`, 1}, {69.07014406283025`, 
    52.74046973016765`, 1}, {67.94685547711617`, 46.67857410673128`, 
    0}, {70.66150955499435`, 92.92713789364831`, 1}, {76.97878372747498`, 
    47.57596364975532`, 1}, {67.37202754570876`, 42.83843832029179`, 
    0}, {89.6767757507208`, 65.79936592745237`, 1}, {50.534788289883`, 
    48.85581152764205`, 0}, {34.21206097786789`, 44.20952859866288`, 
    0}, {77.9240914545704`, 68.9723599933059`, 1}, {62.27101367004632`, 
    69.95445795447587`, 1}, {80.1901807509566`, 44.82162893218353`, 
    1}, {93.114388797442`, 38.80067033713209`, 0}, {61.83020602312595`, 
    50.25610789244621`, 0}, {38.78580379679423`, 64.99568095539578`, 0}};

myX = Take[data, All, 2];
myy = Take[data, All, -1];

posRows = Flatten[Position[myy, {1}]];
negRows = Flatten[Position[myy, {0}]];

resultsPlot =
 ListPlot[{Partition[Riffle[myX[[posRows, 1]], myX[[posRows, 2]]], 2], 
   Partition[Riffle[myX[[negRows, 1]], myX[[negRows, 2]]], 2]}, 
  PlotMarkers -> {"X", "O"}, PlotLegends -> {"Positive", "Negative"}, 
  Frame -> True]

myX = PadLeft[myX, {Length[myX], 3}, 1];

sigmoid[mat_] := 1 /(1 + E^-mat);

cost[\[Theta]_, X_, y_] := Module[{m, hThetaX},
  m = Length[y];
  hThetaX = sigmoid[X.\[Theta]];
  Flatten[1/
     m*(-y\[Transpose].Log[hThetaX] - (1 - y)\[Transpose].Log[1 - hThetaX])]
  ]

grad[\[Theta]_, X_, y_] := Module[{m, hThetaX},
  m = Length[y];
  hThetaX = sigmoid[X.\[Theta]];
  Flatten[1/m*(hThetaX - y)\[Transpose].X]
  ]

thetaInitial = {0, 0, 0}

cost[thetaInitial, myX, myy][[1]]
grad[thetaInitial, myX, myy]


FindMinimum[cost[\[Theta], myX, myy][[1]], {\[Theta], thetaInitial}, 
 Gradient -> grad[\[Theta], myX, myy]]
$\endgroup$
  • $\begingroup$ Testing your code requires a lot of copy/pasting. Perhaps it would be a good idea to repeat the full code in just one code box at the end of your question $\endgroup$ – Dr. belisarius Nov 14 '13 at 14:24
  • $\begingroup$ Why do you use thetaInitial = {{0}, {0}, {0}} rather than thetaInitial = {0, 0, 0}? $\endgroup$ – m_goldberg Nov 14 '13 at 14:35
  • 1
    $\begingroup$ It is not necessary to differentiate between row vectors and column vectors in Mathematica. Octave and MATLAB do not have real vectors, only matrices, so with those systems it is a necessaity to be aware whether something is a row vector or a column vector. Mathematica can handle arbitrary dimensional tensors, so this is not necessary. Treating vectors as simple vectors will save you some transpositions and flattenings here, and will improve readability. $\endgroup$ – Szabolcs Nov 14 '13 at 16:17
  • 1
    $\begingroup$ Theta is a vector? I dont think FindMinimum will work with a vector unknown, you need to give it the components explicitly eg {{theta1,0},{theta2,0},{theta3,0}} .. $\endgroup$ – george2079 Nov 14 '13 at 22:48
  • 2
    $\begingroup$ Ok, correct myself. You CAN minimize over a vector..however you need to make sure your function doesn't evaluate for a scalar (which yours does i think). Put theta_List in your cost function definition (first Clear[cost]). $\endgroup$ – george2079 Nov 14 '13 at 22:59
6
$\begingroup$

Vector-valued variable's input syntax for FindMinimum

The Documentation states (emphasis mine):

<...> since the value of the function would be meaningless unless x had the correct structure, the definition is restricted to arguments with that structure. For example, if you defined the function for any pattern x_, then evaluating with an undefined symbol x (which is what FindMinimum does) gives meaningless unintended results. It is often the case that when working with functions for vector-valued variables, you will have to restrict the definitions.

The main problem with your code is that your cost function and its gradient are defined for any pattern ϴ_ and give unintended results when evaluated with undefined ϴ. You should restrict them to be evaluated only for the values of the vector variable ϴ with correct structure.

Solution for your particular problem

As george2079 correctly stated in the comments you should put ϴ_List in the cost function and in the gradient function and make the cost function returning a scalar (note First added to the definition):

Clear[cost, grad]
cost[θ_List, X_, y_] := First@Module[{m, hThetaX}, m = Length[y];
   hThetaX = sigmoid[X.θ];
   Flatten[1/m*(-y\[Transpose].Log[hThetaX] - (1 - y)\[Transpose].Log[1 - hThetaX])]]
grad[θ_List, X_, y_] := Module[{m, hThetaX}, m = Length[y];
  hThetaX = sigmoid[X.θ];
  Flatten[1/m*(hThetaX - y)\[Transpose].X]]

Now using the "Newton" (as well as Gradient-free "PrincipalAxis") method everything work nicely:

FindMinimum[cost[θ, myX, myy], {θ, thetaInitial}, 
 Gradient -> grad[θ, myX, myy], Method -> "Newton"]

{0.269, {[Theta] -> {-22.6096, 0.173579, 0.196086}}}

$\endgroup$
  • $\begingroup$ Thanks Alexey and George for pointing the way and thanks to belisarius, m_goldberg and Szabolcs for helping me to frame things to make it easier to get help. Alexey correctly and comprehensively described the solution to my problem - there is absolutely no way I would have come to this solution on my own - thankyou. $\endgroup$ – CrustyNoodle Nov 15 '13 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.