Design matrix rank does not match

Question

I'm attempting to fit a linear model and Mathematica returns the error message: "The rank of the design matrix 9 is less than the number of terms 10 in the model. The model and results based upon it may contain significant numerical error."

Basically I imported a .csv file w/ my data. 10 columns, 6199 rows. My dependent variable is in the last column of the .csv. Is my LinearModelFit the proper syntax?

dat = Import["/Users/Desktop/Final/data.csv"];

Dimensions[dat]


(* {6199, 10} *)

dat1 = Transpose[dat]

Dimensions[dat1]


(* {10, 6199} *)

reg = LinearModelFit[dat, {x1, x2, x3, x4, x5, x6, x7, x8, x9}, {x1, x2, x3, x4, x5, x6, x7, x8, x9}]

During evaluation of In[220]:= LinearModelFit::rank: The rank of the design matrix 9 is less than the number of terms 10 in the model. The model and results based upon it may contain significant numerical error. >>

Answer 1

A matrix ${\bf A}$ and its rank may be thought of as follows. Given a linear function $f({\bf X}) = {\bf A}.{\bf X}$, where ${\bf A}$ gives the coefficients of the variables ${\bf X} = (x_1,\dots,x_m)$, the rank is the maximum number of independent equations of the system $f({\bf X}) = {\bf 0}$ that can be found. The null space of the matrix ${\bf A}$ consists of all vectors ${\bf dX}$ that make zero difference in the output, i.e., $f({\bf X}+{\bf dX})=f({\bf X})$. If the rank is equal the number of variables (which equals the number of columns), the null space will consist of just the zero vector, and there will be a unique solution to the system $f({\bf X}) = {\bf 0}$.

LinearModelFit uses the data to set up a linear function of the $n+1$ parameters of the $n$ basis functions for the model plus the constant basis function $1$. The array of coefficients of the parameters, which are equal to the values of the basis functions on the data points, is called the design matrix. The data also defines a response vector ${\bf Y}$. Then LinearModelFit finds an ${\bf X}$ that minimizes the sum of squares of the differences in the vector $f({\bf X}) - {\bf Y}$. If the rank of the design matrix is less than the number of variables, then the null space contains nonzero vectors ${\bf dX}$, so that $f({\bf X})$ and $f({\bf X}+{\bf dX})$ both minimize the error. There will be infinitely many solutions, which means infinitely many models, but that should be okay because a deficient rank also implies that there are linear relationships between the parameters. The relationships imply that the models are equivalent.

Further the warning message implies that there may numerical error. I suppose this is probably from using approximate, machine numbers. It seems likely that some quantities that should be zero, will only be close to zero. These might lead to significant computational errors.

OP's problem

Some random data, except that it makes the first two parameters b0 and b1, the ones for the functions 1 and x1, satisfy the relation b0 + b1 == constant.

SeedRandom[0];
dat = Prepend[RandomReal[1, {9, 6199}], ConstantArray[1., 6199]] // Transpose;
reg = LinearModelFit[
   dat, {x1, x2, x3, x4, x5, x6, x7, x8, x9}, {x1, x2, x3, x4, x5, x6, x7, x8, x9}];

LinearModelFit::rank: The rank of the design matrix 9 is less than the number of terms 10 in the model. The model and results based upon it may contain significant numerical error. >>

In the OP's case, the rank is deficient by 1, which means the null space is the set of all scalar multiples of a single vector. NullSpace returns a list of vectors that form a basis for the null space. Each vector represents an independent change in the values of the parameters that produces another solution. NullSpace computes its answer with a certain numerical Tolerance, and quantities that should cancel sometimes might not come out to zero. We can use Chop to make small numbers and small differences equal zero.

ns = NullSpace[reg["DesignMatrix"]]
Chop[%]
(*
  {{-0.707107, 0.707107, -7.21645*10^-16, 1.15186*10^-15, 
     2.498*10^-16, -2.69229*10^-15, 2.67841*10^-15, 5.68989*10^-16, 
     5.55112*10^-16, 4.02456*10^-16}}

  {{-0.707107, 0.707107, 0, 0, 0, 0, 0, 0, 0, 0}}
*)

This means we can change the parameters by a*{-0.707107, 0.707107, 0, 0, 0, 0, 0, 0, 0, 0}, for any scalar a to get another solution and model. In general, we can change the parameters by

{a1, a2,..., ak}.ns

for any k scalars a1, a2,... where k is the number of vectors in the basis ns for the null space. This can be verified for our random example as follows. A set of parameters is a solution if it satisfies

reg["DesignMatrix"].parameters == reg["PredictedResponse"]

For instance, the solution reg["BestFitParameters"] satisfies

reg["DesignMatrix"].reg["BestFitParameters"] == reg["PredictedResponse"]
(*  True  *)

We can check that an arbitrary change {a}.NullSpace[reg["DesignMatrix"]] by an element of the null space yields another solution:

reg["DesignMatrix"].
 (reg["BestFitParameters"] + {a}.NullSpace[reg["DesignMatrix"]]) == 
   reg["PredictedResponse"] // Simplify // Chop
(*  True  *)

This problem arises when one of the basis functions is a linear combination of the others. For instance, if the functions include the numbers of males, females, and persons (both genders), then there is the linear relationship, persons = males + females, and the rank of the design matrix will be deficient.