When using LinearModelFit[] we can generate a table of parameter values and statistics for the corresponding model using the property "ParameterTable". Looking in the documentation I can't find an explicit statement of the hypothesis test being performed for the p-values listed in this table. I would like to make sure I am interpreting these things properly. So for example if I produce the table $$\begin{array}{l|llll} \text{} & \text{Estimate} & \text{Standard Error} & \text{t-Statistic} & \text{P-Value} \\ \hline 1 & 0.0116261 & 0.00199362 & 5.83162 & \text{3.092$\grave{ }$*${10}^{\wedge}$-8} \\ \text{x$\$$2374}(1) & -0.00573076 & 0.00207492 & -2.76191 & 0.0064413 \\ \text{x$\$$2374}(2) & -0.00614132 & 0.00209498 & -2.93144 & 0.0038851 \\ \text{x$\$$2374}(3) & -0.000374603 & 0.00240746 & -0.155601 & 0.87655 \\ \text{x$\$$2374}(4) & -0.00289621 & 0.00256897 & -1.12738 & 0.261323 \\ \end{array}$$ how do I interpret these p-values? I would think at a 95% confidence level the first three parameters are statistically significant but the last two are not.
Here's how to interpret statistical significance, here illustrated for a $\chi^2$ distribution:
Such a distribution tells you the expected probability of finding a value of $x$ as the sum of squares of values chosen from $n$ univariate Gaussian distributions (where we call $n$ the degrees of freedom). Of course, the value of $x$ can never be negative. The below Manipulate shows this distribution for different values of the settable degrees of freedom (denoted $df$).
The $x$ corresponding to a particular $p$-value (the "significance") is the value of $x$ for which a percentage of $p$ cases lie above (>$x$) in the distribution. If you obtain an experimental value $x$ above that criterion you can reject the null hypothesis, i.e., that the value could arise if the value were zero "by chance". We write that value for instance $\chi^2_{0.01(3)}$ which means the criterion value of $x$ for which 0.01 of the cases lie above that $x$ for $3$ degrees of freedom. Adjust the $p$ value to $p=0.05$ in the below Maniplate and see that the criterion value must drop.
Manipulate[Module[{xsol, x},
xsol = x /.
NSolve[1 - CDF[ChiSquareDistribution[df], x] == Significance,
x][[1, 1]] // Quiet;
Plot[
PDF[ChiSquareDistribution[df], x], {x, 0, 20},
PlotRange -> {0, .4},
ColorFunction -> Function[x, If [x > xsol, Pink, Lighter[Green]]],
ColorFunctionScaling -> False,
Filling -> Axis,
TicksStyle -> Italic,
Ticks -> {Range[0, 20,
1] \[Union] {{xsol,
Column[{" ",
Style[Subscript[\[Chi]^2,
ToString[Significance] <> "(" <> ToString[df] <> ")"],
Red, 12]}, Center], {-.05, .0}, Red}},
Range[0, .4, .1]},
AxesLabel -> {Text[Style["x", 14 , Italic]],
Text[Style["probability", 14 , Italic]]},
PlotLabel -> TextCell[
Row[{Style[
Subscript[\[Chi]^2,
ToString[Significance] <> "(" <> ToString[df] <> ")"]] ,
" = ", Style[ToString[xsol], Italic, 12]}]
]
]],
{{df, 3, Text[Style["df", Italic, 14]]}, 1, 30, 1,
AppearanceElements -> All}, {Significance, {0.01, 0.05}}]
-
$\begingroup$ Thanks for taking the time to write this code, but this is not quite what I was asking. The p-value in the parameter table is used for some type of hypothesis test. I assume the null hypothesis is the parameter is equal to zero. Based on the p-value we can accept or reject the null hypothesis. @ significance level .05 we would reject the null hypothesis for the first 3 parameters which says they add a statistically significant contribution to the model. I want to know if this interpretation is correct since the mathematica documentation never says what the null hypothesis is. $\endgroup$ – Wintermute Apr 30 '15 at 0:21
-
2$\begingroup$ @wintermute: The documentation does specifiy this. Read the "How to Get Results for Fitted Models" and "Statistical Model Analysis" tutorials, e.g., linked in the documentation for the fit functions... $\endgroup$ – ciao Apr 30 '15 at 0:39
