Wald/T-tests and Relationship to Confidence Intervals
Corresponding reading
Chapter 5 in Agresti and Kateri (2021) covers hypothesis testing, and I would recommend just reading the chapter through, in order, because we will be covering all the material in that chapter. The specific material discussed in this lecture is found
- 5.2.4 for test of mean
- 5.3.1 for test of difference of two means
- 5.6 for duality of CI and significance tests.
- 6.4.3 for regression
For duality of CI, the treatment in Rice (2006) in Chapter 9.3 is more comprehensive.
In the following, we are going to work through the steps of calculating a p-value and a critical value of test statistics using the results we developed for normal-based confidence intervals.
Wald Test Statistics
In our example of comparing the mean of two populations, we’ve seen that we can use our distributional results that we developed for normal confidence intervals to get the null distribution of our data.
In particular, suppose we have an estimate \(\hat\theta\) of \(\theta\), and we know the distribution (either exact or asymptotically) of \[T(\theta)=\frac{\hat\theta -\theta}{\hat{SE}(\hat\theta)}\] where \(\hat{SE}(\hat\theta)=\sqrt{1/I_n(\hat\theta)}\)1
Then if we want to test the null hypothesis \[H_0: \theta=\theta_0\] then we define our test statistic as \(T_0=|T(\theta_0)|\), i.e. \[T_0=\frac{|\hat\theta -\theta_0|}{\hat{SE}(\hat\theta)}\] Again, notice that \(\theta_0\) is the null hypothesis value we have chosen; it’s no longer an unknown parameter. So \(T_0\) is a well-defined statistic.
We define the test statistic as the absolute value, because we want to account for evidence in either direction that is away from our null value \(\theta_0\). However, since we know the distribution of \(T(\theta_0)\) if \(\theta=\theta_0\), we can work out the distribution of \(|T(\theta_0)|\).
Notice that unlike my original example when I introduced p-values, where I just used the numerator \(|\bar{X}-\bar{Y}|\), I will usually standardize the statistic by dividing by \(\hat{SE}(\hat\theta)\) in my definition of \(T\). It doesn’t actually matter for our conclusions (remember what we said about monotone transformations), but it does put our test statistics all on the same \(N(0,1)\) scale, so their values are more interpretable.
P-values
Then if we observe \(t_{obs}=T_0(x_1,\ldots,x_n)\) as our observed value of \(T_0\), our p-value is calculated as \[ \begin{aligned} \text{p-value}&=P_{H_0}(T_0 \geq t_{obs})\\ &\approx P( Z \geq t_{obs})+P( Z \leq - t_{obs}) \quad\text{where }Z\sim N(0,1)\\ \\ &=1-\Phi(t_{obs})+\Phi(-t_{obs})=2(1-\Phi(t_{obs})) \end{aligned} \] where \(\Phi(\cdot)\) is the CDF of the \(N(0,1)\).
Cutoff value \(c\)
We want to find the \(c\) so that \[P_{H_0}(T_0>c)=\alpha\]
\[ \begin{aligned} \alpha=P_{H_0}(T_0>c)&\approx P(Z > c)+ P(Z< -c)\\ &= 1-\Phi(c)+\Phi(-c)=2(1-\Phi(c))\\ \Rightarrow c&=\Phi^{-1}(1-\alpha/2)=z_{\alpha/2}\\ \end{aligned} \]
Notice how standardizing my test statistic makes the calculations simple functions of the standard normal. But if you look back at our original treatment of the difference of the two populations, you will see that we get the same answers either way.
Examples of using the Exact Sampling Distribution
We can also get exact p-value when we know the exact sampling distribution of our test statistic.
T statistics and tests
For example, if we assume our data is normally distributed, we know that in the following settings we have an exact \(t\) distribution for our \(T\) random variable for confidence intervals, and can use them to define test statistics like with the Wald Statistics. These test statistics are called t-statistics and the tests that goes with them are called t-tests. Again when we’re talking about both the random variable and the test statistic, we’ll use \(T_0\) for the test statistic just to distinguish them.
Population Mean Assume \(X_i\) i.i.d. \(N(\mu,\sigma^2)\), then \[T=\frac{\bar{X}-\mu}{S^2/n}\sim t_{n-1}\] So we can define a test statistic for the null hypothesis \(H_0:\mu=\mu_0\) as \[T_0=\frac{|\bar{X}-\mu_0|}{S^2/n}\]
Regression Again, if our normal model is true \[Y_i|X_i=x_i\sim N(\beta_0+\beta_1x_i,\sigma^2)\] then we have \[T=\frac{\hat\beta_1-\beta_1}{\hat{SE}(\hat\beta_1)}\sim t_{n-2}\]
In principle, we can test for an arbitrary \(H_0:\beta_1=\beta_{10}\) \[T_0=\frac{|\hat\beta_1-\beta_{1(0)}|}{\hat{SE}(\hat\beta_1)}.\]
But in practice, we always test \(H_0:\beta_1=0\), which means there is no slope; this means \(X_i\) is not relevant for predicting the expected value of \(Y_i\).
- Comparing two population means If our normal model is true, \(X_i\sim N(\mu_X,\sigma_X^2)\) and \(Y_i\sim N(\mu_Y,\sigma_Y^2)\), then we have \[T=\frac{\hat\Delta-\Delta}{S_{\hat{\Delta}}}=\frac{(\bar{X}-\bar{Y})-(\mu_X-\mu_Y)}{\hat{SE}(\Delta)}\sim t_{\nu}\] If our null hypothesis is true, \(H_0: \mu_X=\mu_Y\), then \[T_0=\frac{|\bar{X}-\bar{Y}|}{\hat{SE}(\Delta)}\sim t_{\nu}\]
P-values
The calculation of p-values is exactly as above, only with the appropriate \(t\) distribution. Let \(t_{obs}=T_0(x_1,\ldots,x_n)\) be our observed value of \(T_0\), our p-value is calculated as \[ \begin{aligned} \text{p-value}&=P_{H_0}(T_0 \geq t_{obs})\\ &= P( T \geq t_{obs})+P( T \leq - t_{obs}) \quad\text{where }T\sim t_{\nu}\\ \\ &=1-F(t_{obs})+F(-t_{obs})=2(1-F(t_{obs})) \end{aligned} \] where \(F(\cdot)\) is the CDF of the \(t_{\nu}\), i.e. with the right degrees of freedom for the particular problem context.
Cutoff value \(c\)
We want to find the \(c\) so that \[P_{H_0}(T_0>c)=\alpha\]
\[ \begin{aligned} \alpha=P_{H_0}(T_0>c)&=P(T > c)+ P(T< -c) \quad\text{where }T\sim t_{\nu}\\ &= 1-F(c)+F(-c)=2(1-F(c))\\ \Rightarrow c&=F^{-1}(1-\alpha/2)=t_{\alpha/2,\nu}\\ \end{aligned} \]
Estimating \(SE(\hat\theta)\) and the Score Test
In our Wald Test, we estimated the \(SE(\theta)\) with our standard estimate in our confidence intervals, \[\hat{SE}(\hat\theta)=\sqrt{1/I_n(\hat\theta)}\]
However, since we are calculating probabilities when the null hypothesis is true, \(H_0:\theta=\theta_0\), then we could instead insert \(\theta_0\) into \(SE(\theta)\) to get the actual variance of \(\hat\theta\) under the null, giving us a test-statistic \[ T_0=\frac{|\hat\theta-\theta_0|}{SE(\theta_0)}\] When the null hypothesis is true, then \[\frac{|\hat\theta -\theta_0|}{SE(\theta_0)}\overset{D}{\rightarrow}|N(0,1)|.\] So we can use all of the above calculations, only with \(SE(\theta_0)\)
Example – Binomial Score Test
A common example, given in the book (5.2.1), is testing for a proportion, like our coin flip example, \[H_0: \pi=\pi_0\] The under the Wald Test, you would use the test statistic \[T_{Wald}=\frac{|\hat\pi-\pi_0|}{\sqrt{\frac{\hat\pi(1-\hat\pi)}{n}}},\] while \[T_{Score}=\frac{|\hat\pi-\pi_0|}{\sqrt{\frac{\pi_0(1-\pi_0)}{n}}}\]
The Score Statistic
The score test is an alternative to the Wald test statistic that is based on the score statistic. We encountered this in our discussion of the asymptotic normality of the MLE and Fisher’s info, but didn’t spend much time on it (you can look back at the lecture notes). The score statistic for i.i.d. data is \[U(X;\theta)=\frac{\partial L(\theta)}{\partial \theta}= \sum_{i=1}^n \frac{\partial \log f(X_i;\theta)}{\partial \theta}\]
We showed there that \(E(U(X;\theta))=0\). Further, it is asymptotically normally distributed with asymptotic variance \(I_n(\theta)\) (similar but the reciprocal of that of the MLE).
Then the score test for testing \(H_0: \theta=\theta_0\) uses the test statistic \[T_0=\frac{|U(X;\theta_0)|}{\sqrt{I_n(\theta_0)}}\] which is asymptotically distributed like the absolute value of a \(N(0,1)\) under \(H_0\).
How is this related to how we choose \(SE(\theta)\) in our test statistic? Under many examples (the Binomial, Normal, Poisson,…) the Score Test simplifies to be the same as the Wald Test, but using \(SE(\theta_0)\) instead of using our estimate \(SE(\hat\theta)\). We will compare these test statistic more when we look at the likelihood ratio test.
Relationship to CI
We’ve seen that we’ve used the distributional theory from CI to develop tests. But in fact, there is a more general relationship between CI and significance testing that goes beyond these normality-based tests. It applies when we are testing the value of a specific parameter \(\theta\), and says that any CI of \(\theta\) corresponds to a hypothesis test and that every hypothesis test of \(\theta=\theta_0\) corresponds to a CI.
Theorem 1 (Constructing a \(100(1-\alpha)\) CI from a test) Suppose you have a test so that for every value \(\theta_0\in \Omega\) (the parameter space), there is a test at level \(\alpha\) of \(H_0:\theta=\theta_0\). Let \(A(\theta_0)\) be the values of \(X\) for which you do not reject the null (the “acceptance” region).
Then the set \[\{\theta: X\in A(\theta)\}\] is a \(100(1-\alpha)\) for \(\theta\)
Interpretation: You can create a confidence interval for \(\theta\) as the collection of all the values of \(\theta_0\) for which you would not reject the null of \(H_0:\theta=\theta_0\) based on this data.
Theorem 2 (Constructing a test from a CI) Suppose that you have a confidence interval given by \((T_L(X),T_U(X))\) which is a \(100(1-\alpha)\)% confidence region for \(\theta\), i.e. for every \(\theta_0\), \[P_{\theta_0}(\theta_0 \in (T_L(X),T_U(X)))=1-\alpha.\]
Then a test of the hypothesis \[H_0:\theta=\theta_0 \quad\text{versus}\quad H_a:\theta\neq \theta_0\] given by \[\begin{cases}\text{Reject }H_0 & \theta_0 \notin (T_L(X),T_U(X))\\ \text{Don't reject } H_0& \theta_0 \in (T_L(X),T_U(X)) \end{cases} \] is a valid \(\alpha\)-level test.
This is quite similar to creating a test from p-values. We have constructed a test from the confidence interval and it is a valid test.
We can see that the inferential ideas of confidence intervals, p-values, and hypothesis tests all fit together in a reasonably coherent way.
Note, however, that there are some tests that are not a test of whether a parameter is a specific value. In that case, there is no confidence interval option
Notice that this duality is not specific to Wald Tests, but any test of \(H_0:\theta=\theta_0\).
Which to use?
How do we use them in practice? If you are working with a specific parameter of interest, it is always preferable to report a confidence interval. It provides more information than a p-value. We see from the above that a confidence interval tells us all the values for which we would reject a null based on that value. That’s much more informative. This can be summarized in terms of preference of what we report as: \[\text{Confidence interval} > \text{p-value} > \text{Decision Reject/Not}\] While this is the best practice, you will often still see in practice a p-value reported instead of a confidence interval.
Despite this, the interpretation that p-values and hypothesis testing brings to confidence intervals in terms of control of error, and decision making enriches the meaning and understanding of confidence intervals.
References
Footnotes
One could also use the observed Fisher’s information here for ease of computation and it would still be a Wald Test↩︎