Confidence intervals for linear regression
The last “standard” example that we will tackle using the exact sampling distribution, before we move purely into asymptotic results, is our regression setting.
Confidence Intervals for \(\hat\beta_1\)
Recall what we have derived about the exact sampling distribution of \(\hat{\beta}_1\)
We have numerous ways we can express the estimator \(\hat{\beta}_1\), \[ \hat\beta_1=\frac{\sum_{i=1}^n Y_iX_i-n\bar{Y}\bar{X}}{\sum_{i=1}^n X_i^2-n\bar{X}^2}=\frac{\sum_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})}{\sum_{i=1}^n(X_i-\bar{X})^2}=\frac{\sum_{i=1}^n(X_i-\bar{X})Y_i}{\sum_{i=1}(X_i-\bar{X})^2} \]
Our standard estimate of the variance \(\sigma^2\) of our \(Y_i\) is \[\hat\sigma^2=\frac{1}{n-2}\sum_{i=1}^n(Y_i-(\hat\beta_0+\hat \beta_1 X_i))^2\]
Using the assumption of our linear model, \(\hat{\beta}_1\) is unbiased, \[ E(\hat{\beta}_1|X_1=x_1,\ldots,X_n=x_n)=\beta_1 \]
Using the additional assumption that our \(Y_i\) have shared variance and are independent from each other, we have \[\begin{aligned} \mathrm{Var}(\hat{\beta}_1|X_1=x_1,\ldots,X_n=x_n) &= \frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\\ &= \frac{\sigma^2}{n}(\hat{\sigma}_{x}^{2})^{-1} \end{aligned} \] And our estimate of the standard error of \(\hat{\beta}_1\) is \[\hat{SE}(\hat{\beta}_1)=\frac{1}{\sqrt{n}}\frac{\hat\sigma}{\hat\sigma_x}\]
If we assume our parametric model, \(Y_i|X\sim N(\beta_0+\beta_1x,\sigma^2)\). Then we have that the exact sampling distribution of \(\hat{\beta}_1\) is \[ \hat{\beta}_1 | X=x \sim N(\beta_1,\frac{\sigma^2}{n}(\hat{\sigma}_{x}^{2})^{-1}) \]
We can put these all together to construct a Confidence Interval for \(\beta_1\). We construct our \(T\) random variable, \[T_{\beta_1}=\frac{\hat\beta_1-\beta_1}{\hat{SE}(\hat\beta_1)}\] Then we have
- For large sample sizes, \[T_{\beta_1}\overset{D}{\rightarrow}N(0,1)\]
- For small sample sizes, with the assumption of normality, we have \[T_{\beta_1}\sim t_{n-2}\]
Confidence Intervals for \(\hat\beta_0\)
We can make the same statements for \(\hat\beta_0\)
- We again have different forms for our estimator \[ \hat\beta_0=\bar{Y}-\hat\beta_1\bar{X}=\frac{\bar{Y}\left(\sum_i X_i^2\right) - \bar{X}\left(\sum_{i}X_iY_i\right)}{\sum_{i}(X_i-\bar{X})^2} \]
- Using the assumption of our linear model, \(\hat{\beta}_0\) is unbiased, \[ E(\hat{\beta}_0|X_1=x_1,\ldots,X_n=x_n)=\beta_0 \]
- Using the additional assumption that our \(Y_i\) have shared variance and are independent from each other, we have \[\mathrm{Var}(\hat{\beta}_0|X_1=x_1,\ldots,X_n=x_n) = \frac{\sigma^2}{n}\left(\frac{\displaystyle\sum_{i=1}^{n} x_i^2}{\displaystyle\sum_{i=1}^{n} (x_i-\bar{x})^2}\right) \] And our estimate of the standard error of \(\hat{\beta}_0\) is \[\hat{SE}(\hat{\beta}_0)= \frac{\hat\sigma}{\sqrt{n}}\sqrt{\frac{\displaystyle\sum_{i=1}^{n} x_i^2}{\displaystyle\sum_{i=1}^{n} (x_i-\bar{x})^2}}\]
- If we assume our parametric model, \(Y_i|X\sim N(\beta_0+\beta_1x,\sigma^2)\). Then we have that the exact sampling distribution of \(\hat{\beta}_0\) is \[ \hat{\beta}_0 \sim N(\beta_0,\frac{\sigma^2}{n}\left(\frac{\displaystyle\sum_{i=1}^{n} x_i^2}{\displaystyle\sum_{i=1}^{n} (x_i-\bar{x})^2}\right) ) \]