Hypothesis Testing: Null hypotheses and P-values
Corresponding reading
- Chapter 5.1-5.4 in Agresti and Kateri (2021)
Rice (2006) treats testing in Chapter 9, but it’s in a different order than we are presenting it, so will not really align with this presentation if you are reading along with lecture. But it is worth reading holistically once we get through the main components of hypothesis testing over the next several modules.
Introduction
We turn from parameter estimation and confidence intervals to another cornerstone topic in statistical inference: hypothesis testing. This is one of the most commonly used statistical techniques. We first consider a model that we think might have generated the sample data, and then use probability calculations based on this assumption (called the null hypothesis) to see if our observed sample data is consistent with this assumption. We compute how likely it is that any observed difference is due to random variation. If it is too unlikely, we conclude that the data is extreme enough to contradict the null hypothesis, and we reject our assumption, that is we reject our null hypothesis.
Here are some questions that might be addressed by hypothesis testing:
- Is a coin/die/roulette wheel fair?
- Does the Covid vaccine work?
- Does the police force of a city reflect the ethnic makeup of the city residents?
- Has the proportion of Americans that believe in the theory of evolution declined over the years?
- Has the American’s population support of Planned Parenthood declined?
- Is the Salk vaccine effective at preventing polio?
- Is this new cancer treatment a significant improvement over an existing one?
- Do voters approve of the US action in the Middle East?
- Is a nurse accused of murdering patients on their shift guilty?
- Does the data come from a Normal population centered at zero or centered somewhere else?
- What is the rate of the exponential distribution that generated our sample?
Example: What is the probability of a selected coin?
Let’s take a very simple example of whether we have a fair coin, i.e. is the probability of getting heads for a coin equal to \(0.5\). For example, we might toss a coin 100 times, and see how many times it lands heads. Would we be comfortable assuming that it is fair if it lands heads 55 times? What about 60 heads? Where would we draw the line? We need to quantify how unusual our sample is given our assumption of fairness.
I am given a coin. How can I decide if it is fair or not?
We want to test the null hypothesis, denoted by \(H_0\), that the coin is fair: \[ H_0: \pi = 0.5 \] How might you decide? Well, you could toss the coin many times and count the fraction of heads. If it is too far from 50%, that would be evidence that it is not a fair coin. But how far is too far? How can we decide?
Suppose I toss the coin ten times and count the number of heads \(X\).
| # Heads | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(\pi=0.5\) | 0.0010 | 0.0098 | 0.0439 | 0.1172 | 0.2051 | 0.2461 | 0.2051 | 0.1172 | 0.0439 | 0.0098 | 0.0010 |
| \(T\) | 0.5 | 0.4 | 0.3 | 0.2 | 0.1 | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |
What would you do if \(X=3\)? \(X=6\)? \(X=8\)?
If you see \(X=6\) heads, then if the null hypothesis is true, then that’s a fairly likely outcome. But both \(X=3\) and \(X=7\) heads are equally likely and neither are very unusual, but less than \(X=6\).
To make the comparison to the null easier, we might look at the proportion \(X/10\) and compare to its expectation under the null, \(\pi=0.5\), \[T=|\hat\pi - 0.5|.\]
- We will use the data to summarize the data in a way that large values will be evidence against quantification how likely it is that the null of a fair coin is. It is random so it will be a statistic so we will call it a test statistic.
- Our statistic \(T\) is larger for data that is less likely under the null hypothesis. We can say that as our statistic gets larger, we have more evidence against the null.
- While for some realizations of the data it may seem obvious what we would decide, we clearly need some kind of cutoff to make a general rule about how we would decide for any possible data.
Notice that once we’ve defined our null hypothesis, we can use that information in defining our test statistic! It’s different than the true unknown parameter \(\pi\); we are picking a value for \(\pi\) to test (it might or might not be the right value, so it might not represent the actual unknown parameter)
The Setup of Hypothesis testing
A hypothesis test is a way of evaluating evidence against some default theory, called the null hypothesis. The null hypothesis is usually some feature of the population data that we want to exclude as a possibility.
In our above example, our null hypothesis was regarding the probability \(\pi\) of getting heads, \[H_0: \pi=0.5\]
Other settings we have considered also lend themselves to setting up a hypothesis test. Consider our example of comparing two populations. A common question we really want to know is whether these populations are different in some way, for example whether the means of the two populations are the same. Our null hypothesis would be that the means of the two populations are the same \[H_0: \mu_X=\mu_Y\] or equivalently \[H_0: \Delta=\mu_X-\mu_Y=0\]
Exercise 1 What would be a possible null hypothesis in our linear regression model?
To answer this question we will construct a function of the data called a test statistic (\(T\)) and consider its sampling distribution under the null distribution.
For example, in our comparison of two populations, we could consider the difference of the two sample means, \[T= |\hat\mu_X - \hat\mu_Y|.\] If the null is true, it is unlikely that we get data
In our coin example, we considered \[T=|\hat\pi - 0.5|. \] If the null is true, we are unlikely to observe values of \(X\) that are very small or very large (and thus large values of \(T\)).
“Extreme” values of the test statistic are considered as evidence against the null hypothesis. Here, extreme \(T\) would be large values, since under the null we do not expect large \(T\).
By convention, we will generally assume large \(T\) is evidence contrary to the null hypothesis, and small \(T\) suggests more compatibility with the null hypothesis (though this is just convention to make it easier to talk about testing in generality).
Notice that in both my settings, we are taking absolute values. This is because evidence against the null comes from both very positive and very negative differences, so by taking the absolute value we account for both. Not all test statistics will have this property, but it’s pretty common when we have a null hypothesis that a parameter takes a specific value.
How to evaluate \(T\)
We saw in thinking about the coin heads example, that even if we have a test statistic, to make an actual decision one way or the other about the null hypothesis, we need to determine at what point is \(T\) too extreme to be feasible under the null?
There are two ways of thinking about this:
We can consider that the main problem with our statistic \(T\) is we cannot easily look at it and understand if it is more extreme than we would “expect” under the null. So we want to “standardize” the scale of \(T\) so that we can report a transformation of \(T\) that is quantitative but on a scale we all understand and can compare regardless of the specific test. In this way you report the degree of evidence against the null and let the reader interpret. This leads us to a p-value paradigm.
We decide we want to make a definite decision one way or the other as to whether it’s a fair coin. So we need a specific cutoff \(c\) so that if \(T>c\) we will decide it’s not a fair coin. This leads us to a decision rule paradigm where we either reject the null or we do not.
In fact these are not two completely different things, as we can see, and we will generally combine these two into the standard hypothesis testing framework. Most statisticians recommend on a practical level reporting p-values – not just “reject” or “not reject” implied by a decision rule – since it gives more information about the degree of evidence you have. But the decision rule paradigm brings in important concepts about how to think about the performance of \(T\) and whether it has good behavior.
The theory of hypothesis testing was mostly developed by three people in the first part of the 20th century: Ronald Fisher, Jerzy Neyman, and Egon Pearson. They had different philosophies, with Fisher advocating for p-values, while Neyman and Pearson formulated the problem as making a decision; and they had vigorous disagreements about the best strategy. Modern statistics uses a hybrid approach and does not distinguish between these philosophies (see Lehmann (1992)).
P-values
We can consider that the main problem with our statistic \(T\) is we cannot easily look at it and understand if it is more extreme than we would “expect” under the null.
Why \(T\) needs to be “standardized”
In our above examples we created a statistic \(T\), and we understand that it tells us the strength of evidence against the null, but we don’t have a sense of how common such a value is.
For example, in the coin flip example, we saw that the chance we get \(T=0.3\) is equivalent to either \(X=2\) or \(X=8\), so a probability \[P_{\pi=0.5}(T_n=0.3 )=0.0879\] That’s not a large probability, but it’s certainly far from impossible.
What if we saw \(T=0.3\) with n=100 flips? We would have seen either 20 or 80 heads. The probability of that happening under the null is \[P_{\pi=0.5}(T_n=0.3 )=8.456e-10\] so it’s very unlikely.
So we need to grapple with the distribution of \(T_n\) under the null to ascertain whether \(T_n\) is unusually large and that \(T_n\) is pretty dependent on \(n\).
We see that instead of thinking about the value of \(T\), what is easy to compare and make sense of is its probability under the null. We are transforming from the scale of \(T\) to the scale of probability, which we can make sense of and interpret much more easily.
P-values
These probability calculations point us to probabilities as a natural way to think about whether \(T\) is large. All the calculations were done assuming the null distribution is true. We asked, assuming the null was true, how likely is our test statistic? The key component is figuring out the sampling distribution of \(T\), conditional on the null hypothesis being true.
This leads us to the definition of the null distribution of a test statistic
Definition 1 (Null distribution) Assume we have a test statistic \(T\). Then the sampling distribution of \(T\) when we assume the null distribution is true is called its null distribution.
We often write probabilities involving the null distribution with the following notation \[P_{H_0}(\cdot)\]
So above in our coin flip example, we calculated that for \(n=10\), \[P_{H_0}(T_n=0.3)=0.0879\] and for \(n=100\) \[P_{H_0}(T_n=0.3)=8.46e-10\]
Then we will define the p-value as the following:
Definition 2 (P-value) Assume we observe our realizations of the data, \(x_1,\ldots,x_n\), and calculate the value of our test statistic, \(t_{obs}\). We define the p-value as \[\text{p-value}=P_{H_0}(T \geq t_{obs})\] In other words, we calculate the probability of seeing a value as extreme as \(t_{obs}\) or more extreme under the null distribution of \(T\).
Examples
Coin Flip
Suppose we observe, as we discussed above, 8 heads out of 10.
Then we’ve already seen that our observed test statistic \(t_{obs}=0.3\). Then our p-value is \[\begin{aligned} \text{p-value}&=P_{H_0}(T\geq 0.3)\\ &=P_{H_0}(X\geq 8 \text{ or } X\leq 2)\\ &=2*\mathtt{pbinom(2,prob=0.5,size=10)}=0.109 \end{aligned}\] Notice this isn’t what we calculated before when we explored the idea – we are not just calculating the probability \(X=8\) but the probability \(X\in\{0,1,2,8,9,10\}\). This is because a p-value is calculated as the probability of getting as extreme as \(t_{obs}\) or more extreme. So we include other more extreme events that we didn’t actually see.
We can see what p-values we would get with possible different data:
| Number of Heads (\(x\)) | \(T=|X/n-0.5|\) | p-value | Conclusion |
|---|---|---|---|
| 0 | 0.5 | 0.002 | Poor fit to \(H_0\) |
| 1 | 0.4 | 0.021 | Poor fit to \(H_0\) |
| 2 | 0.3 | 0.109 | Somewhat poor fit to \(H_0\) |
| 3 | 0.2 | 0.344 | Compatible with \(H_0\) |
| 4 | 0.1 | 0.754 | Compatible with \(H_0\) |
| 5 | 0.0 | 1 | Compatible with \(H_0\) |
| 6 | 0.1 | 0.754 | Compatible with \(H_0\) |
| 7 | 0.2 | 0.344 | Compatible with \(H_0\) |
| 8 | 0.3 | 0.109 | Somewhat poor fit to \(H_0\) |
| 9 | 0.4 | 0.021 | Poor fit to \(H_0\) |
| 10 | 0.5 | 0.002 | Poor fit to \(H_0\) |
Comparing Two Populations
Consider our other hypothesis about two populations, \[H_0: \mu_X=\mu_Y\] with test statistic \[T_n=|\bar{X}-\bar{Y}|\] We know that asymptotically, \[\frac{(\bar{X}-\bar{Y})-(\mu_X-\mu_Y)}{\hat{SE}(\bar{X}-\bar{Y})}\overset{D}{\rightarrow} N(0,1)\] Then if we observe \(t_{obs}\) as our absolute difference in the means, our p-value is \[\text{p-value}=P_{H_0}(|\bar{X}-\bar{Y}| \geq t_{obs})=P_{H_0}\left(\frac{|\bar{X}-\bar{Y}|-(0)}{\hat{SE}(\bar{X}-\bar{Y})} \geq \frac{t_{obs}}{\hat{SE}(\bar{X}-\bar{Y})}\right)\] If we define \[U=\frac{t_{obs}}{\hat{SE}(\bar{X}-\bar{Y})},\] then our p-value is (for large \(n\)) \[\text{p-value}\approx P(N(0,1)\geq U)+P(N(0,1)\leq - U)=2*(1-pnorm(U))\] Notice how it’s important for continuous distributions that my p-value is defined in terms of \(\geq\) – it wouldn’t have any meaning to find the exact probability of the observed value I saw.
Notice in both of these examples, I am looking at the absolute difference, and that is the actual distribution I am interested in. But to calculate those probabilities, I use the fact that I can get the distribution by combining the two tails of the standard distributions for which we know their distributions and have easy R functions to calculate them.
Relationship to \(T\) random variables in CI
There are obvious connections here between the calculations I did for my confidence intervals and the calculations I am doing for the p-value. I am relying on the distribution of what in the CI modules I called the random variable \(T\), \[T=\frac{(\bar{X}-\bar{Y})-(\mu_X-\mu_Y)}{\hat{SE}(\bar{X}-\bar{Y})}\overset{D}{\rightarrow} N(0,1)\] But now in the hypothesis testing setting, I will insert the value implied by my null hypothesis. If I’m going back and forth between these two, I could call my test statistic \(T_0\) to differentiate it, and I have \[T_0=\frac{(\bar{X}-\bar{Y})-(0)}{\hat{SE}(\bar{X}-\bar{Y})}\overset{D}{\rightarrow} N(0,1) \text{ under }H_0.\] For p-values, I am only concerned about the distribution of \(T_0\) when my null hypothesis is true.
We will return to this relationship with our Wald CI, but we can see from this example, that a lot of our work in creating normal-based CI will carry over to helping us find the null distribution of our test statistic.
Notice if I don’t assume I know \((\mu_X-\mu_Y)\), then \(T\) is a random variable, but not a statistic – meaning I can’t calculate \(T\) because it relies on unknown values. But when I input the values of my null hypothesis, I get a statistic. Furthermore, even if the null is not true, \(T_0\) is still a statistic that has a distribution; it just won’t be \(N(0,1)\).
Interpreting P-values
Notice a few things about the p-value
As we noted above, the p-value is not the probability of observing \(t_{obs}\). Instead we are also including the probability of more extreme results (which we did not actually see).
A p-value is not the probability that the null hypothesis is true! The unknown parameter (or more generally the unknown data generation \(F\)) is a fixed quantity. Thus \(H_0\) is either true or not, and there is no randomness about this; we just don’t know whether it’s true or not. This is a common mistaken interpretation of p-values (similar to those we discussed for confidence intervals)
If we see a small p-value it tells only that the data we saw is unlikely to have been created under the null distribution. It tells us nothing about if the data is better described by an alternative hypothesis.
Indeed with our coin flip data, we can see that quite different outcomes (0 heads and 10 heads) have the same p-value but clearly if we don’t believe the coin is fair, then we would have very different likely alternatives for these two different results.
Similarly if we see a larger p-value, it doesn’t tell us that the null distribution is actually what created the data. It just says that the data is compatible with having been created under null hypothesis setup. Indeed, if I see \(X=3\) heads, we have a p-value of 0.344, which is not evidence that the null is not true because this is not an unlikely event if the true success probability is \(\pi=0.5\). But at the same time, it’s also clearly a likely outcome for \(\pi=0.3\), \(\pi=0.4\), etc.
P-values are Random Variables
The p-value is a function of our observed value of \(T\), and thus a function of our data. So it is a random variable and has a distribution. You may get a p-value of 0.04 on one realization of the data, and on another realization of the data get a p-value of 0.15.
If the null is true then the distribution of the p-value is \(U(0,1)\) for a continuous statistic \(T\) (assuming you have the correct null distribution of \(T\)). So we can see that if the null is true, and you repeatedly draw samples, 0.05 proportion of those samples will get a p-value \(\leq 0.05\). More generally, \[P_{H_0}(\text{p-value} \leq \alpha)=\alpha\] So we have a clear understanding of how likely it is to get certain p-values.
Of course, if you have a discrete test statistic, your p-value will also have a discrete distribution and cannot have a \(U(0,1)\) distribution. So, we will say that a p-value is valid if \[P_{H_0}(p-value \leq \alpha) \leq \alpha\] for all \(0\leq\alpha\leq 1\).1
More formally, this property can be considered the most basic definition of a p-value, i.e. any test statistic \(p(X)\) bounded to lie in \([0,1]\) that satisfies this requirement is a valid p-value; in which case, the definition we gave above could be considered a method to construct a p-value.2
The Role of the Null Hypothesis
Notice how we are prioritizing the null distribution. We only care about how well the data fit the null. Null distributions, for this reason, tend to be a baseline hypothesis for how the data was generated that is not “interesting” – a straw-man to compare against. Usually investigators don’t want the null to be true, because it means there is really nothing to see in the data. \(H_0\) generally makes quantitative the scientific question of “Is what we measured in our data might be just due to chance?” For example,
- comparing two populations: the null is that there’s nothing different
- linear regression (e.g. how click-through rate on an ad changes between different ages or ethnicities): the null is that there is no slope, i.e. X doesn’t inform you regarding \(Y\)
- a coin is rigged: the null is that it’s fair
- the effect of a vaccine: the null is that it had no effect
- there are more bad outcomes under one nurse: null would be that all the nurses have the same chance of a bad outcome
- an employee was given bad projects due to bias: the null would be that having bad projects assigned was due to random chance.
Since \(H_0\) usually represents a “default” or “random chance” hypothesis, the scientifically conservative strategy accepted throughout research is to first assure ourselves that the chance that this hypothesis created the data is small (e.g. p-values less than 0.05 or 0.01). Which is why it is important to be clear that if we have larger p-values, then we conclude simply that the data is compatible with having created the data, not that it is “true”.
Exercise 2 Think in each of the above examples about what the real-world implications are in saying the data is compatible with having created the data versus that the null is “true”.