Efficiency and Mean Squared Error
Questions/Goals for this Module
- What properties of the sampling distribution makes a good estimator?
- Understand the role of bias and variance in evaluating an estimator
We’ve discussed already some important sampling distribution properties of estimators, such as bias and variance of an estimator. We’re going to be turning now to consider in detail ideas about how we can evaluate different possible estimators, to consider how we can compare estimators.
Comparing the Variance of Estimators
Example: mean versus median
Suppose we know that \(X_1, \ldots, X_n\) are i.i.d. from a Normal distribution \(N(\theta, \sigma^2)\). Our goal is to estimate \(\theta\) using one of the two estimators: either the sample mean or the sample median. Which one should we use? Let’s look at their expected values and variances.
Sample mean:
\[ \begin{aligned} \hat{\mu} &= \bar{X}_n, \\ \mathbb{E}(\hat{\mu}) &= \mu, \\ \text{Var}(\hat{\mu}) &= \frac{\sigma^2}{n} \end{aligned} \]
Sample median: The sample median, \[\tilde{\mu} = median(X_1,\ldots,X_n)\] is asymptotically normal, with a variance that depends on the distribution of the \(X_i\). For a normal, we have \[ \begin{aligned} \mathbb{E}(\tilde{\mu}) &= \mu, \\ \text{Var}(\tilde{\mu}) &\approx \frac{\pi\sigma^2}{2n}\;\text{ for large } n. \end{aligned} \] Thus, they are both unbiased estimates, but if we compare their variances we see \[ \frac{\pi}{2} > 1 \Rightarrow \text{var}(\tilde{\mu}) > \text{var}(\hat{\mu}) \] We see that though they are both unbiased, the sample median has a larger variance than the sample mean. This could lead us to prefer the mean as the estimator of the true mean. Having low variance is important since it increases the accuracy of the estimator.
Relative Efficiency of Two Estimators
To quantify this idea of “better” estimators, we will define the Relative Efficiency of two estimators: Suppose we want to compare a pair of unbiased estimators of a parameter \(\theta\), denoted by \(\hat{\theta}\), \(\tilde{\theta}\).
Definition 1 The efficiency of unbiased estimators \(\tilde{\theta}\) relative to \(\hat{\theta}\) is given by:
\[ \text{eff}(\hat{\theta}_n,\tilde{\theta}_n) = \frac{\text{var}(\hat{\theta}_n)}{\text{var}(\tilde{\theta}_n)} \]
This quantity is called Relative Efficiency.
What does it mean if \(\text{eff}( \hat{\theta}_n,\tilde{\theta}_n) < 1\)? In that case, we have that \(var(\tilde{\theta}_n) > \text{var}(\hat{\theta}_n) \Rightarrow \hat{\theta}_n\) is a preferred estimator in terms of variance.
Consider our comparison of the median and the mean. We have \[\text{eff}( \hat{\mu}_n,\tilde{\mu}_n)=\frac{\sigma^2/n}{\pi\sigma^2/2n}=2/\pi\approx 0.63\]
Role of Assumptions
Note that our comparison of the mean and median depends on our knowing the data is normal. The variance of the mean is always the same for any distributions if the two distributions have the same variance. However, the variance of the median will depend more closely on the specific distribution of the data.
For example, consider the Laplace distribution with density \[f(x)=\frac{1}{2b}\exp{-\frac{|x-\mu|}{b}}.\] The Laplace distribution is symmetric, like the normal, but is very heavy tailed compared to the normal.
For data that are i.i.d. Laplacian with mean \(\mu\) and variance \(\sigma^2=2b^2\), we have that both \(\hat{\mu}\) and \(\tilde{\mu}\) are still unbiased estimates of the mean, but the asymptotic variance of \(\tilde{\mu}\) is \[var(\tilde{\mu})\approx \frac{\sigma^2}{2n},\] i.e. about half that of the mean, which is still \[var(\hat{\mu})= \frac{\sigma^2}{n},\] making the median the estimator with lower variance. Specifically, we have \[\text{eff}( \hat{\mu}_n,\tilde{\mu}_n)=\frac{\sigma^2/n}{\sigma^2/2n}=2>1\] so \(\tilde{\mu}\) is now more efficient than \(\hat\mu\). This will be generally true for heavy tailed distributions. Ultimately, when there is high probability of the large deviations from the mean, the median will be less variable than the mean.
This discrepancy in which estimator is more efficient is not unusual. Evaluations of the efficiency of estimators will frequently be sensitive to the assumptions of the distribution of the data.
This doesn’t mean efficiency is useless. Using our efficiency analysis of the mean and the mean, we can arrive at an idea of under what circumstances the mean is preferred over the median. Depending on what type of data we have, we can use this information to inform us as to which is the better estimate. In other words, knowing which is the most efficient estimator, under a certain distributional assumption, allows us to then consider which assumptions best match our data.
The Cramér-Rao Inequality
We have compared two estimators, but the question that now naturally arises: is some other estimator more efficient than either of the two we are considering? Can we tell if we have reached the most efficient estimator?
The answer is Yes! There is a famous inequality that gives us a lower bound on the variance of any unbiased estimator. If our estimator achieves this lower bound, it cannot be improved.
Let’s state the inequality for unbiased estimators. This inequality gives us what is called the “Cramér-Rao lower bound for an unbiased estimator”:
Theorem 1 Let \(X_1, X_2, \ldots, X_n\) be IID random variables with density function \(f(x \mid \theta_0)\) (the true parameter value). Suppose \(\hat{\theta}\) is an unbiased estimator of \(\theta\). Then, under smoothness assumptions on \(f\), we have that: \[ \text{Var}(\hat{\theta}_n) \geq \frac{1}{n I(\theta_0)} \]
\(I(\theta_0)\) is called the Fisher Information of the distribution with density \(f\), \[ I (\theta_0) = -E_{\theta_0} \left[ \frac{\partial^2}{\partial \theta^2} \log f(X;\theta)\right]\]
Definition 2 If \(\hat{\theta}\) is an unbiased estimator of \(\theta_0\) such that \(\text{Var}(\hat{\theta}) = \dfrac{1}{nI(\theta_0)}\), then we say that \(\hat{\theta}\) is an efficient estimator.
If we have an efficient estimator, then the variance is minimized, which means the sampling distribution of \(\hat{\theta}\) is as concentrated about the true value \(\theta_0\) as it can be.
Note
In fact, the Cramér-Rao Lower bound exists for all estimators, even biased ones. Specifically, if \(E(\hat\theta)=g(\theta_0)\), then \[ \text{Var}(\hat{\theta}_n) \geq \frac{[g'(\theta_0)]^2}{I_n(\theta_0)} \] For unbiased estimators, \(g(\theta)=\theta\) so \(g'(\theta)=1\), which gives us the lower bound \(1/I_n(\theta_0)\). But as we will see, we will not generally compare biased estimators based on their variances.
MLE estimators and Asymptotic Efficiency
If the Cramér-Rao lower bound looks familiar, it should. It’s the asymptotic variance of MLE estimators, \(1/I_n(\theta)\). In general, we may not be able to have a closed-form expression for the MLE or we may not be able to calculate its variance exactly. But we see that the variance of MLEs are asymptotically \((I_n(\theta))^{-1}\). Estimators whose variance is asymptotically \((I_n(\theta))^{-1}\) are called asymptotically efficient.
Frequently we limit ourselves to considering only estimators that are asymptotically normally, and define asymptotically
More generally, we can define asymptotic relative efficiency of two estimators as follows:
Definition 3 Suppose that our data \(X_1,\ldots,X_n\) are i.i.d. with density \(f(x|\theta)\) and our estimator \(\hat\theta_n\) is asymptotically normal with \[ \sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{D} N(0, \nu(\theta)). \] and \(\nu(\theta)\) is continuous.
Then with sufficient regularity conditions on \(f(x|theta)\), we have that \[\nu(\theta) \geq \frac{1}{I(\theta)}. \] where \(I(\theta)\) is the Fisher information for a single observation.
An estimator that achieves this lower bound is asymptotically efficient.
Ultimately, finite sample efficiency is not a particularly useful property for comparing estimators unless you only consider unbiased estimators, which as we will see is limiting. But asymptotic efficiency is a more relevant notion. This is because while we might allow some small bias in our estimator, we don’t want to have that bias linger asymptotically. If we have infinite amounts of data, we want our estimate to get the right answer!
Why only asymptotically normal estimators?
The Cramér-Rao bound can hold beyond asymptotically normal estimators, but it gets more complicated to describe that class of estimators. So you will generally see discussions of efficiency restricted to asymptotically normal estimators to avoid these complexities.
It also gets more complicated to state when it applies for non-i.i.d. data, though it will frequently be true as well.
Note
We could also say the asymptotic normality assumption of the theorem above more informally as \[\hat\theta \approx N(\theta_0, \frac{\nu(\theta)}{n})\] This is because when we are talking about asymptotics, we can’t have something in the limiting distribution that depends on \(n\). You have to be a bit careful about the formal asymptotic statements here.
Similarly, I can write for an MLE estimator \(\hat\theta_n\) of i.i.d. data it’s approximate large sample distribution as \[\hat\theta_n\approx N(\theta_0,\frac{1}{nI(\theta)})\] but I cannot formally write \[\hat\theta_n \xrightarrow{D} N(\theta_0,\frac{1}{nI(\theta)}) \quad \text{as }n\rightarrow\infty\] because it doesn’t make sense to have something that depends on \(n\) be something we converge to. Instead I could write \[\frac{\hat\theta_n -\theta_0}{1/\sqrt{nI(\theta)}}\xrightarrow{D} N(0,1)\] or frequently you see this written as \[\sqrt{n}(\hat\theta_n -\theta_0)\xrightarrow{D} N(0,\frac{1}{I(\theta)}),\] where you leave the \(\sqrt{n}\) on the left-hand side and write down what remains on the left-hand side. This is what is done above in the statement of the distribution of asymptotic normality in terms of \(\nu(\theta)\).
The bound in the statement of the formula is also written in terms of \(\nu(\theta)\) and \(I(\theta),\) both of which are “for a single sample.” Since we are working with i.i.d. data, the factor of \(n\) cancels out anyway in the bound, but we could obviously write \[\frac{\nu(\theta)}{n} \geq \frac{1}{I_n(\theta)}=\frac{1}{nI(\theta)} \]
Robustness to Assumptions
The Cramér-Rao Inequality is a powerful result, and provides an asymptotic justification for using the MLE. However, it is important to recognize that this result is true only we are correct regarding the distribution of our data is true. As I’ve mentioned before, MLEs will generally be asymptotically normal even if our assumptions about the distribution are incorrect, but most commonly the variance will be different from what is given by the theory.
Proof of the Cramér-Rao Inequality
As mentioned above, we can actually generalize the Cramér-Rao Inequality for any estimate, even if its not unbiased,
Theorem 2 Let \(X_1, \ldots, X_n \sim f(x \mid \theta)\) be an IID random sample, and let \(T = t(X_1, \ldots, X_n)\) be an estimator of \(\theta\), such that \(\mathbb{E}(T) = \mathbb{E}[t(X_1, \ldots, X_n)] = g(\theta)\). Then for sufficiently smooth \(f\),
\[ \text{Var}(T) \geq \frac{[g'(\theta)]^2}{nI(\theta)}. \]
Further, if \(\mathbb{E}(T) = g(\theta) = \theta\), that is, \(T\) is an unbiased estimator of \(\theta\), then \[ \text{Var}(T) \geq \frac{1}{I_n(\theta)} = \frac{1}{nI(\theta)} \]
Remark 1. If \(g(\theta)\) is the identity function, that is \(g(\theta) = \mathbb{E}_\theta(T) = \theta\), we recover the Cramér-Rao Inequality for unbiased estimators.
We sketch the proof below in the case of a continuous density function:
\[\begin{align*} g(\theta) &= \mathbb{E}_\theta(t(X_1, \ldots, X_n)) \\ &= \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} t(x_1, \ldots, x_n)\, f(x_1, \ldots, x_n \mid \theta)\, dx_1 \cdots dx_n\\ &= \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} t(x_1, \ldots, x_n)\, f(x_1 \mid \theta) \cdots f(x_n \mid \theta)\, dx_1 \cdots dx_n \end{align*}\] Taking the derivative with respect to \(\theta\) and changing the order of differentiation and integration, we get: \[\begin{align*} g'(\theta) &= \frac{\partial}{\partial \theta}\int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} t(x_1, \ldots, x_n)\, f(x_1 \mid \theta) \cdots f(x_n \mid \theta)\, dx_1 \cdots dx_n\\ &=\int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} t(x_1, \ldots, x_n) \left[\sum_{i=1}^{n} \frac{\frac{\partial}{\partial\theta} f(x_i \mid \theta)}{f(x_i \mid \theta)}\right] f(x_1 \mid \theta) \cdots f(x_n \mid \theta)\, dx_1 \cdots dx_n \end{align*}\] In the last equality, we multiplied and divided by \(f(x_i \mid \theta)\).
Now let
\[ Y = \sum_{i=1}^{n} \frac{\frac{\partial}{\partial\theta} f(x_i \mid \theta)}{f(x_i \mid \theta)} = \sum_{i=1}^{n} \frac{\partial}{\partial\theta} \log f(x_i \mid \theta), \]
which tells us that \(Y\) is the sum of score functions \(\Rightarrow \mathbb{E}_\theta(Y) = 0\), and \(\text{Var}(Y) = I_n(\theta) = nI(\theta)\).
Back to \(g'(\theta)\):
\[ g'(\theta) = \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} t(x_1, \ldots, x_n) \cdot y(x_1, \ldots, x_n)\, f(x_1 \mid \theta) \cdots f(x_n \mid \theta)\, dx_1 \cdots dx_n \]
This means that \(g'(\theta) = \mathbb{E}_\theta(TY) = \text{Cov}(T, Y)\) since \(\mathbb{E}_\theta(Y) = 0\).
Let \(\rho_{TY} = \dfrac{\text{Cov}(T,Y)}{\sqrt{\text{Var}(T)}\sqrt{\text{Var}(Y)}}\) denote the correlation of \(T, Y\).
\[ \Rightarrow \rho_{TY} = \frac{\mathbb{E}_\theta(TY)}{\sqrt{\text{Var}(T)}\sqrt{\text{Var}(Y)}} = \frac{g'(\theta)}{\sqrt{\text{Var}(T)} \cdot \sqrt{nI(\theta)}} \]
We know that \(\rho_{TY}^2 \leq 1\), so:
\[ [g'(\theta)]^2 \leq \text{Var}(T) \cdot nI(\theta) \]
\[ \Rightarrow \frac{[g'(\theta)]^2}{nI(\theta)} \leq \text{Var}(T) \qquad \blacksquare \]
Mean Square Error
We’ve already discussed how we would like an estimator to be both unbiased and low variance. Recall our archery figure,
In the illustration above, it seems like the right answer is to have unbiasedness and low variance. And relative efficiency allows us to evaluate different estimators based on their variance.
But what if your most efficient unbiased estimator looks more like the bullseye on the left than the right? Maybe we would prefer something with a little bias if we got better variance. (And, while we can have zero bias, we will never have zero variance – why?). In fact it is well known that sometimes accepting a small amount of bias will result in much smaller variance than any unbiased estimator.
So how should we compare estimators if they are potentially biased? In this case the variance estimates \[var(\hat\theta)= E(\hat\theta-E(\hat\theta))^2\] which is not what we are really interested in if \(E(\hat\theta)\neq \theta\).
We would prefer to estimate the difference between the expected value of the estimator and the true value of the population parameter, \[ E(\hat\theta-\theta)^2\]
Definition 4 (Mean Squared Error) The mean squared error is the expected value of the squared difference between the estimator \(\hat{\theta}\) and the true value of the population parameter \(\theta\). We denote it by \(MSE\): \[ MSE = E\left[\left(\hat{\theta} - \theta \right)^2\right] \]
If \(\hat\theta\) is unbiased, then clearly \(var(\hat\theta)=MSE(\hat\theta),\) but for biased estimators this isn’t the case.
We can show that
\[ \text{MSE}(\tilde{\theta}_n) = \left(\text{Bias}(\tilde{\theta}_n)\right)^2 + \text{Var}(\tilde{\theta}_n) \] So the MSE can actually be smaller for the biased estimator if the resulting variance is small. This is often called a bias-variance tradeoff and is particularly common in more complex problems, for example in prediction.
Exercise/Question
Show that \(MSE = \text{Variance} + \text{Bias}^2\).
Check your work
\[\begin{align*} \mathrm{MSE}(\hat\theta) &= E\big[(\hat\theta - \theta)^2\big] \\ &= E\big(\hat\theta ^2\big) -2 \theta E\big(\hat\theta\big) + \theta^2\\ &= \mathrm{Var}\big(\hat\theta\big) + \big[E\big(\hat\theta\big)\big]^2 -2 \theta E\big(\hat\theta\big) + \theta^2\\ &= \mathrm{Var}\big(\hat\theta\big) + \big[E\big(\hat\theta\big) - \theta\big]^2\\ &= \mathrm{Var}(\hat\theta) + \big[\mathrm{Bias}(\hat\theta)\big]^2. \end{align*}\]Example of MSE of a biased estimator
This is a common example used to look at biased estimators (Chihara and Hesterberg (2018)). Let \(X_1,\ldots,X_n\) be Bernoulli(\(\pi\)).
Estimator 1 The sample proportion \(\hat{\pi}_1 = \bar{X}=\sum X_i/n\) is an unbiased estimator of \(\pi\), and we have that \[MSE\left(\hat{\pi}_1\right) = var(\hat{\pi}_1)= \dfrac{\pi(1-\pi)}{n}.\]
Estimator 2 Consider an alternative estimator of \(\pi\), denoted by \(\hat{\pi}_2\), where \[\hat{\pi}_2 = \dfrac{\sum X_i+1}{n+2},\] that is, we add one artificial success and one artificial failure to the data. This is not an uncommon estimator to use, in particular if we are estimating something with a small probability of success. For \(\hat{\pi}_1\), if we don’t observe any \(X_i=1\) out of \(n=50\), then we estimate \(\hat{\pi}_1=0\), i.e. it’s impossible to get a success; but \(\hat{\pi}_2=0.03\), a small boost to our prediction of the chance that you might get a success.
Then we have for \(\hat\pi_2\) \[ \begin{gathered} bias(\hat\pi_2)=\frac{1-2\pi}{n+2} \neq 0\\ var(\hat\pi_2)=\frac{n}{(n+2)^2}\pi(1-\pi)\\ MSE(\hat\pi_2)=\frac{1+(n-4)\pi-(n-4)\pi^2}{(n+2)^2} \end{gathered}\] Notice that the variance of \(\hat\pi_2\) is always lower than our MLE \(\hat\pi_1\). This is despite the fact that the Cramer-Rao inequality tells us that \(\hat\pi_1\) acheives the lowest variance amongst unbiased estimators. \(\hat\pi_2\) is biased, so it’s not included in the set of unbiased estimators.
So to compare these estimators, we want to compare their MSE, which will balance the contribution of bias and variance.
Compare what the MSE looks like for different values of the true \(\pi\),
We see that except when \(\pi\) is on the boundary, we have better MSE with \(\hat\pi_2\), even though it’s slightly biased. The improvement is more striking for smaller sample sizes; for larger \(n\) the difference between the two is much smaller, though still remains. This makes sense, because for large \(n\), adding a single count or two to \(\sum X_i\) or \(n\) is not going to change them much, so \(|\hat\pi_2-\hat\pi_1|\rightarrow 0\).
Exercise/Question
Confirm the calculations given above for the bias, variance, and MSE of \(\hat\pi_2\)
References
Agresti, Alan, and Maria Kateri. 2021. Foundations of Statistics for Data Scientists. Chapman; Hall/CRC.
Chihara, Laura M., and Tim C. Hesterberg. 2018. Mathematical Statistics with Resampling and r. John Wiley & Sons, Hoboken, NJ.
Rice, John A. 2006. Mathematical Statistics and Data Analysis. 3rd ed. Duxbury Press.