Consistency
We have discussed properties of our estimators based on their exact sampling distribution. But for many of our estimators, we can’t calculate their moments exactly, and some of our estimators we can’t write down an equation for the estimator, like in logistic regression.
We’ve seen that MLE estimators are asymptotically normal. We are now going to talk about additional properties similar to bias, efficiency, and MSE, but it will all be asymptotic, i.e. the behavior for large sample sizes.
Review of convergence
Asymptotic analysis of an estimator considers how the estimator behaves with larger and larger sample sizes. We can make statements like “as \(n \rightarrow \infty\), we would like \(\hat{\theta}_n\) to converge to \(\theta\)”. Or “as \(n \rightarrow \infty\), \(\hat{\theta}_n\) converges to a normal distribution”.
This section is going to briefly review different types of convergence of random variables and what they mean.
Types of Convergence
There are three different kinds of ways of interpreting the statement \(\hat{\theta}_n \rightarrow \theta\) that you might encounter while studying probability - well, you will mostly encounter the first two, but you might have seen the third in a course on real analysis or measure theory.
Let \(U_1, U_2, \ldots\) be a sequence of random variables, such that \(U_n\) has cdf \(F_n\), and let \(U\) be another random variable with CDF \(F\).
\(U_n\) converges in distribution to \(U\), written as \(U_n \overset{D}{\rightarrow} U\), if \(P(U_n \le x) \rightarrow P(U \le x)\) as \(n \rightarrow \infty\) at all points \(x\) at which the function \(F(x) = P(U \le x)\) is continuous. This means that the CDFs of the \(U_n\) converge to the CDF of \(U\) (\(F_n \rightarrow F\) as \(n\rightarrow \infty\)).
\(U_n\) converges in probability to \(U\), written as \(U_n \overset{P}{\rightarrow} U\), if, for all \(\varepsilon > 0\), we have that \(P(\lvert U_n - U \rvert > \varepsilon) \rightarrow 0\) as \(n \rightarrow \infty\). This isn’t saying that we guarantee that \(U_n\) will get very close to \(U\) for large \(n\). We are saying that it is very likely.
\(U_n\) converges almost surely, written as \(U_n \overset{a.s.}{\rightarrow} U\), if \(\displaystyle P\left(\lim_{n\rightarrow \infty} U_n = U\right) =1\). That is, the event \(\{\omega \in \Omega: \displaystyle \lim_{n\rightarrow \infty} U_n(\omega) = U(\omega)\}\) has probability 1. This convergence guarantees that \(U_n\) will converge to \(U\) except on a set of measure 0 (for example, on finitely many points in a continuous interval). This is put down for completeness, but is usually only encountered in graduate probability.
You can see that the first type is the weakest, and the last is the strongest. \[ \left(U_n \overset{a.s.}{\rightarrow} U\right) \Rightarrow \left(U_n \overset{P}{\rightarrow} U\right) \Rightarrow \left(U_n \overset{D}{\rightarrow} U\right) \] In the above statement the \(\Rightarrow\) means that convergence of the type on the left implies convergence of the type on the right. This is what we mean when we say a type of convergence on the left of an arrow is stronger than any of the types of convergences on its right.
The converse is not true
The above arrows do not work in the other direction. We generally see this by constructing counterexamples, where we have convergence of a weaker type and show that it doesn’t imply convergence of the stronger type. Here’s an example from Grimmett and Stirzaker (2001) where convergence in distribution does not imply convergence in probability.
Suppose \(U\sim Bernoulli(1/2)\), and the sequence of random variables \(U_1, U_2, \ldots\) consists of identical random variables, with each \(U_n = U\) for all \(n\). Now the \(U_n\) are not independent (since they are all equal to each other), but \(U_n \overset{D}{\rightarrow}U\). If \(V = 1-U\), then \(U\) and \(V\) have the same distribution, which means that \(U_n \overset{D}{\rightarrow} V\). But \(U_n\) cannot converge to \(V\) in any other sense because \(\lvert U_n - V\rvert = 1\) always.
Convergence of \(\bar{X}\)
We will now revisit two famous limit theorems for the sample mean \(\bar{X}\) that you know from probability, each of which demonstrates these different kinds of convergence. In these theorems, the convergence of interest is for \(U_n=\bar{X}\).
The Central Limit Theorem
The CLT states that for large \(n\), the sample mean, suitably standardized, will have a CDF that approaches the CDF of the standard Normal. That is, the standardized sample mean converges in distribution to the standard Normal.
If \(X_1, X_2, \ldots, X_n\) is an independent and identically distributed sample from a population with mean \(\mu\) and variance \(\sigma^2\), then:
\[ \left(\frac{\overline{X}-\mu}{\frac{\sigma}{\sqrt{n}}}\right) = \sqrt{n}\left(\frac{\overline{X}-\mu}{\sigma}\right) \overset{D}{\longrightarrow}\mathcal{N}(0,1) \text{ as } n\longrightarrow \infty \]
A Note on Notation
You have to be a bit careful about how you write formal asymptotic statements here. We might write \[\bar{X}\approx N(\mu,\frac{\sigma^2}{n})\quad\text{for large }n.\] But we cannot write \[\bar{X} \xrightarrow{D} N(\mu,\frac{\sigma^2}{n}) \quad \text{as }n\rightarrow\infty,\] because it doesn’t make sense to have something that depends on \(n\) be something to which we converge! This is generically true, but particularly comes up when we are looking at convergence in distribution to normality, where the variance term will generally depend on \(n\).
Instead I could write \[\frac{\bar{X} -\mu}{\sigma/\sqrt{n}}\xrightarrow{D} N(0,1)\] or frequently you see this written as \[\sqrt{n}(\bar{X} -\mu)\xrightarrow{D} N(0,\sigma^2),\] where you leave the \(\sqrt{n}\) on the left-hand side and write down what of the variance term doesn’t depend on \(n\) on the right-hand side.
Similarly, you cannot have anything that depends on the data in the distribution you converge to (the right-hand side), so if we replace \(\sigma^2\) with an estimate of \(\sigma^2\), like \(S^2\), then \(S^2\) will always be on the left-hand side of the convergence statement, as part of the random sequence (note that \(S^2=S_n^2\) depends on \(n\), as will any function of the data, so this is really a special case of the general rule)
The Weak Law of Large Numbers
Let \(X_1, X_2, \ldots, X_n\) be a sequence of iid random variables with mean \(\mu\) and variance \(\sigma^2\), then if \(\overline{X}_n \left(= \dfrac{1}{n} \displaystyle\sum_{i=1}^{n}X_i \right)\) is the sample mean, then for every \(\varepsilon > 0\), we have that \(\lim_{n \rightarrow \infty} P(\lvert \overline{X}_n - \mu \rvert > \varepsilon )=0\).
That is, the sample mean converges in probability to the true mean. Notice that in this case, the “random variable” \(U\) to which our sequence converges is the constant \(\mu\).
Properties of convergence
The following properties can be shown for any of the convergences above, but we will only write them down for convergence in probability (which you can easily work out yourself from the definition):
If \(U_n \xrightarrow{P} U\) and \(V_n \xrightarrow{P} V\), then
\[ U_n + V_n \xrightarrow{P} U + V. \]If \(U_n \xrightarrow{P} U\) and \(c\) is a constant, then
\[ c U_n \xrightarrow{P} c U. \]If \(U_n \xrightarrow{P} a\) and the real-valued function \(g\) is continuous at \(a\), then \[ g(U_n) \xrightarrow{P} g(a). \]
If \(U_n \xrightarrow{P} U\) and \(V_n \xrightarrow{P} V\), then
\[ U_n V_n \xrightarrow{P} U V. \] Less obvious properties:Continuous Mapping Theorem: If we have a sequence of random variables \({U_i}\) such that \(U_n \xrightarrow{P} U\) and \(g\) is a continuous function, then \(g(U_n) \xrightarrow{P} g(U)\).
The above is a special case of this if we let \(U=a,\) a constant. The continuous mapping theorem is also true for the other types of convergence, not just convergence in probability.
Slutsky’s Theorem: if \(U_n \xrightarrow{D} U\) and \(V_n \xrightarrow{P} c\), then \[\begin{gathered} U_n/V_n \xrightarrow{D} U/c \\ U_nV_n \xrightarrow{D} Uc\\ U_n + V_n \xrightarrow{D} U+c . \end{gathered} \] Notice that Slutsky’s theorem is mixing up two types of convergence: the assumptions are that \(U_n\) is converging in distribution while \(V_n\) is converging in probability, while the resulting convergence of combining them is in distribution.
Tail Inequalities
In order to prove the WLLN, we need Chebyshev’s inequality. In order to prove Chebyshev’s inequality, we need Markov’s inequality. These are both called tail inequalities because they bound the tail of the distribution. That is, they put bounds on the probability that the random variable takes very large values.
Markov’s inequality
If \(X\) is a nonnegative random variable (\(P(X\ge 0 ) = 1\)) such that \(E(X) < \infty\) (that is, the expectation exists), then for any \(c> 0\), \[ P(X \ge c) \le \dfrac{E(X)}{c}. \]
Notice that this is a very simple inequality, and requires nothing but that the random variable be nonnegative. The proof is very simple - we just have to write the expected value as a sum over the region where \(X < c\) and where \(X \ge c\). Try it.
Chebyshev’s inequality
If \(X\) is a random variable with mean \(\mu\) and variance \(\sigma^2\), then we have that for any \(c>0\),
\[ P(\lvert X-\mu\rvert> c)\le \dfrac{\sigma^2}{c^2}. \]
Exercise/Question
Apply Markov’s inequality to \(\lvert X - \mu\rvert^2\) to prove Chebyshev’s inequality.
Consistency
Returning to estimation, a fundamental concept using asymptotics considers whether as our sample size gets larger and larger (we get more data), does our estimator become more accurate? That seems like a minimal asymptotic goal. The convergence here is what we call convergence in probability, which basically means that in the long run, the probability of \(\hat{\theta}_n\) being very close to \(\theta\) goes to 1.
Definition 1 (Consistency) Let \(\hat{\theta}_n\) be an estimator of a parameter \(\theta\), based on a sample of size \(n\). We say that \(\hat{\theta}_n\) is a consistent estimator of \(\theta\) if \(\hat\theta\) converges in probability to \(\theta\) for all values of \(\theta\), \[\hat{\theta}_n \overset{P}{\rightarrow} \theta, \forall \theta.\] That is, for any \(\varepsilon > 0\), we have that \(\displaystyle \lim_{n \rightarrow \infty} P\left(\lvert \hat{\theta}_n -\theta \rvert > \varepsilon \right)=0\).
This is saying that for any acceptable amount of error epsilon, the probability of an error in our estimation that is greater than epsilon goes to 0. This means that the error might be large on events of very low probability, and these probabilities get lower as \(n\) gets larger. If the sample is large enough, therefore, the estimator is very likely to be close to its target.
Moreover, the continuous mapping theorem tells us that if \(\hat\theta\) is a consistent estimator of \(\theta\), then \(g(\hat\theta)\) is a consistent estimator of \(g(\theta)\) if \(g\) is a continuous function.
Notice that for consistency, we are using convergence in probability to a constant, i.e. to the true unknown parameter we are trying to estimate. Convergence in probability is a more general concept, since it is defined more broadly for convergence to another random variable.
We are also saying that the convergence of this estimator is true for any valid value of \(\theta\).
Example
For this discussion, we follow Casella and Berger (2002):
Suppose \(X_i \sim N(\theta, 1)\), so that \(f(x \mid \theta) = \dfrac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}(x-\theta)^2}\).
If \(\hat{\theta}_n = \overline{X}_n = \dfrac{\sum_{i=1}^n X_i}{n}\), then:
\[\hat{\theta}_1 = \overline{X}_1 = X_1\]
\[\hat{\theta}_2 = \overline{X}_2 = \frac{X_1 + X_2}{2}\]
\[\hat{\theta}_3 = \overline{X}_3 = \frac{X_1 + X_2 + X_3}{3}, \quad \text{etc.}\]
Consistency says this sequence of estimators \(\{\hat{\theta}_n\}\) is a consistent sequence for the parameter \(\theta\) if for every \(\varepsilon > 0\) and every \(\theta \in \Theta\) (the parameter space): \[\lim_{n \to \infty} P\!\left(|\hat{\theta}_n - \theta| < \varepsilon\right) = 1\] Moreover, the definition of consistency means this is true for the entire family of distributions indexed by \(\theta\).
We can prove consistency of the estimator directly, using the definition and the distributional assumption of normality (rather than rely on the WLLN).
Proof. If \(X_1, \ldots, X_n \overset{\text{IID}}{\sim} N(\theta, 1)\), then \(\overline{X}_n \sim N\!\left(\theta, \tfrac{1}{n}\right)\).
We want to show \(P(|\overline{X}_n - \theta| < \varepsilon) \to 1\). Note that since \(\overline{X}_n \sim N\!\left(\theta, \tfrac{1}{n}\right)\), its density is \(f_{\overline{X}_n}(x) = \sqrt{\dfrac{n}{2\pi}}\, e^{-\frac{n(x-\theta)^2}{2}}\).
\[P(|\overline{X}_n - \theta| < \varepsilon) = P(\theta - \varepsilon < \overline{X}_n < \theta + \varepsilon)\]
\[= \int_{\theta-\varepsilon}^{\theta+\varepsilon} f_{\overline{X}_n}(x \mid \theta)\, x = \int_{\theta-\varepsilon}^{\theta+\varepsilon} \sqrt{\frac{n}{2\pi}}\, e^{-\frac{n(x-\theta)^2}{2}}\, dx\]
We will do two variable changes: First, let \(y = x - \theta\), so \(dy = dx\); when \(x = \theta - \varepsilon\), \(y = -\varepsilon\):
\[= \sqrt{\frac{n}{2\pi}} \int_{-\varepsilon}^{\varepsilon} e^{-\frac{ny^2}{2}}\, dy\]
Next, let \(t = \sqrt{n}\, y\), so \(dt = \sqrt{n}\, dy\), \(t^2 = ny^2\); when \(y = -\varepsilon\), \(t = -\varepsilon\sqrt{n}\):
\[\Rightarrow \frac{1}{\sqrt{2\pi}} \int_{-\varepsilon\sqrt{n}}^{\varepsilon\sqrt{n}} e^{-t^2/2}\, dt = P(-\varepsilon\sqrt{n} < Z < \varepsilon\sqrt{n})\]
As \(n \to \infty, \varepsilon\sqrt{n} \to \infty\), and \(P(-\infty < Z < \infty) = 1\), where \(Z \sim \mathcal{N}(0,1)\).
Therefore, \(\overline{X}_n \text{ is a consistent (sequence of) estimator(s) of } \theta. \quad \blacksquare\).
Consistency of Sample moments
An important consequence of the WLLN concerns the sample moments. Recall the \(k\)th moment of a distribution is given by \[E(X^k)=\mu_k.\] The sample moment of data is given by \[\frac{1}{n}\sum_{i=1}^n X^k.\] Then the WLLN tells us that the sample moments converge in probability to the population moments \(\mu_k\), and thus are consistent estimators.
The WLLN has very minimal assumptions, like the CLT, so the consistency of the sample moments requires very minimal knowledge about the distribution of the data. Some important implications of the consistency of sample moments are:
- the sample mean (\(\overline{X}_n\)) is a consistent estimator of the mean \(\mu=\mu_1\), which is just the first moment.
- the sample variance is a consistent estimator of the variance.
- more broadly, in statistics we will often approximate or estimate an expectation with a sum of i.i.d. quantities. We see this in the bootstrap, for example.
Exercise/Question
Recall the sample variance \(\displaystyle S^2 = \dfrac{1}{n-1}\sum_{i=1}^n \left(X_i - \overline{X}_n\right)^2\).
Is \(S^2\) a consistent estimator of \(\sigma^2\)?
What about \(\hat{\sigma}^2 = \dfrac{n-1}{n}S^2\)?
Asymptotic Normality and Estimating the Variance
We have frequently had an estimator \(\hat\theta\) which is asymptotically normal, \[\frac{\hat\theta-\theta}{SE(\hat\theta)} \overset{D}{\rightarrow}N(0,1).\] But \(SE(\hat\theta)\) is unknown – it is the true standard error of \(\hat\theta\) and generally depends on \(\theta\). To create confidence intervals, we need an estimate of that quantity and we need that we still have asymptotic normality, i.e. \[\frac{\hat\theta-\theta}{\hat{SE}(\hat\theta)} \overset{D}{\rightarrow}N(0,1).\] We have just been stating this as a fact for our estimates of \(SE(\theta)\), but this is actually due to an important general concept. If we have a consistent estimator of \(SE(\hat\theta)\), say \(\hat{SE}(\hat\theta)\), then if \[\frac{\hat\theta-\theta}{SE(\hat\theta)} \overset{D}{\rightarrow}N(0,1),\] then by Slutsky’s theorem \[\frac{\hat\theta-\theta}{\hat{SE}(\hat\theta)} \overset{D}{\rightarrow}N(0,1)\] So we can always replace the standard error with a consistent estimate of it. This is what we have repeatedly done so far in creating our asymptotic confidence intervals.
Relationship of Consistency to MSE
Theorem 1 Let \(\hat\theta_n\) be an estimator of \(\theta\). If the MSE of an estimator \(\hat\theta_n\) goes to 0, the estimator must be a consistent estimator of \(\theta\).
This is often easier to show than computing the probability that we need, and that is why the theorem is so useful.
Moreover, if we can show that, as \(n \to \infty\):
- \(\operatorname{Var}(\hat{\theta}_n) \to 0\)
- \((\operatorname{Bias}(\hat{\theta}_n))^2 \to 0\)
then \(\operatorname{MSE}(\hat{\theta}_n) \to 0\), which implies \(\hat{\theta}_n\) is consistent. In the above example, since \(\overline{X}_n \sim \mathcal{N}\left(\theta, \dfrac{1}{n}\right),\) it is immediate that the MSE \(\to 0\), and therefore \(\overline{X}_n\) is a consistent estimator of \(\theta\).
Proof. Recall the following definitions for an estimator \(\hat \theta_n\) of the parameter \(\theta\).
\[\text{Bias}(\hat{\theta}_n) = E(\hat{\theta}_n) - \theta\]
\[\text{Var}(\hat{\theta}_n) = E\left[(\hat{\theta}_n - E(\hat{\theta}_n))^2\right]\]
\[\text{MSE}(\hat{\theta}_n) = E\left[(\hat{\theta}_n - \theta)^2\right] = \text{Bias}(\hat{\theta}_n)^2 + \text{Var}(\hat{\theta}_n)\]
We want to show that \(\text{MSE}(\hat{\theta}_n) \to 0 \Rightarrow \hat{\theta}_n \xrightarrow{P} \theta\),
that is, \(\forall \varepsilon > 0\), \(\displaystyle \lim_{n \to \infty} P(|\hat{\theta}_n - \theta| > \varepsilon) = 0\).
Applying Markov’s inequality to \(Y = (X - a)^2 \geq 0\) for some \(a\), \(c >0\), we get:
\[P(Y > c^2) = P\!\left((X-a)^2 > c^2\right) = P(|X-a| > c) \leq \frac{E\!\left[(X-a)^2\right]}{c^2} \]
Now let \(X = \hat{\theta}_n\), \(a = \theta\), \(c = \varepsilon > 0\). We get:
\[P(|\hat{\theta}_n - \theta| > \varepsilon) \leq \frac{E\left[(\hat{\theta}_n - \theta)^2\right]}{\varepsilon^2} = \frac{\text{MSE}(\hat{\theta}_n)}{\varepsilon^2}\]
Since the MSE of \(\hat\theta_n\) goes to \(0\) as \(n \to \infty\), this implies that \[ \lim_{n\to\infty} P(|\hat{\theta}_n - \theta| > \varepsilon) = 0 \Rightarrow \hat{\theta}_n \xrightarrow{P} \theta. \] We have shown that if \(MSE(\hat\theta_n) \to 0\), then \(\hat\theta_n\) must be a consistent estimator for \(\theta\).
\(\blacksquare\)
Example
Let \(X_1, \dots, X_n \overset{IID}{\sim} \mathcal{N}(\mu, \sigma^2)\). Since \(E(X_i) = \mu\) and \(\mathrm{Var}(X_i) = \sigma^2\).
Consider an alternative to our usual estimator of \(\mu\), \[\hat\theta_n = \overline{X}_n + \dfrac{1}{n}.\] Is it a consistent estimator of \(\mu\)?
\(E(\hat\theta_n) = E\left(\overline{X}_n + \dfrac{1}{n}\right) = \mu + \dfrac{1}{n}\), the bias of \(\hat\theta_n\) must be \(\dfrac{1}{n}\): \[ \mathrm{Bias}(\hat\theta_n) = E(\hat\theta_n) - \theta = \mu + \dfrac{1}{n} - \mu = \dfrac{1}{n}, \] since \(\theta = \mu\) in this example. Therefore, \(\mathrm{Bias}(\hat\theta_n) \to 0\) as \(n\to \infty\). Note that \(\left(\mathrm{Bias}(\hat\theta_n)\right)^2 = \dfrac{1}{n^2} \to 0\) as \(n \to \infty\).
Further, \[ \mathrm{Var}(\hat\theta_n) = \mathrm{Var}\left(\overline{X}_n + \dfrac{1}{n}\right) = \mathrm{Var}\left(\overline{X}_n \right) = \dfrac{\sigma^2}{n} \to 0 \text{ as } n \to \infty. \] As a consequence of the statements above, we have that the MSE of \(\theta_n\) goes to 0 as \(n \to \infty\), and therefore, \(\hat\theta_n\) is a consistent estimator of \(\mu\).
Reverse is not true: Consistency \(\not\Rightarrow\) MSE \(\to 0\)
Suppose we are trying to estimate \(\mu\). Let:
\[\hat{\theta}_n = \begin{cases} \overline{X}_n & \text{w.p. } 1 - \tfrac{1}{n} \\ n & \text{w.p. } \tfrac{1}{n} \end{cases}\]
With probability \(1 - \frac{1}{n}\), \(\hat{\theta}_n = \overline{X}_n \xrightarrow{P} \theta = \mu\), so \(\hat{\theta}_n \xrightarrow{P} \theta\).
But with probability \(\frac{1}{n}\), \(\hat{\theta}_n = n\), and the MSE satisfies:
\[\operatorname{MSE} = E\!\left[(\hat{\theta}_n - \theta)^2\right]\]
\[= E\!\left[(\overline{X}_n - \theta)^2 \cdot \mathbf{1}(\hat{\theta}_n = \overline{X}_n)\right] + E\!\left[(n - \theta)^2 \cdot \mathbf{1}(\hat{\theta}_n = n)\right]\]
\[\geq E\!\left[(n - \theta)^2 \cdot \frac{1}{n}\right] = \frac{(n-\theta)^2}{n} \to \infty \text{ as } n \to \infty\]
Therefore, consistency does not imply that \(\operatorname{MSE} \to 0\).
Consistency of the MLE of \(\theta\)
As we might guess, the maximum likelihood estimator from an IID sample with PDF \(f(x\vert \theta)\), is consistent for \(\theta\) (under appropriate smoothness conditions on \(f\), which means that we can take as many continuous derivatives of \(f\) as we need -it won’t have any sharp corners etc.).
In the following, we will assume \(f\) is smooth. This will ensure that it is “well-behaved” and we can interchange the orders of integration and differentiation.
Theorem 2 Let \(\widehat{\theta}_n\) be the MLE of \(\theta_0\). Then \(\hat \theta_n\) is consistent for \(\theta_0\). That is, \(\hat \theta_n \overset{P}{\rightarrow}\theta_0\). This means that, for any \(\epsilon > 0\), \[ \lim_{n \rightarrow \infty} P(\lvert \hat \theta_n - \theta_0\rvert > \epsilon) = 0. \]
Summary of the Consistency Argument for i.i.d. data
We let \(U_i=\log f(X_i|\theta)\) and \(U=U(\theta)=\log f(X|\theta)\), for a generic single \(X\) from the same distribution.
- \(\hat{\theta}_n\) maximizes \(L(\theta)= \sum_{i=1}^n U_i.\)
- \(\theta_0\) maximizes \(E[U] \equiv E[U(\theta)]= E\left[\log f(X\mid \theta)\right]\)
- By WLLN \[ \frac{1}{n}L(\theta)=\frac{1}{n} \sum_{i=1}^n U_i \xrightarrow{P} E[U] \forall \theta. \]
- Apply the continuous mapping theorem to give that the maximizers \(\hat{\theta}_n\) of the left-hand side will converge to the maximizer \(\theta_0\) of the right-hand side.
Proof of Consistency
In our proof below, \(\theta_0\) is the true unknown value of the parameter \(\theta\), and \(f(x \mid \theta_0)\) is the true PDF (\(\theta_0\) is some fixed, but unknown value).
Recall that \(X_1, \dots, X_n \overset{\text{IID}}{\sim} f(x \mid \theta)\) is a random sample, and \[ \ell(\theta) = \ell(\theta \mid X_1, \dots, X_n) = \prod_{i=1}^n f(X_i \mid \theta) \]
Define \(L(\theta) = \log \ell(\theta)\). Then, \[ \frac{1}{n} L(\theta) =\frac{1}{n} \log \ell(\theta) = \frac{1}{n} \sum_{i=1}^n \log f(X_i\mid \theta). \]
For ease of writing, we will write \(L'(\theta) = \dfrac{\partial l }{\partial \theta}\) and \(L''(\theta) = \dfrac{\partial^2 L }{\partial \theta^2}\) where possible.
Proof. \[ \Rightarrow \quad \frac{1}{n} L(\theta) = \frac{1}{n} \sum_{i=1}^n \log f(X_i \mid \theta). \]
The RHS is the sample mean of the IID random variables \[ U_i = \log f(X_i \mid \theta). \]
By the WLLN, \[ \frac{1}{n} \sum_{i=1}^n U_i \xrightarrow{P} E[U]. \] Now, since \(U \equiv U(\theta)= \log f(X\mid \theta)\), we have
\[ E[U] = E[\log f(X \mid \theta)] = \int \log f(x \mid \theta) \cdot f(x \mid \theta_0)\, dx \] where \(f(x \mid \theta_0)\) is the true PDF from which the sample \(X_1, \dots, X_n\) was generated.
By the WLLN, for large \(n\), with high probability, \(\dfrac{1}{n} \sum_{i=1}^n U_i\) and \(E[U]\) will be close, which means that \(\dfrac{1}{n} L(\theta)\) and \(E[U]\) will be close.
The maximizer of \(L(\theta)\) is \(\hat{\theta}_n\), and by the above, this should be close to the maximizer of \(\displaystyle \int \log f(x \mid \theta) \cdot f(x \mid \theta_0)\, dx\).
Let’s maximize \(E[U]\) (note that \(E \equiv E_{\theta_0}\)): \[\begin{align*} E[U] &= E[\log f(X \mid \theta)] \\ &= \int \log f(x \mid \theta) \cdot f(x \mid \theta_0)\, dx \\ \end{align*}\] Taking the derivative with respect to \(\theta\) on both sides (and using the fact that \(f\) is smooth to justify changing the order of operations) we get: \[\begin{align*} \frac{\partial}{\partial \theta} E[\log f(X \mid \theta)] &= \frac{\partial}{\partial \theta} \int \log f(x \mid \theta) \cdot f(x \mid \theta_0)\, dx\\ &= \int \frac{\partial}{\partial \theta}\log f(x \mid \theta)\cdot f(x \mid \theta_0)\, dx\\ &= \int \frac{\frac{\partial}{\partial \theta} f(x \mid \theta)} {f(x \mid \theta)} \cdot f(x \mid \theta_0)\, dx \end{align*}\]
Now we will show that \(\theta_0\) maximizes \(E[U]=E[\log f(X \mid \theta)]\), by plugging \(\theta_0\) into the right-hand side above to get:
\[ \int \frac{\frac{\partial}{\partial \theta}f(x \mid \theta_0) } {f(x \mid \theta_0)} \cdot f(x \mid \theta_0)\, dx = \int\frac{\partial f(x \mid \theta_0)}{\partial \theta} \, dx = \frac{\partial}{\partial \theta} \int f(x \mid \theta_0)\, dx = 0. \]
We get \(0\) at the end because \(\int f(x \mid \theta)\, dx = 1\), since \(f(x \mid \theta)\) is a density function and integrates to 1. Therefore, we see that \(\theta_0\) is a stationary point for \(E[\log f(X \mid \theta)]\).
We can show that \(\theta_0\) is a max by showing that \(E[\log f(X \mid \theta)] - E[\log f(X \mid \theta_0)] \le 0\) for all \(\theta\).
\[\begin{align*} E[\log f(X \mid \theta)] - E[\log f(X \mid \theta_0)] &= \int \log f(x \mid \theta) \cdot f(x \mid \theta_0) \, dx - \int \log f(x \mid \theta_0) \cdot f(x \mid \theta_0) \, dx\\ &= \int \left[ \log f(x \mid \theta) - \log f(x \mid \theta_0) \right] \cdot f(x \mid \theta_0) \, dx\\ &= \int \log \left[ \frac{f(x \mid \theta)}{f(x \mid \theta_0)} \right] \cdot f(x \mid \theta_0) \, dx \\ \end{align*}\]
Note that \(\log t \leq t - 1\). This implies that (with \(t = \displaystyle \frac{f(x \mid \theta)}{f(x \mid \theta_0)}\) )
\[\begin{align*} E[\log f(X \mid \theta)] - E[\log f(X \mid \theta_0)] &\leq \int \left[\frac{f(x \mid \theta)}{f(x \mid \theta_0)} - 1 \right] \cdot f(x \mid \theta_0) \, dx\\ &= \int f(x \mid \theta) \, dx \;-\; \int f(x \mid \theta_0) \, dx \\ &= 1 - 1 = 0 \end{align*}\]
Therefore, \[ E[\log f(X \mid \theta)] - E[\log f(X \mid \theta_0)] \leq 0 \quad \text{for all } \theta \]
Which means that \(\theta_0 \text{ maximizes } E_{\theta_0} \log f(X \mid \cdot)\).
So by the Weak Law of Large Numbers:
\[\frac{1}{n} \sum_{i=1}^{n} \log f(X_i \mid \theta) \xrightarrow{\quad} E_{\theta_0} [ \log f(X \mid \theta)]\]
- The left side is \(\dfrac{1}{n}L(\theta)\), maximized by \(\hat{\theta}_n\) (the MLE).
- The right side is \(E[U(\theta)]\), maximized by \(\theta_0\) (the true parameter).