Functions of Estimators and the Delta Method

Corresponding reading

  • Chapter 3.4, 4.2.6 (invariance of MLE) in Agresti and Kateri (2021)
  • Chapter 4.6 (page 162) in Rice (2006)

The discussion of the delta method draws extensively from material from Shobhana Stoyanov’s lecture notes from Spring 2026, Lecture 8.

Questions/Goals for this Module

  • How do you estimate a function of your MLE?
  • How does the Delta Method allow you to find the asymptotic variance of these estimators?
  • Why these estimators are also asymptotically normal

Estimating a function of Parameter with MLE

So far all of our examples where we used the MLE, what we wanted to estimate was the actual parameter of the parametric model.

But consider the exponential distribution, with density \[f(x|\lambda)= \lambda e^{-\lambda x},\] which was an exercise in the MLE section. We found a MLE for the rate parameter \(\lambda\), \[\hat\lambda=\frac{1}{\bar{X}}.\] But what if that isn’t what we are interested in, and we want the MLE for the mean of the distribution, let’s call it \(\mu\)? That is given by \[\mu=E(X)=\frac{1}{\lambda}\] But \(\mu\) isn’t part of the density of the exponential… Can we find a MLE for \(\mu\)?

In fact, yes, and there is a very simple solution. If you know the MLE for the parameters of a distribution, and you want the MLE for a function of those parameters, you can just substitute the MLE into the function. So for an exponential, \[\hat{\mu}=\frac{1}{\hat{\lambda}}=\bar{X}\] Which is both simple, and quite logical – you see almost every time, MLE is going to tell you that the best way to estimate the mean of the distribution is the mean of the data, irrespective of the distribution you assume.

Specifically, we have the following Theorem:

Theorem 1 (The MLE of functions of \(\theta\) (Invariance of MLE)) Let \(\tau = g(\theta)\) be a function of \(\theta\). Let \(\hat\theta\) be the MLE of \(\theta\). Then \(\hat\tau = g(\hat\theta)\) is the MLE of \(\tau\).

Example: standard deviation

Another example is the parameter \(\sigma^2\) in our normal. We emphasized that we will make our life easier by working with \(\sigma^2\) as our parameter, not \(\sigma\). But now we see that the MLE of \(\sigma\) will just be the square-root of our MLE of \(\sigma^2\), which justifies our approach!

Exercise/Question

Consider our Bernoulli example for when we have binary data \(X_1,\ldots,X_n\). What if instead of estimating the probability of success, \(\pi\), I want to estimate the log-odds, \[\tau=\log(\frac{\pi}{1-p}).\] What would be the MLE?

Exercise/Question

Consider what happens if we assume our data is i.i.d. \(N(\mu,\sigma^2)\), and we want to estimate the coefficient of variation (CV): \(\mu/\sigma\). What is the MLE for the CV?

Exercise/Question

Suppose you decided to model your data with the Gamma(\(\alpha, \beta\)) distribution, where \(\beta\) is the rate parameter. In other words, you choose to model \(X_1,\ldots,X_n\) as each being independent with distribution Gamma and shared parameters \(\alpha,\beta\). Since there are two ways to parameterize the Gamma distribution, let’s be specific, and write the density as, \[ f(x)=\frac{\beta^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}; \alpha,\beta >0 \] Suppose you had a MLE for \(\alpha\) and \(\beta\), what is \(g\) and what would be the MLE for the mean of the distribution?

Exercise/Question

What characteristics would you want of your data \(X_1,\ldots,X_n\) to make Gamma a good choice for modeling your data?

Proof

The theorem holds for all \(g\), but it’s pretty easy to prove in the case that \(g\) is one-to-one (i.e. invertible). In this case, we could rewrite the density with \(\tau\) the new parameter of the density by replacing \(\theta\) with \(g^{-1}(\tau)\),1, calculate the likelihood \(\ell(\tau)\), and find a \(\tau\) that maximizes it. This would give you the MLE of \(\tau\). This theorem says you don’t need to go through this process, because \(\hat\tau=g(\hat\theta)\) is the MLE of \(\tau\), i.e. maximizes \(\ell(\tau)\).

Let’s assume that \(\hat{\theta}\) is the MLE for \(\theta\), i.e. it is the value \(\theta\) that maximizes \(\ell(\theta)\). Then for for any particular \(\tau_0\), there is a single \(\theta_0=g^{-1}(\tau_0)\) and we can calculate the likelihood of our data using the likelihood with \(\tau_0\) or with \(\theta_0\), and these values will be the same, \[\ell(\tau_0)=\ell(g^{-1}(\tau_0))=\ell(\theta_0)\] So for any particular \(\tau_0\) and corresponding \(\theta_0\), we have \[\ell(\tau_0)=\ell(\underbrace{g^{-1}(\tau_0)}_{\theta_0}) \leq \ell(\hat\theta) \] because of the definition of \(\hat\theta\) as the value that maximizes \(\ell(\theta)\).

But because \(g\) is one-to-one, \(\hat\theta\) corresponds to a unique \(\hat\tau=g(\hat\theta)\) and \[\ell(\hat\theta_n)=\ell(\hat\tau_n)\] so that means \[\ell(\tau_0) \leq \ell(\hat\tau) \] This is true for any \(\tau_0\), so therefore \(\hat\tau = g(\hat{\theta})\) maximizes \(\ell\) over all other possible values of \(\tau\). This means \(\hat\tau\) is the MLE for \(\tau\).

The distribution of \(g(\hat\theta)\)

We have the MLE of \(g(\theta)\) is given by \(g(\hat\theta)\), but how do we find it’s variance or distribution?

This type of situation is common in statistics, and not at all limited to MLEs. We have a random variable \(U\) for which we know the mean and variance. But we are actually interested in a different random variable, \(V = g(U)\), and we want the mean and variance of \(Y\).

The problem is that \(g\) is not a linear function, so \(E\left(g(U)\right) \ne g\left(E(U)\right)\).

We are going to discuss a method to approximate the distribution of \(Y\) called the delta method or the propagation of error formula.

Taylor Series

The main idea behind the delta method is that we use a Taylor series expansion of \(g\), so let’s remind ourselves what the Taylor series expansion is.

Theorem 2 (Taylor series expansion) Let \(k\geq 1\) be an integer and let the function \(g: R \rightarrow R\) be \(k\) times differentiable at the point \(a\in R\). Then there exists a function \({\displaystyle h_{k}:R\rightarrow R }\) such that

\[{\displaystyle g(x)=\sum _{i=0}^{k}{\frac {g^{(i)}(a)}{i!}}(x-a)^{i}+R_k(x),} \] where \[R_k(x)=h_{k}(x)(x-a)^{k}\] and \[{\displaystyle \lim_{x\to a}h_{k}(x)=0.}\] \(R_k(x)\) is often called the remainder term.

This is called the Taylor series expansion of \(g\) around \(a\).

This means that if \(x\) is sufficiently near \(a\), we can approximate \(g(x)\) by a \(n\)th order polynomial. So for example, this allows us to say \[g(x)\approx b_0+b_1 x\] or \[g(x)\approx b_0+b_1x+b_2x^2\] with coefficients that depend on the point \(a\) where we are trying to approximate the function \(g\). Moreover, these coefficients are of the form \((x-a)^k\), so if \(x\) is near \(a\), the higher order terms of the polynomial become much smaller.

Expectation and Variance of \(g(\hat\theta)\)

Why is this useful for random variables? We are going to apply this to our function \(g\) so that we can say something about \(Y = g(X)\), when \(X\) is near \(a=\mu_X\).

If we do that, it will be easy to determine the approximate expectation and variance of \(Y\) because we assume we know the expectation and variance of \(X\). For example, if \(Y=a+bX\), we know \[E(a+bX)=b\mu_X \qquad var(a+bX)=b^2var(X).\] With a first order expansion of \(g(x)\), we will be able to do this approximately. This method of linearizing \(Y\) in order to approximate its mean and variance is called the Delta method.

First-order expansion

Let’s do the first order expansion: \[ Y = g(\mu_X) + (X-\mu_X)g'(\mu_X) + R. \]

\(R\) is the remainder term that will be negligible if \(X\) is closely concentrated around its mean.

This implies that \[\displaystyle Y \approx g(\mu_X) + (X-\mu_X)g'(\mu_X),\] and by linearity we get: \[\begin{aligned} E(Y) &\approx E\left[g(\mu_X)\right]\quad\text{since }E(X-\mu_X) = 0\\ var(Y) &\approx \left[g'(\mu_X)\right]^2 var(X) \end{aligned} \]

Second-order expansion

Note we just said \(E(g(X))\neq g(E(X))\). Delta’s method only says this is approximately true. We can also do a second order Taylor’s series expansion to improve our approximation and see what we are ignoring in our approximation. \[ Y = g(X) \approx g(\mu_X) + (X-\mu_X)g'(\mu_X) + \dfrac{1}{2}(X-\mu_X)^2 g''(\mu_X) \] And we see indeed that \(E(g(X))\neq g(E(X))\) if you have more careful approximation, and taking expectations gives us that \[ E(Y) \approx g(\mu_X) + \dfrac{1}{2} \sigma_X^2 g''(\mu_X). \] This additional term tells us that how good the approximation \(E(Y) \approx E\left[g(\mu_X)\right]\) will depends on:

  1. How nonlinear \(g\) is in a neighborhood of \(\mu_X\) (because of \(g''(\mu_X)\)) and
  2. On the variance of \(X\) (because of the \(\sigma_X^2\)).

Note that we can apply the delta method in a multivariate setting, but here we will only consider univariate functions.

Delta Method Theorem for Asymptotic Normality

Now we have the mean and the variance (approximately) of \(g(X)\), and we are ready for its asymptotic distribution.

Theorem 3 Suppose we have a sequence of random variables \(\{U_n\}\) that is asymptotically normal. In other words, there exist constants \(\mu, \sigma\), where \(\sigma > 0\), such that \[ \sqrt{n} \left(\dfrac{U_n -\mu}{\sigma} \right)\overset{D} \longrightarrow \mathcal{N}(0,1). \] That is, the distribution of \(U_n\), suitably standardized, converges to a standard normal distribution. Then, for any differentiable function \(g\), such that \(\lvert g'(\mu)\rvert \ne 0\), we have: \[ \sqrt{n} \left(\dfrac{g(U_n) -g(\mu)}{\lvert g'(\mu)\rvert \sigma} \right)\overset{D} \longrightarrow \mathcal{N}(0,1). \] That is, \(g(U_n)\) is also approximately normal for large \(n\): \[ g(U_n) \approx \mathcal{N}\left(g(\mu), \frac{\lvert g'(\mu)\rvert^2 \sigma^2}{n}\right) \]

Sketch of Proof

Using Taylor expansion (or the delta method) of \(g\) about \(\mu\), we have that \[ g(U_n) = g(\mu) + (U_n - \mu)g'(\mu) + R, \] where \(R\) is a remainder term that goes to \(0\) as \(n \rightarrow \infty\).

Subracting \(g(\mu)\) from both sides and multiplying both sides by \(\sqrt{n}\) we get: \[ \sqrt{n}\left[g(U_n) - g(\mu)\right] = \sqrt{n} (U_n - \mu)g'(\mu) + R \] Notice that multiplying the remainder term by \(\sqrt{n}\) makes no difference, it will still go to \(0\) as \(n \rightarrow \infty\), so we just continue to write it as \(R\).

We know that \(\sqrt{n} (U_n - \mu)/\sigma\) is approximately standard normal(that is \(\sqrt{n} (U_n - \mu)/\sigma \approx \mathcal{N}(0, 1))\) which implies that \(\sqrt{n} (U_n - \mu)\) is approximately normal with mean \(0\) and variance \((\sigma\cdot g'(\mu))^2\). Therefore the lefthand side also must be approximately normal with the same parameters: \[ \sqrt{n}\left[g(U_n) - g(\mu)\right] \approx \mathcal{N}\left(0, [\sigma\cdot g'(\mu)]^2\right). \]

which gives us the result of the theorem (by dividing by \(\lvert \sigma\cdot g'(\mu)\rvert\)).

Distribution of functions of our MLE

Returning to our problem. Instead of \(\{U_n\}\), we have an MLE estimator \(\hat\theta_n\) which is asymptotically normally distributed \[ \hat{\theta}_n \approx N\!\left(\theta_0,\ \frac{1}{ n I(\theta_0)}\right), \] So \(I(\theta_0)^{-1}\) will take on the role of \(\sigma^2\) in our earlier theorem, and \(\theta_0\) will take on the role of \(\mu\).

We want to estimate \(\tau_0=g(\theta_0)\), and we know from earlier that \(\hat\tau_n=g(\hat\theta_n)\) is the MLE for \(\tau_0\).

From the delta method, the asymptotic variance of \(\hat\tau_n\) is given by \[ SE(\hat\tau)=\lvert g'(\theta_0)\rvert I(\theta_0)^{-1/2}/\sqrt{n}\] Note I call this quantity \(SE(\hat\tau)\) with the understanding that it is its asymptotic standard error.

We can also use the delta method theorem to get the asymptotic distribution of \(\hat\tau_n\), \[ \left(\frac{\hat\tau_n - \tau_0}{SE(\hat\tau)}\right)=\sqrt{n}\left(\frac{g(\hat\theta_n) - g(\theta_0)}{\lvert g'(\theta_0)\rvert I(\theta_0)^{-1/2}}\right)\overset{D}{\longrightarrow}\mathcal{N}(0,1). \] This implies that \[ \hat\tau \approx \mathcal{N}\left(g(\theta_0), \frac{(g'(\theta_0))^2}{nI(\theta_0)}\right) \] This is perhaps a slightly confusing statement, since it only involves \(\theta_0\) and nothing about the parameter we are estimating, \(\tau_0\). But remember, we have defined \(\tau_0=g(\theta_0)\), so we can write it in terms of either one.

Example: Log-Odds

Above we considered trying to estimate the for when we have binary \(X_1,\ldots,X_n\) and model our data as Bernoulli(\(\pi_0\)). The log odds is defined as \[\tau=g(\pi)=\log(\frac{\pi}{1-\pi}).\] Our MLE of \(\pi\) is \(\hat{\pi}=\bar{X}\), which is asymptotically normally distributed, \(N(\pi_0, \frac{\pi_0(1-\pi_0)}{n})\).

The MLE of \(\tau\) by the invariance principle is \[\hat\tau=\log(\frac{\bar{X}}{1-\bar{X}})\] and by the Delta Theorem, \[\hat\tau=g(\hat\pi) \approx \mathcal{N}\left(g(\pi_0), \lvert g'(\pi_0)\rvert^2 \frac{\pi_0(1-\pi_0)}{n}\right)\] Then \[ \begin{aligned} g'(\pi)&=\frac{1-\pi}{\pi}\left(\frac{1}{1-\pi}+\frac{\pi}{(1-\pi)^2} \right)\\ &=\frac{1}{\pi}+\frac{1}{1-\pi}\\ &=\frac{(1-\pi)+\pi}{\pi(1-\pi)}=\frac{1}{\pi(1-\pi)}. \end{aligned}\] So the asymptotic variance of \(\hat\tau\) is given by \[\frac{1}{[\pi_0(1-\pi_0)]^2}\frac{\pi_0(1-\pi_0)}{n}=\frac{1}{n\pi_0(1-\pi_0)},\] and \[\hat\tau \approx \mathcal{N}\left(log(\frac{\pi_0}{1-\pi_0}),\frac{1}{n\pi_0(1-\pi_0)}\right)\] We can notice that if we invert \(g\), we have that \[\pi_0=\frac{e^{\tau_0}}{1+e^{\tau_0}}\] so that \[\pi_0(1-\pi_0)=\frac{e^{\tau_0}}{1+e^{\tau_0}}(1-\frac{e^{\tau_0}}{1+e^{\tau_0}})=\frac{e^{\tau_0}}{1+e^{\tau_0}}\frac{1}{1+e^{\tau_0}}=\frac{e^{\tau_0}}{(1+e^{\tau_0})^2} \] so we can write this in terms of \(\tau_0\), \[\hat\tau \approx \mathcal{N}\left(\tau_0,\frac{e^{\tau_0}}{n(1+e^{\tau_0})^2}\right)\]

Confidence Intervals

We will of course, have to estimate the unknown quantity \(\theta_0\) to get our estimate of standard error.

\[ \hat{SE}(\hat\tau)=\lvert g'(\hat\theta_0)\rvert I(\hat\theta_0)^{-1/2}/\sqrt{n}\] We can substitute in \(\hat{SE}(\hat\tau)\) and the normal asymptotics still apply, \[ \frac{\hat\tau_n - \tau_0}{\hat{SE}(\hat\tau)}\overset{D}{\longrightarrow}\mathcal{N}(0,1).\]

This allows us to make asymptotic Wald confidence intervals, \[\hat\tau \pm z_{\alpha/2}\hat{SE}(\hat\tau)\]

Back to Example: Log-Odds

We can estimate the asymptotic standard error of \(\hat\tau\) with \[\hat{SE}(\hat\tau)= (1+e^{\hat\tau})\sqrt{\frac{e^{\hat\tau}}{n}}\] to get a confidence interval, \[\hat\tau \pm z_{\alpha/2}(1+e^{\hat\tau})\sqrt{\frac{e^{\hat\tau}}{n}}\]

References

Agresti, Alan, and Maria Kateri. 2021. Foundations of Statistics for Data Scientists. Chapman; Hall/CRC.
Rice, John A. 2006. Mathematical Statistics and Data Analysis. 3rd ed. Duxbury Press.

Footnotes

  1. Note you can do this with parameters. You can’t do this with random variables↩︎