Delta Method

Often estimators are functions of other random variables, for example in the method
of moments. These functions of random variables can sometimes inherit a normal ap-
proximation from the underlying random variables. Earlier in the course we obtained a
result where a continuous function of a sequence of consistent estimators also inherited
the property of being consistent estimators. The delta method allows a normal approx-
imation (a normal central limit type or result, that is convergence in distribution to a
normal distribution) for a continuous and differentiable function of a sequence of r.v.s
that already has a normal limit in distribution.

Example : Method of Moments for Exponential Distribution.

Xi , i = 1, 2, . . . , n are iid exponential, λ with pdf

f (x; λ) = λe−λx I(x > 0)

The first moment is then µ1 (λ) = λ1 . The the method of moments estimator is

1
λ̂n =
X̄n

Notice this is of the form λ̂n = g(X̄) where g : R+ 7→ R+ with g(x) = x1 .

Theorem 1 (Delta Method) Suppose X̄n has an asymptotic normal distribution, that
is
√
n(X̄n − µ) → N (0, γ 2 )

1
in distribution as n → ∞. Suppose g is a function that is continuous and also has a
derivative g ′ at µ, and that g ′ (µ) ̸= 0. Then
√ ( )
n g(X̄n ) − g(µ) → N (0, (g ′ (µ))2 γ 2 )

Remark : The condition g ′ (µ) ̸= 0 is actually only needed so that

√ ( )
n g(X̄n ) − g(µ)
→ N (0, 1)
|g ′ (µ)|γ
in distribution as n → ∞.

Remark : Theorem 1 is called the delta method.

Proof (Outline)

The first order Taylor approximation of g about the point µ, and evaluated at the
random variable X̄n is ( )
g(X̄n ) ≈ g(µ) + g ′ (µ) X̄n − µ
√
Subtract g(µ) from both sides and multiply by n gives
√ ( ) √ ( )
n g(X̄n ) − g(µ) ≈ g ′ (µ) n X̄n − µ → N (0, (g ′ (µ))2 γ 2 )

Remark : A more careful study of Taylor’s formula with remainder is needed to

justify all steps in this approximation. For our purposes in this course these details are
not needed.

Remark : Notice this delta method is an extension of the idea used earlier to approx-
imate moments, specifically to approximate means and variances.

Example Continued : For the exponential example we have

√ ( 1
)
1
n X̄n − → N (0, 2 )
λ λ
Using the function g(x) = 1/x expanded about µ = λ1 , and noting
1
λ̂n = g(X̄n ) = and g(µ) = λ
X̄n

2
we thus have as an application of the delta method (Theorem 1)
( ) ( )
√ ( ) 1√ 1 λ4
n λ̂n − λ ≈ − 1 n X̄n − → N 0, 2 = N (0, λ2 )
λ2
λ λ

In our discussion of the Central Limit Theorem (CLT) we studied in the case of iid
sampling from a distribution with 2 moments that
√
n(X̄n − µ)
√
σ2
converges in distribution to N (0, 1). The student should refer back to that section of the
text and the notes to review the notation and conditions and the statement of the CLT.
We also discussed, but without proof, that one can replace the population variance σ 2
in the denominator with the r.v. Sn2 (the sample variance) and still obtain convergence
in distribution to N (0, 1). This is due to Sn2 being a consistent estimator of σ 2 , that is
Sn2 → σ 2 in probability as n → ∞.

One can replace the population variance σ 2 in the denominator with any other consis-
tent estimator. Therefore in the exponential example above we have two random variables
that may be used
√
n(X̄n − λ1 )
Z1,n = √
1
λ̂2n
√
n(X̄n − λ1 )
Z1,n = √
Sn2

both of which converge in distribution to N (0, 1). Both of these may be used to construct
an approximate or asymptotic confidence interval for µ = E(X). This will be discussed
later in the course.

3
 There is also a delta method for random vectors. It is described in the same fashion.
For simplicity of writing and since we only apply it in our course for dimension 2, we
write it in this form. For this we need a preliminary theorem.

Theorem 2 Suppose Xi , i ≥ 1 are iid with finite 4-th moments. Write

1∑ n
1∑ n
µ̂1,n = Xi and µ̂2,n = X2
n i=1 n i=1 i

and let µk be the k-th population moments. Then

√
n (µ̂1,n − µ1 , µ̂2,n − µ2 ) → BVN(0, A)

in distribution as n → ∞, where BVN means bivariate normal with mean vector 0 and
variance (or covariance) matrix
 
µ2 − µ21 µ3 − µ1 µ2 
A=
µ3 − µ1 µ2 µ4 − µ22

Remark : This is a consequence of a multivariate central limit theorem, and is not

proven in the course.

In the Theorem above the matrix A is a variance matrix. What are some properties
of this matrix that we can use below?

Suppose that b1 , b2 are two constants. Set b = (b1 , b2 ), a 1 by 2 matrix (row vector).
Suppose also that (X, Y )T is a 2 by 1 matrix (column vector) with components the r.v.s
X, Y and variance matrix A. Then
 
X 
b = b1 X + b2 Y .
Y

Then

Var(b1 X + b2 Y ) = b21 Var(X) + 2b1 b2 Cov(X, Y ) + b22 Var(X)

= bAbT .

4
The student should write out the matrix multiplication and verify this.

Similarly if B is a matrix of appropriate size then W = B(X, Y )T is a random vector.

In this case the random vector W has variance matrix

BAB T .

End of Remark

5
 Theorem 3 (below) is the delta method applied to a function of (µ̂1,n , µ̂2,n ). We state
this rather than the general delta method to avoid more complicated notation. The
idea is the same as used in Theorem 1, but is based on working with bivariate normal
distributions, and more generally with multivariate normal distributions.

Theorem 3 Suppose the conditions of Theorem 2. Suppose g is a function of two vari-

ables mapped to two variables, that is continuous and also has a derivative g ′ at (µ1 , µ2 ),
and that g ′ (µ1 , µ2 ) is non zero. Then
√
n (g(µ̂1,n , µ̂2,n ) − g(µ1 , µ2 )) → N (0, g ′ (µ1 , µ2 )Ag ′ (µ1 , µ2 )T )

Remark :

If g maps pairs into R, then we interpret g ′ (µ1 , µ2 ) as a row vector of length 2. In this
case
g ′ (µ1 , µ2 )Ag ′ (µ1 , µ2 )T
is a (1 × 2) × (2 × 2) × (2 × 1) = 1 × 1 matrix, that is a real number.

If g maps pairs into R × R, then we interpret g ′ (µ1 , µ2 ) as a 2 by 2 matrix.

g ′ (µ1 , µ2 )Ag ′ (µ1 , µ2 )t

is a product of a 2 by 2 matrix g ′ , a 2 by 2 matrix A and 2 by 2 matrix (g ′ )T (the
transpose of the first 2 by 2 matrix). Since variance matrices are positive definite this
resulting matrix is also positive definite, that is it is a variance matrix.

End of Remark

For an application of this result, see the rainfall data example and the method of
moments for that example. Part of this example is discussed in more detail later in
this handout. In that example we use the following fact, that for a bivariate normal
distribution the marginal distribution of each component is normal. For example for the
first component we have a normal distribution with variance given by the (1,1) component
of the variance matrix.

6
 When we discussed the central limit theorem (CLT) we stated without proof, that
one can replace the population variance σ 2 with a consistent estimator of σ 2 , in that case
s2n the sample variance, and still retain the convergence in distribution to N(0,1). This
same property carries over more generally. In our delta method this corresponding result
allows one to replace (the first two for g(X̄n ) and the last two for g(µ̂1,n , µ̂2,n ))

• σ 2 = Var(X) by s2n

• g ′ (µ) by g ′ (X̄n )

• The matrix A = Var(X, X 2 ) by

 
µ̂2,n − µ̂21,n µ̂3,n − µ̂1,n µ̂2,n 
Ân = 
µ̂3,n − µ̂1,n µ̂2,n µ̂4,n − µ̂22,n

• g ′ (µ1 , µ2 ) by g ′ (µ̂1,n , µ̂2,n )

We do not write this in a technically proper manner in this course.

7
 Example : Delta Method applied to Gamma method of moment estimator

Recall method of moments estimator for the Gamma parameters are

µ̂1,n
λ̃n =
µ̂2,n − µ̂21,n
µ̂1,n µ̂1,n
α̃n = λ̃n µ̂1,n =
µ̂2,n − µ̂21,n

Thus we have
λ̃n = g(µ̂1,n , µ̂2,n )
where the function g is given by
x
g(x, y) = .
y − x2
Thus we find
∂g y + x2
=
∂x (y − x2 )2
∂g x
= −
∂y (y − x2 )2

Thus the matrix (or row vector) for the partial derivative of g is

∂g(x, y)
g ′ (x, y) =
∂(x, y)
( )
y + x2 x
= , − .
(y − x2 )2 (y − x2 )2

Aside If one writes the partial derivative as a column vector

 
y+x2
g ′ (x, y) =  (y−x2 )2 
− (y−x
x
2 )2

8
then one has to be careful to use the appropriate transposes in the matrix multiplications
above. It does not matter as long as one is consistent. In this handout we will use the
row vector.

End of Aside

Recall λ = g(µ1 , µ2 ), by the construction of our method of moments estimator.

The first order Taylor’s formula approximation for λ̃n is then

λ̃n − λ = g(µ̂1,n , µ̂2,n ) − g(µ1 , µ2 )

 
µ̂1,n − µ1 
≈ g ′ (µ1 , µ2 ) 
µ̂2,n − µ2

Then
 
√ √ µ̂1,n − µ1 
n(λ̃n − λ) ≈ g ′ (µ1 , µ2 ) n 
µ̂2,n − µ2
⇒ g ′ (µ1 , µ2 )W

as n → ∞, and where W has the bivariate normal distribution with mean vector 0
and covariance matrix A (see earlier section). The symbol ⇒ is used her for shorthand
convergence in distribution. Thus
√
n(λ̃n − λ) ⇒ g ′ (µ1 , µ2 )W
∼ N (0, g ′ (µ1 , µ2 )Ag ′ (µ1 , µ2 )T ) .

√
In particular we have that for large n, the r.v. n(λ̃n − λ) has an approximate normal
distribution. Thus our estimator has an asymptotic normal distribution approximation.

In particular we can use this to construct confidence intervals for λ.

9
 There are a few additional ideas that are needed to make use of the delte method,
Theorem 3, in practice. Recall that it gives a limit normal distribution
√
n (g(µ̂1,n , µ̂2,n ) − g(µ1 , µ2 )) → N (0, g ′ (µ1 , µ2 )Ag ′ (µ1 , µ2 )T )

so that
g ′ (µ1 , µ2 )Ag ′ (µ1 , µ2 )T (1)
is the variance matrix for the limiting normal distribution. We also need a sample esti-
mate of this; this will play the role of the sample variance in our simpler 1 dimensional
standardized sample mean √
n(X̄ − µ)
t= √ . (2)
S2
If the Xi are iid normal then (2) has a student’s t distribution. If not, but the sample
size n is large, then (2) has approximately a standard normal distribution.

In (1) we estimate A by using the sample momengts

 
µ̂2 − µ̂2 µ̂3 − µ̂1 µ̂2 
 = 
µ̂3 − µ̂1 µ̂2 µ̂4 − µ̂22

and estimate g ′ (µ1 , µ2 ) by also using the sample moments

g ′ (µ̂1 , µ̂2 ) .

Here the estimate of the sample moment with data x1 , x2 , . . . , xn is

1∑ n
µ̂k = xki
n i=1

Earlier when we needed to keep track of the sample size in our notation we used µ̂k,n , and
as we noted earlier we drop the extra subscript n except where we need to pay attention
to this sample size.

10