INTRODUCTION TO VECTOR

AND MATRIX DIFFERENTIATION

Econometrics 2
Heino Bohn Nielsen
September 21, 2005

T
his note expands on appendix A.7 in Verbeek (2004) on matrix differenti-
ation. We first present the conventions for derivatives of scalar and vector
functions; then we present the derivatives of a number of special functions
particularly useful in econometrics, and, finally, we apply the ideas to derive the
ordinary least squares (OLS) estimator in the linear regression model. We should
emphasize that this note is cursory reading; the rules for specific functions needed in
this course are indicated with a (∗).

1 Conventions for Scalar Functions

Let β = (β 1 , ..., β k )0 be a k ×1 vector and let f (β) = f (β 1 , ..., β k ) be a real-valued function
that depends on β, i.e. f (·) : Rk 7−→ R maps the vector β into a single number, f (β).
Then the derivative of f (·) with respect to β is defined as
⎛ ∂f (β) ⎞
∂β 1
∂f (β) ⎜ .. ⎟
=⎜
⎝ . ⎟.
⎠ (1)
∂β
∂f (β)
∂β k

This is a k × 1 column vector with typical elements given by the partial derivative ∂f∂β(β) .
i
Sometimes this vector is referred to as the gradient. It is useful to remember that the
derivative of a scalar function with respect to a column vector gives a column vector as
the result1 .
1 ∂f (β)
We can note that Wooldridge (2003, p.783) does not follow this convention, and let ∂β
be a 1 × k
row vector.

1
 Similarly, the derivative of a scalar function with respect to a row vector yields the
1 × k row vector
∂f (β) ³ ∂f (β) ∂f (β)
´
= · · · .
∂β 0 ∂β 1 ∂β k

2 Conventions for Vector Functions

Now let ⎛ ⎞
g1 (β)
⎜ .. ⎟
g(β) = ⎜
⎝ . ⎟⎠
gn (β)
be a vector function depending on β = (β 1 , ..., β k )0 , i.e. g(·) : Rk 7−→ Rn maps the k × 1
vector into a n × 1 vector, where gi (β) = gi (β 1 , ..., β k ), i = 1, 2, ..., n, is a real-valued
function.
Since g(·) is a column vector it is natural to consider the derivatives with respect to a
row vector, β 0 , i.e. ⎛ ∂g (β) ⎞
1 ∂g1 (β)
∂β 1 · · · ∂β k
∂g(β) ⎜ ⎜ .. .. .. ⎟
⎟,
0 = ⎝ . . . ⎠ (2)
∂β
∂gn (β)
∂β · · · ∂g∂β n (β)
1 k

where each row, i = 1, 2, ..., n, contains the derivative of the scalar function gi (·) with
respect to the elements in β. The result is therefore a n × k matrix of derivatives with
typical element (i, j) given by ∂g∂β
i (β)
. If the vector function is defined as a row vector, it
j
is natural to take the derivative with respect to the column vector, β.
We can note that it holds in general that
µ ¶
∂ (g(β)0 ) ∂g(β) 0
= , (3)
∂β ∂β 0

which in the case above is a k × n matrix.

Applying the conventions in (1) and (2) we can define the Hessian matrix of second
derivatives of a scalar function f (β) as
⎛ ∂ 2 f (β) ∂ 2 f (β)
⎞
∂β 1 ∂β 1 ··· ∂β 1 ∂β k
∂ 2 f (β) ∂ 2 f (β) ⎜ .. .. .. ⎟
= =⎜
⎝ . . . ⎟,
⎠
∂β∂β 0 ∂β 0 ∂β
∂ 2 f (β) ∂ 2 f (β)
∂β k ∂β 1 ··· ∂β k ∂β k

∂ f (β)2
which is a k × k matrix with typical elements (i, j) given by the second derivative ∂β .
i ∂β j
Note that it does not matter if we first take the derivative with respect to the column or
the row.

2
 3 Some Special Functions
First, let c be a k × 1 vector and let β be a k × 1 vector of parameters. Next define the
scalar function f (β) = c0 β, which maps the k parameters into a single number. It holds
that
∂ (c0 β)
= c. (∗)
∂β
To see this, we can write the function as

f (β) = c0 β = c1 β 1 + c2 β 2 + ... + ck β k .

Taking the derivative with respect to β yields

⎛ ∂(c β +c β +...+c ⎞ ⎛ ⎞
1 1 2 2 k βk )
∂β 1 c1
∂f (β) ⎜ .. ⎟ ⎜ . ⎟
=⎜
⎝ . ⎟ = ⎜ .. ⎟ = c,
⎠ ⎝ ⎠
∂β
∂(c1 β 1 +c2 β 2 +...+ck β k )
∂β k ck

which is a k × 1 vector as expected. Also note that since β 0 c = c0 β, it holds that

¡ ¢
∂ β0c
= c. (∗)
∂β
Now, let A be a n × k matrix and let β be a k × 1 vector of parameters. Furthermore
define the vector function g(β) = Aβ, which maps the k parameters into n function values.
g(β) is an n × 1 vector and the derivative with respect to β 0 is a n × k matrix given by

∂ (Aβ)
= A. (∗)
∂β 0
To see this, write the function as
⎛ ⎞
A11 β 1 + A12 β 2 + ... + A1k β k
⎜ .. ⎟
g(β) = Aβ = ⎜
⎝ . ⎟,
⎠
An1 β 1 + An2 β 2 + ... + Ank β k

and find the derivative

⎛ ∂(A β +...+A ⎞ ⎛ ⎞
11 1 1k β k ) ∂(A11 β 1 +...+A1k β k )
∂β 1 ··· ∂β k A11 · · · A1k
∂g(β) ⎜ .. .. .. ⎟ ⎜ . .. .. ⎟
=⎜
⎝ . . . ⎟ = ⎜ ..
⎠ ⎝ . . ⎟
⎠ = A.
∂β 0
∂(An1 β 1 +...+Ank β k ) ∂(An1 β 1 +...+Ank β k )
∂β 1 ··· ∂β k An1 · · · Ank

Similarly, if we consider the transposed function, g(β) = β 0 A0 , which is a 1 × n row vector,

we can find the k × n matrix of derivatives as
¡ ¢
∂ β 0 A0
= A0 . (∗)
∂β

This is just an application of the result in (3).

3
 Now consider a quadratic function f (β) = β 0 V β for some k×k matrix V . This function
maps the k parameters into a single number. Here we find the derivatives as the k × 1
column vector ¡ ¢
∂ β0V β
= (V + V 0 )β, (∗)
∂β
or the row variant ¡ ¢
∂ β0V β
= β 0 (V + V 0 ). (∗)
∂β 0
If V is symmetric this reduces to 2V β and 2β 0 V , respectively. To see how this works,
consider the simple case k = 3 and write the function as
⎛ ⎞⎛ ⎞
³ ´ V11 V12 V13 β1
⎜ ⎟⎜ ⎟
β0V β = β 1 β 2 β 3 ⎝ V21 V22 V23 ⎠ ⎝ β 2 ⎠
V31 V32 V33 β3
= V11 β 21 + V22 β 22 + V33 β 23 + (V12 + V21 )β 1 β 2 + (V13 + V31 )β 1 β 3 + (V23 + V32 )β 2 β 3 .

Taking the derivative with respect to β, we get

⎛ ⎞
∂(β 0 V β)
¡ 0 ¢
∂ β Vβ ⎜ ∂(β∂β0 V1 β) ⎟
= ⎜ ⎝ ∂β0 2 ⎠
⎟
∂β ∂(β V β)
∂β 3
⎛ ⎞
2V11 β 1 + (V12 + V21 )β 2 + (V13 + V31 )β 3
⎜ ⎟
= ⎝ 2V22 β 2 + (V12 + V21 )β 1 + (V23 + V32 )β 3 ⎠
2V33 β 3 + (V13 + V31 )β 1 + (V23 + V32 )β 2
⎛ ⎞⎛ ⎞
2V11 V12 + V21 V13 + V31 β1
⎜ ⎟⎜ ⎟
= ⎝ V12 + V21 2V22 V23 + V32 ⎠ ⎝ β 2 ⎠
V13 + V31 V23 + V32 2V33 β3
⎛⎛ ⎞ ⎛ ⎞⎞ ⎛ ⎞
V11 V12 V13 V11 V21 V31 β1
⎜⎜ ⎟ ⎜ ⎟⎟ ⎜ ⎟
= ⎝⎝ V21 V22 V23 ⎠ + ⎝ V12 V22 V32 ⎠⎠ ⎝ β 2 ⎠
V31 V32 V33 V13 V23 V33 β3
= (V + V 0 )β.

4 The Linear Regression Model

To illustrate the use of matrix differentiation consider the linear regression model in matrix
notation,
Y = Xβ + ,

where Y is a T × 1 vector of stacked left-hand-side variables, X is a T × k matrix of

explanatory variables, β is a k × 1 vector of parameters to be estimated, and is a T × 1
vector of error terms. Here k is the number of explanatory variables and T is the number
of observations.

4
 One way to motivate the ordinary least squares (OLS) principle is to choose the esti-
bOLS of β, as the value that minimizes the sum of squared residuals, i.e.
mator, β
T
X
bOLS = arg min
β b2t = arg min b0b.
e
β e
β
t=1

Looking at the function to be minimized, we find that

³ ´0 ³ ´
b0b = Y − X β b Y − Xβb
³ ´³ ´
= Y0−β b0 X 0 Y − X β b

= Y 0Y − Y 0X β b−βb0 X 0 Y + βb0 X 0 X β
b
b+β
= Y 0 Y − 2Y 0 X β b0 X 0 X β,
b

where the last line uses the fact that Y 0 X βb and β b0 X 0 Y are identical scalar variables.
Note that b0b is a scalar function and taking the first derivative with respect to β b yields
the k × 1 vector
³ ´
¡ 0 ¢ b+β
0 Y − 2Y 0 X β b0 X 0 X βb
∂ bb ∂ Y
= b
= −2X 0 Y + 2X 0 X β.
b
∂β ∂βb

∂ (e0e)
Solving the k equations, e = 0, yields the OLS estimator
∂β
¡ 0 ¢−1 0
b
β OLS = X X X Y,

provided that X 0 X is non-singular.

To make sure that β bOLS is a minimum of b0b and not a maximum, we should formally
take the second derivative and make sure that it is positive definite. The k × k Hessian
matrix of second derivatives is given by
³ ´
¡ 0 ¢ b
0 Y + 2X 0 X β
2
∂ bb ∂ −2X
= = 2X 0 X,
b β
∂ β∂ b0 ∂βb0

which is a positive definite matrix by construction.

References
[1] Verbeek, Marno (2004): A Guide to Modern Econometrics, Second edition, John
Wiley and Sons.
[2] Wooldridge, Jeffrey M. (2003): Introductory Econometrics: A Modern Approach,
2nd edition, South Western College Publishing.