Gioele Zardini Control Systems II FS 2018

Lecture 9: Linear Quadratic Regulator

1 LQR
Motivation:
Last week, we introduced the concept o state feedback, with the idea of using the states
of the system and its dynamics to synthesize a controller using desired pole-placement.
This week we introduce Linear Quadratic control (LQ) and the special case of the Linear
Quadratic Regulator (LQR). The concept behind this control strategy has a key role for
control theory and is worth a detailed explanation.
Moreover, pole placement has some drawbacks:

• Does not work well with model uncertainty.

• Does not allow specific tuning of desired trade-offs (e.g. cost vs. performance).

1.1 Problem Definition

Given the dynamics of a system
d
x(t) = A · x(t) + B · u(t), A ∈ Rn×n , B ∈ Rn×m , x(t) ∈ Rn , u(t) ∈ Rm (1.1)
dt
find a controller
u(t) = f (x(t), t), t ∈ [0, ∞] (1.2)
that brings x(t) asymptotically to zero (with x(0) 6= 0). In other words it should hold:

lim x(t) = 0, (1.3)

t→∞

i.e. the cost Z ∞

JLQR (x(t), u(t)) = kz(t)k22 + ρku(t)k22 dt, ρ ∈ R+ (1.4)
0
is minimized, where ρ allows trade-off between energy of the input and energy of the
controlled signal and z(t) = Ex(t) + F u(t) can be chosen to contain the state variables
of interest. The LQR standard control loop is reported in Figure 1.

z(t) ∈ Rk
r(t) e(t) u(t) ẋ(t) = Ax(t) + Bu(t)
KLQR
- y(t) = Cx(t) + Du(t)
x(t)

Figure 1: LQR Problem: Closed loop system.

1
Gioele Zardini Control Systems II FS 2018

1.2 General Form

1.2.1 Preliminary Definitions
Since the cost function introduced in Equation 1.4 contains the euclidean norm, one recalls
its definition:
ku(t)k22 = u(t)| u(t)
(1.5)
X
= ui (t)2 .
i

The weighted euclidean norm reads

ku(t)k2R,2 = u(t)| Ru(t). (1.6)

Definition 1. A pair (A, C) is observable if and only if rank(O) = n = dim(A), where

 
C
 CA 
O =  ..  . (1.7)
 
 . 
CAn−1

Definition 2. A pair (A, C) is detectable if all the unobservable modes are stable.
Definition 3. A real matrix M is said to be positive definite (denoted as M > 0) when
the associated quadratic form V (z) is non-negative, i.e.
X
V (z) = mi,j zi zj = z | M z ≥ 0, ∀z 6= 0. (1.8)
i,j

Definition 4. A real matrix M is said to be positive semi-definite (denoted as M ≥ 0)

when the associated quadratic form V (z) is positive, i.e.
X
V (z) = mi,j zi zj = z | M z > 0, ∀z 6= 0. (1.9)
i,j

How can we check if a matrix is positive definite?

Eigenvalue Test
A real, symmetric matrix is (semi)-positive definite if and only if it has all (non-negative)
positive eigenvalues.

Sylvester’s Criterion
An symmetric matrix M ∈ Rm×m is positive definite if and only if all the upper-left i × i
submatrices (principal minors), i ∈ 1, . . . , m have positive determinant. In the case of
 
a b
A= , (1.10)
b d
the conditions are
1. a > 0.
2. ad − b2 > 0.

2
Gioele Zardini Control Systems II FS 2018

1.2.2 Weighted LQR

With these definitions, one can write (by dropping the time dependency for simplicity)
the weighted LQR problem as
Z ∞
JLQR (x(t), u(t)) = kz(t)k2Q̄,2 + ρku(t)k2R̄,2 dt
Z0 ∞
= z | Q̄z + ρu| R̄udt
0
Z ∞
= (Ex + F u)| Q̄ (Ex + F u) + ρu| R̄udt
Z0 ∞
(x| E | + u| F | ) Q̄Ex + Q̄F u + ρu| R̄udt


=
Z0 ∞
= x| E | Q̄Ex + x| E | Q̄F u + u| F | Q̄Ex + u| F | Q̄F u + ρu| R̄udt
Z0 ∞
x| E | Q̄E x + u| F | Q̄F + ρR̄ u + 2x| E | Q̄F udt




=
Z0 ∞
= x| Qx + u| Ru + 2x| N udt,
0
(1.11)

where Q̄, R̄ are symmetric and positive definite, ρ ∈ R+ , u(t) ∈ Rm×1 , z(t) ∈ Rk×1 ,
x ∈ Rn×1 and
R = F | Q̄F + ρR̄, R ∈ Rm×m
Q = E | Q̄E, Q ∈ Rn×n (1.12)
N = E | Q̄F.

Example 1. You are given the criterion

Z ∞
x21 + 6 · x1 · x2 + 100 · x22 + 6 · u21 + 10 · u22 dt.


J(x, u) = (1.13)
0

Matrix N is the zero matrix and matrices Q and R are

 
1 3
Q= (1.14)
3 100

and  
6 0
R= . (1.15)
0 10

1.3 Solution
If

• The system (A, B) is stabilizable (all unstable modes are reachable). Intuition: this
is necessary for state feedback to work (stabilizable means that the unstable modes
must be controllable). Controllability is the same as reachability for continuous
time systems).

3
Gioele Zardini Control Systems II FS 2018

• the pair (Ã, Q̃) = (A − BR−1 N | , Q − N R−1 N | ) is detectable. Intuition: this is

necessary to ensure internal stability, i.e. that the closed loop is asymptotically
stable. If the system input were known a priori to stabilize the closed loop, then
this condition would not be necessary. For now, just remember that internal stabil-
ity corresponds to input-output stability when no unstable zero/pole cancellations
occur.

then
uLQR (t) = − R−1 (N + P B)| x(t), (1.16)
| {z }
KLQR

where P is the real, symmetric, positive definite solution of the algebraic Riccati
equation
(N + P · B) · R−1 · (N | + B | · P ) − P · A − A| · P − Q = 0. (1.17)

1.3.1 Solving the ARE

Hamiltonian Method
Starting from Equation 1.17, one can rearrange as

(N + P · B) · R−1 · (N | + B | · P ) − P · A − A| · P − Q = 0
(N R−1 + P BR−1 )(N | + B | P ) − P · A − A| · P − Q = 0
N R−1 N | + N R−1 B | P + P BR−1 N | + P BR−1 B | P − P · A − A| · P − Q = 0
− (A − BR−| N | )| P − P (A − BR−1 N | ) +P (BR−1 B | ) P − (Q − N R−1 N | ) = 0.
| {z } | {z } | {z } | {z }
Ã| Ã R̃ Q̃
(1.18)

Hence, one gets the Riccati equation

Ã∗ P + P Ã| + P R̃P + Q̃ = 0, (1.19)

with the unknown quadratic matrix P . Note that this equation can be rewritten as
  

à R̃ I
P −I ∗ = 0, (1.20)
−Q̃ −Ã P
| {z }
H∈R2n×2n

where H is the hamiltonian matrix. In order to find the solution of the ARE, we assume
two things:

1. H has no eigenvalues on the imaginary axis, i.e. it wil have n eigenvalues in the
LHP and n in the RHP. Let the subspace spanned by the eigenvectors associated
to the stable eigenvalues (i.e. in the LHP) be
 
X1
XH = Im , (1.21)
X2

where X1 , X2 ∈ Cn×n .

2. X1 is invertible.

4
Gioele Zardini Control Systems II FS 2018

If the two assumptions are met, one says that the Hamiltonian belongs to the domain of
the Riccati operator, i.e. H ∈ dom(Ric). With these two assumptions, the solution of the
ARE can be computed as
P = X2 X1−1 . (1.22)
Remark. The ARE has in general more than one solution, but only one is stabilizing, i.e.
it makes the closed loop asymptotically stable (see assumption before).

Theorem 1. H ∈ dom(Ric) if there exists symmetri matrices P, H ∈ Rn×n with stable

H such that    
X1 X1
H = H, (1.23)
X2 X2
where

a) P = X2 X1−1 is real and symmetric.

b) P satisfies the ARE.

c) The matrix à + R̃P is stable (all eigenvalues are in the open LHP).

1.3.2 Direct Method

The direct method consists in introducing a matrix P with the correct dimensions and
unknowns, apply the Riccati equation and solve the system of equations.

5
Gioele Zardini Control Systems II FS 2018

1.4 Examples
Example 2. You have to design a LQ regulator for a plant with 2 inputs, 3 outputs and
6 state variables.

(a) What are the dimensions of A, B, C and D?

(b) What is the dimension of the transfer function u → y?

(c) What is the dimension of the matrix Q of JLQR ?

(d) What is the dimension of the matrix R of JLQR ?

(e) What is the dimension of the matrix K?

Solution.

(a) One can find the solution by analyzing the meaning of the matrices:

• Since we are given 6 states variables, the matrix A should have 6 rows and 6
columns, i.e. A ∈ R6×6 .
• Since we are given 2 inputs, the matrix B should have 2 columns and 6 rows,
i.e. B ∈ R6×2 .
• Since we are given 3 outputs, the matrix C should have 6 columns and 3 rows,
i.e. C ∈ R3×6 .
• Since we are given 2 inputs and 3 outputs, the matrix D should have 2 columns
and 3 rows, i.e. D ∈ R3×2 .

(b) Since we are dealing with a system with 2 inputs and 3 outputs, P (s) ∈ R3×2 .
Moreover, P (s) should have the same dimensions of D because of its formula.

(c) From the formulation of Q one can easily see that its dimensions are the same of
the dimensions of A, i.e. Q ∈ R6×6 .

(d) From
u(t) = −K · x(t).
we can see that K should have 6 columns and 2 rows, i.e. K ∈ R2×6 .

6
Gioele Zardini Control Systems II FS 2018

Example 3. A system is given as

ẋ1 (t) = 3 · x2 (t)

1
ẋ2 (t) = 3 · x1 (t) − 2 · x2 (t) + · u(t) (1.24)
2
7
y(t) = 4 · x1 (t) + · x2 (t).
3
a) Solve the LQR problem for the criterion
Z ∞
1
J(x(t), u(t)) = 7 · x1 (t)2 + 3 · x2 (t)2 + · u(t)2 dt (1.25)
0 4
and find the state feedback controller K using the direct method.

b) Solve a) using the Hamiltonian method.

c) Find the eigenvalues of the closed-loop system with the LQ regulator K.

d) Does the new criterion

Z ∞
10 2
Jnew = 70 · x21 + 30 · x22 + · u dt (1.26)
0 4
affect the solution for K?

Solution.
a) Using quadratic forms, one can identify Q̃ and R̃ to be (using the null matrix for
N)  
7 0
Q= (1.27)
0 3
and
1
R= . (1.28)
4
The state-space description of the system can be re-written in standard form as
      
ẋ1 (t) 0 3 x1 (t) 0
= + 1 u(t)
ẋ2 (t) 3 −2 x2 (t) 2
| {z } | {z }
A
  B (1.29)
x1 (t)
y(t) = 4 73


+ |{z}
0 u(t).
| {z } x2 (t)
D
C

In order to find the controller K, one has to compute the symmetric, positive definite
solution of the Riccati equation related to this problem. First, one has to look at
the form that this solution should have. Here B ∈ R2×1 . This means that since
Φ = Φ| we are dealing with Φ ∈ R2×2 of the form
 
ϕ1 ϕ2
Φ= . (1.30)
ϕ2 ϕ3

7
Gioele Zardini Control Systems II FS 2018

With the Riccati equation, it holds

Φ · (B · R−1 · B | ) · Φ − Φ · (A − BR−1 N | ) − (A − BR−1 N | )| Φ − Q = 0
 
−1 | | 0 0
Φ·B·R ·B ·Φ−Φ·A−A ·Φ−Q=
0 0
         
0 0 3 0 3 7 0 0 0
Φ · 1 · 4 · 0 12 · Φ − Φ ·


− ·Φ− =
2
3 −2 3 −2 0 3 0 0
         
2ϕ2 3ϕ2 3ϕ1 − 2ϕ2 3ϕ2 3ϕ3 7 0 0 0
· ϕ22 ϕ23 −


− − =
2ϕ3 3ϕ3 3ϕ2 − 2ϕ3 3ϕ1 − 2ϕ2 3ϕ2 − 2ϕ3 0 3 0 0
 2     
ϕ2 ϕ2 ϕ3 6ϕ2 + 7 3ϕ1 − 2ϕ2 + 3ϕ3 0 0
2 − = .
ϕ2 ϕ3 ϕ3 3ϕ1 − 2ϕ2 + 3ϕ3 6ϕ2 − 4ϕ3 + 3 0 0
(1.31)

Hence, one gets 3 equations (two elements are equal because of symmetry):

ϕ22 − 6 · ϕ2 − 7 = 0 (I)
ϕ2 · ϕ3 − 3 · ϕ1 + 2 · ϕ2 − 3 · ϕ3 = 0 (II) (1.32)
ϕ23 − 6 · ϕ2 + 4 · ϕ3 − 3 = 0 (III).

Sylvester’s Criterion: An Hermitian (here symmetric) matrix M ∈ Cm×m is positive

definite if and only if all the upper-left i×i submatrices (leading minors), i ∈ 1, . . . , m
has positive determinant. Applying this to Φ one gets the conditions:

(a) ϕ1 > 0.
(b) ϕ1 ϕ3 − ϕ22 > 0.

From the Equation (I), one gets

√
6± 64
ϕ2 =
2 (1.33)
= {−1, 7}.

Since we cannot discard a specific value, we pursue with ϕ2,1 = −1 and ϕ2,2 = 7:

Case ϕ2,1 = −1:

Plugging this into the Equation (III), one gets

ϕ23 + 6 + 4ϕ3 − 3 = 0
(1.34)
ϕ23 + 4ϕ3 + 3 = 0,

and
√
−4 ± 4
ϕ3 =
2 (1.35)
= {−3, −1}.

In order for these two values to fulfill the second Sylvester condition, it should hold
ϕ1 < 0, which violates the first condition. For this reason this is not a possible
choice.

8
Gioele Zardini Control Systems II FS 2018

Case ϕ2,1 = 7:

Plugging this into the Equation (III), one gets

ϕ23 − 42 + 4ϕ3 − 3 = 0
(1.36)
ϕ23 + 4ϕ3 − 45 = 0,
and
√
−4 ± 196
ϕ3 =
2 (1.37)
= {−9, 5}.
ϕ3 = 5 is the only value which does not violate the two Sylverster’s conditions.
Plugging the values into Equation (II) one gets

35 − 3ϕ1 + 14 − 15 = 0
34 (1.38)
ϕ1 = .
3
The solution of the Riccati equation hence is
 34 
7
Φ= 3 . (1.39)
7 5
The controller K can be computed as
K = R−1 B | Φ
 34 
1

7
=4· 0 2 · 3 (1.40)
7 5


= 14 10 .

b) Before applying the Hamiltonian method, one need to check

• (A, B) stabilizable (all unstable modes are reachable). The reachability matrix
for this pair


R = B AB
0 32 (1.41)
 
= 1
2
−1
has full rank, hence the system is reachable.
• (A, Q) detectable. The observability matrix for the pair
 
Q
O=
QA
 
7 0 (1.42)
0 3 
=0 21 


9 −6
has full column rank, hence the system is observable.

9
Gioele Zardini Control Systems II FS 2018

In order to use the Hamiltonian method, one needs to build the Hamiltonian matrix
 
à R̃
H= , (1.43)
−Q̃ −Ã|
where
à = A − BR−1 N |
R̃ = −BR−1 B | (1.44)
Q̃ = Q − N R−1 N | .
For this specific example (N = 0), one has
à = A.
 
0 1 1


R̃ = − 1 1 0 2
 2  4
(1.45)
0 0
= .
0 −1
Q̃ = Q.
Therefore, the Hamiltonian is
 
0 3 0 0
 3 −2 0 −1
H=
−7 0
. (1.46)
0 −3
0 −3 −3 2
In order to have the eigenvalues, one computes
 
−λ 3 0 0
 3 −2 − λ 0 −1 
det (H − λI) = det 
 −7

0 −λ −3 
0 −3 −3 2 − λ
   
−2 − λ 0 −1 3 0 −1
= −λ det  0 −λ −3  − 3 det −7 −λ −3 
−3 −3 2 − λ 0 −3 2 − λ
    
−λ −3 0 −λ (1.47)
= λ (2 + λ) det + det
−3 2 − λ −3 −3
    
−λ −3 −7 −λ
− 9 det − 3 det
−3 2 − λ 0 −3
 
−λ −3
= (λ2 + 2λ − 9) det − 3λ2 + 63
−3 2 − λ
= (λ2 − 9 − 2λ)(λ2 − 9 + 2λ) − 3λ2 + 63
= λ4 − 25λ2 + 144.
Therefore, the eigenvalues are
λ1,2 = ±3, λ3,4 = ±4. (1.48)
Since we only care about stable eigenvalues (in LHP), we compute the eigenvectors
for λ2 = −3 and λ4 = −4.
It holds:

10
Gioele Zardini Control Systems II FS 2018

• Eλ2 = E−3 : from (H − λ2 I) · x = 0 one gets the linear system of equations

 
3 3 0 0 0
 3
 1 0 −1 0  .
 −7 0 3 −3 0 
0 −3 −3 5 0
Using the first row as reference and subtracting the correct multiples of it from
the other rows, one gets the form
 
3 3 0 0 0
 0 −2 0 −1 0 
 .
 0 7 3 −3 0 
0 −3 −3 5 0
Using the second row as reference and subtracting the correct multiples of it
from the other rows, one gets the form
 
3 3 0 0 0
 0 −2 0 −1 0 
 0 0 3 − 13 0  .
 
2
0 0 0 0 0
Since one has one zero row, one can introduce a free parameter. Let x4 = s,
then x3 = 13
6
s, x2 = − 2s , x1 = 2s , s ∈ R. This defines the first eigenspace,
which is (multiplying everything by 6)
 
3
n −3 o
E−3 =   13  .
 (1.49)
6

• Eλ4 = E−4 : from (H − λ4 I) · x = 0 one gets the linear system of equations

 
4 3 0 0 0
 3
 2 0 −1 0  .
 −7 0 4 −3 0 
0 −3 −3 6 0
Using the first row as reference and subtracting the correct multiples of it from
the other rows, one gets the form
 
4 3 0 0 0
 0 −1 0 −4 0 
 0 21 16 −12 0  .
 

0 −3 −3 6 0
Using the second row as reference and subtracting the correct multiples of it
from the other rows, one gets the form
 
4 3 0 0 0
 0 −1 0 −4 0 
 0 0 16 −96 0  .
 

0 0 0 0 0

11
Gioele Zardini Control Systems II FS 2018

Since one has one zero row, one can introduce a free parameter. Let x4 = s,
then x3 = 6s, x2 = −4s, x1 = 3s, s ∈ R. This defines the second eigenspace,
which is (multiplying everything by 6)
 
3
n −4 o
E−4 =  6 .
 (1.50)
1

Stacking the eigenvectors one gets

 
  3 3
X1 −4 −3
=
 6 13  .
 (1.51)
X2
1 6

It holds
Φ = X2 X1−1
  −1
6 13 3 3
=
1 6 −4 −3
(1.52)
  
1 6 13 −3 −3
=
3 1 6 4 3
 34 
7
= 3 ,
7 5

which confirms the result of a).

c) The closed-loop matrix to analyse is

   
0 3 0

A−B·K = − 1 · 14 10
3 −2
  2 
0 3 0 0
= − (1.53)
3 −2 7 5
 
0 0
= .
−4 −7

The eigenvalues of the closed loop system are given by

det((A − B · K) − λ · I) = 0
λ2 + 7λ + 12 = 0

from which it follows: λ1 = −3 and λ2 = −4.

d) No. Since it holds Jnew = 10 · J, K remains the same.

12
Gioele Zardini Control Systems II FS 2018

Example 4. You design with Matlab a LQ Regulator:

1 A = [1 0 0 0; 1 1 0 0; 1 1 1 0; 0 0 1 1];
2 B = [1 1 1 1; 0 1 0 2];
3 C = [0 0 0 1; 0 0 1 1; 0 1 1 1];
4
5 nx = size(A,1); Number of state variables of the plant, in Script: n
6 nu = size(B,2); Number of input variables of the plant, in Script: m
7 ny = size(C,1); Number of output variables of the plant, in Script: p
8
9 q = 1;
10 r = 1;
11 Q = q*eye(###);
12 R = r*eye(###);
13
14 K = lqr(A,B,Q,R);

Fill the following rows:

11 : ### =
12 : ### =

Solution. The matrix Q is a weight for the states and the matrix R is a weight for the
inputs. The correct filling is

11 : ### = nx
12 : ### = nu

13
Gioele Zardini Control Systems II FS 2018

References
[1] Essentials of Robust Control, Kemin Zhou.

[2] Karl Johan Amstroem, Richard M. Murray Feedback Systems for Scientists and En-
gineers. Princeton University Press, Princeton and Oxford, 2009.

[3] Sigurd Skogestad, Multivariate Feedback Control. John Wiley and Sons, New York,
2001.

14