Note - 1

CHAPTER 1: INTRODUCTION TO OPTIMAL CONTROL PROBLEM

1.1. The basic problem

1.2. Some examples
1.3. A geometric solution
1.4. Overview

1.1 THE BASIC PROBLEM.

DYNAMICS. We open our discussion by considering an ordinary differential

equation (ODE) having the form

ẋ(t) = f (x(t)) (t > 0)
(1.1) 0
x(0) = x .

We are here given the initial point x0 ∈ Rn and the function f : Rn → Rn . The un-
known is the curve x : [0, ∞) → Rn , which we interpret as the dynamical evolution
of the state of some “system”.

CONTROLLED DYNAMICS. We generalize a bit and suppose now that

f depends also upon some “control” parameters belonging to a set A ⊂ Rm ; so that
f : Rn ×A → Rn . Then if we select some value a ∈ A and consider the corresponding
dynamics:

ẋ(t) = f (x(t), a) (t > 0)
0
x(0) = x ,
we obtain the evolution of our system when the parameter is constantly set to the
value a.

The next possibility is that we change the value of the parameter as the system
evolves. For instance, suppose we define the function α : [0, ∞) → A this way:

 a 1 0 ≤ t ≤ t1
α(t) = a2 t1 < t ≤ t2

a3 t2 < t ≤ t3 etc.

for times 0 < t1 < t2 < t3 . . . and parameter values a1 , a2 , a3 , · · · ∈ A; and we then
solve the dynamical equation

ẋ(t) = f (x(t), α(t)) (t > 0)
(1.2) 0
x(0) = x .

The picture illustrates the resulting evolution. The point is that the system may
behave quite differently as we change the control parameters.
3
 α = a4

t3

α = a3

t2
time

trajectory of ODE
α = a2

t1

α = a1

x0

Controlled dynamics

More generally, we call a function α : [0, ∞) → A a control. Corresponding to

each control, we consider the ODE

ẋ(t) = f (x(t), α(t)) (t > 0)
(ODE) 0
x(0) = x ,

and regard the trajectory x(·) as the corresponding response of the system.

NOTATION. (i) We will write

 
f 1 (x, a)

f (x, a) =  .. 
. 
f n (x, a)

to display the components of f , and similarly put

 1 
x (t)
 .. 
x(t) =  .  .
xn (t)

We will therefore write vectors as columns in these notes and use boldface for
vector-valued functions, the components of which have superscripts.
(ii) We also introduce

A = {α : [0, ∞) → A | α(·) measurable}

4
to denote the collection of all admissible controls, where
 1 
α (t)
α(t) =  ...  .
 

αm (t)
Note very carefully that our solution x(·) of (ODE) depends upon α(·) and the initial
condition. Consequently our notation would be more precise, but more complicated,
if we were to write
x(·) = x(·, α(·), x0),
displaying the dependence of the response x(·) upon the control and the initial
value. 

PAYOFFS. Our overall task will be to determine what is the “best” control for
our system. For this we need to specify a specific payoff (or reward) criterion. Let
us define the payoff functional
Z T
(P) P [α(·)] := r(x(t), α(t)) dt + g(x(T )),
0

where x(·) solves (ODE) for the control α(·). Here r : Rn × A → R and g : Rn → R
are given, and we call r the running payoff and g the terminal payoff. The terminal
time T > 0 is given as well.

THE BASIC PROBLEM. Our aim is to find a control α∗ (·), which maximizes
the payoff. In other words, we want
P [α∗ (·)] ≥ P [α(·)]

for all controls α(·) ∈ A. Such a control α∗ (·) is called optimal.

This task presents us with these mathematical issues:
(i) Does an optimal control exist?
(ii) How can we characterize an optimal control mathematically?
(iii) How can we construct an optimal control?
These turn out to be sometimes subtle problems, as the following collection of
examples illustrates.

1.2 EXAMPLES

EXAMPLE 1: CONTROL OF PRODUCTION AND CONSUMPTION.

Suppose we own, say, a factory whose output we can control. Let us begin to
construct a mathematical model by setting
x(t) = amount of output produced at time t ≥ 0.
5
We suppose that we consume some fraction of our output at each time, and likewise
can reinvest the remaining fraction. Let us denote

α(t) = fraction of output reinvested at time t ≥ 0.

This will be our control, and is subject to the obvious constraint that

0 ≤ α(t) ≤ 1 for each time t ≥ 0.

Given such a control, the corresponding dynamics are provided by the ODE

ẋ(t) = kα(t)x(t)
x(0) = x0 .
the constant k > 0 modelling the growth rate of our reinvestment. Let us take as a
payoff functional
Z T
P [α(·)] = (1 − α(t))x(t) dt.
0
The meaning is that we want to maximize our total consumption of the output, our
consumption at a given time t being (1 − α(t))x(t). This model fits into our general
framework for n = m = 1, once we put

A = [0, 1], f (x, a) = kax, r(x, a) = (1 − a)x, g ≡ 0.

α* = 1

α* = 0

0 t* T

A bang-bang control

As we will see later in §4.4.2, an optimal control α∗ (·) is given by


∗ 1 if 0 ≤ t ≤ t∗
α (t) =
0 if t∗ < t ≤ T
for an appropriate switching time 0 ≤ t∗ ≤ T . In other words, we should reinvest
all the output (and therefore consume nothing) up until time t∗ , and afterwards, we
should consume everything (and therefore reinvest nothing). The switchover time
t∗ will have to be determined. We call α∗ (·) a bang–bang control. 

EXAMPLE 2: REPRODUCTIVE STATEGIES IN SOCIAL INSECTS

6
 The next example is from Chapter 2 of the book Caste and Ecology in Social
Insects, by G. Oster and E. O. Wilson [O-W]. We attempt to model how social
insects, say a population of bees, determine the makeup of their society.
Let us write T for the length of the season, and introduce the variables
w(t) = number of workers at time t
q(t) = number of queens
α(t) = fraction of colony effort devoted to increasing work force
The control α is constrained by our requiring that

0 ≤ α(t) ≤ 1.

We continue to model by introducing dynamics for the numbers of workers and

the number of queens. The worker population evolves according to

ẇ(t) = −µw(t) + bs(t)α(t)w(t)
w(0) = w0 .
Here µ is a given constant (a death rate), b is another constant, and s(t) is the
known rate at which each worker contributes to the bee economy.
We suppose also that the population of queens changes according to

q̇(t) = −νq(t) + c(1 − α(t))s(t)w(t)
q(0) = q 0 ,
for constants ν and c.
Our goal, or rather the bees’, is to maximize the number of queens at time T :

P [α(·)] = q(T ).

So in terms of our general notation, we have x(t) = (w(t), q(t))T and x0 = (w0 , q 0 )T .
We are taking the running payoff to be r ≡ 0, and the terminal payoff g(w, q) = q.

The answer will again turn out to be a bang–bang control, as we will explain
later. 

EXAMPLE 3: A PENDULUM.
We look next at a hanging pendulum, for which

θ(t) = angle at time t.

If there is no external force, then we have the equation of motion


θ̈(t) + λθ̇(t) + ω 2 θ(t) = 0
θ(0) = θ1 , θ̇(0) = θ2 ;
the solution of which is a damped oscillation, provided λ > 0.
7
 Now let α(·) denote an applied torque, subject to the physical constraint that

|α| ≤ 1.

Our dynamics now become


θ̈(t) + λθ̇(t) + ω 2 θ(t) = α(t)
θ(0) = θ1 , θ̇(0) = θ2 .
Define x1 (t) = θ(t), x2 (t) = θ̇(t), and x(t) = (x1 (t), x2 (t)). Then we can write the
evolution as the system
     
ẋ1 θ̇ x2
ẋ(t) = = = = f (x, α).
ẋ2 θ̈ −λx2 − ω 2 x1 + α(t)
We introduce as well Z τ
P [α(·)] = − 1 dt = −τ,
0
for
τ = τ (α(·)) = first time that x(τ ) = 0 (that is, θ(τ ) = θ̇(τ ) = 0.)
We want to maximize P [·], meaning that we want to minimize the time it takes to
bring the pendulum to rest.
Observe that this problem does not quite fall within the general framework
described in §1.1, since the terminal time is not fixed, but rather depends upon the
control. This is called a fixed endpoint, free time problem. 

EXAMPLE 4: A MOON LANDER

This model asks us to bring a spacecraft to a soft landing on the lunar surface,
using the least amount of fuel.
We introduce the notation
h(t) = height at time t
v(t) = velocity = ḣ(t)
m(t) = mass of spacecraft (changing as fuel is burned)
α(t) = thrust at time t
We assume that
0 ≤ α(t) ≤ 1,
and Newton’s law tells us that

mḧ = −gm + α,

the right hand side being the difference of the gravitational force and the thrust of
the rocket. This system is modeled by the ODE
 α(t)
 v̇(t) = −g + m(t)
 ḣ(t) = v(t)
ṁ(t) = −kα(t).
8
 height = h(t)

moonÕs surface

A spacecraft landing on the moon

We summarize these equations in the form

ẋ(t) = f (x(t), α(t))

for x(t) = (v(t), h(t), m(t)).

We want to minimize the amount of fuel used up, that is, to maximize the
amount remaining once we have landed. Thus

P [α(·)] = m(τ ),

where
τ denotes the first time that h(τ ) = v(τ ) = 0.
This is a variable endpoint problem, since the final time is not given in advance.
We have also the extra constraints

h(t) ≥ 0, m(t) ≥ 0.

EXAMPLE 5: ROCKET RAILROAD CAR.

Imagine a railroad car powered by rocket engines on each side. We introduce
the variables
q(t) = position at time t
v(t) = q̇(t) = velocity at time t
α(t) = thrust from rockets,
where
−1 ≤ α(t) ≤ 1,
9
 rocket engines

A rocket car on a train track

the sign depending upon which engine is firing.

We want to figure out how to fire the rockets, so as to arrive at the origin 0
with zero velocity in a minimum amount of time. Assuming the car has mass m,
the law of motion is
mq̈(t) = α(t).

We rewrite by setting x(t) = (q(t), v(t))T . Then

  
 ẋ(t) = 0 1 x(t) + 0
α(t)
1
0 0

x(0) = x0 = (q0 , v0 )T .
Since our goal is to steer to the origin (0, 0) in minimum time, we take
Z τ
P [α(·)] = − 1 dt = −τ,
0

for
τ = first time that q(τ ) = v(τ ) = 0.

1.3 A GEOMETRIC SOLUTION.

To illustrate how actually to solve a control problem, in this last section we

introduce some ad hoc calculus and geometry methods for the rocket car problem,
Example 5 above.

First of all, let us guess that to find an optimal solution we will need only to
consider the cases a = 1 or a = −1. In other words, we will focus our attention only
upon those controls for which at each moment of time either the left or the right
rocket engine is fired at full power. (We will later see in Chapter 2 some theoretical
justification for looking only at such controls.)

CASE 1: Suppose first that α ≡ 1 for some time interval, during which

q̇ = v
v̇ = 1.
Then
v v̇ = q̇,
10
and so
1 2
(v )˙ = q̇.
2
Let t0 belong to the time interval where α ≡ 1 and integrate from t0 to t:
v 2 (t) v 2 (t0 )
− = q(t) − q(t0 ).
2 2
Consequently

(1.1) v 2 (t) = 2q(t) + (v 2 (t0 ) − 2q(t0 )) .

| {z }
b

In other words, so long as the control is set for α ≡ 1, the trajectory stays on the
curve v 2 = 2q + b for some constant b.

v-axis

α =1

q-axis

curves v2=2q + b

CASE 2: Suppose now α ≡ −1 on some time interval. Then as above


q̇ = v
v̇ = −1,
and hence
1 2
(v )˙ = −q̇.
2
Let t1 belong to an interval where α ≡ −1 and integrate:

(1.2) v 2 (t) = −2q(t) + (2q(t1 ) − v 2 (t1 )).

| {z }
c

Consequently, as long as the control is set for α ≡ −1, the trajectory stays on the
curve v 2 = −2q + c for some constant c.
11
 v-axis

α =-1

q-axis

curves v2=-2q + c

GEOMETRIC INTERPRETATION. Formula (1.1) says if α ≡ 1, then (q(t), v(t))

lies on a parabola of the form
v 2 = 2q + b.
Similarly, (1.2) says if α ≡ −1, then (q(t), v(t)) lies on a parabola

v 2 = −2q + c.

Now we can design an optimal control α∗ (·), which causes the trajectory to jump
between the families of right– and left–pointing parabolas, as drawn. Say we start
at the black dot, and wish to steer to the origin. This we accomplish by first setting
the control to the value α = −1, causing us to move down along the second family of
parabolas. We then switch to the control α = 1, and thereupon move to a parabola
from the first family, along which we move up and to the left, ending up at the
origin. See the picture.

1.4 OVERVIEW.

Here are the topics we will cover in this course:

• Chapter 2: Controllability, bang-bang principle.

In this chapter, we introduce the simplest class of dynamics, those linear in both
the state x(·) and the control α(·), and derive algebraic conditions ensuring that
the system can be steered into a given terminal state. We introduce as well some
abstract theorems from functional analysis and employ them to prove the existence
of so-called “bang-bang” optimal controls.

• Chapter 3: Time-optimal control.

12