CHAPTER 2 Dynamic programming
Dynamic programming – The concept
When you complete this module you should be able
to:
• Understand the concept of dynamic programming
• Know characteristics of dynamic programming
• Know applications of dynamic programming
Dynamic programming – The concept
Complex problems are sometimes solved quickly if approached in a
sequential manner.
Dynamic Programming (DP) is a sequential or multistage decision
making process. i.e. in DP decisions are made at a number of stages.
DP transforms a complex problem into a sequence of simpler problems.
The concept of DP is:
Problem is divided into smaller sub-problems
Optimize these sub-problems without losing the integrity of the original
problem.
Decisions made in a stage will affect the decisions at the subsequent stages
but are independent on each other.
Example: An N variable problem is represented by N single variable problems.
These problems are solved successively to get the optimal value of the original
problem.
Dynamic programming – The concept
Consider a single stage decision process
Here,
S1 is the input state variable
S2 is the output state variable
X1 is the decision variables, and
NB1 is the net benefit
The transformation function for the input and output is
S2 = g(X1, S1)
Net benefits are expressed as a function of decision variables and
input variable
NB1 = h(X1, S1)
Dynamic programming – The concept
Now, consider a multistage decision process consisting of T stages
For the tth stage the stage transformation and the benefit functions are
St+1 = g(Xt, St)
NBt = h(Xt, St)
Objective of this multistage problem is to find the optimum values of
all decision variables X1, X2,…, XT such that the individual net benefits
of each stage that is expressed by some objective function, f(NBt) and
the total net benefit which is expressed by f(NB1, NB2,…, NBT) should
be maximized.
Dynamic programming – The concept
Therefore, dynamic programming can be applied to this multistage
problem if the objective function is separable and monotonic.
An objective function is separable, if it can be decomposed and
expressed as a sum or product of individual net benefits of each stage.
Satisfying the equations,
St+1 = g(Xt, St)
NBt = h(Xt, St) for t = 1,2,…,T
Application areas of DPP
Allocating scarce resources. E.g. water allocation to users
Shortest route problem. E.g Water supply distribution
Production scheduling problem E.g. Critical path in CPM and PERT
Capacity expansion problem. E.g. Road expansion, Water supply
Reservoir operation problem
Equipment replacement problem
Approach to DPP
The process of multi-stage optimization is based on the Bellman’s
principle of optimality which states that
“An optimal policy (or a set of decisions) has the property that
whatever the initial state and initial decision are, the remaining
decisions must constitute an optimal policy with regard to the
state resulting from the first decision.”
How to solve a DPP?
Recursive equations are used to solve a DPP in sequence.
Recursive equations are used to structure a multistage decision
problem as a sequential process.
Each recursive equation represents a stage at which a decision is
required.
A series of equations are successively solved, each equation
depending on the output values of the previous equations.
Recursive equations for a multistage decision process can be
formulated in
a backward manner, and
a forward manner
Backward recursion - A problem is solved by writing equation first for
the final stage and then proceeding backwards to the first stage.
Forward recursion - The problem is solved by starting from the stage 1
and proceeding towards the last stage.
How to solve a DPP? – Backward Recursion
S3 = S2-X2
S2 = S1-X1
How to solve a DPP? – Forward Recursion
S1 = S2-X2
S2 = S3-X3
How to solve a DPP?
To explain the concept of backward and forward recursion
approach, take the following Water Allocation Problem
• Consider a canal supplying water for three different Fields
for crop production.
• Maximum capacity of the canal is Q units of water
• Amount of water allocated to each field as Xi
How to solve a DPP?
• Let net benefits (returns) from producing the crops are
expressed as a function of the water allocated.
NB1 x1 5 x1 0.5 x12
NB2 x2 8 x2 1.5 x22
NB3 x3 7 x3 x32
• Optimization Problem: Determine the optimal allocations
xi to each crop that maximizes the total net benefits from
all the three crops.
How to solve a DPP?
• Objective function: To maximize the net benefits
3
Max NBi xi
i 1
• Subjected to the constraints
x1 x2 x3 Q
0 xi Q for i 1, 2, 3
• Let f1(Q) be the maximum net benefits that can be
obtained from allocating water to crops 1, 2 and 3
f1 (Q) Max NBi xi
3
x1 x2 x3 Q
i 1
x , x , x 0
1 2 3
How to solve a DPP?
A. Backward Recursion:
How to solve a DPP?
A. Backward Recursion:
f1 (Q) Max NB1 x1
0 x1 Q S1
Max NB2 x2 Max NB3 x3
S1 Q
0 x2 Q x1 S 2
0 S 2 Q 0 x3 S 2 x2 S 3
0 S 3 Q
Transforming this into three problems each having only one
decision variable:
• Now starting from the last stage, let f3(S3) be the maximum
net benefits from crop 3
• State variable S3 for this stage can vary from 0 to Q
How to solve a DPP?
• Thus,
f 3* ( S3 ) Max NB3 x3
0 x3 S3
0 S 3 Q
• But S3=S2-x2. Therefore, f3(S3) = f3(S2-x2)
• Hence, the maximum benefits derived from crops 2 and 3
for a given quantity S2 which can vary between 0 and Q
f 2* ( S 2 ) Max NB x f
0 x2 S 2
2 2
*
3
( S 2 x2 )
0 S 2 Q
How to solve a DPP?
• Again,S2=S1-x1= Q-x1. Therefore, f2(S2)=f2(S1-x1). And, S1 =Q.
• Hence, the maximum benefits derived from crops 1, 2 and
3 for a given quantity S1 which is equal to Q
f 1* ( S1 ) f 1* (Q) Max NB1 x1 f 2* ( S1 x1 )
0 x1 Q
S1 Q
Note:
Once the value of f3(S3) is calculated the value of f2(S2) can be
determined from which f1(Q) can be determined.
How to solve a DPP?
A. Forward Recursion:
How to solve a DPP?
A. Forward Recursion:
• Let the function fi(Si)be the total benefit from crops 1 to i
for a given input of Si which is allocated to those crops.
• Considering the first stage,
f 1* ( S1 ) Max NB1 x1
0 x1 S1
0 S1 Q
• But S1=S2-x2
• Considering the first two crops, for an available quantity of
S2 , f2(S2) can be written as:
f 2* ( S 2 ) Max
0 x2 S 2
NB x f
2 2 1
*
( S 2 x2 )
0 S 2 Q
How to solve a DPP?
• But S2=S3-x3
• Considering the whole system, f3(S3) can be expressed as:
f 3* ( S3 ) Max NB3 x3 f 2* ( S3 x3 )
0 x3 S3
S 3 Q
Numerical Example 1 . Resource Allocation Problem
Numerical Example 1 . Resource Allocation Problem
• Net benefits from producing the crops can be expressed
as a function of the water allotted. They are calculated as
below.
NB1 ( x1 ) 5 x1 0.5 x12
NB2 ( x2 ) 8 x2 1.5 x22
NB3 ( x3 ) 7 x3 x32
xi NB1(x1) NB2(x2) NB3(x3)
0 0.0 0.0 0.0
1 4.5 6.5 6.0
2 8.0 10.0 10.0
3 10.5 10.5 12.0
4 12.0 8.0 12.0
Numerical Example 1 . Resource Allocation Problem
Representation of the problem as a set of nodes and links
Numerical Example 1 . Resource Allocation Problem
Numerical Example 1 . Resource Allocation Problem
Case A. Backward Recursion
• Sub-optimization function for the 3rd crop:
f 3 ( S3 ) Max NB3 x3
0 x3 S 3
0 S 3 Q
Numerical Example 1 . Resource Allocation Problem
• Considering the last two stages together, the sub-
optimization function is
f 2* ( S 2 ) Max NB x f
0 x2 S 2
2 2
*
3
( S 2 x2 )
0 S 2 Q
Numerical Example 1 . Resource Allocation Problem
Numerical Example 1 . Resource Allocation Problem
• Considering all the three stages together,
f 1* (Q) Max NB1 x1 f 2* ( S1 x1 )
0 x1 Q
S1 Q
Numerical Example 1 . Resource Allocation Problem
Check :
NB1 ( x1 ) 5 x1 0.5 x12 5 1 0.5 12 4.5
NB2 ( x2 ) 8 x2 1.5 x22 8 1 1.5 12 6.5
NB3 ( x3 ) 7 x3 x32 7 2 2 2 10
3
Max NBi 4.5 6.5 10 21
i 1
Numerical Example 1 . Resource Allocation Problem
Case B. Forward Recursion
• Start from the first stage and proceed towards the final
stage
• Sub-optimization function for the first stage
f 1* ( S1 ) Max NB1 x1
0 x1 S1
0 S1 Q
• Range of values for S1 is from 0 to 4.
• The calculations are shown in the table
Numerical Example 1 . Resource Allocation Problem
Numerical Example 1 . Resource Allocation Problem
NB2 x2
f (S2 ) Max *
*
0 x2 S 2 f ( S x )
2
0 S 2 Q 1 2 2
Numerical Example 1 . Resource Allocation Problem
f 3* ( S3 ) Max NB3 x3 f 2* ( S3 x3 ) with S3 4
0 x3 S3
S 3 Q
Numerical Example 1 . Resource Allocation Problem
Example 2. Short Route Problem
Consider A traveler wishes to minimize the length of a journey
from town A to J.
Can you find shorter route?
The total number of routes to be tested: 3321=18
Example 2. Short Route Problem
Stage 1:
S1 : Node in stage 1
x1 : node from which S1 is reached
d(x1,S1) : distance between nodes x1 and S1
f*1(S1) : Minimum distance from source node A to node S1
: min[d(x1,S1)]
S1 x1 d(x1,S1) f1* (S1) x1*
B A 2 2 A
C A 4 4 A
D A 3 3 A
Example 2. Short Route Problem
Stage 2:
S2 : Node in stage 2
x2 : node from which S2 is reached
d(x2,S2) : distance between nodes x2 and S2
f *2(S2) : Minimum distance from source node A to node S2
: min[d(x2,S2)+ f *1(x2)]
Example 2. Short Route Problem
S2 x2 d(x2,S2) f1* (x2) d(x2,S2)+f1* (x2) f2* (S2) x2*
E B 7 2 9
C 3 4 7 7 C, D
D 4 3 7
F B 4 2 6
C 2 4 6 4 D
D 1 3 4
G B 6 2 8
C 4 4 8 8 B,C,D
D 5 3 8
Example 2. Short Route Problem
Stage 3:
S3 : Node in stage 3
x3 : node from which S3 is reached
d(x3,S3) : distance between nodes x3 and S3
f3(S3) : Minimum distance from source node A to node S3
: min[d(x3,S3)+ f *2(x3)]
Example 2. Short Route Problem
S3 x3 d(x3,S3) f2* (x3) d(x3,S3)+f2* (x3) f3* (S3) x3*
H E 1 7 8
F 6 4 10 8 E
G 3 8 11
I E 4 7 11
F 3 4 7 7 F
G 3 8 11
Example 2. Short Route Problem
Stage 4:
S4 : Node in stage 4
x4 : node from which S4 is reached
d(x4,S4) : distance between nodes x4 and S4
f4(S4) : Minimum distance from source node A to node S4
: min[d(x4,S4)+ f *3(x4)]
S4 x4 d(x4,S4) f3* (x4) d(x4,S4)+f3* (x4) f4* (S4) x4*
J H 3 8 11
I 4 7 11 11 H,I
Example 2. Short Route Problem
Backtrack the shortest path
2 7
0 4 4 11
3 8
Example 2. Short Route Problem
The optimal route is traced back:
➢ From the last table, to reach the destination J, we must come
from the node x4*= H, I take, H
➢ From stage 3 with S3=H, we reach x3*= E
➢ From stage 2 with S2=E, we reach x2*= C, D Take C
➢ From stage 1 with S1= C, we reach x1*= A
➢ The shortest path is thus, A – C – E – H – J and the shortest
distance is 11.
➢ But alternatively, we have A – D – F – I – J
Example 2. Short Route Problem
2 7
0 4 4 11
3 8
Route A-C-E-H-J: 4+3+1+3=11
THANK YOU
Assignment
1. A total of 6units of water is to be allocated optimally to three users. The
allocation is made in discrete steps of one unit ranging from 0 to 6. With the
three users denoted as user 1, user 2 and user 3 respectively. The returns
obtained from the users for a given allocation are given in the following table.
Find allocations to the three users so that the total return is maximized.
Amount of water allocated Return from
User 1 User 2 User 3
B1 (x) B2 (x) B3 (x)
0 0 0 0
1 5 5 7
2 8 6 12
3 9 3 15
4 8 -4 16
5 5 -15 15
6 0 -30 12
Answer: x1 = 2; x2 = 1; x3 = 3 and Maximum return = 28
Assignment
2. Find the minimum cost of travel from start to End. The Petrol
and the hotel cost is given.
Assignment
3. The government is planning to implement a mix of four
prioritized developmental projects with 3 million euros
obtained from Africa Development Bank. The profits that can
be gained by implementing the projects with 1 to 3 million
euros allocated are shown below. What is the mix of projects
to develop so that the return will be maximized?
x Proj #1 Proj #2 Proj #3 Proj #4
0 0 0 0 0
1 2 5 4 2
2 8 6 4 4
3 9 7 4 5
Assignment
4. You are asked for an advice to find the mix of development of
urban area with proposed projects of housing, retail, office and
hotel areas. The available total area is 7ha. Rental income (in
x10,000birr) is a function of the floor areas allocated to each
activity. What is the mix of development that will maximize the
return?