Chapter 1
Introduction
In this chapter we introduce the main concepts in
reinforcement learning. We start by looking at some simple
examples to build intuitions about the core components of a
reinforcement learning problem, namely agent and
environment.
In particular, we will look at how an agent interacts with an
environment to optimize an objective. We will then define these
more formally and formulate reinforcement learning as a
Markov Decision Process. This is the theoretical foundation of
reinforcement learning.
Next we introduce the three primary functions an agent can
learn — a policy, value functions, and a model. We then see how
learning these functions give rise to different families of deep
reinforcement learning algorithms.
Finally, we give a brief overview of deep learning which is the
function approximation technique used throughout this book,
and discuss the main differences between reinforcement
learning and supervised learning.
1.1 REINFORCEMENT LEARNING
Reinforcement learning (RL) is concerned with solving
sequential decision making problems. Many real world
problems can be framed in this way — playing video games,
WOW! eBook
[Link]
sports, driving, optimizing inventory, robotic control. These are
things that humans and our machines do.
When solving each of these problems we have an objective or
goal — winning the game, arriving safely at our destination,
minimizing the cost of building products. We take actions and
get feedback from the world about how close we are to achieving
the objective — the current score, distance to our destination,
price per unit. Reaching our goal typically involves taking many
actions in sequence and each action changes the world around
us. We observe these changes in the world as well as the
feedback we receive before deciding on the next action to take as
a response.
Imagine the following scenario: you are at a party and a friend
brings out a flag pole and challenges you to balance it on your
hand for as long as possible. Because you have never held a flag
pole before, your initial attempts will not be very successful. You
may spend the first few moments using trial and error to get a
feel of the flag pole, but it keeps falling over.
These mistakes allow you to collect valuable information and
gain some intuition about how to balance the flag pole — where
is its center of gravity, how fast does it tilt over, how quickly
should you adjust, what is the angle at which it falls over, etc.
You use this information to make corrections for your next
attempts, improve, make further adjustments, and before you
know it, you can start balancing it for 5 seconds, 10 seconds, 30
seconds, 1 minute, and longer.
This process illustrates how reinforcement learning works. In
reinforcement learning, you are what is called the “agent”, and
WOW! eBook
[Link]
the flag pole and your surroundings are called an
“environment”. In fact, the first environment we will learn to
solve with reinforcement learning is a toy version of this
scenario called CartPole, as shown in Figure 1.1. An agent
controls a sliding cart along an axis in order to balance a pole
upright for a long time. In reality, a human does much more.
For example, we may apply our existing intuition about physics,
or transfer skills from similar tasks such as balancing a tray full
of drinks, but the problems are essentially the same in
formulation.
Figure 1.1: CartPole-v0 is a simple toy
environment. The objective is to balance a pole for
200 time steps by controlling the left-right motion
of a cart.
Reinforcement learning studies problems of this form and
methods by which artificial agents learn to solve them. It is a
subfield of artificial intelligence that dates back to optimal
control theory and Markov decision processes (MDPs). It was
WOW! eBook
[Link]
first worked on by Richard Bellman in the 1950s in the context
of dynamic programming and quasilinear equations[5]. We will
see this name again when we study a famous equation in
reinforcement learning — the Bellman equation.
RL problems can be expressed as a system consisting of an
agent and an environment. An environment produces
information which describes the state of the system. This is
known as a state. An agent interacts with an environment by
observing the state and using this information to select an
action. The environment accepts the action and transitions into
the next state. It then returns the next state and a reward to the
agent. When the cycle of (state → action → reward) completes,
we say that one time step has passed. The cycle repeats until the
environment terminates, for example when the problem is
solved. This entire process is described by the control loop
diagram in Figure 1.2.
WOW! eBook
[Link]
WOW! eBook
[Link]