This training step is repeated until the network’s outputs stop
changing or the loss has minimized and plateaued, i.e. the
network has converged.
In reinforcement learning, neither the network inputs x nor the
correct outputs y are given in advance. Instead, these values are
obtained through agent interactions with an environment which
produce the states and rewards it observes. This setting
presents a particular challenge for training neural networks in
reinforcement learning, and will be the subject of much
discussion throughout this book.
The difficulties in data generation and evaluation are due to the
fact that the functions which we try to learn are tightly coupled
with the MDP loop. The data exchange between an agent and an
environment is interactive and the process is inherently limited
by the time needed for an agent to act and for an environment
to transition. There is no shortcut to generating data for
training — an agent has to experience every time step. The data
collection and training cycle run repeatedly, with every training
step (potentially) waiting for new data to be collected.
Furthermore, because the current state of an environment and
the actions an agent takes affect the future states it experiences,
states and rewards at any given point in time are not
independent of states and rewards in previous time steps. This
violates an assumtion of gradient descent, that data is
identically and independently distributed (i.i.d.). The speed at
which a network converges and the quality of the final result can
be adversely affected. Significant research effort has gone into
minimizing this effect, and some techniques for achieving this
are discussed later in this book.
WOW! eBook
[Link]
Despite these challenges, deep learning is a powerful technique
for function approximation in reinforcement learning. It is
worth persevering to overcome the difficulties of applying it to
reinforcement learning, as the benefits greatly outweigh the
costs.
1.6 REINFORCEMENT LEARNING AND
SUPERVISED LEARNING
At the core of deep reinforcement learning is function
approximation. This is something it shares with supervised
learning (SL)6. However reinforcement learning is unlike
supervised learning in a number of ways. There are three main
differences.
6
The A.I. community does love abbreviations. We will see many more later, since
nearly all of the algorithms and component names have abbreviations
• Lack of an oracle7
7
In computer science, an oracle is a hypothetical blackbox with provides the
correct answers to questions asked.
• Sparsity of feedback
• Data generation
1.6.1 Lack of an Oracle
A major difference between reinforcement learning and
supervised learning is that for reinforcement learning problems,
the “correct” answer for each model input is not available,
whereas in supervised learning we have access to the correct or
optimal answer for each example. In reinforcement learning the
equivalent of the correct answer would be access to an “oracle”
which tells us the optimal action to take at every time step so as
WOW! eBook
[Link]
to maximize the objective.
The correct answer can convey a lot of information about a
datapoint. For example, the correct answer for classification
problems contains many bits of information. It not only tells the
right class for each training example, but also implies that the
training example does not belong to any of the other classes. If a
particular classification problem has 1000 classes (as in the
ImageNet dataset[28]), an answer contains 1000 bits of
information per example (1 positive and 999 negative).
Furthermore, the correct answer does not have to be a category
or a real number. It can be a bounding box, or a semantic
segmentation, each of which contains many bits of information
about the example at hand.
In reinforcement learning, after an agent takes actions a in state
s is only has access to the reward it received. The agent is not
told what the best action to take was. Instead it is only given an
indication, via the reward, of how good or bad a is. Not only
does this convey less information than having the right answer
would have done, but the agent only learns about rewards for
the states it experiences. To learn about (s, a, r), an agent must
experience the transition (s, a, r, s′). Consequently, an agent
may have no knowledge about important parts of the state and
action space because it hasn’t experienced them.
One way to deal with this problem is to initialize episodes to
start in states we want an agent to learn about. However it is not
always possible to do this for two reasons. First, we may not
have full control over an environment. Second, states can be
easy to describe but difficult to specify. Consider a simulation of
a humanoid robot learning to do a backflip. To help an agent
WOW! eBook
[Link]
learn about the reward for succesfully landing, we can initialize
an environment to start just as the robot’s feet make contact
with the floor after a “good” flip. The reward function in this
part of the state space is critical to learn about, since this is
where the robot may either retain its balance and successfully
execute the flip, or fall over and fail. However, it is not
straightforward to define the precise numerical position and
velocity of each of the robot’s joint angles, or the force being
exerted so as to initialize the robot in this position. In practice,
to reach this state an agent needs to execute a long, very specific
sequence of actions to first flip then almost land. There is no
guarantee that an agent will learn to do this, so this part of state
space may not be explored.
1.6.2 Sparsity of Feedback
In reinforcement learning a reward function may be sparse, so
the scalar reward is often 0. This means that most of the time,
an agent is receiving no information about how to change the
parameters of the network so as to improve performance.
Consider again the backflipping robot and suppose an agent
only receives a non-zero reward of +1 after successfully
executing a backflip.
Almost all actions that it takes will result in the same reward
signal of 0 from the environment. Under these circumstances
learning is extremely challenging because an agent will receive
no guidance about whether the intermediate actions it takes
help reach the goal. Supervised learning doesn’t have this
problem. All input examples are paired with a desired output
which conveys some information about how a network should
WOW! eBook
[Link]
perform.
The combination of sparse feedback and the lack of an oracle
means that much less information is received per time step in
the environment in reinforcement learning when compared
with training examples in supervised learning[29]. As a result,
all reinforcement learning algorithms tend to be significantly
more sample inefficient.
1.6.3 Data
In supervised learning, data is typically generated
independently from algorithm training. Indeed, the first step of
in applying supervised learning to a problem is often to find or
construct a good dataset. In reinforcement learning, data must
be generated by an agent interacting with an environment. In
many cases, this data is generated as training progresses in an
iterative manner, with alternating phases of data gathering and
training. Data and algorithm are coupled. The quality of an
algorithm affects the data it is trained on, which in turn affects
an algorithm’s performance. This circularity and bootstrapping
requirement does not occur in supervised learning.
RL is also interactive — actions made by the agent actually
change the environment, which then changes the agent’s
decisions, which change the data the agent sees, and so on. This
feedback loop is the hallmark of reinforcement learning,
whereas it is nonexistent in supervised learning problems.
There is no equivalent notion of an agent in supervised learning
which could change the data that an algorithm is trained on.
A final and more minor difference between reinforcement
learning and supervised learning is that neural networks in
WOW! eBook
[Link]