ACCELERATED M ETHODS FOR D EEP R EINFORCEMENT L EARNING

Adam Stooke 1 Pieter Abbeel 1

A BSTRACT
Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a
key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern
computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value
arXiv:1803.02811v2 [[Link]] 10 Jan 2019

learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible
to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity
or final performance. We leverage these facts to build a unified framework for parallelization that dramatically
hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both
data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari
games in mere minutes, using both synchronous and asynchronous algorithms.

1 I NTRODUCTION 2017). To provide calibrated results, we test our implemen-

tations in the heavily benchmarked Atari-2600 domain via
Research in deep reinforcement learning (RL) has re- the Arcade Learning Environment (ALE) (Bellemare et al.,
lied heavily on empirical evaluation, making experiment 2013).
turnaround time a key limiting factor. Despite this critical
bottleneck, many reference implementations do not fulfill We found that highly parallel sampling using batched in-
the potential of modern computers for throughput, unlike in ferences can accelerate experiment turn-around time of all
supervised learning (see e.g. (Goyal et al., 2017)). In this algorithms without hindering training. We further found
work, we study how to adapt deep RL algorithms–without that neural networks can learn using batch sizes consid-
changing their underlying formulations–to better leverage erably larger than are standard, without harming sample
multiple CPUs and GPUs in one machine. The result is a sig- complexity or final game score.
nificant gain in efficiency and scale of hardware utilization Beyond exploring these new learning regimes, we lever-
and hence in learning speed. age them to dramatically speed up learning. For example,
Today’s leading deep RL algorithms have roughly clustered policy gradient algorithms ran on an 8-GPU server learned
into two families: (i) Policy gradient methods, of which successful game strategies in under 10 minutes, rather than
Asynchronous Advantage Actor-Critic (A3C) (Mnih et al., hours. We similarly reduced the duration of some standard
2016) is a representative example, (ii) Q-value learning Q-value-learning runs from 10 days to under 2 hours. Al-
methods, a representative example being Deep Q-Networks ternatively, independent RL experiments can run in parallel
(DQN) (Mnih et al., 2015). Traditionally, these two families with high aggregate throughput per computer. We believe
appear in distinct implementations and use different hard- that these results promise to accelerate research in deep
ware resources; in this paper we unify them under the same RL, and we suggest directions for further investigation and
framework for scaling. development.
Our contribution is a framework for parallelized deep RL
including novel techniques for GPU acceleration of both 2 R ELATED W ORK
inference and training. We demonstrate multi-GPU versions Efforts to parallelize and accelerate deep RL algorithms
of the following algorithms: Advantage Actor-Critic (Mnih have been underway for several years. Gorila (Nair et al.,
et al., 2016), Proximal Policy Optimization (PPO) (Schul- 2015) parallelized DQN using distributed computing. It
man et al., 2017), DQN (Mnih et al., 2015), Categorical achieved significant although sub-linear speedups using
DQN (Bellemare et al., 2017), and Rainbow (Hessel et al., hundreds of computing units as samplers or learners, with
1
Department of Computer Science, University of Berke- central parameter servers for managing parameter updates.
ley, Berkeley, California. Correspondence to: Adam Stooke This effort suffered in sample complexity relative to single-
<[Link]@[Link]>. threaded DQN. More recently, (Horgan et al., 2018) showed
 Accelerated Methods for Deep Reinforcement Learning

that a distributed, prioritized replay buffer can support faster sampled experience.
learning while using hundreds of CPU cores for simula-
Q-value learning methods instead parameterize the Q-
tion and a single GPU for training. The same work used
function Q(s, a; θ), which in DQN (Mnih et al., 2015) is
increased batch sizes, with a brief study of the effect of 2
learning rate. regressed against an objective as: E[(yi − Q(ai |si ; θ)) ],
where yi is the data-estimated Q-value given by yi =
The policy gradient method A3C is itself a parallelized al- ri + γ maxa Q(a|si+1 ; θ− ). The target network θ− is peri-
gorithm. In GA3C (Babaeizadeh et al., 2016), a speedup odically copied from θ. Training data is selected randomly
over CPU-only A3C was achieved by using a GPU. It was from a replay buffer of recent experiences, each to be used
employed asynchronously, with “predictor” and “trainer” multiple times. Categorical DQN (Bellemare et al., 2017)
threads queuing observations and rewards for batched in- discretizes the possible Q-values into a fixed set and learns
ferences and training updates. GA3C induced a “policy a distribution for each Q(a|s; θ). The use of distributional
lag” between generation and consumption of training data, learning was combined with five other enhancements under
worsening sample complexity. In independent work simul- the name Rainbow: 1) Double-DQN (Van Hasselt et al.,
taneous to ours, (Espeholt et al., 2018) extended policy 2016), 2) Dueling Networks (Wang et al., 2016), 3) Priori-
gradient methods to a distributed setting, enabling an al- tized Replay (Schaul et al., 2016), 4) n-step learning (Peng &
ternative approach to multi-GPU training called IMPALA. Williams, 1994), and 5) NoisyNets (Fortunato et al., 2018).
They introduced a more heavily modified algorithm, V-trace, In our experiments, we use the -greedy version of Rainbow,
to mitigate policy lag–which we avoid–and did not employ without parameter noise: -Rainbow. We refer the inter-
GPU inference. In PAAC (Clemente et al., 2017), the au- ested reader to the original publications for further details
thors explored the use of many simulators and increased on these algorithms.
batch sizes learning rates in (single-GPU) batched A2C–
ideas central to our studies. Our contributions to actor-critic
4 PARALLEL , ACCELERATED RL
methods exceed this work in a number of ways, chiefly:
improved sampling organization, tremendously enhanced F RAMEWORK
scale and speed using multiple GPUs, and inclusion of asyn- We consider CPU-based simulator-environments and poli-
chronous optimization. cies using deep neural networks. We describe here a com-
plete set of parallelization techniques for deep RL that
3 RL A LGORITHM BACKGROUND achieve high throughput during both sampling and opti-
mization. We treat GPUs homogeneously; each performs
In a standard RL formulation as a Markov Decision Process, the same sampling-learning procedure. This strategy scales
a learning agent aims to maximize the sum of discounted straightforwardly to various numbers of GPUs.
rewardsPexperienced while interacting with an environment:
∞
Rt = k=0 γ k rt+k , where r is the reward and γ ≤ 1 the 4.1 Synchronized Sampling
discount factor. The value of a state, V (st ) = E [Rt |st ], is
defined as the expected return under a given policy. The Q- We begin by associating multiple CPU cores with a single
value, Q(st , at ) = E [Rt |st , at ] is the same but first using GPU. Multiple simulators run in parallel processes on the
action at to advance. CPU cores, and these processes perform environment steps
in a synchronized fashion. At each step, all individual ob-
In policy gradient methods, the policy is directly param- servations are gathered into a batch for inference, which
eterized as a distribution over actions, as π(a|s; θ). The is called on the GPU after the last observation is submit-
Advantage Actor-Critic algorithm (see, e.g. (Mnih et al., ted. The simulators step again once the actions are returned,
2016)) learns to estimate state values V (s; θ), and it- and so on, as in (Clemente et al., 2017). System shared
eratively optimizes the policy on fresh environment ex- memory arrays provide fast communication between the
perience using gradient steps as E [∇θ log π(at |st ; θ)At ], action-server and simulator processes.
where A(s, a) = Q(s, a) − V (s) is the advantage, es-
timated as Rt − V (st ). Proximal Policy Optimization Synchronized sampling may suffer slowdowns due to the
(PPO) (Schulman et al., 2017) maximizes a surrogate objec- straggler effect–waiting for the slowest process at each step.
tive E [ρt (θ)At ], where ρt (θ) = π(at |st ; θ)/π(at |st ; θold ) Variance in stepping time arises from varied computation
is the likelihood ratio of the recorded action between loads of different simulator states and other random fluctua-
the updated and sampled policies. We use the clipping- tions. The straggler effect worsens with increased number
objective version of PPO, which optimizes the expression of parallel processes, but we mitigate it by stacking multiple,
E [min (ρt (θ)At , clip(ρt (θ), 1 − , 1 + )At )] under hyper- independent simulator instances per process. Each process
parameter  < 1. Unlike A3C, PPO performs multiple steps all its simulators (sequentially) for every inference
parameter updates using (minibatches from) each set of batch. This arrangement also permits the batch size for
 Accelerated Methods for Deep Reinforcement Learning

inference to increase beyond the number of processes (i.e. rately, each with its own lock (steps 2-3 become a loop over
CPU cores). A schematic is shown in Figure 1(a). Slow- chunks). This balances update call efficiency against lock
downs caused by long environment resets can be avoided by contention and can provide good performance.2
resetting only during optimization pauses.1
If simulation and inference loads are balanced, each compo- 5 E XPERIMENTS
nent will sit idle half of the time, so we form two alternating
We used the Atari-2600 domain to study the scaling char-
groups of simulator processes. While one group awaits
acteristics of highly parallelized RL, investigating the fol-
its next action, the other steps, and the GPU alternates be-
lowing: 1) How efficient is synchronized sampling, and
tween servicing each group. Alternation keeps utilization
what speeds can it achieve? 2) Can policy gradient and Q-
high and furthermore hides the execution time of whichever
learning algorithms be adapted to learn using many parallel
computation is the quicker of the two.
simulator instances without diminishing learning perfor-
We organize multiple GPUs by repeating the template, allo- mance? 3) Can large-batch training and/or asynchronous
cating available CPU cores evenly. We found it beneficial to methods speed up optimization without worsening sample
fix the CPU assignment of each simulator process, with one complexity?
core reserved to run each GPU. The experiments section
In all learning experiments, we maintained the original train-
contains measurements of sampling speed, which increases
ing intensity–meaning average number of training uses of
with the number of environment instances.
each sampled data point. For A3C, PPO, and DQN+variants,
the reference training intensities are 1, 4, and 8, respectively.
4.2 Synchronous Multi-GPU Optimization
All learning curves shown here are averages over at least
In our synchronous algorithms, all GPUs maintain iden- two random seeds. For policy gradient methods, we tracked
tical parameter values. We leverage the data-parallelism online scores, averaging over the most recent 100 com-
of stochastic gradient estimation and use the well-known pleted trajectories. For DQN and variants, we paused every
update procedure, on every GPU: 1) compute a gradient 1-million steps to evaluate for up to 125,000 steps, with
using locally-collected samples, 2) all-reduce the gradient maximum path length of 27,000 steps, as is standard. The
across GPUs, 3) use the combined gradient to update local appendices contain learning curves and experiment details
parameters. We use the NVIDIA Collective Communication beyond those we highlight here, including additional hyper-
Library for fast communication among GPUs. parameter adjustments.

4.3 Asynchronous Multi-GPU Optimization 5.1 Sampling

In asynchronous optimization, each GPU acts as its own A series of sampling-only measurements demonstrated that
sampler-learner unit and applies updates to a central parame- despite the potential for stragglers, the synchronized sam-
ter store held in CPU memory. Use of accelerators compels pling scheme can achieve good hardware utilization. First,
a choice of where to perform the parameter update. In our we studied the capacity of a single GPU at serving infer-
experience, applying common update rules to the network ences for multiple environments. Figure 1(b) shows mea-
is faster on the GPU. Our general update procedure includes surements running a trained A3C-Net policy on a P100 GPU
three steps: 1) compute the gradient locally and store it on while playing B REAKOUT. Aggregate sampling speed, nor-
the GPU, 2) pull current central parameters onto the GPU malized by CPU core count, is plotted as a function of the
and apply the update rule to them using the pre-computed number of (sequential) Atari simulators running on each
gradient, 3) write the updated parameters back to the central core.3 The minimum was 2 simulators per core by the alter-
CPU store. After this sequence, the local GPU parameters nating scheme. Different curves represent different numbers
are in sync with the central values, and sampling proceeds of CPU cores running simulations. For reference, we in-
again. Following (Mnih et al., 2016), we also centralize the clude the sampling speed of a single core running without
update rule parameters. inference–the dashed line for a single process, and the dotted
Rather than add update increments to the central parameters, line one process on each of the two Hyperthreads. Running
which requires CPU computation, we overwrite the values. with inferences and with a single core, the sampling speed
Therefore, we employ a lock around steps (2) and (3) above, increased with simulator count until the inference time was
preventing other processes from reading or writing param- completely hidden. Synchronization losses appeared for
eter values concurrently. We divide the parameters into a 2
e.g., for 8 workers and 3 chunks in A3C, we observed less
small number of disjoint chunks which are updated sepa- time spent blocked than updating.
3
1 Intel Turboboost was disabled for this test only, keeping the
For example, one may either 1) ignore a simulator in need of
clock speed of every core at 2.2 GHz.
reset or 2) immediately swap in a fresh instance held in reserve.
 Accelerated Methods for Deep Reinforcement Learning

higher core count. But at as little as 8 environments per evaluate here. Interestingly, scaling affects synchronous and
core, the GPU supported even 16 CPU cores running at asynchronous learning somewhat differently.
roughly 80% of the inference-free speed.
Starting State Decorrelation: Learning failed very early
in some policy gradient experiments with many simulators.
We found correlation in starting game states to result in
large but poorly informed learning signals, destabilizing
early learning. We correct this by stepping every simulator
through a random number of uniform-random actions during
experiment initialization. When taking this measure, we
found learning rate warmup (Goyal et al., 2017) to have no
further effect. While training, game resets proceed as usual.
A2C: The optimization batch size grows with the number
of simulators (keeping the sampling horizon fixed). Cor-
respondingly fewer parameter update steps are made per
(a) sample gathered. Unlike in (Clemente et al., 2017), we
found that increasing the learning rate with the square root
of the batch size worked best across a test set of games. The
top panel of Figure 2 shows learning curves vs total sample
count, with simulator count ranging from 16 to 512 (batch
size 80 to 2,560). Game scores were largely unchanged,
although a gradual decay in sample efficiency remained for
large simulator counts.
A3C: An asynchronous adaptation we tested used a 16-
environment A2C agent as the base sampler-learner unit.
Figure 2 shows learning curves vs aggregate sample count
for numbers of learners ranging from 1 to 32,4 correspond-
ing to 16 to 512 total simulators. The resulting learning
(b) curves were nearly indistinguishable in most cases, although
some degraded at the largest scales.
Figure 1. Synchronized sampling (a) schematic: n parallel sim-
PPO: The large batch size already used to benchmark PPO
ulation processes, each with m sequential simulator instances,
interacting synchronously with GPU-based action-server process
(8-simulator x 256-horizon = 2,048) provides a different
(alternation not shown) (b) speed vs number of simulators per route to learning with many simulators: we decreased the
core, using 1 GPU. Running multiple simulators per core mitigates sampling horizon such that the total batch size remained
synchronization losses and hides NN inference time, resulting in fixed. Figure 2 shows learning curves vs sample count for
higher throughput. simulator counts ranging from 8 to 512, with corresponding
sampling horizons from 256 down to 4 steps. Successful
learning continued to the largest scale.
Next, we measured the sampling-only speed of the same
A3C-Net playing B REAKOUT parallelized across an entire APPO: We also experimented with an asynchronous ver-
8-GPU, 40-core server. At simulator counts of 256 (8 per sion of PPO, using an 8-simulator PPO agent as the base
core) and above, the server achieved greater than 35,000 learner unit. The bottom panel in Figure 2 shows learning
samples per second, or 500 million emulator frames per curves from a study of 8 learners running on 8 GPUs, with
hour, confirming scalability. The appendix contains a table varying communication frequency. Standard PPO uses 4 gra-
of results for other simulator counts. dient updates per epoch, and 4 epochs per optimization; we
experimented with 1-4 gradient updates between synchro-
5.2 Learning with Many Simulator Instances nizations (update rule provided in supplementary material).
We found it helpful to periodically pull new values from
To leverage the high throughput of parallel sampling, we the central parameters during sampling, and did this with
investigated ways to adapt existing deep RL algorithms to a horizon of 64 steps in all cases (thus decreasing policy
learn with many simulator instances. The following findings lag inherent in asynchronous techniques, lag made acute
show that only minor changes suffice to adapt all algorithms 4
and maintain performance. We experimented with differ- Learner counts in excess of 8 were run with multiple separate
learners sharing GPUs.
ent techniques for each algorithm, which we describe and
 Accelerated Methods for Deep Reinforcement Learning

by PPO’s less frequent but more substantial updates). In

Table 1. Average and median human-normalized scores across 49
several games, the learning remained consistent, showing it
games, including baseline and scaled configurations. Scaled ver-
is possible to reduce communication in some asynchronous sions tend to match un-scaled versions. PG: 25M steps, DQN:
settings. 50M steps.
DQN + Variants: We organized the experience replay Algorithm-Scale Average Median
buffer by simulator. The total buffer size remained at 1 A2C 16-env 2.5 0.65
million transitions, so a correspondingly shorter history was A2C 128-env 8.0 0.48
held for each simulator. We observed learning performance A3C 3×16-env 5.4 0.53
to be largely independent of simulator count up to over 200,
provided the number of update steps per optimization cycle PPO 8-env 2.8 1.2
is not too high (large batch size ameliorates this). PPO 128-env 9.8 0.97
APPO 8×8-env 8.8 1.2
5.3 Q-Value Learning with Large Training Batches DQN-32* 2.4 0.96
DQN-512 9.8 1.4
DQN: We experimented with batch sizes ranging from the
standard 32 up to 2,048. We found consistent learning CatDQN-32** 25.5 2.1
performance up to 512, beyond which, it became difficult to CatDQN-2048 22.1 2.9
find a single (scaled) learning rate which performed well in -Rainbow-512 22.4 2.5
all test games. In several games, a larger batch size improved *
from (Van Hasselt et al., 2016)
learning, as shown in Figure 3. We also found asynchronous **
from (Bellemare et al., 2017)
DQN to learn well using up to 4 GPU learners, each using
batch size 512.
Categorical DQN: We found Categorical DQN to scale fur-
Table 2. Hours to complete 50 million steps (200M frames) by
ther than DQN. The lower panel of Figure 3 shows learning GPU and CPU count. A2C/A3C used 16 environments per GPU,
curves for batch sizes up to 2,048, with no reduction in PPO/APPO used 8 (DQN batch sizes shown).
maximum scores. This was possibly due to richer content # GPU (# CPU)
of the gradient signal. Notably, learning was delayed for A LGO 1 (5) 2 (10) 4 (20) 8 (40)
the largest batch sizes in the game S EAQUEST, but greater
A2C 3.8 2.2 1.2 0.59
maximum scores were eventually reached. Due to use of the A3C – 2.4 1.3 0.65
Adam optimizer, scaling of the learning rate was not neces- PPO 4.4 2.6 1.5 1.1
sary, and we return to study this surprising result shortly. APPO – 2.8 1.5 0.71
A LGO -B.S. 1 (5) 2 (10) 4 (20) 8 (40)
-Rainbow: Despite its use of distributional learning, -
*
Rainbow lost performance above batch size 512 in some DQN-512 8.3 4.8 3.1/3.9 2.6
games. Scores at this batch size roughly match those re- -R NBW-512 14.1 8.6 6.6 6.4
C AT DQN-2 K 10.7 6.0 2.8 1.8
ported in the literature for batch size 32 (Hessel et al., 2017)
*
(curves shown in appendix). Asynchronous

5.4 Learning Speed

cores, DQN and -Rainbow completed 50 million steps (200
We investigated the learning speeds obtainable when run-
million frames) in 8 and 14 hours, respectively–a significant
ning an 8-GPU, 40-core server (P100 DGX-1) to learn a
gain over the reference times of 10 days. These learning
single game, as an example large-scale implementation. Fig-
speeds are comparable to those in (Horgan et al., 2018),
ure 4 shows results for well-performing configurations of
which used 1 GPU and 376 CPU cores (see e.g. Figure 2
the policy gradient methods A2C, A3C, PPO, and APPO.
therein for 10-hour learning curves). Using multiple GPUs
Several games exhibit a steep initial learning phase; all
and more cores sped up our implementations. By virtue of
algorithms completed that phase in under 10 minutes. No-
a larger batch size, Categorical-DQN scaled best and com-
tably, PPO mastered Pong in 4 minutes. A2C with 256
pleted training in under 2 hours using the entire server, a
environments processed more than 25,000 samples per sec-
speedup of over 6x relative to 1 GPU. DQN and -Rainbow,
ond, equating to over 90 million steps per hour (360 mil-
however, experienced diminishing returns beyond 2 GPUs.
lion frames). Table 2 lists scaling measurements, showing
We were unable to find asynchronous configurations that
greater than 6x speedup using 8 GPUs relative to 1.
further boosted learning speed without curbing performance
We ran synchronous versions of DQN and its variants, with in some games (we only tested fully-communicating algo-
training times shown in Table 2. Using 1 GPU and 5 CPU rithms). Opportunities may exist to improve on our scaling.
 Accelerated Methods for Deep Reinforcement Learning

Figure 2. Scaling investigations for policy gradient algorithms: game scores vs aggregate sample count. Top) A2C with various batch
sizes (proportional to environment count), Upper) A3C with various numbers of 16-environment learner processes, Lower) PPO with
varied number of simulators, Bottom) Asynchronous PPO, 8 learners with varied communication period. In most cases, the scaled/adapted
versions match the baseline performance.

5.5 Effects of Batch Size on Optimization to the Q-value estimates near initialization and became too
off-policy to learn. In the same test using two 256-learners,
Possible factors limiting training batch sizes include: 1)
their scores matched. Had the 2048-secondary-learner out-
reduced exploration, since fewer different networks are ex-
paced the 2048-sampler-learner, it would have suggested
ercised in the environment, and 2) difficulties in numerical
exploration to be a more important factor than optimization.
optimization of network weights. We conducted experi-
See the supplementary materials for figures.
ments to begin to identify these factors.
Update Rule: We conducted an experiment to isolate the
Secondary-Learner Experiment: We configured a sec-
effect of update rule on optimization in Categorical DQN.
ondary DQN learner to train using only the replay buffer
We found the Adam (Kingma & Ba, 2014) formula to be
of a normal DQN agent. The secondary learner was ini-
superior to RMSProp (Tieleman & Hinton) in providing
tialized with the same parameter values as the primary,
large-batch learners with capability to traverse parameter-
“sampler-learner”, and the two networks trained simultane-
space during learning. When comparing agents achieving
ously, at the same rate of data consumption. Each sampled
the same learning curves, those using smaller batch sizes
its own training batches. In the game of B REAKOUT, 64-
(and hence performing more update steps) tended to have
and 2048-sampler-learners achieved the same score, but
larger parameter vector-norms at all points in training. Un-
the 2048-learner required more samples, despite using the
like RMSProp, the Adam rule resulted in a fairly tight spread
fastest stable learning rate (the number refers to training
in parameter norms between batch sizes without changing
batch size). When training a 64-secondary-learner using a
the learning rate. This explains the lack of need to scale the
2048-sampler-learner, the secondary learner’s score tracked
learning rate in Categorical DQN and -Rainbow, and indi-
that of the primary. In the reverse scenario, however, the
cates that the update rule plays an important role in scaling.
2048-secondary-learner failed to learn. We posit this was
Further details, including trends in convolutional and fully
due to the slower optimization of the decreased number of
connected layers, appear in an appendix.
parameter updates–it was unable to track the rapid changes
 Accelerated Methods for Deep Reinforcement Learning

Figure 3. Scaling investigations for DQN (top) and Categorical-DQN (bottom): game scores vs sample count. Both learn well using
training batch sizes as large as 512; Categorical-DQN succeeds using up to 2,048.

Figure 4. Policy gradient algorithms using an entire 8-GPU, 40-core server for a single learning run: game scores vs time, in minutes.
Asynchronous and synchronous versions learn successful game strategies in under 10 minutes.

Gradient Estimate Saturation: Using A2C, we measured likely to scale favorably to more sophisticated neural net-
the relation between the normal, full-batch gradients and work agents, due to GPU acceleration of both inference
gradients computed using only half of the batch, at each iter- and training. Moreover, as network complexity increases,
ation. For small-batch agents, the average cosine-similarity scaling could become easier, as GPUs may run efficiently
between
√ the full- and half-batch gradients measured near with smaller batch sizes, although communication overhead
1/ 2. This implies the two half-batch gradients were could worsen. Reduced-precision arithmetic could hasten
orthogonal, as are zero-centered random vectors in high- learning–a topic yet to be explored in deep RL due to use of
dimensional spaces. For large-batch learners (e.g. 256 CPU-based inference. The current, single-node implemen-
environments), however,
√ the cosine similarity increased sig- tation may be a building block for distributed algorithms.
nificantly above 1/ 2. Saturation of the gradient estimate
Questions remain as to the extent of parallelization possible
was clearly connected to worsened sample efficiency as in
in deep RL. We have not conclusively identified the limiting
the learning curves in the top panel of Figure 2.
factor to scaling, nor if it is the same in every game and
algorithm. Although we have seen optimization effects in
6 C ONCLUSIONS AND D ISCUSSION large-batch learning, other factors remain possible. Lim-
its to asynchronous scaling remain unexplored; we did not
We have introduced a unified framework for parallelizing
definitively determine the best configurations of these algo-
deep RL that uses hardware accelerators to achieve fast
rithms, but only presented some successful versions. Better
learning. The framework is applicable to a range of al-
understanding may enable further gains in scaling, which is
gorithms, including policy-gradient and Q-value learning
a promising direction for the advancement of deep RL.
methods. Our experiments show that several leading algo-
rithms can learn a variety of Atari games in highly parallel
fashion, without loss of sample complexity and at unprece- ACKNOWLEDGEMENTS
dented wall-clock times. This result indicates a promising
Adam Stooke gratefully acknowledges the support of the
direction to significantly boost experiment scale. We will
Fannie & John Hertz Foundation. The DGX-1 used for
release the code-base.
this research was donated by the NVIDIA Corporation. We
We note several directions for extension of this frame- thank Frédéric Bastien and the Theano development team
work. First is to apply it to domains other than Atari, espe- (Theano Development Team, 2016) for their framework
cially ones involving perception. Second, our framework is and helpful discussions during development of GPU-related
 Accelerated Methods for Deep Reinforcement Learning

methods. Thanks to Rocky Duan et al for the rllab code- Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve-
base (Duan et al., 2016) out of which this work evolved. ness, J., Bellemare, M. G., Graves, A., Riedmiller, M.,
Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C.,
R EFERENCES Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wier-
stra, D., Legg, S., and Hassabis, D. Human-level control
Babaeizadeh, M., Frosio, I., Tyree, S., Clemons, J., and through deep reinforcement learning. Nature, 518(7540):
Kautz, J. GA3C: gpu-based A3C for deep reinforcement 529–533, 2015.
learning. arXiv preprint arXiv: 1611.06256, 2016.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap,
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn-
The arcade learning environment: An evaluation platform chronous methods for deep reinforcement learning. In
for general agents. Journal of Artificial Intelligence Res., International Conference in Machine Learning, 2016.
47:253–279, 2013.
Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon,
Bellemare, M. G., Dabney, W., and Munos, R. A distri- R., De Maria, A., Panneershelvam, V., Suleyman, M.,
butional perspective on reinforcement learning. arXiv Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu,
preprint arXiv: 1707.06887, 2017. K., and Silver, D. Massively parallel methods for deep re-
inforcement learning. arXiv preprint arXiv: 1507.04296,
Clemente, A. V., Martı́nez, H. N. C., and Chandra, A. Ef- 2015.
ficient parallel methods for deep reinforcement learning.
CoRR, abs/1705.04862, 2017. URL [Link] Peng, J. and Williams, R. J. Incremental multi-step q-
org/abs/1705.04862. learning. In Machine Learning Proceedings 1994, pp.
226–232. Elsevier, 1994.
Duan, Y., Chen, X., Houthooft, R., Schulman, J., and
Abbeel, P. Benchmarking deep reinforcement learning Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Priori-
for continuous control. CoRR, abs/1604.06778, 2016. tized experience replay. In International Conference on
URL [Link] Learning Representations, 2016.

Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, Klimov, O. Proximal policy optimization algorithms.
I., Legg, S., and Kavukcuoglu, K. IMPALA: Scalable arXiv preprint arXiv: 1707.06347, 2017.
Distributed Deep-RL with Importance Weighted Actor- Theano Development Team. Theano: A Python framework
Learner Architectures. ArXiv e-prints, February 2018. for fast computation of mathematical expressions. arXiv
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Hessel, M., e-prints, abs/1605.02688, May 2016. URL http://
Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., [Link]/abs/1605.02688.
Pietquin, O., Blundell, C., and Legg, S. Noisy networks Tieleman, T. and Hinton, G. RMSprop Gradient Op-
for exploration. In International Conference on Learning timization. URL [Link]
Representations, 2018. edu/˜{}tijmen/csc321/slides/lecture_
slides_lec6.pdf.
Goyal, P., Dollr, P., Girshick, R., Noordhuis, P., Wesolowski,
L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, Van Hasselt, H., Guez, A., and Silver, D. Deep reinforce-
large minibatch sgd: Training imagenet in 1 hour, 2017. ment learning with double q-learning. In AAAI, vol-
ume 16, pp. 2094–2100, 2016.
Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostro-
vski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot,
Silver, D. Rainbow: Combining improvements in deep re- M., and de Freitas, N. Dueling network architectures for
inforcement learning. arXiv preprint arXiv: 1710.02298, deep reinforcement learning. In International Conference
2017. on Machine Learning, 2016.
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel,
M., van Hasselt, H., and Silver, D. Distributed Prioritized
Experience Replay. ArXiv e-prints, March 2018.

Kingma, D. P. and Ba, J. Adam: A method for stochastic

optimization. CoRR, abs/1412.6980, 2014. URL http:
//[Link]/abs/1412.6980.
 Accelerated Methods for Deep Reinforcement Learning

Supplementary Materials
A E XPERIMENT D ETAILS
A.1 Atari Frame Processing
Our frame pre-processing closely resembles that originally described in the original DQN publication. The sole difference is
that we abandon the square frame dimensions in favor of simply downsizing by a factor of 2, which provides crisp images at
minimal computational cost. Before downsizing, we crop two rows, making the final image size 104 × 80. This simplifies
selection of convolution size, stride, and padding. Otherwise, we keep all standard settings. For Q-learning experiments, we
used the standard 3-convolutional-layer network (DQN-Net) or its algorithm-specific variants, and for policy gradients the
standard 2-convolutional-layer feed-forward network of (A3C-Net). The second (and third) convolution layers have padding
1, so the convolution output is always 12 × 9.

A.2 DGX-1 Sampling Speed

Table 3 shows results of DGX-1 sampling speed, playing B REAKOUT using a trained A3C-Net, for various total simulator
counts. In the synchronized setting, a barrier was placed across all GPU processes every five steps (mimicking the
optimization in A2C). Otherwise, each GPU and its associated cores ran independently.

Table 3. Sampling speeds on the DGX-1 with A3C-Net, by total simulator count, in thousands of samples per second.

# S IMS ( PER C ORE ) S YNC A SYNC

64 (2) 29.6 31.9
128 (4) 33.0 34.7
256 (8) 35.7 36.7
512 (16) 35.8 38.4

A.3 Hyperparameters for Scaling

A2C: We used RMSProp with a base learning rate of 7 × 10−4 for 16 environments (e.g., scaled up to 3 × 10−3 for 512
environments).
A3C: We found no hyperparameter adjustments to be necessary.
PPO: We did not change any optimization settings.
APPO: Each 8-simulator asynchronous agent used the same sampling horizon (256) and update sequence as original PPO.
Relative to PPO, we introduced gradient-norm-clipping, reduced the learning rate by a factor of four, and removed the
learning rate schedule, all of which benefited learning.
DQN: When growing the simulator count and batch size in DQN, we maintained training intensity by adjusting the sampling
horizon and the number of update steps per optimization phase. For the batch sizes 32, 512, and 1024, we used learning
rates 2.5, 7.5, 15 × 10−4 , respectively. Hyperparameters other than learning rate were as in original publication.
Categorical DQN: We used a learning rate of 4.2 × 10−4 at all batch sizes 256 and above. We employed the published
setting for epsilon in the Adam optimizer: 0.01/L, where L is the batch size.
-Rainbow: We used the published hyperparameters without scaling the learning rate. We scaled epsilon in the Adam
update rule as 0.005/L, with L the batch size.
 Accelerated Methods for Deep Reinforcement Learning

B U PDATE RULE FOR M ULTI -S TEP A SYNCHRONOUS A DAM

Our asynchronous PPO experiments used the update rule described here, which permits multiple local gradient steps per
synchronization with the central parameters. The usual Adam update rule (Kingma & Ba, 2014) is the following. It has
fixed hyperparameters r, β1 , β2 , and ; g stands for the gradient; and all other values except the network parameters θ are
initialized at 0:

t←t+1
p
1 − β2t
a←r
1 − β1t
m ← β1 m + (1 − β1 )g
v ← β2 v + (1 − β2 )g 2
am
s← √
v
θ ←θ−s .

We kept these rules for making local updates and introduced the additional, local accumulation variables, also zero-initialized:

ag ← β1 ag + g
ag2 ← β2 ag2 + g 2
as ← as + s .

When applying an update to the central parameters, denoted with a tilde, we used the following assignments:

θ, θ̃ ← θ̃ − as
m, m̃ ← β1n m̃ + (1 − β1 )ag
v, ṽ ← β2n ṽ + (1 − β2 )ag2
ag , ag2 , as ← 0

where n is the number of local gradient steps taken between synchronizations. This rule reduces to the usual Adam update
rule in the case of a single learner thread.
 Accelerated Methods for Deep Reinforcement Learning

C F IGURES FOR S ECONDARY-L EARNER E XPERIMENT (DQN)

(a) (b)

Figure 5. Learning the game B REAKOUT with a secondary-learner using only the replay buffer of the normal, sampler-learner, both using
DQN. a) The 64-batch-size secondary-learner kept pace with its 2048-batch-size sampler-learner, but b) the 2048-batch-size secondary
learner failed to track its 64-batch-size sampler-learner or even learn at all. (Curves averaged over two random trials.)

(a) (b)

Figure 6. Neural network parameter vector-norms (l-2) during training. In both cases, the large-batch learner lagged behind the small
batch learner. In b) the parameters of large-batch secondary-learner continued to grow while its game score remained nil.

Figure 7. Learning the game B REAKOUT, where a secondary-learner using the same batch-size as the sampler-learner tracked (albeit
imperfectly) the game score, learning successfully. (Curves averaged over two random trials.)
 Accelerated Methods for Deep Reinforcement Learning

D O BSERVATIONS ON U PDATE RULES AND BATCH S IZE S CALING

We present observations of the effects of scaling the training batch size on neural net optimization under two different
parameter update rules: Adam and RMSProp (RMSProp without momentum and only direct accumulation of the squared
gradients, see e.g. [Link] We
trained agents on the game Q*B ERT with learning rates adjusted to yield very similar performance curves for all settings,
and we tracked the L-2 vector-norms of several quantities during learning. These included the gradients, parameter update
steps, and the parameter values themselves. As in all DQN experiments in this paper, the training intensity was fixed at 8, so
that the number of parameter update steps during learning scaled inversely with the batch size. Two random seeds were run
for each setting.
Although the game scores roughly matched throughout training, the exact solutions found at any point did not, as evidenced
by the differing parameter norms. No regularization was used. The following paragraphs follow the panels in Figure 8,
where curves are labeled by batch-size and learning rate. When viewing the network as a whole (i.e. norms of all weights
and biases as a single vector), trends reflected those seen in FC-0, where most of the weights are.
i) Learning Curves: We controlled for game score, adjusting the learning rate as needed. For the case of RMSProp with
batch-size 64, we included a learning rate that was slightly too low (1 × 10−4 ), yielding slow learning and lower final score,
and a learning rate that was slightly too high (5 × 10−4 ), yielding lower final score due to instability–these are the dashed
lines in all panels.
ii) Fully-Connected-0 Weights-Norm: The Adam optimizer yielded fairly tight grouping despite using the same learning
rate for all settings. The RMSProp learner, on the other hand, needed to scale the learning rate by 20x between batch sizes
64 and 1,024, which then produced very similar norms. At batch-size 64, slow / unstable learning was characterized by
small / large norms, respectively. The large norm of the batch-size 256 runs suggests this learning rate was likely near the
upper limit of stability.
iii) Fully-Connected-0 Gradients-Norm: Under both update rules, large batch sizes always produced smaller gradient
vectors–reduced variance led to reduced magnitudes. We also observed this pattern in policy gradient methods, when
looking at the total gradient norm. Here, the magnitude of the gradients depended inversely on the parameter norm; see the
RMSProp 64-batch-size curves. This effect was opposed and outweighed by the effect of batch size.
iv) Fully-Connected-0 Step-Norm: The Adam optimizer yielded significantly bigger step sizes for the bigger batch
learners, despite the smaller gradients. RMSProp required an adjusted learning rate to produce the same effect. Under
both update rules, the amount of step size increase did not fully compensate for the reduction in step count, indicating
that the larger batch learners followed straighter trajectories through parameter space. RMSProp led to significantly larger
steps overall, but despite this ended learning at smaller weights–its learning trajectories were apparently less direct, more
meandering.
v) Convolution-0 Weights-Norm: The Adam optimizer gave much greater spread in norms here than in the FC-0 layer; as
batch size increased, the learning emphasis shifted away from Conv-0. But in RMSProp the increased learning rate led the
first convolution layer to grow larger for larger batch sizes, placing more emphasis on this layer.
vi) Convolution-0 Gradients-Norm: The Adam update rule produced an intriguing cross-over in gradient norm; the large
batch learner actually started higher, bucking the trend seen in other cases. The pattern under RMSProp matched that for
FC-0.
vii) Convolution-0 Step-Norm: Unlike for FC-0, the step norm did not change significantly with batch size under Adam.
RMSProp yielded a similar pattern as in FC-0.
Overall, the Adam optimizer appeared to compensate for batch size in the FC-0 layer, but less so in the Conv-0 layer, leading
to de-emphasized learning in Conv-0 for large batches. The increased learning rate in RMSProp compensated for batch
size in the FC-0 layer and increased the emphasis on learning in Conv-0. This sort of pattern could have implications
for learning representations vs game strategies. Further study of these clear trends could yield insights into the causes of
learning degradation and possible solutions for large batch RL.
 Accelerated Methods for Deep Reinforcement Learning

Figure 8. L-2 vector-norms of the parameters, gradients (average), and parameter steps (average) for the first convolution layer and the
first fully connected layer while learning to play Q*B ERT with Categorical DQN at various batch sizes: Adam vs RMSProp.
 Accelerated Methods for Deep Reinforcement Learning

E A DDITIONAL L EARNING C URVES

Figure 9. Learning curves for Advantage Actor-Critic: baseline (A2C-16env) and scaled configurations, including synchronous and
asynchronous (to 25M steps = 100M frames). Only in ATLANTIS, G OPHER, and possibly K RULL does the baseline stand out above both
scaled versions.
 Accelerated Methods for Deep Reinforcement Learning

Figure 10. Learning curves for Proximal Policy Optimization: baseline (PPO-8env) and scaled configurations, including synchronous and
asynchronous (to 25M steps = 100M frames). Only in A STEROIDS and B OWLING does the baseline stand out above both scaled versions.
 Accelerated Methods for Deep Reinforcement Learning

Figure 11. Learning curves for scaled versions of DQN (synchronous only): DQN-512, Categorical-DQN-2048, and -Rainbow-512,
where the number refers to training batch size (to 50M steps = 200M frames). The anomalously low scores for -Rainbow in B REAKOUT
also appeared for smaller batch sizes, but was remedied when setting the reward horizon to 1 or with asynchronous optimization (cause
unknown; reward horizon 3 usually helped).