Two Level Control of Non-Player Characters for

Navigation in 3D Games Scenes: A Deep

Reinforcement Learning Approach
Gilzamir Gomes Creto A. Vidal
Department of Computing Department of Computing
Federal University of Ceará (UFC) Federal University of Ceará (UFC)
Fortaleza, Brazil Fortaleza, Brazil
gilzamir@[Link] cvidal@[Link]

Joaquim B. Cavalcante-Neto Yuri L. B. Nogueira

Department of Computing Department of Computing
Federal University of Ceará (UFC) Federal University of Ceará (UFC)
Fortaleza, Brazil Fortaleza, Brazil
joaquimb@[Link] yuri@[Link]

Abstract—This paper presents a deep reinforcement learning the shortest path between any two nodes. The found path is
approach for navigation problem coupled with traditional ani- typically smoothed to look more realistic to the player.
mation processes in games. Deep Reinforcement Learning (DRL) However, NavMesh lack the scalability and the flexibility
is a promising approach for this problem. So, we design a Non-
Player Character (NPC) as an autonomous agent guided by a necessary to provide more versatile navigation behaviors that
neural controller. Our approach works with the control of virtual enable the character to follow a diverse variation of paths.
game characters at two different levels of abstraction. A neural This is because adding new abilities to a preprogrammed agent
controller produces high-level actions, which are performed by requires adding new edges to the navigable polygon graph.
both a character controller and an animation controller. We While this is not an issue for simple skills, adding complex
test our approach on a three-dimensional game scene. We find
that our approach achieves promising results in navigation of skills often requires a redraw of the navigation map prone to
animated characters in a game scene. failure and rework.
Index Terms—non-player characters, reinforcement learning, Alonso and Peter [2] show that reinforcement learning can
navigation problem. be more flexible than NavMesh in generating complex naviga-
tion behaviors for NPCs. They also argue that reinforcement
I. I NTRODUCTION learning is a promising approach to point-to-point navigation
in three-dimensional mesh environments. Reinforcement learn-
Generating navigation behaviors is a typical problem when ing works for the navigation problem usually use very simple
dealing with current Non-Player Characters (NPCs). A first characters, without character part animations for skills such as
approach to game navigation was the Waypoint graph [1], walking or jumping (see reference [2] and Mirowski’s classic
which consists of a set of points of interest connected to work [3]). Furthermore, if we consider the success rate of
each other. This approach has several main drawbacks: its hitting the target, for predefined environments, NavMesh and
manual construction is prone to human errors, it can not classic search (A*) achieved maximum success rate. But that
control dynamic objects, it is expensive to build since it needs approach is not flexible in the sense that new agent scenarios
to check all n(n-1) combinations of paths, and the paths tend and skills would require manual readaptation of the agent.
to not look realistic since all agents follow the same set of Thus, their results were obtained using only simple avatars
constrained paths [2]. (often a box with a texture) and basic movements (without
NavMesh is the currently dominant approach for navigation articulation of limbs, for example). Training a reinforcement
problem and it has overcome the main waypoint graph prob- learning model to learn from scratch how to produce anima-
lems. NavMesh divides the game map into a set of convex tions (e.g, articulated limbs for walking) is computationally
regions, witch can each be trivially navigated within [2]. Once expensive. However, that feature is essential to viable NPCs
the polygons have been placed, a graph is created by using in modern games. So, without an appropriate decision-making
the polygons as nodes and by connecting adjacent polygons architecture, even the most complex behaviors of NPCs lack
with edges [2]. In each polygon, navigation is trivial. So, animation expressiveness in their actions and are visually
search algorithms (such as A* and Dijkstra’s algorithm) find cumbersome.
 In this work, we use a model-free reinforcement learning magnitude greater than those normally used in the Deep RL
since it does not force us to make strong assumptions about the literature. The authors reported success rates above 90% in all
environment, and, therefore, it helps to reduce the modeling tested scenes, including a real game scene.
time of the problem. We tried an approach for learning Previous work used model-free RL for navigation, either in
navigation behavior in three-dimensional games in terms of simple 2D environments [12], [13] or in complex navigation
computational resources, which was easier to integrate with problems [2]. Recent works, to overcome the absence of
game animation modeling tools, and minimizes memory con- planning in model-free RL, used a hierarchical architecture,
sumption. Thus, we use A3C algorithm for training because where intermediate goals are given to a controller by a high-
we do not make assumptions about heavy use of hardware. In level planner [14]–[16]. As navigation in a visually complex
fact, we want that our approach is able to run in many different environment is usually modeled as a partially observable
types of hardware. In addition, we adopted curriculum learning Markov decision process, the importance of memory use
and an informed reward function according to the approach has been previously recognized [3]. Although unstructured
presented in [2]. memory like LSTM (Long-Short Term Memmory) [17] can
Thus, our approach train a humanoid character through two be used, architectures involving spatially structured memory
levels of control. In the first level, a neural network sends have also been explored [18], [19]. The use of auxiliary tasks
an abstract action (for example, walking) to a programmed to accelerate the learning of RL problems based on challenging
character controller; and, in the second level, the character goals was also a subject of study [3], [20], [21]. Alonso et al.
controller interprets the received action in terms of velocity, [2] used the SAC (Soft Actor Critic) algorithm as an end-
and triggers an animation produced by an animation controller to-end solution to the navigation problem over a 3D map.
in a synchronized manner. This procedure makes it possible The agent is controlled by a neural network whose input
to obtain movement with convincing animations, which are consists of a 3D occupation map, a 2D depth map and a
designed by human modelers. We show how to encapsulate linear input containing information such as the agent’s position
the state of the animation to preserve the property of the and the target position in the game’s environment. The agent
Markovian state necessary for an agent to learn to navigate in uses a neural network with an LSTM layer to handle partial
a three-dimensional environment, while exhibiting convincing information.
animation of its body. Therefore, our work differs from those in the literature
because it combines reinforcement learning with a neural
II. R ELATED W ORKS controller specially designed to interact with a character’s
The use of reinforcement learning for training NPCs has controller. So, the agent has two levels of control in which
aroused the interest of the academic community for about a higher level controller selects abstract actions (e.g., walk)
a decade [4], [5], [6], [7], [8], but only recently, robust that are effectively performed by a preprogrammed lower level
results for AAA games have been achieved [2]. Still, only controller. Moreover, for the reinforcement learning problem,
recently game development tools have started to support agent we combine navigation learning using the A3C algorithm
training using environments built with the tool, even though combined with curriculum learning and an informed reward
this support is primarily for testing algorithms, techniques function (which is the opposite of a sparse function). For the
and machine learning approaches using games as test beds control of the NPC, we use a neural network architecture that
for artificial intelligence [9], [10]. In 2020, it was shown acts according to environment state perceived by the agent.
that the navigation behaviors, in 3D videogames, obtained
with reinforcement learning are superior to those obtained III. M ETHODS
with traditional approaches [2]. The training of an NPC with Here, we shown our Deep Reinforcement Learning (DRL)
reinforcement learning capable of expressing emotions and System for character control in navigation problem in games.
varying the NPC’s behavior according to its emotional state is The main objective of the agent is to reach a given target
shown by Gomes et al. [11]. position randomly placed in the game’s scene. For this, the
Navigation is what takes an NPC from one point to another agent’s architecture is endowed with four main components:
on a game map, so it is an essential component of an (1) a neural controller, (2) an agent body with actuators and
NPC. NavMesh is the game industry’s preferred approach sensors, (3) a controller that receives actions from agent’s
to deal with the navigation problem. Unfortunately, complex actuators and translates those actions into agent velocity that
navigation skills that extend the character’s movement ability are applied to the agent in a manner that is consistent with the
increase NavMesh’s complexity, making it unmanageable in agent’s body animations, and (4) an animation controller that
many practical settings [2]. Game designers are therefore runs the agent’s body animation. The environment interaction
restricted to only adding skills that can be manipulated by interface consists of sensors and actuators, which are modeled
NavMesh. Thus, Alonso et al. [2] proposed the use of Deep with the AI4U tool [10], a public tool available to facilitate
Reinforcement Learning (Deep RL) model-free to learn nav- the development of game environments with support for rein-
igation through 3D maps using any navigation skill. They forcement learning algorithms through the Unity Game Engine
further tested that approach in complex 3D environments using [9]. We also detail the components of the agent’s architecture
the Unity Game Engine [9], and maps that were one order of and the way to integrate them to generate navigation behaviors
with expressive animations. The overall DRL agent is shown the input increases the performance of networks that learn
in Fig. 1. sequential decisions, but at a higher computational cost.
C. The Agent’s Interface with the Environment
A. Neural Network Controller
The agent has sensors to perceive the environment and
The neural network controller implements the agent’s pol- actuators to control the NPC. The NPC is controlled from a
icy. The neural network (NN) receives perceptions of virtual player’s perspective, however, autonomously, because instead
sensors, and, in response to those perceptions, it outputs a of being controlled by a human, the character is controlled by
discrete probability distribution p = pa1 , pa2 , ..., pak , where a script that makes decisions based on an underlying neural
pai is the probability of the action ai . The agent take an network model. Thus, the NPC runs on the Unity Engine ar-
observation ot at time step t, and selects the action ai in a chitecture and in the AI4U framework, which allows control of
probabilistic manner according to the probability distribution the NPC through neural network. The agent’s neural network
p. selects actions based on the current state. The interdependence
of the components of this architecture is shown in Fig. 5.
B. A Neural Network Architecture
The actions generated by the neural network of the agent are
The agent’s neural network has a feedforward architecture applied to a physical controller and to an animation controller
with two input layers, two hidden layers, and two output based on unity’s animation mechanisms. More specifically, the
layers. The amount of neurons in the input and output layers neural network outputs an abstract action, such as ”walk”,
depend on the type of problem you want to solve. To determine to the physical controller, which moves the agent’s avatar,
the amount of input neurons, it was taken into account the say, forward, while the animation controller produces coherent
agent’s perception system. In addition, one of the output avatar’s movements, for example, the avatar’s legs’ animation.
layers produces the actions’ probabilities, and the other output That two-level control is essential to apply reinforcement
layer estimates the state value. The agent uses the actions’ learning to games, because it integrates reinforcement learning
probability distribution to select the current action. Estimated with animations produced by artists.
state values are used during the agent’s training. Two kinds of sensors were provided to the agent: a linear
The network’s inputs constitute an observation ot at time sensor, and a two-dimensional visual sensor. The linear sensor
step t, which contains two groups of perceptions: a visual captures global features, such as the agent’s world orientation,
data, and a linear data. The visual data is a sequence of relative orientation, distance to target, and touch on environ-
bidimensional matrices Rk shaping a three-dimensional signal ment’s objects. The two-dimensional visual sensor returns an
[I3 , I2 , I1 , I0 ], where Ik is a visual signal perceived by the array representing the local image seen by the agent at a
agent at time step t − k. Hence, to represent the visual data, given time step. While the linear sensor provides information
we use frame stacking strategies based on [22]. The image Ik more directly related to the goal of the agent’s navigation,
is a matrix with the codes of the seen objects in each cell, the two-dimensional visual sensor provides an input channel
in this case. The linear input consists of the agent’s forward that allows the agent to perceive nearby objects, allowing the
direction, orientation relative to the target, distance to target, agent to learn how to interact with objects in a manner that is
jump status, and touch signals. In the next section, we describe consistent with the agent’s view.
this inputs in more details. The visual sensor uses ray-casting to generate an image It
To process those two types of input data, the network has at time step t. In our model, a 30×30-pixel image is generated
two feature extractors: a convolutional bidimensional feature by casting rays that define a symmetrical frustum with a 90
extractor, and a linear feature extractor. The convolutional fea- degree field of view, as shown in Fig. 6. Therefore, as a result
ture extractor has two layers that use the ReLu [23] activation of its artificial vision, at each instant, the agent receives a
function: the first layer consists of 128 (4 × 4) filters with matrix of 30 × 30 float-point numbers representing the visual
stride equals to 2, and the second layer consists of 64 (2 × 2) information.
filters with stride equals to 1. The linear feature extractor has In turn, at each time step t, the agent’s linear sensor
two dense ReLu layers containing 30 neurons each. The input captures:
features are concatenated to be processed by the deeper layers • dt = (G − Pt )/||(G − Pt )||, the unit vector from the
(see Fig. 2). agent’s current position, Pt , to the target’s position, G,
We use two configurations of the deeper layers. The simpler which does not change during an episode;
one is shown in Fig. 3. The more complex configuration, • ft , the agent’s view direction (a unit vector);
shown in Fig. 4, uses two stacked LSTM (Long-Short Term • δt = ft · dt , the agent’s orientation relative to the target
Memory) [17] layers. The purpose of the latter configuration orientation;
is to deal with aspects of the environment that are not • St = (gt , ht ), the animation controller’s status, where gt
directly observable. Thus, the size of the visual input has is a Boolean variable, which is 0 if the agent is on a flight
also been increased to a sequence of size eight so that the mode, and it is 1 if the agent is in contact with a floor,
recurrent neural network LSTM can learn from sequential data. ht is the absolute height of the agent with respect to the
Experimentally, we found that the use of larger sequences in ground floor; and
 Fig. 1. An overall description of our DRL System.

Fig. 5. Map of Interrelations between the components of the approach used

in this work. The NPC implements its functions based on AI4U and Unity.
AI4U uses the functions of Unity and provides a specific view or form of use
of Unity for those who use AI4U.

Fig. 2. Convolutional layers for extracting features from the agent’s visual
input.

Fig. 6. Ray casting with an aperture angle of 90 degrees and symmetrical

perspective projection.

• Tt = (Tw , Tg ), the touch sensor’s signal, where the

variables Tw = Tg = 0 if no touch is detected, Tw = 1
when the agent touches a wall, and Tg = 1 when the
Fig. 3. Deeper layers of the network. In this configuration, we use traditional
fully connected feedforward layers.
agent reaches the target.
In addition to information captured by the linear sensors,
the model receives the pair, (at−1 , rt ), where at−1 is the
previous action performed by the agent, and rt is the reward
the agent received for that action. Therefore, the linear input
is represented by the tuple Lt = (dt , ft , δt , St , Tt , at−1 , rt ).
The actual input to the agent’s neural network at time t
consists of the last four sequences of data from both sensors,
Ot = (Lt−3 , Lt−2 , Lt−1 , Lt , It−3 , It−2 , It−1 , It ).
To select actions, the agent has a virtual actuator that phys-
ically controls the NPC in the three-dimensional environment
of the game. Virtual actuator has two levels of control. In first
Fig. 4. Deeper layers of the network. In this configuration, we use two stacked level, the agent’s neural controller sends a high-level command
LSTM layers. action ∈ [0, 1, 2, 3, 4, 5] to the second level. Then, the second
level translates the received action into an action encoding
(numerical array), and sends it to the character’s controller. success rate. The necessary information is the height of the
The character’s controller applies a velocity proportional to agent with respect to the ground reference plane and the feet
the value of the sent command, and activates an animation contact status with the floor.
controller to produce a corresponding animation based on
the received action. Specifically, at every time step t, the
character’s controller translates the command action into a
vector a = (f b, lr, ju, jf ), where f b is a real number that
is used in Equation (1) to compute the agent’s forward and
backward velocity; lr is a real number that represents the
agent’s left or (right) rotation; ju = 1 if the agent is supposed
to perform an upward jump and ju = 0 otherwise; and jf = 1
if the agent needs to perform a forward jump and jf = 0
otherwise. The command list supported by the agent’s neural
controller is defined as:
• Forward (command 0): [f, 0, 0, 0], where v is a fixed
positive real number;
• Backward (command 1): [-b, 0, 0, 0], where b is a fixed
positive real number; Fig. 7. The boxes labeled with a white or gray background indicate the
components of our reinforcement learning system. The blue arrow indicates
• Turn Left (command 2): [0, l, 0, 0], where l is a fixed the sending of commands or actions. When the character controller receives
positive real number; an action, it repeats that action k times in the environment. The black boxes
• Turn Right (command 3): [0, -r, 0, 0] where r is a fixed indicate the time steps in which the actions were applied. The blue boxes
indicate the actions applied in their corresponding time steps. The green
positive real number; arrows indicate the duration of the effects of each action. Note that the jump
• Jump (command 4): [0, 0, 1, 0]; and action affects the effects of the subsequent actions.
• Jump Forward (command 5): [0, 0, 0, 1].
For example, if the actuator controller receives a command E. The Reward Function
code 0 (Forward movement), it translates it into an action We use the AI4U [10] tool’s description of reward functions
vector a = (f, 0, 0, 0). The character controller translates this to annotate events in the environment with functions that
action into the agent’s velocity generate reward. More specifically, each agent’s action was
v = (ft × f × µ)/∆t, (1) noted with a reward function based on [2] and shown in
Equation 2:
where ∆t is the time step between two successive frames, and
µ is a multiplier factor to adjust speed to different time scales. Rt = max(min∀i∈[0,t−1] E(t, i), goal), 0) − α + 100touch(agent,goal) , (2)
Then, the animation controller produces expressive movements where E(t, i) = Di (agent, goal) − Dt (agent), Dt is the
synchronized with the agent’s velocity. Euclidean distance between the positions of its arguments
D. Deep Reinforcement Learning in Two Time Steps at time t, α is a positive value that represents a penalty
for each performed action (the agent is expected to perform
In real games, the human-like character’s movements are
the least amount of actions possible for reach its goal), and
expected to be animated to resemble human movements.
touch(agent, goal) is a predicate that is true when the agent
Instead of training the character to learn how to solve its main
touches the object that represents the position it has to reach,
navigation objective while learning to articulate its limbs in a
and false otherwise. Note that 100touch(agent,goal) is equal to
realistic manner, we train the character to learn how to solve its
one hundred (100) only if the predicate touch(agent, goal) is
main objective while being animated by traditional animation
true, otherwise it produces a 0 (zero).
mechanisms, which provide a rich library of ready-made
We use the AI4U tool to implement Equation (2) in order
animations designed by professional modelers and artists.
to visually associate events in the environment with the gen-
Joint training of the agent to obtain its main behavior while
eration of annotated rewards. So, AI4U manages the agent’s
managing an animation controller (which affects the charac-
reward at each time step t. At the beginning of a time step t,
ter’s movement) adds a level of extra time to the result of the
that is, before the environment receives the agent’s next action,
agent’s actions. This is because the actions generate effects
the value of r is equal to zero.
with different duration (see Fig. 7), especially when using an
Predefined objects (which in Unity are called prefabs) are
animation controller. For example, when the agent performs
associated with events in the environment and increases or
a jumping action, no other movement actions will produce
decreases r as those events occur. We set up an event that is
any effect when the agent is in the air, except the rotation of
fired whenever the environment receives an action sent by the
certain body parts, such as the turning of the head. We simply
agent. In this case, the event adds the first part of Equation
expand the state of the environment with information from
(2) to r, that is,
the animation controller to allow the reinforcement learning
(1)
algorithm to learn how to solve the problem with a high Rt = max(min∀i∈[0,t−1] (E(t, i), goal), 0) − α. (3)
 Then, we associate the collision event with the target object, TABLE I
which indicates that the agent reaches the the main goal. The R EINFORCEMENT L EARNING S YSTEM C ONFIGURATIONS . FF IS THE
NEURAL NETWORK ARCHITECTURE SHOWN IN F IG . 3 AND LSTM IS THE
reward generation function produces ARCHITECTURE SHOWN IN F IG . 4
(2)
Rt = 100touch(agent,goal) . (4) Configuration Model Architecture State
Base FF It includes animation status
In practice, this means that r is equal to the result shown in VAR1 FF It does not include animation status
(1) (2) VAR2 LSTM (One Layer) It includes animation status
Equation 2, that is, r = Rt = Rt + Rt .
VAR3 LSTM (Two Layers) It includes animation status
F. Agent’s Training
Our approach was partially based on the work of Alonso et
shown in Table I was tested in the simulated environment with
al. [2], with necessary adaptations for achieving efficiency and
the same set of initial seeds for the different configurations.
openness. To achieve efficiency we make as few assumptions
We obtained results with a confidence level of over 95%.
as possible about the hardware requirements when using
the training algorithm and the action execution model. To V. R ESULTS AND D ISCUSSIONS
achieve openness we consider a training algorithm with public
implementations adaptable to the demands of our work. Thus, The test results show that the proposed approach obtained
since the A3C algorithm has widely used and tested public NPC navigation behaviors with animated avatar. The graph in
implementations, we adapted it to run on low-cost hardware, Fig. 9 shows the agents’ success rate’s evolution per episode
as described in section III-B over 6, 000 episodes during training. Fig. 10 shows the success
We use the A3C algorithm for training in a point-to-point rate’s moving average per episode for each approach after
navigation scenario. The agent learns in two steps. In the first the training of the agent over 100 episodes. In this case, we
step, the agent learns an easy problem, and in the second assess whether the agent maintains performance after training,
step, a more difficult problem. In the performed experiments, that is, in a testing phase (in analogy to what is done in
a distance greater than 4.5 units of distance (ud) substantially supervised learning). In the test phase, the agent has already
increased the convergence time of the algorithm, so when the been trained and it is no longer learning. In addition, the
target is placed at a maximum distance of 4.5ud, the problem is environment is initialized differently from the training phase,
considered easy to solve by RL. The target position is chosen such that the position of obstacles is randomly placed in the
to be placed randomly in one of the buildings in the scene. scene. And in this case, the pseudo-random number generator
Then, the agent trains on an easy problem first until it reaches a is initialized with different seeds in relation to the training
50% success rate. Next, the agent trains on a difficult problem, phase. During the test phase, it is natural that the success
until it reaches a 90% success rate or up to approximately rate at the beginning is high and decreases with time until it
6,000 training episodes. stabilizes (see Fig. 10), given that the probability of the agent
making a mistake increases with time.
IV. E XPERIMENTS The results show that the agent was able to learn in a scene
To demonstrate the capacity of our DRL system, we used a with obstacles. The scene’s topology contains ramps, walls
map with dimensions of 400m × 400m × 35m shown in Fig. 8. and stairs arranged in a way that looks like a city street,
The evaluated configurations are found in Table I. To show but containing a wide open space that could cause the agent
that our reinforcement learning system is capable of learning to wander in the empty region. Fig. 11 shows, qualitatively,
how to solve the problem of navigation in games when we the the agent’s behavior of avoiding obstacles, jumping stairs,
have two different levels of control, we tested a base case going up a ramp and successfully reaching the target.
and three variations of our system. In the first configuration, The experiments showed that all variations of the proposed
we used a feedforward neural network model (neural network approach were able to improve their success rates during
architecture shown in Fig. 3), providing each required input to training. The Base configuration with a feedforward neural
that model. In the second configuration, we used a feedforward network obtained the best performance, both qualitatively
neural network model, but we do not use information from and quantitatively. The comparison between Base and VAR1
the animation controller (jump status and NPC height). The models tells us that the information from the animation con-
purpose of this configuration is to verify the hypothesis that troller and the height of the agent are two essential pieces
the agent only learns with a state description that takes into of information for the agent to learn a refined control of its
account information from the animation controller. Next, we actions.
tested two configurations with LSTM networks. One with The agents with LSTM network (VAR3 and VAR4) obtained
a single layer and one with two stacked layers. The two a much worse result than the agents Base and VAR1. Note
stacked layers contain 128 neurons each. The single-layer that the number of episodes was stipulated a priori, consid-
LSTM network is shown in Fig. 4. Based in [3], LSTM hidden ering a configuration of experiments with few computational
units output LSTM hidden activation signals. resources. The problem is modeled as a partially observable
During the training phase, each configuration of our DRL problem, because at each moment, the neural network has
system was trained for 6,000 episodes. Each configuration the global information of the target’s location and the local
 Fig. 8. Map used in the experiments (a three-dimensional map with dimensions of 400m × 400m × 35m, containing ramps, obstacles and towers).

Fig. 9. The success rate’s moving average of the last hundred episodes during Fig. 10. The success rate’s moving average during testing of the agents.
training of the agents.

egy that combines easy and difficult episodes. This training

information of the obstacles, including the ground touch strategy using the A3C algorithm proved to be efficient in
sensors and the agent’s height sensor. Thus, at each instant, obtaining policies with a higher entropy, which is necessary
the agent has all the relevant information to reach the target for environments with different scenes.
position. In this scenario, LSTM networks are not essential. We also showed that managing an animation control, while
Thus, given the same amount of training, it is natural for a the agent navigates the environments, adds extra complexity,
feedforward network to converge faster than a more complex since an animation controller can invalidate the Markovian
model like a LSTM network. state’s assumption, hindering the learning convergence. This
was solved simply by adding information from the animation
VI. C ONCLUSION
controller to the state perceived by the agent. This approach
The reinforcement learning approach presented in this work achieved greater performance in the tests.
proved to be effective in solving the navigation problem of Finally, although we have shown that reinforcement learning
animated NPCs in games, being able to combine global in- systems are efficient for the point-to-point navigation problem
formation, directly related to the goal, with visual information of animated virtual characters, there is still a number of
to promote navigation behavior that avoids different types of problems that need to be solved if reinforcement learning
obstacles. We also used a direct and effective training strat- systems are viable candidates to replace the approaches of the
Fig. 11. The agent’s behavior evading obstacles, jumping stairs and going up a ramp. The captured frames are in a temporal sequence, but with variable
capture time between them. The red line indicates the agent’s trajectory. The blue line indicates a jump and the straight line indicates that the agent performed
the walk action. In this case, the agent reached the target (greenish cylinder).

classic NavMesh. Among those problems, we can mention: [8] A. Dobrovsky, U. Borghoff, and M. Hofmann, “Applying and
generalization of scene (adapting agents that are trained in augmenting deep reinforcement learning in serious games through
interaction,” Periodica Polytechnica Electrical Engineering and
one scene to function in other scenes), behavior variability, Computer Science, vol. 61, no. 2, pp. 198–208, 2017. [Online].
navigation problem with sub-problems along the way, and Available: [Link]
navigation in open worlds games. [9] A. Juliani, V.-P. Berges, E. Vckay, Y. Gao, H. Henry, M. Mattar, and
D. Lange, “Unity: A general platform for intelligent agents,” arXiv
preprint arXiv:1809.02627, 2020.
R EFERENCES [10] G. Gomes, C. A. Vidal, J. B. Cavalcante-Neto, and Y. L. Nogueira,
“Ai4u: A tool for game reinforcement learning experiments,” in 2020
[1] L. Lidén, “Strategic and tactical reasoning with waypoints,” in AI Game 19th Brazilian Symposium on Computer Games and Digital Entertain-
Programming Wisdom, S. Rabin, Ed. Hingham, MA, USA: Charles ment (SBGames). IEEE, 2020, pp. 19–28.
River Media, 2002, pp. 211–220. [11] G. Gomes, C. A. Vidal, J. B. Cavalcante Neto, and Y. L. B. Nogueira,
[2] E. Alonso, M. Peter, D. Goumard, and J. Romoff, “Deep reinforce- “An emotional virtual character: A deep learning approach with rein-
ment learning for navigation in aaa video games,” arXiv preprint forcement learning,” in 2019 21st Symposium on Virtual and Augmented
arXiv:2011.04764, 2020. Reality (SVR), Oct 2019, pp. 223–231.
[3] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, [12] P. Mirowski, “Learning to navigate,” in 1st International Workshop on
M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu et al., “Learning to Multimodal Understanding and Learning for Embodied Applications,
navigate in complex environments,” arXiv preprint arXiv:1611.03673, 2019, pp. 25–25.
2016. [13] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva,
[4] S. Phon-Amnuaisuk, “Learning chasing behaviours of non-player char- and D. Batra, “Dd-ppo: Learning near-perfect pointgoal navigators
acters in games using sarsa,” in Applications of Evolutionary Com- from 2.5 billion frames,” in International Conference on Learning
putation, C. Di Chio, S. Cagnoni, C. Cotta, M. Ebner, A. Ekárt, Representations, 2019.
A. I. Esparcia-Alcázar, J. J. Merelo, F. Neri, M. Preuss, H. Richter, [14] S. Bansal, V. Tolani, S. Gupta, J. Malik, and C. Tomlin, “Combining op-
J. Togelius, and G. N. Yannakakis, Eds. Berlin, Heidelberg: Springer timal control and learning for visual navigation in novel environments,”
Berlin Heidelberg, 2011, pp. 133–142. in Proceedings of the Conference on Robot Learning, ser. Proceedings of
[5] T. Barron, M. Whitehead, and A. Yeung, “Deep reinforcement learn- Machine Learning Research, L. P. Kaelbling, D. Kragic, and K. Sugiura,
ing in a 3-d blockworld environment,” Deep Reinforcement Learning: Eds., vol. 100. PMLR, 30 Oct–01 Nov 2020, pp. 420–429.
Frontiers and Challenges, IJCAI, vol. 2016, p. 16, 2016. [15] B. Eysenbach, R. R. Salakhutdinov, and S. Levine, “Search on the replay
[6] F. G. Glavin and M. G. Madden, “Learning to shoot in first person buffer: Bridging planning and reinforcement learning,” in Advances in
shooter games by stabilizing actions and clustering rewards for reinforce- Neural Information Processing Systems, 2019, pp. 15 246–15 257.
ment learning,” in 2015 IEEE Conference on Computational Intelligence [16] X. Meng, N. Ratliff, Y. Xiang, and D. Fox, “Scaling local control to
and Games (CIG), Aug 2015, pp. 344–351. large-scale topological navigation,” arXiv preprint arXiv:1909.12329,
[7] J. Beck, K. Ciosek, S. Devlin, S. Tschiatschek, C. Zhang, and K. Hof- 2019.
mann, “Amrl: Aggregated memory for reinforcement learning,” in In- [17] S. Hochreiter and J. Schmidhuber, “Lstm can solve hard long time lag
ternational Conference on Learning Representations, 2019. problems,” in Advances in neural information processing systems, 1997,
 pp. 473–479.
[18] E. Parisotto and R. Salakhutdinov, “Neural map: Structured memory for
deep reinforcement learning,” arXiv preprint arXiv:1702.08360, 2017.
[19] E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf, “Egomap:
Projective mapping and structured egocentric memory for deep rl,” in
European Conference on Machine Learning and Principles and Practice
of Knowledge Discovery in Databases (ECML-PKDD), 2020.
[20] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder,
B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience
replay,” in Proceedings of the 31st International Conference on Neural
Information Processing Systems, ser. NIPS’17. Red Hook, NY, USA:
Curran Associates Inc., 2017, p. 5055–5065.
[21] D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devin, B. Eysenbach, and
S. Levine, “Learning to reach goals without reinforcement learning,”
arXiv preprint arXiv:1912.06088, 2019.
[22] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wier-
stra, and M. A. Riedmiller, “Playing atari with deep reinforcement
learning,” CoRR, vol. abs/1312.5602, 2013.
[23] H. Ide and T. Kurita, “Improvement of learning for cnn with relu
activation by sparse regularization,” 05 2017, pp. 2684–2691.