Mixed Deep Reinforcement Learning-behavior Tree for Intelligent

Agents Design

a b
Lei Li , Lei Wang, Yuanzhi Li and Jie Sheng
Department of Automation, University of Science and Technology of China, Hefei 230027, Anhui, China

Keywords: Reinforcement Learning, Behavior Tree, Intelligent Agents, Option Framework, Unity 3D.

Abstract: Intelligent agent design has increasingly enjoyed the great advancements in real-world applications but most
agents are also required to possess the capacities of learning and adapt to complicated environments. In this
work, we investigate a general and extendable model of mixed behavior tree (MDRL-BT) upon the option
framework where the hierarchical architecture simultaneously involves different deep reinforcement learning
nodes and normal BT nodes. The emphasis of this improved model lies in the combination of neural net-
work learning and restrictive behavior framework without conflicts. Moreover, the collaborative nature of two
aspects can bring the benefits of expected intelligence, scalable behaviors and flexible strategies for agents.
Afterwards, we enable the execution of the model and search for the general construction pattern by focusing
on popular deep RL algorithms, PPO and SAC. Experimental performances in both Unity 2D and 3D environ-
ments demonstrate the feasibility and practicality of MDRL-BT by comparison with the-state-of-art models.
Furthermore, we embed the curiosity mechanism into the MDRL-BT to facilitate the extensions.

1 INTRODUCTION these constrained behaviors have difculty responding

towards dynamically changing environments.
Designing an intelligent agent confronted with com- Reinforcement Learning (RL) as one of the
plex tasks in diverse environments is generally known paradigms and methodologies of machine learning
as an intractable challenge. A universally accepted based on Markov Decision Process (MDP) has been
definition of intelligent agents in (Wooldridge and scaled up to a variety of challenging domains, such as
Jennings, 1995) indicates that the agent can operate AlphaGo (Silver et al., 2016) and AlphaGo Zero (Sil-
automatically, perceive environments reactively and ver et al., 2017), Atari game (Mnih et al., 2013), Sim-
exhibit goal-settled acts initiatively, which demands ulated Robotic Locomotion (Lillicrap et al., 2015),
for the ability of observing, learning and behaving. StarCraft (Vinyals et al., 2017), even Vehicle Energy
The techniques of employing such intelligent agents Management (Liessner et al., 2019). Accordingly,
have profound impacts on a wide range of appli- deep RL gradually emerges with the significant ad-
cations including computer games, scenario simula- vance of neural network. Compared with behavior
tions, robot locomotion. trees, agents augmented with RL not only can po-
Behavior Tree (BT), expressed by (Dromey, 2003) tentially take adaptive strategies, but also learns in-
in the mid2000s, is a well-defined and graphical crementally a complex policy. But deep RL mod-
framework for modelling AI decision behaviors. As a els are always accompanied with poor sampling ef-
replacement of Finite State Machines (FSM) and a fa- ficiency and limited convergence, lacking a guarantee
vorable AI approach utilized inherently in games, BT of an optimal result. Another cause for the bounded
owns the features of re-usability, readability and mod- applicability is the difficulty of designing a reward
ularity. However, an excellent design of BT needs function that encourages the desired behaviors all
enough experience and efforts when the behaviour through training. With respect to hyperparameters,
representations of agents become increasingly com- most methods depend on special settings and easily
plicated. It’s apparent in the fact that agents with get brittle with a small change.
Whether an appropriate model can implement an
a [Link] intelligent agent with given demands is contingent
b [Link] upon effective design mechanisms and applicable ex-

113
Li, L., Wang, L., Li, Y. and Sheng, J.
Mixed Deep Reinforcement Learning-behavior Tree for Intelligent Agents Design.
DOI: 10.5220/0010316901130124
In Proceedings of the 13th International Conference on Agents and Artificial Intelligence (ICAART 2021) - Volume 1, pages 113-124
ISBN: 978-989-758-484-8
Copyright c 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
ICAART 2021 - 13th International Conference on Agents and Artificial Intelligence

ecution. As discussed in prior contents, there ex- tions of this paper is intended to contain the follow-
ist ubiquitous shortcomings in practicability above. ing aspects: firstly, we demonstrate a general model
It’s significant for us to embed the deep RL into MDRL-BT combined flexibly with different deep op-
BT and offer a valid model pattern. According to tions and a simple training procedure to design an in-
(de Pontes Pereira and Engel, 2015), a series of sub- telligent agent. Besides, we investigate potential traits
tasks can be abstractly transformed into a reinforce- of MDRL-BT for an effective training model. The la-
ment learning node and in turn BT, profiting from its tent variable generative models and primitive process
hierarchical architecture, can be enhanced reasonably of learning can be strengthened with mixed deep RL
by absorbing these nodes. In both theoretical and ex- algorithms including PPO and SAC by comparative
perimental aspects at last, the model named MDRL- experiments. Furthermore, we set up experiments on
BT can availably incorporate heteogeneous deep RL Unity 3D environment for high quality physics sim-
nodes and normal BT nodes to produce a considerable ulations and revise a simple and unified reward func-
improvement in intelligent agents design. tion about scores and time. Finally, the MDRL-BT
model with curiosity is implemented practically in an
empirical 2D application.
The remainder of this paper is structured in the
2 RELATED WORK following: The introduction of intelligent agents and
corresponding research are presented firstly. After-
The fundamental theories of BT arouse out of (Mateas wards, the theories of BT and RL and analysis of
and Stern, 2002; Isla, 2005; Florez-Puga et al., 2009). MDRL-BT architecture are introduced in detail. In
(Mateas and Stern, 2002) provided a behavior lan- the model section, we outline the framework based
guage designed specifically for authoring believable on options and bring the RL nodes into BT. At the
agents with rich personality as a primitive forerunner. same time, we facilitate the execution of the model.
(Isla, 2005) centering on scalable decision-making In the experiments, we build up some experiments
used BT to handle complexity in the Halo2 AI. (Sub- to search for better performance and draw some con-
agyo et al., 2016) enriches behavior tree with emotion clusions from the results. Finally, we summarize the
to simulate multi-behavior NPCs in re evacuation. work and look forward to the future research direc-
The deep RL originates from the paper (Mnih tion.
et al., 2013) with the enforcement of CNN network
directly. (Mnih et al., 2015) has stricken a great suc-
cess by developing a deep Q-network (DQN) in this
field. (Schulman et al., 2015) proposes Trust Region 3 PRELIMINARIES
Policy Optimization (TRPO) in policy optimization.
On the basis of TRPO, Proximal Policy Optimization Formally speaking, a behaviour tree is composed of
(PPO) (Schulman et al., 2017) takes the minibatch up- some nodes and directed edges where internal nodes
date and optimizes a surrogate objective function with called composite nodes and leaf nodes known as ac-
stochastic gradient ascent. Soft actor-critic (SAC) are tion nodes are connected by edges.
proposed by Haarnoja (Haarnoja et al., 2018) to max- Each node is classified by the execution strategy
imize expected reward and entropy in Actor-Critic in the following. Sequence, analogies to logical-and
(AC). operation, returns Failure once one of the children
The concept of integrating RL into BT has been fails, otherwise Success. Note that Fallback nodes,
put forward to alleviate the endeavors of manual pro- equivalent to logical-or, are appropriate for executing
gramming in some research. (Zhang et al., 2017) the first success nodes. Condition nodes represent a
combines BT with MAXQ to induce constrained and proposition check and instantly return Success if the
adaptive behavior generation. (Dey and Child, 2013) condition holds or Failure if not yet. All Action nodes
presents Q-learning behaviour trees (QL-BT). In the having specific codes return Success if the action cor-
(de Pontes Pereira and Engel, 2015), a formal den- rectly completes, Failure if it is impossible to con-
ition of learning nodes is applicable to address the tinue and Running when the process is ongoing.
problem of learning capabilities in constrained agents. The execution of BT operates with a tick gener-
Yanchang Fu in (Fu et al., 2016) carries out simu- ated by a root node at a given frequency, which prop-
lation experiments including 3 opponent agents with agates in depth first. When receiving the signal, the
RL-BT. In (Kartasev, 2019) there are detailed descrip- node invokes its execution, enables the corresponding
tions of Hierarchical reinforcement learning and Semi behaviors or traverses the tick to children. Each node
Markov Decision Processes in BT. except root completes with the return status of Run-
Compared with the relevant works, the contribu- ning, Success or Failure, which is transferred to the

114
 Mixed Deep Reinforcement Learning-behavior Tree for Intelligent Agents Design

parent for determining the next routine. Until the root regularization to maximize a trade-off between explo-
node terminates, a new tick always comes into being ration and exploitation with the acceleration of the
from root with a cyclical loop. learning process. The agent at every time step obtains
The key problem of RL aims at maximizing cumu- an augmented reward proportional to the expected en-
lative rewards in the interactions with environments. tropy of the policy over ρπ with trade-off coefficient
At time step t, the agent in a given state st ∈ S de- α:
cides on an action at ∈ A with respect to a mapping
T
relationship called the policy π : S → A, then receives
a reward signal rt and reaches a new state st+1 ∈ S.
J(π) = ∑ E(st ,at )∼ρπ [r(st , at ) + αH (π(·|st ))] (4)
t=0
In a given environment, S is a complete description
of state space and A often represents the set of all
valid actions. Generally speaking, the entire sequence 4 MODEL
of states, actions and reward can be considered as an
infinite-horizon discounted Markov Decision Process
(MDP), defined by the tuple (S, A, p, r). The state The general approach for maintaining the superiority
transition probability p demonstrates the probability of RL and BT together is to apply the option frame-
density of the next state st+1 in the condition of the work to BT. On this basis, deep RL algorithms can be
current state st ∈ S and action at ∈ A. To represent imbedded unaffectedly in the learning nodes to obtain
the long-term cumulative reward, the discount fac- observation information and make decisions in accor-
tor γ is considered to avoid the infinite total reward: dance with the learned policy. In the meantime, the
Rt = ∑Ti=t γi−t ri (si , ai ). learning nodes are claimed to keep the feasible and
In Q-learning, to evaluate the expected return of a constrained characteristics of normal nodes. Deriv-
policy, a value function is defined: V π (s) = Eπ [Rt |st = ing from recursive BT, the generated model MDRL-
s] and the state-action value function is the expected BT stresses on a relatively simple and efficient real-
return for an action a performed at state s : Qπ (s, a) = ization and implements an optimized execution with
Eπ [Rt |st = s, at = a]. From the Bellman equation, these nodes.
the recursive relationship can be shown: Qπ (st , at ) =
Eπ [rt+1 + γQπ (st+1 , at+1 )|st = s, at = a]. In DQN 4.1 The Option Framework in BT
(Mnih et al., 2015), a deep convolutional neural net-
work is used to approximate the optimal action-value
function as follows: The central trait of MDRL-BT focuses on an idea that
a big task can be decomposed into multiple smaller
Q∗ (s, a) = max E[rt + γrr+t + ...|st = s, at = a, π] (1) tasks in BT and several nodes associated with a task
π
can aggregate into a learning node. Each divided
In the policy network (Silver et al., 2014) , log task reduces the non-linear increase of dimensional-
loss and discount reward are used to update the pol- ity with the size of the observation and action space,
icy guide gradient. The policy can be updated as the similar to the essence of Hierarchical Reinforcement
equation: Learning (HRL) and Semi Markov Decision Pro-
∇θ J(µθ ) = Es∼ρµ [∇θ µθ (s)∇a Qµ (s, a)|a=µθ (s) ] (2) cesses (SMDP) based on the option framework pro-
posed in (Sutton et al., 1998). In the framework, a
Mathematically, the advantage function which is primary option is initialized in a certain state and then
crucially important for policy gradient methods is de- a sub-option is adopted by the learning strategy. After
fined by Aπ (s, a) = Qπ (s, a) −V π (s). Proximal policy that, the sub-option proceeds until it terminates and
optimization (PPO) (Schulman et al., 2017) breaks another option continues like the running process of
down the return function into the return function by BT.
the old strategy plus other terms with the monotonic An option is a 3-tuple consisting of three elements
improvement guarantee and gives a definition of the < I , π, β > where : I ⊆ S indicates the initial state of
probability ratio: rt (θ) = π πθ (a(at |st |st )t ) and rt (θold ) = 1. option, π : S × O → [0, 1](O = s∈S Os ) represents the
S
θold
The main objective of PPO is the following: semi-markov policy which is a probability distribu-
tion function based on state space and option space,
L(θ) = Êt [min(rt (θ)Ât , clip(rt (θ), 1 − ε, 1 + ε)Ât )] µ : S × O × A → [0, 1] with additional action space
(3) defines the intra-option policy. For each state s, the
Soft Actor Critic(SAC) (Haarnoja et al., 2018), an available options are represented by O (s). When the
extended stochastic off-policy optimization approach present state s is an element of I , a corresponding op-
based on actor-critic formulation, centers on entropy tion is successfully initialized. The bellman equation

115
ICAART 2021 - 13th International Conference on Agents and Artificial Intelligence

for the value of an option o in state s can be expressed:

µ
QπO (s, o) = QO (s, o) + ∑ P(s0 |s, o) ∑ π(s0 , o0 )QπO (s0 , o0 )
s0 o0 ∈Os
(5)
The current option chooses the next option o with
the probability π(s, o) during the execution and then
the state can change to s0 . Moreover, the definition of Figure 1: The transformation from several normal nodes to
the intra-option value function is: a learning node with observations, reward functions, policy
and actions.
µ
QO (s, o) = ∑ µ(a|s, o)QU (s, o, a) (6)
a
sub-options can be modelled as an option with an in-
where QU : S × O × A → R represents the action put set I = S, an output order (ci1 , ci2 ..., cin ), a termi-
value in a state-option pair according to (Bacon et al., nation β = 1 if any Tick(ci ) ∈ Success or all Tick ∈
2017): Failure and a policy π.
QU (s, o, a) = r(s, a) + γ ∑ P(s0 |s, a)U(o, s0 ) (7) The learning Fallback nodes can query the rele-
s0 vant children every episode in learned priority instead
β : S → [0, 1] is the termination condition and β(s) of the constant order. The corresponding learned pol-
means that state s has the probability β(s) of termi- icy π, according to the observations mainly correlated
nating and exiting the current option. The value of o with the state of the environment, devotes to decid-
upon the arrival of state s0 with the probability β(s0 ) ing one of the children and then holding a series of
of option termination, U(o, s0 ) is written as: updates during execution. The learning Sequence re-
sembling learning Fallback just differs in the termi-
µ
U(o, s0 ) = (1 − β(s0 ))QO (s0 , o) + β(s0 )VO (s0 ) (8) nation condition β = 1 if any tick(ci ) ∈ Failure or all
ticks ∈ Success. The example of learning composite
In the context of BT, the termination condition β
nodes involving SAC is illustrated in algorithm 1.
is bound up with the return status of Failure or Suc-
cess. A new episode starts when an option is acti- Definition 2. A learning action node can be modelled
vated by a signal tick for a timestep and ends up with as an option with an input set I = S, actions a ∈ As , a
the termination of option. With regard to Running, termination condition β, and a policy µ.
it accounts for the process of an option node with a In most cases, MDRL-BT abstracts a subtree into
consecutive series of uninterrupted ticks in BT. This a learning action node for potential performance and
would imply that the ticks complete the synchroniza-
tion with RL algorithm. As far as an MDP problem Algorithm 1: SAC composite nodes with N children.
is concerned, the option collects the actions, rewards Input: an input set I = S , initial state value func-
and states stored in trajectories D , which updates the tion parameters φ, φ and soft Q-function paremeters
option-option policy π or intra-option policy µ. In ψ, tractable policy θ parameters, count steps k = 0.
general, traditional composite and decorator nodes in Output: Failure, Suceess, or Running
BT have a fixed policy for calling their children se- 1: state is Failure if Sequence, Success if Fallback
quentially. In this paper, we remove the limitations for 2: if a tick arrives then
the utilization of option framework so that the policy 3: collect global state sk , run policy πθ , take ac-
π could rearrange the execution order of children. tion ak ∈ A and get an index order i1 , i2 ..., iN
4: for j ← 1 to N do
4.2 Reinforcement Learning Nodes 5: childstatus ← Tick(child(i j ))
6: if childstatus=Running then
Based on the previous theorem (de Pontes Pereira and 7: return Running
Engel, 2015), learning action and composite nodes 8: else if childstatus=state then
are referred to as the extensions of the normal BT 9: goto → line 11
nodes. These learning nodes not only successfully are
10: state ← ∼state
equipped with the learning capacities , but also main-
11: receive reward rk , add tuple (sk , ak , rk , sk+1 ) to
tain the readability and modularity in hierarchical BT
trajectories Dk , update the parameters(i ∈ {1, 2}):
framework. For learning composite nodes, the notion
φ ← φ − λV ∇ˆ φ JV (φ), ψi ← ψi − λQ ∇
ˆ ψ JQ (ψi )
of learning fallback nodes is defined as follows. i
ˆ θ Jπ (θ), φ ← τφ + (1 − τ)φ, k ← k + 1
θ ← θ − λπ ∇
Definition 1. A learning fallback node below at-
tached with children c1 , c2 ..., cn as possible choosing 12: return state

116
 Mixed Deep Reinforcement Learning-behavior Tree for Intelligent Agents Design

Figure 2: The structure of MDRL-BT contains four categories. The blue option nodes refer to the two kinds of learning nodes
with deep RL algorithms. The blank nodes indicate the normal BT nodes and the green rectangle with imaginary line is not a
tree node but the input of external state of environment or initial design settings.

simplification. In figure 1, the entire BT can be turned 4.3 MDRL-BT Architecture

into a learning action node for an enemy attack in
this simple case. In short, the learning action nodes In this section, we will systematically illustrate the ex-
carry through an MDP to define the evolution of agent tended architecture of MDRL-BT and analyze respec-
states. The learning action nodes with PPO can be tively the different functions of every node area. As
summarized in algorithm 2. stated in figure 2, it is a recursive BT as a whole with a
core option and four types of divided areas, in keeping
with the hierarchies of option framework. It deserves
Algorithm 2: PPO learning action nodes.
to be mentioned that every periodic tick represents a
Input: an input set I = S, initial policy parameters temporal level of timescale signal propagating from
θ, value function parameters φ, count steps k = 0. top to down and activates the running courses of trig-
Output: Failure, Suceess, or Running gered nodes. The core option as a representative of
1: if a tick arrives then the core logic abstraction from complicated tasks can
2: collect local state sk , run policy πθold , take ac- also be replaced with normal composite nodes with a
tion ak ∈ A, and receive reward rk fixed execution setting. The children of core option is
3: if the task goal is finished then roughly classified on the grounds of types of nodes.
4: return Success Although the four areas are distinct, every area is di-
5: else if impossible to finish then rectly connected with the core option and can be inter-
6: return Failure spersed disorderly with every independent individual
7: else of other areas. The parameter N in every area is also
8: add (sk , ak , rk , sk+1 ) to the trajectories Dk , different, ranging from zero to infinity.
compute rewards-to-goes R̂t , and compute the ad- The blue option nodes are the learning compos-
vantage estimates Ât , based on value function Vφk . ite nodes with discrete outputs. In conjunction with
9: Update the policy parameter: global state input relative to the local state, this type
θ = argmax Êt [min(rt (θ)Ât , clip(rt (θ), 1 − ε, 1 + ε)Ât )] of nodes can distill global observations to dispense
θ
the order index of children. As seen in figure 2, the
φ = argmin Êt [(Vφ (st ) − R̂t )2 ] recursive sub-tree can follow the option node with the
φ
same framework of four areas generation after gener-
10: θold ← θ, φold ← φ, k ← k + 1
ation. So the recusive sub-tree can be large or small
11: return Running
depending on specific tasks and in this perspective the

117
ICAART 2021 - 13th International Conference on Agents and Artificial Intelligence

MDRL-BT combined with this type of node can be in ably with the demonstrations for solving complicated
possess of flexible structure and have certain general- problems and improving learning efficiency of the
ity in applications. tree.
The blue action nodes are the learning action A myriad of flexible classical algorithms are in-
nodes which collects the local state to improve pol- corporated concurrently when different blue nodes are
icy with continuous or discrete actions. The nodes adopted, giving rise to the nature of BT. It’s plausible
with meticulous reward function can facilitate imple- that MDRL-BT can reap the advantage of BT and RL
mentation of subtask and simplify large-scale archi- and can vary with practical applications to cater for
tecture. In the presence of several learning action designers’ needs. The next part would introduce the
nodes with the similar action space, reward function execution process and effective reward function.
and task goals, it is recommended that a RL brain can
be independent of these nodes and keep parallel con- 4.4 MDRL-BT Execution
nection for reusability and time saving, such as details
in Experiment 1. Accompanying the MDRL-BT with There are N learning children of core option with
this blue action nodes in essence extends the BT to unfixed execution time interval Tn , policy πn , the
MDRL-BT. corresponding state Sn , action space An , reward Rn
(n ∈ [1, N]) and M normal nodes with time interval
Composite nodes and action nodes are subsumed
Tm and degree of goal completion Gm . Uniformly the
together into blank nodes. For the blank option nodes
core option has Tc ,πc ,Sc ,Ac ,Rc . Along with the begin-
they can be conventional identified types of Fallback,
ning of MDRL-BT, the core option gathers observa-
Sequence, Parallel, Decorator mentioned above. The
tion sc ∈ Sc and take an action ac ∈ Ac , get the tuple
priority setting, p(s) → R mapping the state to pri-
results of index order (i1 , i2 ..., iN ). At the time t, we
ority value, is the function of the execution order de-
can get the following equation.
signed initially as the input. The blank nodes of Ac-
tion or Condition with typical commands are com- (i1 , i2 ..., iN )t = atc |πc (stc ), atc ∈ Ac , stc ∈ Sc , (9)
mon indivisible units. As the granularity of MDRL-
BT, the executable action nodes can be defined prefer- The execution flow of MDRL-BT can be summa-
rized detailedly in figure 3. Up to now, the tuple
< stc ,atc ,rtc ,st+1
c >, where sc is the global state of next
t+1
episode, can be stored in its buffer trajectories τ for
experience replay. The subsequent procedure may be
easily adapted to other learning options with a plural-
ity of subspaces because of the recursive inference for
subtree. In the local time interval, the children moti-
vate MDPs, considered explicitly as the sub episode
of the high level.
It’s an assumption that the first k children nodes
has finished with Success status and the ik+1 node
aborts the next execution. It is derived that the total
time of core option is the sum of first k normal exe-
cution time where ε is the total error of transfroming
time and Tc corresponds to the amount of the whole
tree execution time.
k
Tc = ∑ Ti j + ε (10)
j=1

The reactive rewards function needs to reflect the

tendency of goal-achieving in a sense. In the ordi-
nary way, the rewards are tied to the execution time
and degree of task completion, and should be normal-
ized theoretically for training performance. In this pa-
Figure 3: The execution process of MDRL-BT. It is actu-
ally a nested process with inter-option and intra-option pol- per, the abort status doesn’t exist in scenarios where
icy distinguished by having children. The traditional nodes the core option can execute the all children nodes
suffer from stationary policies and are ignored selectively. k = M + N. Hereby, the mixture rewards of core op-

118
 Mixed Deep Reinforcement Learning-behavior Tree for Intelligent Agents Design

tion can be computed in the following.

(
f 1 (Ti (t)) + fi2 (Gi (t)) i ∈ normal nodes
rti = i1
fi (Ti (t)) + Ri (t) i ∈ learning nodes
(11)
M+N
rtc = Normalize( ∑ rti ), rti ∈ Ri (12)
i=1
where fi1 is always a piecewise function for reducing
the time error ε and fi2 is a mapping function for goal (a) experiment 1 2 env. (b) experiment 3 env.
achievement. The equation is not the only rewards de- Figure 5: The plane environments. Every tagged objects
sign but can sometimes be a more reasonable choice would be labelled by arrows. Four discrete actions only be
than the others. taken by the agent to achieve the taskmove forward, move
Thus far, MDRL-BT has made use of option backward, turn left, turn right. Furthermore, the agent is
framework to ascertain usability in a theoretical man- provided by a view fan field composed by a number of ray
ner. This conjugated model with this hierarchical de- sensors as the primary observations which can detect the
corresponding objects. The apparent information of relative
sign for the intelligent agents, compatible with the positions, rotations and distances, are added collectively up
structure of BT, mixes nodes together and can over- to 109 and 1490 observations.
come the weakness of the RL and BT. In the model, it
turns out to be that the learning can be undertaken in to real-world situations for generality and veracity. In
tandem by mixed RL nodes and the constrained run- order to take a deep dive into the traits of MDRL-BT,
ning is solely in the charge of BT. MDRL-BT with the training models are configured with different con-
the underlying option framework has circumvented structions and components as comparison. The Unity
the conflicts between BT and RL and can be easily ML-Agents Toolkit (Juliani et al., 2018), accessible to
altered for different targets. In the next section, we the wider research, serves as an open-source project
will do some experiments for comparison and mani- and can be used to train intelligent agents through a
fest operability. simple-to-use Python API. In (Noblega et al., 2019),
adaptable NPC-agent with PPO has been devised in
Unity ML-Agents Toolkit environments as an enlight-
5 EXPERIMENTS enment of experimental simulations.

Several valuable experiments with various complex- 5.1 Description

ity are performed in this section to identify the char-
acteristics of MDRL-BT and validate the intelligent Inspired by (Sakr and Abdennadher, 2016) in which
agents. Most experiments are consequently con- rescue and saving simulation involves task plan-
ducted and measured on Unity 3D environments close ning and realistic estimations, experiment 1 is es-
tablished by extending simulated fire control scenar-
ios (de Pontes Pereira and Engel, 2015) to a 3D en-
vironment in figure 4(a)(b) for agent training. Ex-
periment 1 and 2 almost take place in an identical
surroundings where independent RL can accomplish
(a) experiment 1 2 env. (b) experiment 3 env.
the benchmarks and the proposed MDRL-BT would
combat the challenges of baselines. With regard to
complex relationship in experiment 3, significant ad-
vances in performance are made by MDRL-BT irre-
spective of training and testing. And an agent attached
(c) corgi env in global. (d) corgi env in local. with MDRL-BT on a 2D game about corgi engine is
heightened by the driven-curiosity learning for a fur-
Figure 4: The unity 3D environments (a)(b) of experiments ther expansion.
and the high fidelity makes it possible for agents to bring The standard of scores acquired by agent for eval-
evidence towards the application of model. Details and dis-
uation is measured by the degree of target completion
cussion of experiments are released in the description sec-
tion. Corgi environment (c)(d) with an extendable engine and the frequency of collisions with the walls. Simul-
is a friendly 2D game where agents can finish some simple taneously, the total time of each episode is collected
tasks. separately for the estimation of completion speed.

119
ICAART 2021 - 13th International Conference on Agents and Artificial Intelligence

are implemented separately to establish a test base-

line and opsive behavior designer is integrated by
Unity NavMeshAgent to build a normal BT with-
out the need of training (BT). To employ the ver-
satile framework of MDRL-BT appropriately, learn-
ing action nodes at first are assigned as children of a
sequence node in BT which is identical to the con-
(a) model in exp1. (b) model in exp2. struction in figure 6(a) except option nodes. The se-
Figure 6: The models of two experiments. The solid lines quence node as a core attempts to control the main
are ticks flowing between nodes and the imaginary lines process and the action nodes with SAC (BTsac ) and
proceed with an interaction between blue nodes and green PPO (BT ppo ) opt to act from the learned strategy. In
brain. The RL option in dark blue is a learning compos- the cost of more observations and changing inputs, the
ite node in all experiments and the RL nodes in watery blue different action nodes can be connected with a shared
are learning action nodes aiming at sub-tasks like saving the RL brain to speed up training in terms of the similar
victim. Other blank nodes are nothing but normal nodes.
strategy.
Undoubtedly, the sequence node above with a
constrained querying pattern may lead to a sub-
optimal consequence on account of hardly inevitable
order, which is also the universal self-imposed restric-
tions in general BT. We replace the sequence node
by RL option node as the figure 6(a) shows, which
is indicative of the breakthrough point of the limited
structure. The core option node serves as a global
decision maker to explore a better consequence, cal-
culating specific target value and scheduling the ex-
pected queries with learning policy. Accordingly,
there are two promising alternative options with PPO
(Option ppo ) and SAC (Optionsac ) for further training.
Figure 7: The mixed model described is composed of 12 For removing the impacts of learning action nodes,
watery blue PPO learning action nodes with a shared brain, PPO action nodes aren’t modified in the two models.
3 dark blue SAC composite learning options with different
configurations and another 9 blank normal BT nodes. Taking the time of training into consideration, we also
use a previously trained PPO action node in the start
In order to assess the results equally and exactly, to apply the option with SAC(Option pre ).
the mean values of every experiment upon thirty-two Experiment 2. On the foundation of the preced-
thousand times are calculated. Consistent with the ing subject in experiment 1, an extinguisher marked
general learning process, incremental steps and re- green is placed as a vital part of the environment and
wards of feedback during the training are kept track the sequential order of three independent tasks is de-
of to understand the convergence. manded to confirm the positive features of BT in ex-
Experiment 1. As enumerated in plane figure periment 2. The agent certainly acquires an extin-
5(a), there are four types of objects characterized by guisher intended for addressing the fire issue ahead
victim, fire, criminal and agent. The task refers to it of time, otherwise approaching the fire within certain
that the agent is bound to save victim, extinguish the distances leads to a punishment of reward and score.
fire and catch criminal as soon as possible. For ev- The model described in figure 6(b) is dominated by a
ery episode, the agent is commanded to accomplish sequence node in this restrictive and flexible circum-
the task spontaneously but the four objects are ini- stance. For comparative analysis, the training holds
tially placed or reset in a random pattern to eliminate fixed steps of 3 ∗ 107 .
the training contingency. This scenario is surrounded Experiment 3. An increasingly complicated re-
markedly by high walls in figure 5(a)(b) to prohibit quest arises that the agent struggles to undertake three
stepping outside. Afterwards, the ground without victims saving and put out two types of fire previ-
friction appears so smooth that the agent with man- ously, then catch a criminal and enter the door to
ual control is indeed difficult to manipulate in discrete restart a period lastly in figure 5(b). Meanwhile,
action spaces. three extinguishers with corresponding tags are as-
As aforementioned, the state-of-arts of off-policy sociated with the assumption that extinguisher1 only
SAC (labelled as SAC) and on-policy PPO (PPO) deals with fire1, extinguisher2 only for fire2 but extin-

120
 Mixed Deep Reinforcement Learning-behavior Tree for Intelligent Agents Design

(a) exp1 training. (b) exp1 training. (c) exp2 training.

(d) exp3 training. (e) exp3 training. (f) corgi training.

Figure 8: The training curve of experiments. Because learning composite options are different from learning action nodes
in training frequency. So experiment 1 and experiment 3 would have two graphs. Experiment 2 and corigi without learning
composite nodes only have one graph. Every curve with corresponding tag means an algorithm with the same conditions in a
graph.

guisher3 for fire1 and fire2 both. That’s the same case quent movement manipulation of agents than SAC
that every object is generated arbitrarily in any cor- which is the same difference between PPO(BT ppo )
ner of the environment and more walls are added to and SAC(BTsac ). PPO(BT ppo ) appears more stable
hamper the movement of the agent. What’s more, the and behaves as well as the independent RL on the ta-
victims, extinguishers and fire may be randomly ab- ble 1 but it takes more steps to converge. Adopting
sent at the outset of every episode. This highlights the PPO nodes can yield the better objective of scores and
dynamically changing of the environment and reaches time.
the number of 28 kinds of different situations totally.
MDRL-BT vs RL. According to the quantitative
In addition, the active criminal is moving all the time
analysis of training steps in experiment 2, the frame-
with a slow velocity and will stay away from the agent
within a certain distance in the face of agent.
Table 1: Evaluation statistics. Exp is about experiment type.
S and T respectively show scores and time. Mean means an
5.2 Results and Analysis average value. The full score reaches 100 and the unit of T
is seconds.
In this section, the performances for the trials are sub- Exp Model Mean S Mean T Train
jected to contrastive analysis. The examinations are BT 92.4365 13.2821 -
carried out to corroborate the benefit of mixed com- PPO 96.5761 9.0265 107
ponents. SAC 95.4983 13.1198 107
PPO vs SAC. Figure 8(a) indicates that indepen- BT ppo 96.5513 9.5955 107
1
dent PPO and SAC can rapidly converge. Although BTsac 96.1467 11.2166 107
the reward of SAC with the preponderance of sample- Option ppo 97.4605 8.8977 5 ∗ 106
efficient learning can get close to one in a short time Optionsac 97.7533 8.5394 3 ∗ 106
and in contrast it takes a long time for PPO to ar- Option pre 97.7208 8.2931 1 ∗ 106
rive, it is apparent in table 1 and figure 9 that in the
PPO 71.2845 12.2972 3 ∗ 107
perspective of scores and time the PPO outperforms
2 SAC 84.8081 17.5410 3 ∗ 107
readily SAC due to the influence of the sensitive hy-
BT ppo 94.2582 12.3749 3 ∗ 107
perparameters tuning and frequent policy updates of
SAC in discrete action space. Therefore, it is hy- other - - 1 ∗ 108
3
pothesized that PPO is more applicable to the fre- mixed 90.2595 19.5666 1 ∗ 108

121
ICAART 2021 - 13th International Conference on Agents and Artificial Intelligence

(a) exp1 score. (b) exp1 time.

(c) exp2 score. (d) exp2 time.

Figure 9: The distribution of 32000 tests suggests the average value and the standard deviation. The corresponding conclusion
can be drawn clearly on the grounds of the results. In experiment 3, only the mixed model can get a positive score in a limited
time. So the result isn’t shown.

work for modeling constrained yet adaptive agents re- 2D. On the basis of 3D experiment, we explore
veals the character of BT and surpasses the behavior the 2D game environment in figure 4(c)(d) built by
of PPO and SAC quite a few. The sequential strategy corgi engine to verify the other features of MDRL-
time-consuming for the independent RL algorithms is BT model. The corgi agent with BT ppo is required
exhibited favorably by MDRL-BT for simplification to collect the coins scattered all over the corners in a
and efficiency. limited time. What’s more, curiosity is employed in
Option Nodes. Table 1 especially shows that the MDRL-BT and in figure 8(f) both models can be
all models with learning core option and PPO ac- trained well enough but curiosity (Burda et al., 2018)
tion nodes can outperform the other methods and can get a little better result.
present an optima. Respectively, Optionsac with less
training steps but thoroughly performs better than
Option ppo from figure 8(b) and table 1, because SAC 6 CONCLUSIONS
is particularly appropriate for low frequency updates
and its sample efficiency consequently exceeds PPO. This paper has researched a mixed model for invent-
Option pre with pre-trained action nodes can almost ing an intelligent agent. We do some surveys on con-
keep in line with Optionsac in scores with the less textual backgrounds and related studies to explore the
steps. It is indicated that Option pre can be a supe- promotion of agent designs. Enough efforts about
rior choice in complicated models for the reduction of the combination of deep RL and BT have been made
training steps. To sum up, learning action nodes with by digging deep into the theoretical basis and ex-
PPO deals with complex environmental dynamics and isting correlations. As a specialization of option-
option nodes with SAC quickly handle planning and framework, MDRL-BT architecture is refined on the
scheduling in the construction of MDRL-BT. strength of deep learning nodes and BT construction.
MDRL-BT vs Others. The behavior of wall We accomplish the execution synchronization of RL
touching or breaking rules with a reduction of scores and BT and define an appropriate rewards function to
and rewards in experiment 3 makes the scenario no- prescribe the desired decisions. Several virtual simu-
ticeably troublesome and the agent must be in posses- lations are implemented on Unity 2D and 3D environ-
sion of some intelligence to manage and conduct its ments to employ semantics and structure of MDRL-
behavior across the environment. In figure 8(d), the BT. The mixed model also varies slightly with the
independent PPO and SAC which beforehand fall into complexity for displaying the special attributes.
a local dilemma hardly proceed with training and a The insights gained from results may be of assis-
simple model of BT ppo is incapable of achieving good tance to intelligent agents. MDRL-BT, reflecting the
performance due to the dynamic rewards and the pun- integrated advantage in the theorem, empirically out-
ishment of far too much walls touching. Nevertheless, weights the BT and RL and can be successfully ap-
the MDRL-BT with the model in figure 7 successfully plied to 2D and 3D environments. When especially
addresses the issues as table 1 and figure8 (e) shows. faced with complicated affairs or sequential tasks, the

122
 Mixed Deep Reinforcement Learning-behavior Tree for Intelligent Agents Design

MDRL-BT keeps the mind of hierarchies by decom- ICLR 2018 : International Conference on Learning
posing a main issue into several simple questions to Representations 2018.
provide a rational alternative solution. As evident Isla, D. (2005). Gdc 2005 proceeding: Handling complexity
from the result, MDRL-BT doesn’t need elaborate re- in the halo 2 ai. Retrieved October, 21:2009.
ward design to guarantee the training convergence rel- Juliani, A., Berges, V., Vckay, E., Gao, Y., Henry, H., Mat-
ative to general RL algorithms. In the design of mixed tar, M., and Lange, D. (2018). Unity: A general plat-
models, its a better choice to use PPO action nodes form for intelligent agents. arXiv:1809.02627.
with a shared brain and SAC composite nodes, even Kartasev, M. (2019). Integrating reinforcement learning
pre-train nodes. So as to a real available application, into behavior trees by hierarchical composition.
general RL algorithms or normal BT can be used for Liessner, R., Schmitt, J., Dietermann, A., and Bker, B.
simple tasks and by the way, MDRL-BT can be a can- (2019). Hyperparameter optimization for deep re-
inforcement learning in vehicle energy management.
didate for complex problems. In Proceedings of the 11th International Conference
MDRL-BT has a certain extensibility because of on Agents and Artificial Intelligence - Volume 2:
recusive BT framework and RL foundations. Some- ICAART,, pages 134–144. INSTICC, SciTePress.
times further exploration for extending MDRL-BT by Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T.,
importing other mechanisms such as curiosity in the Tassa, Y., Silver, D., and Wierstra, D. (2015). Contin-
sparse reward distribution can be an exciting avenue. uous control with deep reinforcement learning. arXiv
However, there will be enormous work to finish from preprint arXiv:1509.02971.
the unconspicuous consequence. In the future work, Mateas, M. and Stern, A. (2002). A behavior language for
the correlative theory and applicable scene about the story-based believable agents. IEEE Intelligent Sys-
additional algorithms can be investigated for better tems, 17(4):39–47.
performance. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A.,
Antonoglou, I., Wierstra, D., and Riedmiller, M. A.
(2013). Playing atari with deep reinforcement learn-
ing. arXiv preprint arXiv:1312.5602.
REFERENCES Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve-
ness, J., Bellemare, M. G., Graves, A., Riedmiller,
M., Fidjeland, A. K., Ostrovski, G., Petersen, S.,
Bacon, P.-L., Harb, J., and Precup, D. (2017). The option- Beattie, C., Sadik, A., Antonoglou, I., King, H., Ku-
critic architecture. In Thirty-First AAAI Conference maran, D., Wierstra, D., Legg, S., and Hassabis, D.
on Artificial Intelligence. (2015). Human-level control through deep reinforce-
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Dar- ment learning. Nature, 518(7540):529–533.
rell, T., and Efros, A. A. (2018). Large-scale Noblega, A., Paes, A., and Clua, E. (2019). Towards adap-
study of curiosity-driven learning. arXiv preprint tive deep reinforcement game balancing. In Proceed-
arXiv:1808.04355. ings of the 11th International Conference on Agents
de Pontes Pereira, R. and Engel, P. M. (2015). A framework and Artificial Intelligence - Volume 2: ICAART,, pages
for constrained and adaptive behavior-based agents. 693–700. INSTICC, SciTePress.
arXiv preprint arXiv:1506.02312. Sakr, F. and Abdennadher, S. (2016). Harnessing super-
Dey, R. and Child, C. (2013). Ql-bt: Enhancing behaviour vised learning techniques for the task planning of am-
tree design and implementation with q-learning. In bulance rescue agents. In Proceedings of the 8th In-
2013 IEEE Conference on Computational Inteligence ternational Conference on Agents and Artificial In-
in Games (CIG), pages 1–8. telligence - Volume 1: ICAART,, pages 157–164. IN-
Dromey, R. G. (2003). From requirements to design: for- STICC, SciTePress.
malizing the key steps. In International Conference Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and
on Software Engineering and Formal Methods. Abbeel, P. (2015). Trust region policy optimization.
Florez-Puga, G., Gomez-Martin, M., Gomez-Martin, P., arXiv preprint arXiv:1502.05477.
Diaz-Agudo, B., and Gonzalez-Calero, P. (2009). Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Query-enabled behavior trees. IEEE Transactions Klimov, O. (2017). Proximal policy optimization al-
on Computational Intelligence and AI in Games, gorithms. arXiv preprint arXiv:1707.06347.
1(4):298–308. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Fu, Y., Qin, L., and Yin, Q. (2016). A reinforcement learn- Den Driessche, G. V., Schrittwieser, J., Antonoglou,
ing behavior tree framework for game ai. In 2016 In- I., Panneershelvam, V., Lanctot, M., et al. (2016).
ternational Conference on Economics, Social Science, Mastering the game of go with deep neural networks
Arts, Education and Management Engineering, pages and tree search. Nature, 529(7587):484–489.
573–579. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D.,
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). and Riedmiller, M. (2014). Deterministic policy gra-
Soft actor-critic: Off-policy maximum entropy deep dient algorithms. In Proceedings of the 31st In-
reinforcement learning with a stochastic actor. In ternational Conference on International Conference

123
ICAART 2021 - 13th International Conference on Agents and Artificial Intelligence

on Machine Learning - Volume 32, ICML’14, page

I387I395. [Link].
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I.,
Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M.,
Bolton, A., et al. (2017). Mastering the game of go
without human knowledge. Nature, 550(7676):354–
359.
Subagyo, W. P., Nugroho, S. M. S., and Sumpeno, S.
(2016). Simulation multi behavior npcs in fire evacu-
ation using emotional behavior tree. In 2016 Interna-
tional Seminar on Application for Technology of Infor-
mation and Communication (ISemantic), pages 184–
190.
Sutton, R. S., Precup, D., and Singh, S. P. (1998). Intra-
option learning about temporally abstract actions. In
Proceedings of the Fifteenth International Conference
on Machine Learning, ICML ’98, page 556564, San
Francisco, CA, USA. Morgan Kaufmann Publishers
Inc.
Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhn-
evets, A. S., Yeo, M., Makhzani, A., Kttler, H., Aga-
piou, J., Schrittwieser, J., Quan, J., Gaffney, S., Pe-
tersen, S., Simonyan, K., Schaul, T., van Hasselt,
H., Silver, D., Lillicrap, T., Calderone, K., Keet, P.,
Brunasso, A., Lawrence, D., Ekermo, A., Repp, J.,
and Tsing, R. (2017). Starcraft ii: A new challenge
for reinforcement learning.
Wooldridge, M. and Jennings, N. R. (1995). Intelligent
agents: theory and practice. The Knowledge Engi-
neering Review, 10(2):115152.
Zhang, Q., Sun, L., Jiao, P., and Yin, Q. (2017). Combin-
ing behavior trees with maxq learning to facilitate cgfs
behavior modeling. In 2017 4th International Confer-
ence on Systems and Informatics (ICSAI), pages 525–
531.

124