1

Intelligent Routing Algorithm over SDN: Reusable

Reinforcement Learning Approach
Wang Wumian, Sajal Saha, Anwar Haque, and Greg Sidebottom

Abstract—Traffic routing is vital for the proper functioning of To provide better network performance for increasing de-
the Internet. As users and network traffic increase, researchers mands over a variety of applications, new network architec-
try to develop adaptive and intelligent routing algorithms that tures and traffic routing methods have been proposed. Software
can fulfill various QoS requirements. Reinforcement Learning
(RL) based routing algorithms have shown better performance Defined Networking (SDN) is an architecture that decouples
arXiv:2409.15226v1 [[Link]] 23 Sep 2024

than traditional approaches. We developed a QoS-aware, the control plane and data plane of a router [7], [8], [9].
reusable RL routing algorithm, RLSR-Routing over SDN. Traditional network routers have their own control and data
During the learning process, our algorithm ensures loop-free plane [8], and in a protocol like OSPF, each router maintains
path exploration. While finding the path for one traffic demand its own forwarding table [3]. For SDN, the control plane is
(a source destination pair with certain amount of traffic),
RLSR-Routing learns the overall network QoS status, which in a logically central controller which has a global view of
can be used to speed up algorithm convergence when finding the the network; and the SDN controller enables better interaction
path for other traffic demands. By adapting Segment Routing, with applications, Quality of Service (QoS) provisioning,
our algorithm can achieve flow-based, source packet routing, network monitoring and so on [7], [8]. SDN can also be
and reduce communications required between SDN controller applied to the wireless network in addition to traditional
and network plane. Our algorithm shows better performance
in terms of load balancing than the traditional approaches. It Wide Area Network [9]. Segment Routing (SR) [10] is a
also has faster convergence than the non-reusable RL approach source packet routing architecture in that each packet encodes
when finding paths for multiple traffic demands. a list of Segments in its header to guide the packet travel
through the specified path [10], [11]. As stated in [12], SR’s
Keywords: Reinforcement Learning (RL), Traffic Routing, ability to control packet forwarding can provide better network
Software Defined Networking (SDN), Segment Routing (SR),
Quality of Service (QoS). programmability, fast reroute, load balancing etc. In addition
to the new routing architecture, researchers also try to apply
Reinforcement Learning (RL) in traffic routing [13]. One
I. I NTRODUCTION recent implementation of RL in SDN can also be found in
Over the past decade, the number of people who have [14].
access to the Internet has been steadily increasing. CISCO’s
report predicts that by 2023, there will be 5.3 billion Internet
A. Motivation and Objectives
users, which is about two third of the World’s population [1].
Routing, which is the process of determining and forwarding Reinforcement Learning has been studied to develop adap-
packets from their source to destination, is vital for the trans- tive, intelligent routing algorithms over the past three decades
mission of data and information between network of networks, [13]. It has shown extraordinary flexibility to combine with
a.k.a., the Internet [2]. Traditionally, traffic routing relies on other techniques for traffic routing over a variety of network
protocols like OSPF, which aims to find the shortest path types, with different QoS requirements [13], [15]. Compared
between source and destination [3], or Routing Information with traditional routing methods which are based on assump-
Protocol (RIP), which is a distance-vector based algorithm tions about network conditions, applying RL allows routing
that mainly considers hop count [4]. OSPF is one of the algorithms to automatically learn the dynamic of the network,
most widely used Interior Gateway Protocol, and the shortest such as traffic flows, link quality and so on [15]. Therefore,
paths between source and destination pairs are calculated based on RL’s self-learning ability, some of the previous
based on assigned weights to every link of a network [5]. To research’s objectives are to improve service quality to end
obtain good network performance instead of only considering users, while optimizing network resources [15]. However,
number of hops along a routing path, the links weights should many previously proposed algorithms do not fully exploit the
be optimized. However, a previous study has proved that global view of SDN controller, or the ability to control the
optimizing OSPF weights is a NP-complete problem [6]. whole forwarding path (i.e., flow-based routing) by combining
with technique like SR [15]. In addition, many previous works
W. Wumian, and A. Haque are with the Department of Computer Science, only focus on one traffic demand at a time, and some require
Western University, London, ON N6A 5B7 Canada e-mail: {wwang457,
ahaque32}@[Link]. pre-defined algorithms (like OSPF) or initial paths as input
S. Saha is with the Department of Computer Science, University of [15].
Northern British Columbia, Prince George, BC V2N 4Z9 Canada e-mail: Our objective is to develop a RL-based routing algorithm
{[Link]}@[Link]
G. Sidebottom is with the Juniper Networks, Kanata, ON K2K 3E7 Canada for SDN, such that it can address previous work’s limitations.
email: {gsidebot}@@[Link] Our algorithm should be light-weighted, efficient, and allow
 2

users to customize their QoS requirements for definition of II. P ROBLEM S TATEMENT AND R ELATED W ORK
users’ preferred routing paths. While the algorithm aiming In this section, we formally define our research problem
to minimize costs during the path finding process, the final and present a literature review of recent work related to our
routing path from the algorithm should be as good as possible problem and existing solutions. This chapter also highlights
in terms of satisfying user defined QoS requirements. the current research gap and sets transitions to introducing
our proposed methodology.
B. Contribution
We developed a RL-based, SR-based, QoS-aware routing A. Problem Statement
algorithm for SDN, RLSR-Routing. The algorithm considers We focused on traffic routing in Software Defined Net-
various QoS factors, which can be customized by users, to working (SDN); therefore, we define our network as a set
find a user preferred path for traffic demands. The main of SDN switches with an SDN controller that has a global
contributions of this research are listed below: view of the network. This allows the controller to know the
• We modified State-Action-Reward-State-Action topology of the network, including every node and link. The
(SARSA), an on-policy RL method [13], so that controller has dedicated links to communicate with network
our routing algorithm aggregates action selections first. nodes, but neither the controller nor these links are used for
Such modification can further reduce the number of routing packets between the network’s end users. Our routing
message exchanges (e.g., sending packets, receiving algorithm interacts with the SDN controller to perform path
QoS info) required between SDN controller and network calculation and installs defined routing paths into the network.
switches. From the routing algorithm’s point of view, a network can
• In addition, during the learning process, our algorithm be abstracted as a directed graph, 𝐺 (𝑉, 𝐸), where 𝑉 is the
ensures that no packet will be stuck in a loop, which set of nodes that each corresponds to a physical device (SDN
means that packets either reach the destination or stop switch), and 𝐸 is the set of links that connects nodes. 𝑙𝑖, 𝑗
being forwarded if all next-hop nodes have been visited denotes the unidirectional link from node 𝑖 to 𝑗 where 𝑖, 𝑗 ∈ 𝑉.
by themselves. In other words, a packet will be sent to We define a traffic demand as a flow of data transmitted from a
any node in the network at most once. source node to a destination node. One source and destination
• We divided an action’s reward into local and global pair can have multiple traffic demands, each with a specified
rewards. Local rewards are used for finding a path for amount of data as demand traffic (expressed in bits/second).
one traffic demand, whereas global rewards are purely However, each traffic demand can only be assigned one routing
based on network status like link utilizations. Global path, which is defined as a list of nodes that connects from
rewards can be used to initialize local Q-table to speed source to destination.
up algorithm convergence. Each link and node of a network has a maximum capacity.
• RLSR-Routing does not require prior knowledge of the For a link, we use the term “maximum bandwidth” (in
network. In addition, it does not require pre-defined paths bits/second) to represent the maximum traffic that can travel
or initial link weights as a starting point. The algorithm through the link, and “used bandwidth” (in bits/second) to
can self-learn network status and assign a list of traffic represent the current amount of data on the link. A node’s
demands over the network in sequence. processing rate (in bits/second) is the maximum amount of
We compared our RLSR-Routing with a routing algorithm incoming data it can handle. In addition, all links have a field
currently used by one of the major telecom solutions providers. called “link reliability”, which varies from 0% to 100%. The
A mesh topology network is setup with randomly generated higher the link reliability, the lower the chances of link failure
traffic to be placed over the network. We compared the quality will occur.
of selected paths for generated traffic, in terms of maxi- Based on the above settings, our research objective is
mum link utilization. Our experiment result shows that the to give a network with SDN architecture with all nodes
proposed RLSR-Routing approach contributes to minimizing supporting Segment Routing (SR) and develop an RL-based
link utilization in the network. In addition, when tested on routing algorithm that can find user-preferred paths for traffic
a sequence of traffic demands, the proposed RLSR-Routing demands. Our definition of ‘user-preferred’ paths are those
approach speeds up the algorithm convergence by applying paths found by our routing algorithm based on weights of
knowledge of the network from its previous learning processes. one or more Quality of Service (QoS) parameters, which are
The structure of this paper is outlined below. In section II, adjusted by our algorithm’s users. The following assumptions
we define our problem, network environment, and introduce are made:
previous researchers that use RL approach to solve traffic • Communications between SDN controllers and network
routing problems. Section III presents our methodology, in- switches are through dedicated links, in a reliable transfer
cluding overall architecture, pseudocode of RLSR-Routing, manner.
and rationale behind our algorithm designs. In section IV, • SDN controllers are not involved in routing packets
we summarize our experiment results, including a comparative that originated from the network’s end users. For better
study with a non-RL routing algorithm. In section V, we con- visibility, sometimes we do not draw the controllers and
clude our study and propose potential updates and objectives those dedicated links on network topology diagrams in
for future work. this research.
 3

• All SDN switches of the networks considered in this to calculate vehicle movements. QGeo proposed in [22] also
research support SR, i.e., they can handle packet’s SR considers nodes mobility: each node exchanges their location
header and execute segments properly. information periodically through HEELO packets [22].
• All nodes are reliable, and if total incoming traffic does Authors in [23] proposed a routing algorithm based on fuzzy
not exceed a node’s processing rate; no packets will be logic, game theory and RL. Fuzzy logic groups vehicles (as
dropped by nodes. nodes) into clusters and selects cluster heads, and game theory
• For each pair of adjacent nodes 𝑖 − 𝑗 that connects with coordinates commutations between nodes and their cluster
each other, at most two links exist: link 𝑖 → 𝑗 and link heads. Nodes are enforced to use cluster heads for multi-hop
𝑗 → 𝑖, that directly connect 𝑖 and 𝑗. transmission and RL is used for path evaluation. Similarly, Q2-
• If a link’s used bandwidth does not exceed its maximum R proposed in [24] has additional bootstrapping steps before
bandwidth, and link failure does not happen, no packet RL routing. Q-learning starts with initial paths that discov-
will be lost when traveling on that link. ered from bootstrapping steps, and agent gradually improves
paths’ Q-value during learning process. A hierarchical routing
scheme is proposed in [25], which first divides geological area
B. Related Work into grids without the selection of cluster heads. Q-learning is
1) Reinforcement Learning on Non-Software Defined Net- used for selecting next grid and can be used to select the next
works: Similar to [16], several recent research focused on RL hop (vehicle) inside the optimal grid.
approach for routing in WSNs [17], [18], [19]. One character Author in [28], [29] focused on Wireless Mesh Networks
of WSNs is network devices are often powered by battery, (WMNs), in which nodes with multiple interfaces serve as a
therefore, energy consumption or energy conservation is an gateway that connects other nodes to the Internet [15], [29]. In
important factor to be optimized when designing RL-based [28]’s framework, each node predefined a set of routing algo-
routing algorithm. Both [17], [18]’s work involves exchanging rithms, and Q-learning is used for adaptively selecting routing
information related to energy conservation, which is accom- algorithms to execute. In [29], gateway nodes periodically
plished by adding extra fields in data packets. In [17], the broadcast their traffic load, each node first selects the gateway
sender node’s current Q-value is sent to a packet’s next hop router, then uses RL to select next-hop node to send packets
node, along with information about previous nodes energy to the selected gateway. Authors in [30] focused on multi-
level. In [19], the authors combine Q-learning and transmission hop wireless networks, particularly for video streaming. The
gradient for optimizing energy consumption over the WSNs. proposed RLOR algorithm combines RL and Opportunistic
For transmission gradient, each node maintains an estimated Routing, which exploit the broadcast nature of the wireless
number of transmissions for sending a packet from the node to network and determines packets’ next-hop on the fly.
the base station [19]. Estimating the number of transmissions 2) Reinforcement Learning on Software Defined Networks:
and energy level are used for calculating rewards; and nodes As illustrated in [14], when acting as an RL agent, SDN
use status packets to send their Q-value and estimated number controller(s) can find optimal path from source to destination,
of transmissions to other nodes [19]. in addition to next-hop node. To address the scalability issue
Another type of network which many studies focused on is of a single controller, the authors of [31] proposed QAR, a
Ad-hoc Networks (ANETs). In general, ANETs do not have a QoS-aware routing algorithm on hierarchical SDN. Nodes are
centralized controller, nodes are wirelessly connected, and net- divided into subnets, and each subnet has a domain controller,
work topology is dynamic. Mobile Ad-hoc Networks (ANETs) assisted by one or more “slave” controllers. When a routing
is a subtype of ANETs that nodes have mobility; common request’s source and destination are at different subnets, the
types of MANETs include VANETs for vehicles moving in main controller with a global view calculates a subnet path
certain area like a city [15]. In this review, all 6 recent that connects the source and destination’s subnets. Then each
papers which focus on ANETs used packet loss/retransmission subnet along the path runs RL on their domain controller to
rate/delivery ratio as one of metric for performance evaluation travel packets within subnets, and such path finding can be
[20], [21], [22], [23], [24], [25]. Unlike early work that is done in parallel. Another notable feature of QAR is that it
only based on RL, many of these ANETs related papers uses SARSA, an on-policy RL algorithm that compared with
combine RL with other techniques or have additional steps Q-learning, updates of a state-action pair’s Q-value depends
in addition to RL algorithm. Authors in [20] proposed QL- on action performed on the next state. [32], [33] is also based
MAODV, a combination of Q-learning and AODV protocol. on SARSA, but on a non-hierarchical SDN. VS-routing in [32]
AODV is defined in RFC3561, it is a routing protocol for applies a variable 𝜀-greedy for action selection, such that the
MANETs, and one notable feature is that AODV ensures loop probability of randomly selecting an action instead of a greedy
free routing [26]. QLMAODV considers stability when Q- approach varies with hop count and a dynamic factor [32].
learning evaluates a path to achieve a stable route [20]. In In addition, VS-routing has no immediate rewards, Q-value
[21], the authors proposed PFQ-AODV, an extension of their updates are based on the next state’s associated Q-values and
previous work in [27]. PFQ-AODV also combines AODV 𝜀. A multiple controllers SDN is considered in [33], and its
with Q-learning. In addition, it considers bandwidth, node’s objective is load-balancing for communication between nodes
movement, and link quality in route selection by fuzzy logic and controllers. also improved 𝜀-greedy action selection, by
[21]. Compared with their previous work, PFQ-AODV can using the Bayesian method: during exploration, agent tends to
use neighbor information, in addition to position information, select frequently selected actions.
 4

For Q-learning based routing algorithms in SDN, we also routing paths. ENERO proposed in [41] also combines Seg-
observed diversity in terms of network architecture and al- ment Routing and its objective is also to minimize maximum
gorithms’ workflows. Authors of [34] focused on congestion link utilization. Taking initial OSPF weights and a set of
control in SDN. Learning is not based on selected actions traffic demands as input, ENERO uses DRL to assign each
and immediate rewards but update all edges reward in each traffic demand to an intermediate SR node to optimize links
episode of the routing algorithm. Congestion identification is utilizations. To further improve the result, local search (LS) is
based on the current bandwidth of potential bandwidth, same applied to DRL’s output in ENERO [41].
for updating each edge’s corresponding reward. In [35], Q- In [42], the authors applied Deep Q-learning for routing
learning based routing is used for load-balancing in Wireless in data center SDN. The authors classified different traffic
SDN. In general, a Wireless SDN still has a central controller flows into two types: the mice-flows and the elephant-flows
and set of SDN switches, in addition, SDN base stations are based on the amount of data and duration of traffic. [42]
used to connect with SDN mobile devices. States are assigned thus builds two Deep Q-networks to assign paths for two
to users and depend on whether users’ demands are satisfied, traffic flows, respectively, and each flow type has a different
actions are connecting every user’s traffic flow to a base QoS objective. [43] proposed a hierarchical deep double Q-
station, and rewards are calculated based on available resource routing algorithm, which groups nodes into different clusters
and a fairness function. QR-SDN in [36] also adapts flow- at different hierarchal levels, each cluster with a group leader.
based routing. A flow is defined as data transmission between During the recursive route-finding process, the source’s routing
a given source and destination pair, and each flow is assigned request is sent to the highest level’s cluster leader, such that the
one path for routing traffic. QR-SDN will not compute the source and destination nodes are at two different sub-clusters.
path for a given flow, instead, it selects a path from a set The cluster leader uses DRL to select one link that connects
of pre-calculated possible paths. During the learning process, two sub-clusters, and the process is repeated until a whole path
QR-SDN can take a list of flows, change path assignment is built. The double Q-learning technique is adapted from [44];
for one or more flows in each episode, and gradually learns when combined with deep learning, two Deep Q-networks are
optimal path assignments. interchangeably used for action selection and evolution.
Author in [37], [38] use Q-learning, but on a “distributed”
SDN. In [37], the authors implemented a “distributed” SDN
over several computers (PC), with each PC containing a part of C. Analysis and Current Research Gap
SDN network with one controller. Links are initially assigned Most of our reviewed papers proposed an RL-based rout-
weights, and the objective of Q-learning is to optimize link ing algorithm, although the DRL approach has attracted re-
weights assignments. SDCIV proposed in [38] aims to achieve searchers’ interest dramatically in the past 5 years. Among
adaptive routing in SDN based Internet of Vehicles (IoV). The studies that focus on non-SDN networks, 85.7% are on kind
logical central controller is made up with a SDN controller of wireless network and/or Ad-hoc network. In addition to
cluster, and thus we classify it as a “distributed” operation latency-related factors (like delay or hop-count), RL-based
mode. SDCIV defines states as average vehicle speeds and approaches can consider a variety of factors that affect net-
densities, and the learning agent gradually learns the best work performance. However, when working on a non-SDN
routing protocol to execute for different states. Similar to [28], network like WSNs or ANETs, applying RL-based routing
RL is not used for routing or path finding directly, but to find may generate additional commutations overhead. For example,
an appropriate protocol to execute from pre-defined routing [17], [18], [19] routing algorithms require exchanging energy
algorithms. consumption or device residual energy information among
3) Deep Reinforcement Learning based Routing: Using RL network nodes. With SDN controllers’ global view of their
in traffic routing started in 1990s, in contrast, DRL based networks, or a hierarchical network with some nodes being
approach is a relatively emerging field. Fig. 3 shows a number cluster heads (like [23], [43]) could reduce commutations
of published papers related to DRL and routing between 2017 overhead placed by RL-based routing over a network.
and 2021, from Web of Science database. Since this literature 61.5% of reviewed papers proposed a composite routing
review focuses on traffic routing, we do not cover how neural protocol, where RL or DRL is part of the whole routing proto-
network is trained in different DRL based routing algorithms. col. Common reasons of combining RL with other techniques
In [39], the authors proposed DROM, a routing optimization are: 1. To speed up the algorithm’s convergence (like QAR
algorithm for SDN. DROM takes a traffic matrix of current in [31]); 2. To provide additional control, such as loop free
network load as input, during the learning process, the DRL routing [20], [21] or source packet routing [41], [40]. It should
agent changes some links’ weight so that some traffics alters be noticed that for some studies on SDN, like QAR, their
their routing path [39]. WA-SRTE in [40] works on a partially routing algorithm does not guarantee loop-free routing during
deployed Segment Routing IPv6 network, and it has two the learning phase, even though SDN controller(s) are RL
phases. In offline network design phases, WA-SRTE uses agents [31]. The usage of RL is not limited to finding the best
DRL to optimize OSPF link weights and SR-enabling nodes’ next-hop node to forward a packet. RL agents can learn the
deployment based on historical traffic data. Once the offline whole path from a source to a destination or assign a path to
phase is complete, link weights and SR nodes deployment traffic flows from a set of possible paths. In addition, several
is fixed. In the online routing optimization phase, WA-SRTE studies focus on OSPF optimization by using RL approach,
uses other techniques like linear programming to optimize which we define as the indirect usage of RL in traffic routing.
 5

Most papers’ framework only focuses on forwarding one-

packet, or one source-destination pair’s traffic. None of our
reviewed literature, which is based on RL over SDN, use RL
to directly find the path for a set of traffic flows in parallel.
In summary, RL has been studied to develop adaptive,
intelligent routing algorithms over the past three decades.
It has shown extradentary flexibility to combine with other
techniques for traffic routing over a variety of network types,
with different QoS requirements. However, currently proposed
algorithms do not fully exploit the global view of the SDN
controller, or the ability to control the whole forwarding path
(i.e., flow-based routing) by combining with a technique like
SR. Additional work needs to be done in order to develop
a RL-based, QoS aware flow-based routing over SDN, such
that RL agent can find path for multiple traffic demands in
parallel. Alternatively, while RL agent still focuses on one
traffic demand at a time, the agent can reuse the network status
knowledge it has learned previously to speed up algorithm Fig. 1: High-level Overview of RLSR-Routing Architecture
convergence.
III. P ROPOSED M ETHODOLOGY
As the number of Internet users and types of services that
rely on networking increases, traditional distance vector or
shortest path based algorithms may not be able to provide
optimal network performance. Previous studies demonstrate
the potential of using RL or DRL to develop an intelligent
routing algorithm that is adaptive to dynamic network topology
and fulfills a variety of QoS requirements. To the best of
our knowledge, existing approaches do not fully exploit SDN
architecture combined with a flow-based source packet routing
paradigm. In this research, we propose RLSR-Routing, an RL-
based routing algorithm over SR-enabled SDN architecture to
address the current research gap. RLSR-Routing reduces net-
work operation costs during the path-finding process and can
have faster convergence than the previous RL-based approach. Fig. 2: High-level Overview of RLSR-Routing Components
This section described the RLSR-Routing framework and the
rationale behind RLSR-Routing’s design.
information like QoS data, and topology information
A. RLSR-Routing Architecture from the SDN controller.
Fig.1 presents an abstract view of our framework. RL
routing algorithm is one component of the SDN controller,
and it relies on the SDN control plane to provide network
topology and link state information. Although different lit-
erature may categorize routing as an application plane or
control plane’s function [2], [7], such difference does not affect Fig. 3: Trivial Topology 1 (T1)
RLSR-Routing’s framework. Because in either case RLSR-
Routing does not directly interact with network devices. Fig. 2
describes RLSR-Routing’s components in greater detail. These
components can be classified into three categories: B. State, Action, Reward, and Modifications
1) Storage of information: such as Global Q-table which The RL algorithm used in RLSR-Routing can be viewed
represents previously learned network status, Default as an extension to previous work, particularly QAR from
parameters, and Network topology. [31]. RLSR-Routing applies modified SARSA, an on-policy
2) RL algorithm: this is the central part of RLSR-Routing. RL algorithm and a complete Markov Decision Process is
It finds paths for specified traffic demands, as well as included in RLSR-Routing. In this subsection, we define the
learns network status during execution. state space, the action space, and rewards’ QoS considerations
3) Communication: these components interact with the for RL algorithm used in RLSR-Routing. We also explain the
SDN controller to help other components send instruc- rationale for two major modifications we made in our RL
tions to the SDN controller, and to collect network algorithms.
 6

1) State and Action Space: RLSR-Routing focuses on values are considered. The calculation of current traffic inten-
routing for one traffic demand at a time: during each episode sity (𝑅inten ) and estimated traffic intensity (𝑅inten-est ) related
of algorithm execution, RLSR-Routing instructs the SDN reward are shown below:
controller to send one packet from the traffic demand’s source
to its destination. Based on the above settings, our definition Node 𝑗 current total incoming traffic
of state space and action space is as follows. 𝑅inten = 1 − (4)
Node 𝑗 processing rate
• State space: let 𝑆 denotes the set of all possible states
Node 𝑗 estimated total incoming traffic
of a network. |𝑆| = |𝑉 | where 𝑉 is set of nodes of the 𝑅inten-est = 1 − (5)
network. A state 𝑠 ∈ 𝑆 means node 𝑠 is holding the packet. Node 𝑗 processing rate
Notations 𝑠𝑡 and 𝑠𝑡+1 are also used, which represent the Calculation of current link utilization (𝑅util ) and estimated
state at time 𝑡 and time 𝑡 + 1, respectively. link utilization (𝑅util-est ) related reward are shown below:
• Action space: let 𝐴 denotes the set of all possible actions
Link 𝑙 current used bandwidth
that can be performed over a network. | 𝐴| = |𝐸 | where 𝑅util = 1 − (6)
Link 𝑙 maximum bandwidth
𝐸 is set of edges of the network. An action 𝑎 𝑖, 𝑗 ∈ 𝐴
represents sending the packet along link 𝑎 𝑖, 𝑗 , from its Link 𝑙 estimated used bandwidth
𝑅util-est = 1 − (7)
source node 𝑖 to its destination node 𝑗. Notations 𝑎 𝑡 and Link 𝑙 maximum bandwidth
𝑎 𝑡+1 are also used, which represent selected action at time Eq. 1 – 7 ensure that all QoS-related rewards values are
𝑡 and time 𝑡 + 1, respectively. no greater than 1. The closer a reward’s value is towards 1,
Fig. 3 shows a trivial topology 𝑇1 with 5 nodes, and the better quality of the performed action is in terms of the
5 unidirectional links (followed by the arrows). The SDN reward. Take T1 as an example; suppose a packet is sent from
controller and its connections with nodes are not shown in node 0 to 4 following the path 0 → 1 → 2 → 3 → 4. All
the figure. If a packet is sent from node 0 to node 5, at time links have 10 Mb/s maximum bandwidth, 95% of the time are
𝑡 0 , node 0 holds the packet, the chosen action is 𝑙 0,1 . At time working; all nodes have 50 Mb/s processing rate. Currently,
𝑡 1 , the packet reaches node 1, and therefore the state at 𝑡1 is there is 5 Mb/s traffic sending from node 3 to node 4, and the
node 1 holding the packet. traffic demand from node 0 to 4 has 0.5 Mb/s estimated traffic.
2) QoS Considerations for Actions’ Rewards: Previous If we evaluate the action of sending packet from node 3 to 4,
studies have demonstrated the flexibility of RL-based routing the corresponding rewards are:
that multiple QoS metrics can be used to calculate an action’s 1
• 𝑅hop = 4 = 0.25 (Since link 𝑙 3,4 is the fourth hop the
reward. In RLSR-Routing, we considered five QoS parameters packet has travelled).
which can be categorized into three groups: latency; reliability; 2
• 𝑅transmission = 𝜋 × arctan(50 Mb/s) = 0.9873 (Round to 4
and load balancing. Assuming the performed action is sending decimals).
one packet from node 𝑖 to node 𝑗 through link 𝑙, the calculation • 𝑅reliability = 95% = 0.95.
of each QoS parameter’s reward is presented below. For 5 Mb/s
• 𝑅inten = 1 − 50 Mb/s = 0.9 (Currently, there is 5 Mb/s traffic
latency-related QoS metrics, we considered ”number of hops” towards node 4).
and ”transmission rate”. 5+0.5 Mb/s
• 𝑅inten-est = 1 − 50 Mb/s = 0.89 (After placed demand’s
Number of hops related reward (𝑅hop ) is calculated as traffic, node 4 are expected to have 5.5 Mb/s total incom-
follow: ing traffic).
1 5 Mb/s
𝑅hop = (1) • 𝑅util = 1 − 10 Mb/s = 0.5.
number of hops 5+0.5 Mb/s
• 𝑅util-est = 1 − 10 Mb/s = 0.45.
Transmission rate related reward (𝑅transmission ) reflects 3) RL Modification (Aggregate Action Selection): Here, we
the transmission delay of a node, and it is calculated as describes and explains one of the major modifications we made
follows. The higher a node’s processing rate, the less time it for SARSA used in RLSR-Routing - aggregate action selection
requires to transmit all bits of a packet to the link. Therefore, of one episode. Based on our literature review, the two most
the related action will receive a higher reward. commonly used RL algorithms in traffic routing are Q-learning
2 and SARSA. Both Q-learning and SARSA follow a similar
𝑅transmission = × arctan(node 𝑖 processing rate) (2)
𝜋 workflow during each episode of algorithm execution.
For reliability-related QoS metric, we considered 1) link 1) At time 𝑡, from current state 𝑠𝑡 , select an action 𝑎 𝑡 based
reliability. Since, in our assumption, all network switches and on action selection policy.
SDN controllers are 100% reliable, only the possibility of 2) Perform action 𝑎 𝑡 .
link failure is considered. Calculation of link reliability-related 3) Observe and/or calculate action’s reward, and new state
reward (𝑅reliability ) is: 𝑠𝑡+1 at time 𝑡 + 1.
4) Update 𝑄(𝑠𝑡 , 𝑎 𝑡 ).
𝑅reliability = link 𝑙 ′ 𝑠 reliability value (3) 5) Time 𝑡 ← 𝑡 + 1.
For load balance-related QoS metrics, we considered 1) 6) Current state 𝑠𝑡 ← 𝑠𝑡+1 .
traffic intensity, 2) link utilization. For these two metrics, both The general workflow described above for Q-learning is
current and estimated (i.e., estimated traffic intensity and link similar to a ”stop and wait” format: Until the reward is
utilization after placing the traffic demand’s flow on the link) observed, and the state action pair’s Q-value has been updated,
 7

RL learning agent cannot choose another action to perform. explained earlier, an update of any state-action pair’s Q-value
In our modified RL with aggregation of action selection, the does not affect previously updated state-action pairs. Based on
workflow in each episode is as follows: the above observations, we think that our modification will not
1) Let current state be 𝑠0 , select an action 𝑎 0,1 which leads affect the accuracy of Q-values updates. On the other hand,
to a never reached state 𝑠1 , if the action is successfully aggregate action selection can reduce network costs during the
performed. learning process, particularly communication costs between
2) Add 𝑎 0,1 to a list {𝑎 0,1 , . . . }. network nodes and the SDN controller which is explained
3) Update the current state to be 𝑠1 and repeat step 1 and 2. below.
Stop repeating when the selected action leads to traffic
demand’s destination, or to avoid stuck in a loop.
4) Perform the actions {𝑎 0,1 , 𝑎 1,2 , 𝑎 2,3 , . . . } in order.
5) Observe and calculate rewards and store them in an
ordered list.
6) Updated corresponding state-action pairs, follow the Fig. 4: Comparing Number of Communications
order of performed actions.
Compared with the unmodified workflow, in RLSR- Using the same example, Fig. 4 illustrates the communi-
Routing’s approach, the RL agent first selects all actions that cation patterns between the controller (labelled as 𝐶) and
will be performed during one episode. Then, the agent instructs network nodes under two different scenarios: without action
the network to perform all the actions in order, and passively selection aggregation (on the left) and with action selection
waits for returning QoS data which are used to calculate each aggregation (on the right). Without aggregation, the controller
action’s reward. The reason why we can aggregate action must send a new packet to the network for every newly
selection before updating Q-values are based on the following selected action. In previous RL-based routing implementations
observations. like QAR in [31], the RL agent selects and performs only one
Observation I: Update 𝑄(𝑠𝑡 , 𝑎 𝑡 ) does not affect action action during each episode. For instance, even if the path from
selection at state 𝑠𝑡+1 , if selected action 𝑎 𝑡+1 leads to a never- node 0 to node 4 is straightforward, when the current state is
reached state during this episode. “node 0 holds the packet”, the RL agent can only instruct node
Observation II: Update 𝑄(𝑠𝑡+1 , 𝑎 𝑡+1 ) does not affect updat- 0 to send the packet to node 1. After this action is executed and
ing 𝑄(𝑠𝑡 , 𝑎 𝑡 ), if Q-values are updated in order of performed the packet reaches node 1 (updating the current state to “node
actions, and path is non-cyclic. 1 holds the packet”), the RL agent must select and perform
When an SARSA’s learning agent updates a state-action a new action, sending an instruction to node 1 regarding the
pair’s Q-value, it uses the following equation: next hop for the packet.
With aggregation and Source Routing (SR) technique, the
𝑄 𝑡+1 (𝑠𝑡 , 𝑎 𝑡 ) = (1 − 𝛼) × 𝑄 𝑡 (𝑠𝑡 , 𝑎 𝑡 ) + 𝛼 × (𝑅 + 𝛾 × 𝑄 𝑡 (𝑠𝑡+1 , 𝑎 𝑡+1 )) controller can encode all selected actions into the packet’s
(8) header, initiate routing, and then passively wait for Quality of
𝑅 stands for rewards, 𝛼 represents the learning rate, and Service (QoS) data to return from the network, assuming no
𝛾 indicates the importance of long-term rewards. Update of packet loss occurs. Generally, if 𝑛 actions are to be performed
(𝑠𝑡 , 𝑎 𝑡 )’s Q-value at time 𝑡 + 1 depends on its old Q-value at during a single RL episode, the absence of action selection
time 𝑡 (i.e., 𝑄 𝑡 (𝑠𝑡 , 𝑎 𝑡 )) and old Q-value for state-action pair aggregation would result in 2𝑛 communications between the
at the next state (i.e., 𝑄 𝑡 (𝑠𝑡+1 , 𝑎 𝑡+1 )). When the path a.k.a. the controller and nodes. With action selection aggregation, this
list of actions, is non-cyclic, each state will only be included number is reduced to 𝑛 + 1. Moreover, if the RL approach
at most once in the path. If the update of Q-values follows the does not incorporate any techniques to prevent infinite loop
same order as actions performed, previously updated state- formation, the number 𝑛 is more likely to equal the maximum
action pairs’ Q-values will not be affected by updating the Time-To-Live (TTL) value a packet is allowed within the
subsequent state-action pairs’ Q-values. network. Conversely, our modification avoids infinite loops by
Take topology 𝑇1 as an example. Suppose now the aggre- ensuring the RL agent is aware of the states already included
gated selected actions are {𝑎 0,1 , 𝑎 1,2 , 𝑎 2,3 , 𝑎 3,4 }, which is a in the path during the aggregated action selection process.
non-cyclic path from node 0 to node 4. Starting from time 𝑡, 4) RL Modification (Local and Global Reward):: In addi-
the update of Q-values are as follows: tion to the aggregate action selection process, we also aimed to
• 𝑄 𝑡+1 (𝑠0 , 𝑎 0,1 ) = (1 − 𝛼) × 𝑄 𝑡 (𝑠0 , 𝑎 0,1 ) + 𝛼 × (𝑅0,1 + 𝛾 × use what the RL agent has learned during each episode more
𝑄 𝑡 (𝑠1 , 𝑎 1,2 )) efficiently to speed up algorithm convergence. Taking QAR,
• 𝑄 𝑡+1 (𝑠1 , 𝑎 1,2 ) = (1 − 𝛼) × 𝑄 𝑡 (𝑠1 , 𝑎 1,2 ) + 𝛼 × (𝑅1,2 + 𝛾 × proposed in [31], as an example, its RL algorithm workflow
𝑄 𝑡 (𝑠2 , 𝑎 2,3 )) can be summarized as follows:
• 𝑄 𝑡+1 (𝑠2 , 𝑎 2,3 ) = (1 − 𝛼) × 𝑄 𝑡 (𝑠2 , 𝑎 2,3 ) + 𝛼 × (𝑅2,3 + 𝛾 × 1) Take source and destination node from a traffic demand.
𝑄 𝑡 (𝑠3 , 𝑎 3,4 )) 2) Initialize Q-table with all entries set to 0.
• 𝑄 𝑡+1 (𝑠3 , 𝑎 3,4 ) = (1 − 𝛼) × 𝑄 𝑡 (𝑠3 , 𝑎 3,4 ) + 𝛼 × (𝑅3,4 + 𝛾 × 1) 3) In each episode, repeat the Markov Decision Process.
Since action 𝑎 3,4 is the last one performed, here we just 4) After RL converges, output the routing path from the
use 1 to represent its next state-action pair’s Q-value. As updated Q-table.
 8

of RLSR-Routing, as illustrated in Fig. 2.

Topology 𝑇2 in Fig.5 is sufficient to demonstrate the ad-
vantage of using the dual rewards scheme. All links represent
the two unidirectional links between two end nodes. The RL
algorithm finds the best path between node 0 and 3 based on
the number of hops and link reliability, and it has reached
convergence. Assuming the final local and global Q-tables are
as in Table I and II respectively.

TABLE I: Local Q-table after Convergence (Hypothesized)

Node 0 Node 1 Node 2 Node 3 Node 4
Node 0 X -1.5 -1.8 X X
Node 1 0 X -1.1 -1.5 -1.2
Fig. 5: A Simple Topology (T2) Node 2 0 -1.0 X -0.8 -1.3
Node 3 X 0 0 X X
Node 4 X -1.2 -1.1 X X

The Q-table is updated after the algorithm above converges,

including but not limited to entries that together form a path TABLE II: Global Q-table after Convergence (Hypothesized)
with maximal rewards from the source to the destination. Since Node 0 Node 1 Node 2 Node 3 Node 4
Q-values represent estimated cumulative rewards, the updated Node 0 X -1.9 -1.7 X X
Q-table can be viewed as containing knowledge about some Node 1 0 X -1.5 -1.4 -1.1
Node 2 0 -0.2 X -0.9 -0.9
of the network’s QoS status, assuming the reward function is Node 3 X 0 0 X X
based on those QoS parameters. However, as the RL agent Node 4 X -0.4 -0.7 X X
handles a new traffic demand and reinitializes the Q-table’s
entries to 0, the previously learned knowledge about network Each row represents a state for Table I and II, and each
QoS status is lost, requiring the RL agent to explore the column represents an action. The letter “X” indicates that the
network from scratch again. state-action pair does not exist, i.e., there is no direct link
Inspired by previous work, which provides some starting from the source to the destination of the node pair. A value
point during the RL agent’s initialization [24], we propose of 0 indicates that the agent did not explore the state-action
a “dual rewards scheme” for our modified RL algorithm to pair during the learning process. Based on the local Q-table,
improve the efficacy of using RL results. Instead of one reward a non-cyclic path with maximal reward from node 0 to 3 will
for one action, our algorithm calculates two rewards for each be: 0 → 1 → 2 → 3. While the RL agent was exploring the
action: local and global rewards. The concept of local reward best path for the traffic demand, it also updated the global
is similar to an action’s reward in traditional Q-learning or Q-table based on link utilization, traffic intensity, and link
SARSA: the RL agent uses local rewards to find actions that reliability. Suppose now the RL agent needs to find a path
lead to maximum expected rewards (a.k.a. towards a preferred for a traffic demand from node 0 to node 4, focusing on link
path defined by the user’s customized QoS requirements). On utilization and traffic intensity instead of the number of hops. If
the other hand, global rewards preserve the knowledge the RL the agent initializes all entries of the local Q-table to 0, it has
agent learned about the network, which can be used in future zero knowledge about network load-balancing status before
learning processes. the learning process begins. In contrast, if it uses the global
Q-table to initialize the local Q-table, the agent will have some
As a result of using two kinds of reward, RLSR-Routing
prior knowledge about the network, e.g., the first action should
needs to construct two Q-tables: a local Q-table and a global
better choose to send the packet from node 0 to 2. We believe
Q-table. At the initialization phase, users of RLSR-Routing
that in some cases, a priori knowledge of network QoS status
decide whether to use a global Q-table to initialize the local
will speed up algorithm convergence.
Q-table, instead of setting every entry to a random value like
0. During the learning process, the agent only uses the local Q-
table to select actions, since the main objective is to find a path C. RLSR-Routing Workflow
for given traffic demands rather than learning overall network In subsection III-A and III-B, we demonstrated RLSR-
status. By adjusting weights of different QoS parameters, Routing’s architecture and explained the rationale behind some
users of RLSR-Routing can customize the calculation of local of our modifications to RL algorithm. In this subsection, we
rewards, and thus customize the evaluation of a path’s quality describe RLSR-Routing’s workflow in pseudo-code, especially
for different traffic demands. Meanwhile, the calculation of on how our RL routing takes traffic demand from input and
global rewards and updating the global Q-table are hidden produces a user preferred path for a given demand.
from incoming traffic demands. To support potential future RL findRoute() in Algorithm 1 describes the overall work-
upgrades of RLSR-Routing that enables multi-threading, i.e., flow of the RL algorithm component: the user of RLSR-
running multiple RL algorithm components to find a path for Routing initiates a routing request by providing traffic demand
multiple traffic demands concurrently, we separated the global and optional customized factors. The RL algorithm explores
Q-table from the RL algorithm and made it another component the network and gradually learns the user-preferred path during
 9

Algorithm 1 Find Route using Reinforcement Learning

1: function RL FIND ROUTE(TrafficDemand Demand
Boolean B, QoSWeights W, Hyperparameters H)
2: // Initialization Phase
3: get Graph 𝐺 from RLSR-Routing;
4: get QTable 𝐺𝑇 from RLSR-Routing;
5: get QoSWeights 𝑊 𝑑 from RLSR-Routing;
6: get Hyperparameters 𝐻𝑑 from RLSR-Routing;
7: QTable 𝑙𝑜𝑐𝑎𝑙𝑇 = I NIT LTABLE (𝐺, 𝐺𝑇, 𝐵);
8: QoSWeights 𝑤𝑙 = (𝑊 = NULL)?𝑊 𝑑 : 𝑊;
9: Hyperparameters ℎ𝑙 = (𝐻 = NULL)?𝐻𝑑 : 𝐻;
10: 𝑒 𝑝𝑖𝑠𝑜𝑑𝑒𝑠 = 0
11: while 𝑒 𝑝𝑖𝑠𝑜𝑑𝑒𝑠 < ℎ𝑙.𝐸 do
12: Path 𝑡𝑒𝑚 𝑝𝑃𝑎𝑡ℎ = F IND T EMP PATH (𝐷𝑒𝑚𝑎𝑛𝑑, 𝑙𝑜𝑐𝑎𝑙𝑇, ℎ𝑙)
13: P USH PATH FOR ROUTING(tempPath)
14: QoSData[] 𝑞𝐷𝑎𝑡𝑎, int 𝑐𝑜𝑢𝑛𝑡 = C OLLECT Q O S DATA (𝑡𝑒𝑚 𝑝𝑃𝑎𝑡ℎ)
15: Reward[] 𝑙 𝑅𝑒𝑤𝑎𝑟 𝑑𝑠 = C ALCULATE LR EWARDS (𝑞𝐷𝑎𝑡𝑎, 𝑤𝑙, 𝑐𝑜𝑢𝑛𝑡, 𝐷𝑒𝑚𝑎𝑛𝑑)
16: Reward[] 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠 = C ALCULATE GR EWARDS (𝑞𝐷𝑎𝑡𝑎, 𝑊 𝑑, 𝑐𝑜𝑢𝑛𝑡)
17: U PDATE TABLE(localT, lRewards, hl, count)
18: U PDATE TABLE(GT, gRewards, Hd, count)
19: 𝑒 𝑝𝑖𝑠𝑜𝑑𝑒𝑠+ = 1
20: end while
21: return F IND F INAL PATH(Demand, localT, hl)
22: end function

the learning phase. Once the algorithm converges or finishes On the other hand, users can initialize the local Q-table
the learning process, the final path is retrieved from the local from scratch: for every existing link, the corresponding value
Q-table. In practice, RLSR-Routing will be implemented and is initialized to 0. For links that do not exist, the corresponding
deployed on the SDN controller. By using the controller’s state-action pair’s value is initialized to a minimal integer
API, RLSR-Routing receives requests for finding paths for value. During initialization steps, the RL agent also determines
given traffic demands. Similarly, RLSR-Routing explores the whether to use default QoS weights and/or hyperparameters,
network by using the controller’s API to send packets to SR- based on user input. The structure of QoS weights and
enabled network nodes. As network nodes route the packets hyperparameters is presented in Table III and IV respectively.
based on their header-defined segments, RLSR-Routing waits
for link-state information from network nodes. The final TABLE III: Structure of QoS weights
routing path for a traffic demand will be sent to the source Variable Description
node in the network, in order to achieve flow-based routing. double lConstant; Constant used when calculating global re-
ward
double gConstant; Constant used when calculating local reward
double Wc; Hop-count reward’s weight
D. RL Algorithm: Initialization double Wt; Transmission rate reward’s weight
Before RL agent starts learning process, it needs to extract double Wr; Link reliability reward’s weight
double Wi; Traffic intensity reward’s weight
information from user input as well as other RLSR-Routing double Wu; Link utilization reward’s weight
components. This information is necessary to initialize local
Q-table, QoS weights, hyperparameters and most importantly,
the traffic demand to handle. TABLE IV: Structure of Hyperparameters
InitLTable() in Algorithm 2 is a helper function for ini- Variable Description
tializing the local Q-table. Both global and local Q-tables double 𝜀; For 𝜀-greedy action selection
are implemented as a 2-dimensional array of doubles, where double 𝛼; Learning rate
double 𝛾; Importance of long-term rewards
the array entry [𝑖] [ 𝑗] represents the Q-value for the state- int TTL; Maximum number of hops
action pair “at node 𝑖, sends packet to node 𝑗”. If the user int E; Number of training episodes
opts to apply the global Q-table, the helper function creates a
deep copy of the global Q-table as the starting point for the Users can provide customized weights to emphasize the
local Q-table. It is recognized that, in some cases, users of importance of different factors, and variables 𝑔𝐶𝑜𝑛𝑠𝑡𝑎𝑛𝑡 and
RLSR-Routing may wish to apply the global Q-table but not 𝑙𝐶𝑜𝑛𝑠𝑡𝑎𝑛𝑡 depend on these weights, as illustrated below.
copy every entry from the global to the local Q-table exactly. Hyperparameters contain fields for RL’s action selection and
Future enhancements may include more options for applying Q-value updating (𝜀, 𝛼, 𝛾) and for adjusting training duration
the global Q-table for initialization. (TTL, 𝐸).
 10

Algorithm 2 Initialize Local Q-Table

1: function I NIT LTABLE(Graph 𝐺, QTable 𝐺𝑇, Boolean 𝐵)
2: if 𝐵 is true then
3: QTable localT = D EEP C OPY (𝐺𝑇);
4: return localT;
5: end if
6: QTable localT = new QTable(𝐺. NUMBERO F N ODES)
7: Link[] []links = 𝐺. GET L INK M ATRIX
8: for 𝑖 = 0 to [Link] − 1 do
9: for 𝑗 = 0 to links[0].length − 1 do
10: if link[𝑖] [ 𝑗] == NULL then
11: localT[𝑖] [ 𝑗] = −0𝑥80000000;
12: else
13: localT[𝑖] [ 𝑗] = 0;
14: end if
15: end for
16: end for
17: return localT;
18: end function

if the traffic demand’s destination node is one of the current

𝑙𝐶𝑜𝑛𝑠𝑡𝑎𝑛𝑡 = 𝑊 𝑐 + 𝑊𝑡 + 𝑊𝑟 + 𝑊𝑖 + 𝑊𝑢 + 0.1 (9) node’s neighbors. Considering topology T3 in Fig. 6, all links
are unidirectional, and now there is a traffic demand from
node 0 to node 2, with QoS consideration on link utilization.
𝑔𝐶𝑜𝑛𝑠𝑡𝑎𝑛𝑡 = 𝑊𝑟 + 𝑊𝑖 + 𝑊𝑢 (10) Suppose link 𝑙 0,2 currently has a 99% utilization rate, whereas
link 𝑙 0,1 and 𝑙 1,2 has 0% utilization. Although node 2 is
E. RL Algorithm: Actions Selection adjacent to node 0, the preferred path is 0 → 1 → 2 instead
of 0 → 2.

F. RL Algorithm: Perform Actions & Observe QoS Data

Push_Path_for_Routing() and
Collect_QoS_Data() in Algorithm 1 represent the
process of generating a packet, encoding tempPath into
the packet’s header, sending the packet to the network via
the SDN controller for routing, and waiting for related
link-state QoS data collected by the SDN controller. In this
study, we only focused on simulating the overall workflow.
However, during actual deployment on SDN architecture,
Fig. 6: Edge Case Topology (T3) these functions will be implemented by other components of
RLSR-Routing. Modifications to the SR protocol may also be
FindTempPath() in Algorithm 3 describes how the RL necessary so that packets generated by RLSR-Routing in the
algorithm uses aggregate action selection to produce a non- learning process are distinguished from other packets in the
cyclic path. The function keeps track of every node that network. Therefore, network devices will only send related
has been added to the temporary path, and because RLSR- QoS data to the SDN controller when they process packets
Routing assigns each network node an integer ID that starts created by RLSR-Routing’s RL algorithm. Another approach
from 0, visitedNodes[i] is enough to express whether to modifying the protocol could involve adding a field called
node 𝑖 has been included. Starting from the traffic demand’s “SEND QoS DATA” on all packet headers. Network switches
source node,FindTempPath() builds a consecutive path by would then only send QoS data to the controller for certain
selecting from the current node’s unvisited neighbors. The packets (such as those created by the RL algorithm during
path building process ends due to one of the following: 1) a the learning process) that have the “SEND QoS DATA” field
dead-end, i.e., all current node’s neighbors have been included set.
in tempPath. 2) the path length reaches the time-to-live a Although we assumed that communications between net-
packet is allowed to travel in the network. 3) the path has work devices and the controller are reliable, the returned QoS
reached the traffic demand’s destination node. data may be out-of-order due to factors like physical distance
It should be noted that FindTempPath() follows an between a node and the controller. Collect_QoS_Data()
action selection policy throughout its running process, even will sort incoming data in order of tempPath’s added
 11

Algorithm 3 Find Temporary Path

1: function F IND T EMP PATH(TrafficDemand 𝐷, QTable 𝐿, Hyperparameters 𝐻)
2: Link[] []links = get link matrix from Graph 𝐺, from RLSR-Routing;
3: int[]visitedNodes = new int[𝐺.numberOfNodes];
4: initialize all visitedNodes entries to 0;
5: int currentId = 𝐷.srcId;
6: int nextId = −1;
7: Path tempPath;
8: int remainTTL = 𝐻.TTL;
9: while remainTTL > 0 and nextId ≠ 𝐷.dstId do
10: Node[]neighbors = get current node’s neighbors from Graph;
11: Node[]unvisited = neighbors ∩ (visitedNodes with entry = 0);
12: if unvisited is empty then
13: break;
14: end if
15: if Probability 𝑃 < 𝐻.𝜀 then
16: nextId = randomly selects one unvisited neighbor;
17: else
18: nextId = one with highest Q-value from 𝐿;
19: end if
20: add selected action(next hop) to tempPath, and updates related fields;
21: unvisited[nextId] = 1;
22: currentId = nextId;
23: remainTTL = remainTTL − 1;
24: end while
25: return tempPath;
26: end function

links. Moreover, Collect_QoS_Data() should be able network status. Thus, the current traffic intensity (Rinten) and
to detect packet loss through mechanisms such as a timer. current link utilization (Rutil) are used to calculate global
When packet loss occurs, it signifies that one performed rewards. Eq. 11 ensures all local rewards are no greater than
action failed to complete (i.e., transform the current state to -0.1, and Eq. 12 ensures all global rewards are no greater than
the next state), and no QoS data for the failed action will 0.
be received. The action performing process will terminate, CalculateLRewards() and CalculateGRewards() show the
and this failed action becomes the last action performed. pseudocode for calculating local and global rewards. Both
Subsequently, Collect_QoS_Data() should actively send functions have a similar workflow, applying Eq. 11 and
a request to the controller to explicitly require related link- 12 to calculate local/global rewards for the first performed
state information. To distinguish a failed action from an action until the second last. For the last performed action,
action without packet loss, a field in the QoSData structure, packet loss may occur, necessitating a different approach to
hasLost, has been added to indicate whether packet loss calculate the rewards and potentially applying a punishment
occurred when the action was performed. for unsuccessfully performed actions, so that RL agent learns
to avoid select those actions in the future. However, definition
G. RL Algorithm: Rewards Calculation of “unsuccessfully performed last action” is different from the
local or global reward’s perspective. For local rewards calcu-
The local and global reward calculations are given by:
lation, the last action is considered unsuccessful if it failed
𝑅local = 𝑊𝑐 × 𝑅hop + 𝑊𝑡 × 𝑅transmission + 𝑊𝑟 × 𝑅reliability to deliver the packet to its traffic demand’s destination, even
+ 𝑊𝑖 × 𝑅inten-est + 𝑊𝑢 × 𝑅util-est − 𝑙 Constant (11) if no packet loss happened. For global rewards calculation,
the last action is considered unsuccessful only when packet
𝑅global = 𝑊𝑟 × 𝑅reliability + 𝑊𝑖 × 𝑅inten + 𝑊𝑢 × 𝑅util − 𝑔Constant
loss occurred. In addition to assigning punishment values for
(12)
unsuccessful actions, RL algorithm adopted in RLSR-Routing
Eq. 11 and 12 are calculations of local reward and global has different ways to update related Q-values.
reward, respectively. Each individual QoS factor’s reward is
calculated based on Eq. 1 – 7, and two constant values
calculations (𝑙Constant and 𝑔Constant ) are based on Eq. 9 and 10. H. RL Algorithm: Tables Update
In our traffic routing problem, local rewards can be viewed The RL algorithm uses the UpdateTable() function
as a reflection of path quality for a given traffic demand. On for both local and global Q-table updates. For performed
the other hand, global rewards are used to learn the current actions that are guaranteed to be successful (from the first
 12

Algorithm 4 Calculate Local Rewards

1: function C ALCULATE LR EWARDS(QoSData 𝐷, QoSWeights 𝑊, int 𝐶𝑜𝑢𝑛𝑡, TrafficDemand 𝐷𝑒𝑚𝑎𝑛𝑑)
2: Reward[] 𝑙 𝑅𝑒𝑤𝑎𝑟 𝑑𝑠;
3: for 𝑖 = 0; 𝑖 < 𝐶𝑜𝑢𝑛𝑡 − 1; 𝑖 + + do
4: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝑖].𝑠𝑟𝑐𝐼𝑑 = 𝐷.𝑠𝑟𝑐𝐼𝑑;
5: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝑖].𝑑𝑠𝑡𝐼𝑑 = 𝐷.𝑑𝑠𝑡𝐼𝑑;
6: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝑖].𝑎𝑐𝑡𝑖𝑜𝑛𝑆𝑢𝑐𝑐𝑒𝑠𝑠 = True;
7: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝑖].𝑣𝑎𝑙𝑢𝑒 = use Eq. 11, 𝐷, 𝑊 to calculate;
8: end for
9: if 𝐷.ℎ𝑎𝑠𝐿𝑜𝑠𝑡 OR 𝐷.𝑑𝑠𝑡𝐼𝑑 ≠ 𝐷𝑒𝑚𝑎𝑛𝑑.𝑑𝑠𝑡𝐼𝑑 then
10: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑎𝑐𝑡𝑖𝑜𝑛𝑆𝑢𝑐𝑐𝑒𝑠𝑠 = False;
11: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑣𝑎𝑙𝑢𝑒 = −𝑊 .𝑙𝐶𝑜𝑛𝑠𝑡𝑎𝑛𝑡;
12: else
13: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑎𝑐𝑡𝑖𝑜𝑛𝑆𝑢𝑐𝑐𝑒𝑠𝑠 = True;
14: 𝑙 𝑅𝑒𝑤𝑎𝑟𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑣𝑎𝑙𝑢𝑒 = use Eq. 11, 𝐷, 𝑊 to calculate;
15: end if
16: return 𝑙 𝑅𝑒𝑤𝑎𝑟 𝑑𝑠;
17: end function

Algorithm 5 Calculate Global Rewards

1: function C ALCULATE GR EWARDS(QoSData 𝐷, QoSWeights 𝑊, int 𝐶𝑜𝑢𝑛𝑡)
2: Reward[] 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠;
3: for 𝑖 = 0; 𝑖 < 𝐶𝑜𝑢𝑛𝑡 − 1; 𝑖 + + do
4: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝑖].𝑠𝑟𝑐𝐼𝑑 = 𝐷.𝑠𝑟𝑐𝐼𝑑;
5: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝑖].𝑑𝑠𝑡𝐼𝑑 = 𝐷.𝑑𝑠𝑡𝐼𝑑;
6: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝑖].𝑎𝑐𝑡𝑖𝑜𝑛𝑆𝑢𝑐𝑐𝑒𝑠𝑠 = True;
7: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝑖].𝑣𝑎𝑙𝑢𝑒 = use Eq. 12, 𝐷, 𝑊 to calculate;
8: end for
9: if 𝐷.ℎ𝑎𝑠𝐿𝑜𝑠𝑡 then
10: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑎𝑐𝑡𝑖𝑜𝑛𝑆𝑢𝑐𝑐𝑒𝑠𝑠 = False;
11: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑣𝑎𝑙𝑢𝑒 = −𝑊 .𝑔𝐶𝑜𝑛𝑠𝑡𝑎𝑛𝑡;
12: else
13: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑎𝑐𝑡𝑖𝑜𝑛𝑆𝑢𝑐𝑐𝑒𝑠𝑠 = True;
14: 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠[𝐶𝑜𝑢𝑛𝑡 − 1].𝑣𝑎𝑙𝑢𝑒 = use 4.12, 𝐷, 𝑊 to calculate;
15: end if
16: return 𝑔𝑅𝑒𝑤𝑎𝑟 𝑑𝑠;
17: end function

Algorithm 6 UpdateTable function

1: function U PDATE TABLE(𝑄𝑇 𝑎𝑏𝑙𝑒 𝑇, 𝑅𝑒𝑤𝑎𝑟 𝑑𝑠 𝑅, 𝐻𝑦 𝑝𝑒𝑟 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝑠 𝐻, 𝑖𝑛𝑡 𝐶𝑜𝑢𝑛𝑡)
2: //update Q-values for the first till the second last performed actions
3: for 𝑖𝑛𝑡 𝑖 = 0; 𝑖 < 𝐶𝑜𝑢𝑛𝑡 − 1; 𝑖 + + do
4: 𝑖𝑛𝑡 𝑠𝑡𝑎𝑡𝑒 = 𝑅[𝑖].𝑠𝑟𝑐𝐼𝑑;
5: 𝑖𝑛𝑡 𝑎𝑐𝑡𝑖𝑜𝑛 = 𝑅[𝑖].𝑑𝑠𝑡𝐼𝑑;
6: 𝑇 [𝑠𝑡𝑎𝑡𝑒] [𝑎𝑐𝑡𝑖𝑜𝑛] = use Eq. 8 to update Q-value;
7: end for
8: 𝑖𝑛𝑡 𝑠𝑡𝑎𝑡𝑒 = 𝑅[𝐶𝑜𝑢𝑛𝑡 − 1].𝑠𝑟𝑐𝐼𝑑;
9: 𝑖𝑛𝑡 𝑎𝑐𝑡𝑖𝑜𝑛 = 𝑅[𝐶𝑜𝑢𝑛𝑡 − 1].𝑑𝑠𝑡𝐼𝑑;
10: 𝑑𝑜𝑢𝑏𝑙𝑒 𝑛𝑒𝑥𝑡𝑄𝑉 𝑎𝑙 = 0;
11: if 𝑅[𝑐𝑜𝑢𝑛𝑡 − 1].𝑎𝑐𝑡𝑖𝑜𝑛𝑆𝑢𝑐𝑐𝑒𝑠𝑠 == 𝐹𝑎𝑙𝑠𝑒 then
12: 𝑇 [𝑠𝑡𝑎𝑡𝑒] [𝑎𝑐𝑡𝑖𝑜𝑛]+ = 𝑅[𝐶𝑜𝑢𝑛𝑡 − 1].𝑣𝑎𝑙𝑢𝑒;
13: else
14: 𝑇 [𝑠𝑡𝑎𝑡𝑒] [𝑎𝑐𝑡𝑖𝑜𝑛] = use Eq. 8 and nextQVal to update;
15: end if
16: end function
 13

action to the second last), the function applies SARSA’s reaches the destination node, with the highest estimated cumu-
typical function (Eq. 8) to update corresponding Q-values. lative reward from local Q-table. However, the final path may
For the last action, classified as unsuccessfully performed, not be the best solution based on the user’s QoS requirements,
the associated Q-value accumulates the penalty by adding the and one possible reason could be the action selection policy
reward’s value. The reason UpdateTable() does not use used during the learning process.
Eq. 8 for the unsuccessful actions’ Q-value update is discussed
subsequently. IV. E XPERIMENT S ETUP AND R ESULT A NALYSIS
We provided two ways to create a network topology for
performing experiments. First is to use our implemented
methods to write code, which step by step creates a graph
representation of a network, adds nodes and links, and possibly
assigns the initial used bandwidth of certain links. Another
way is to write a JSON file with a specified format, which
can be parsed using an open-source JSON parser.
Fig. 7: Edge Case Topology (T4) RLSR-Routing is a framework that directly addresses path
finding issues in SDN, without requiring prior knowledge of
Topology T4 in Fig. 7 illustrates an edge case in which the network (e.g., link weights) or external libraries to train
using Eq. 8 to update an unsuccessful action’s Q-value may a neural network. Deployment of RLSR-Routing should be
cause a problem in RLSR-Routing. Consider a scenario where flexible as long as it can interact with the SDN controller
a traffic demand is to send packets from node 0 to node 4. and parse JSON file. This study mainly focused on the RL
Every successfully performed action has a local reward value algorithm component’s implementation, with additional code
of −2; whereas, unsuccessfully performed ones have a local to support network simulation. All code is written in JAVA
reward value of −3. The RL agent uses 𝛼 = 𝛾 = 1, 𝜀 = 0.5, but transfer to other programming languages should not be a
and 𝑇𝑇 𝐿 = 16 for hyperparameters. No packet loss occurred complicated task. This section presents our comparative study
during the learning process. After running enough episodes and validation experiments’ results with discussion.
that the local Q-table converges, the final local Q-table should
look like this: A. Settings for Hyperparameters
Based on the setting of the RL algorithm, when node 0 Authors who proposed QAR in [31] conducted experiments
chooses to send the packet to node 5, such action is regarded to study the effects of learning rate and the importance of
as unsuccessful by local rewards calculation. This is because long-term rewards. We followed their guidelines and unless
from node 5, the RL agent cannot find a never visited next-hop specified explicitly, the hyperparameters used in this experi-
node to deliver the packet to the traffic demand’s destination. ments are summarized in Table V.
If we use Eq. 8 to update 𝑄(𝑠0 , 𝑎 0,5 ), the value will converge
at −3 based on our settings above. As a result, at node 0, TABLE V: Hyperparameters, their values, and description
choosing action 𝑎 0,5 has a higher cumulative reward than Parameter Value Description
choosing action 𝑎 0,1 (−3 vs −8), even if sending a packet 𝜖 0 Exploration rate in the 𝜖 -greedy strategy,
to node 1 is the only way to reach the final destination set to 0 for no exploration.
𝛼 0.9 Learning rate, controls how quickly the al-
node 4. In contrast, the accumulative penalty mechanism in gorithm updates its knowledge with new
UpdateTable() ensures that the more times an action is information.
performed unsuccessfully, the lower Q-value it will receive. 𝛾 0.9 Discount factor, determines the importance
of future rewards in the reinforcement learn-
For example, consider the updates to the Q-values as the agent ing algorithm.
repeatedly chooses action 𝑎 0,5 : TTL 32 Time to Live, may represent the number
• The first time the agent chooses 𝑎 0,5 , 𝑄(𝑠0 , 𝑎 0,5 ) will
of steps or iterations before termination in
specific contexts.
become −3. 𝐸 75 Represent episodes.
• The second time 𝑄(𝑠0 , 𝑎 0,5 ) will become −6.
• The third time 𝑄(𝑠0 , 𝑎 0,5 ) will become −9,
• and so on.
B. Comparative Study with Non-RL routing
We performed a comparative study between RLSR-Routing
I. RL Algorithm: Return final result and a non RL-based routing algorithm, NR-Routing, which is
After all learning episodes are completed, the RL algorithm used by a telecom solutions provider. Similar to our proposed
uses FindFinalPath() to retrieve the final routing path RLSR-Routing, NR-Routing is able to assign paths for given
from the source node to the destination node of the input traffic traffic flows between source and destination pairs; however, it
demand. FindFinalPath() executes FindTempPath(), uses a non-RL based agent to explore the traffic engineering
but with Hyperparameters’ 𝜀 value set to 0 – i.e., using a space. The agent places traffic demands greedily (from largest
greedy selection policy. With an adequate number of learning to smallest) and/or randomly on the network capacity using
episodes, the final path should be non-cyclic and successfully multi-path segment routing based traffic engineering. Due to
 14

TABLE VII: RLSR-Routing Generated Paths for Traffic De-

mands
Source-Destination Path
4 to 14 i. 4 → 7 → 15 → 14
ii. 4 → 6 → 15 → 14
1 to 5 i. 1 → 13 → 5
ii. 1 → 13 → 5
0 to 6 i. 0 → 3 → 15 → 6
ii. 0 → 2 → 14 → 15 → 6
iii. 0 → 12 → 13 → 4 → 6
iv. 0 → 3 → 15 → 7 → 6
v. 0 → 1 → 3 → 15 → 6
13 to 11 i. 13 → 1 → 2 → 14 → 11
ii. 13 → 1 → 3 → 14 → 11
iii. 13 → 4 → 7 → 15 → 11
Fig. 8: 16 Nodes Topology (T7)) iv. 13 → 5 → 6 → 15 → 11
3 to 13 i. 3 → 1 → 13
ii. 3 → 0 → 12 → 13
iii. 3 → 1 → 13
confidentiality issues, we cannot provide additional informa-
tion about NR-Routing’s implementation or other details. The 3 to 4 i. 3 → 15 → 7 → 4
ii. 3 → 2 → 1 → 12 → 8 → 13 → 4
overall objective is to compare the two algorithms’ load bal- iii. 3 → 14 → 15 → 7 → 4
ancing ability in terms of minimizing maximum link utilization iv. 3 → 15 → 6 → 4
over a given network. The testing network topology T7 is v. 3 → 1 → 13 → 4
vi. 3 → 0 → 12 → 13 → 4
shown in Fig. 8, with 16 nodes and each pair of adjacent
nodes has two unidirectional links connecting them. All links
TABLE VIII: NR-Routing’s Links with Traffic
have a 2.0Mb/s maximum bandwidth, 100% reliability rate,
and all nodes have a 20Mb/s processing rate. No. Path Bandwidth (BD)
1 R3(14) → R4(15) 8000000.0
NR-Routing is an executable program embodying two main 2 R4(15) → R3(14) 4000000.0
functions: 3 B2(5) → B3(6) 316666.0
• Generation of traffic engineering problems consisting of 4 R1(12) → R2(13) 14333330.0
5 B3(6) → R4(15) 966666.0
network topologies and sets of traffic demands to place 6 R4(15) → B3(6) 500000.0
on the network capacity. 7 A2(1) → R2(13) 500000.0
• Optimization of traffic demand engineering by placing 8 A1(0) → R1(12) 933333.0
9 A4(3) → R4(15) 466666.0
demands on paths in the network with segment routing 10 R2(13) → B2(5) 866666.0
based traffic steering techniques. 11 B1(4) → B3(6) 516666.0
12 B3(6) → B1(4) 300000.0
We first randomly generated networks and demands and 13 A4(3) → R3(14) 466666.0
let NR-Routing assign one or more paths for each demand. 14 R2(13) → B1(4) 1066666.0
These generated paths were saved as a JSON file. We then 15 A2(1) → R1(12) 500000.0
16 A3(2) → R3(14) 3333330.0
removed unwanted demands, adjusted the amount of data for 17 B1(4) → B4(7) 3500000.0
the remaining, and then used them as NR-Routing’s input for 18 B4(7) → B1(4) 3000000.0
load balancing. This time, the final output is saved in another 19 A4(3) → A1(0) 6000000.0
20 A1(0) → A3(2) 3333330.0
JSON file to be used for comparison. The traffic and paths 21 A4(3) → A2(1) 6000000.0
assignments for selected tunnels presented in Table VI that 22 B4(7) → R4(15) 5000000.0
we used in NR-Routing program’s load balancing test. 23 R4(15) → B4(7) 3000000.0
24 B2(5) → B4(7) 1500000.0
25 R4(15) → PE4(11) 6000000.0
TABLE VI: Traffic distribution among nodes
Source Destination Total Traffic Number of Paths
4 14 400kb/s 2 utilization when calculating local rewards during the learning
1 5 400kb/s 2
0 6 1.0Mb/s 5 process. After the RL algorithm found a path for one traffic
13 11 600kb/s 4 demand, the demand’s traffic was placed over the network
3 13 600kb/s 3 along the path. These steps were repeated until the RL
3 4 1.2Mb/s 6 algorithm found paths for all traffic demands and placed all
demand traffic over the network.
For RLSR-Routing, it interpreted each tunnel as 𝑛 traffic
demands, where 𝑛 equals the number of paths assigned to the
tunnel by NR-Routing. Every traffic demand from the same C. Routing Paths by RLSR-Routing
tunnel split the tunnel’s total traffic equally. For example, for The paths generated for each tunnel (source and destination
tunnels from node 3 to 4, each of the six traffic demands pair) by RLSR-Routing are meticulously listed in Table VII,
had 200kb/s traffic. The RL algorithm only considered link showcasing the algorithm’s ability to produce valid, loop-free
 15

TABLE IX: RLSR-Routing’s Links with Traffic topology was used in both NR-Routing and RLSR-Routing,
No. Path Bandwidth (BD) the labels for each node used in NR-Routing are different.
1 0→1 200000.0 Therefore, we included the translated node’s ID in brackets,
2 0→2 200000.0
3 0→3 400000.0
for example, node with ID R3 in NR-Routing is node with ID
4 0→12 600000.0 (14) in RLSR-Routing. For NR-Routing, the maximum traffic
5 1→2 150000.0 assigned to one link is 1.433333Mb/s, thus the maximum
6 1→3 350000.0
7 1→12 200000.0
link utilization in NR-Routing’s output is 1.433333 / 2.0 =
8 1→13 1000000.0 71.67%. For RLSR-Routing, the maximum traffic assigned
9 2→1 200000.0 to one link is 1.0Mb/s, thus the maximum link utilization in
10 2→14 350000.0
11 3→0 400000.0
RLSR-Routing’s links assignment is 1.0 / 2.0 = 50.0%.
12 3→1 600000.0
13 3→2 200000.0
14 3→14 350000.0 E. Cost of Exploration During Learning Process
15 3→15 1600000.0 We used a 30 nodes network as shown in Fig. 9 to test
16 4→6 400000.0
17 4→7 350000.0 the efficacy of RLSR-Routing in the learning process. In
18 5→6 150000.0 topology T8, each pair of adjacent nodes are connected by two
19 6→4 200000.0 unidirectional links to support bi-directional communications.
20 6→15 350000.0
21 7→4 400000.0 All nodes have a 100Mb/s processing rate, and all links have
22 7→6 200000.0 a 10Mb/s maximum bandwidth. Before any traffic demand’s
23 7→15 350000.0 data been placed over the network, links used bandwidth are
24 8→13 200000.0
25 12→8 200000.0 as follows:
26 12→13 600000.0 • Link 𝑙 6,7 , 𝑙 7,6 , 𝑙 18,19 , 𝑙 19,18 : 0Mb/s used bandwidth;
27 13→1 300000.0
28 13→4 950000.0 • Link 𝑙 1,3 , 𝑙 2,3 : 5Mb/s used bandwidth;
29 13→5 550000.0 • Link 𝑙 0,3 , 𝑙 4,8 , 𝑙 7,11 , 𝑙 10,13 , 𝑙 18,22 : 9Mb/s used bandwidth;
30 14→11 300000.0 • The rest of the links: 1Mb/s used bandwidth.
31 14→15 400000.0
32 15→6 800000.0 In this experiment, we gave RLSR-Routing nine traffic de-
33 15→7 600000.0 mands, each with 0.1Mb/s estimated traffic. The RL algorithm
34 15→11 300000.0
35 15→14 400000.0
found one path for every given traffic demand, with traffic
intensity and link utilization as considered QoS parameters
(𝑊𝑖 = 𝑊𝑢 = 1). A global Q-table was not used during
this experiment since the experiment was not focused on
convergence speed. After one path was returned from the RL
algorithm, corresponding traffic demand data was placed over
the network (i.e., all links on the path added 0.1Mb/s to their
used bandwidth).
1) Final Paths: Table X summarizes the final routing path
returned by RLSR-Routing for every traffic demand. All
routing paths are valid: every node in a path is only included
once, and every path reaches the specified destination. During
the learning process, RLSR-Routing gradually explored the
network, so that highly utilized links (like link 𝑙0,3 , 𝑙 4,8 , 𝑙7,11 ,
𝑙10,13 , 𝑙 18,22 ) and nodes with relatively high traffic intensity
(like node 3, 8, 11, 13, 22) are excluded in final paths. As a
Fig. 9: 30 Nodes Topology (T8) result, all the paths include the same set of nodes, node 4, 7,
6, 10, 14, 18, 19, 23 as intermediate nodes between source
and destination. Although these are not the shortest paths in
routing paths that consist of consecutive nodes connecting the
terms of hop-count, they are the user preferred paths in terms
specified source and destination. All paths produced by RLSR-
of link utilization and traffic intensity.
Routing are validated as efficient routing paths; they are loop-
2) Path Length vs Episode Number: Fig. 10 demonstrated
free and comprise a consecutive series of nodes linking the
when RLSR-Routing was finding the path for each traffic
source and destination nodes for specified traffic demands.
demand, the length (i.e., number of hops) of temporary paths
(denote as TempPath) generated in the first 50 episodes. None
D. Links Utilizations of the temporary paths would result in a packet inside a loop,
Table VIII and IX show the links that placed traffic on and most of the paths have less than 20 nodes included.
it by NR-Routing and RLSR-Routing, respectively. For NR- Although at the beginning of the learning process, especially
Routing, traffic assigned on different links of the network was during the first 20 episodes, some temporary paths failed to
saved in the JSON output file. We extracted the information reach the desired destination, the agent quickly learned to
and saved in another text file. Although the same network avoid choosing actions that lead to a dead end. For most of the
 16

TABLE X: Final Path for Traffic Demands

Traffic Demand Final Path
0 → 26 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
1 → 26 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
2 → 26 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
0 → 27 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
1 → 27 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
2 → 27 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
0 → 28 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
1 → 28 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
2 → 28 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28

(a) TempPath length for Demand 0 → 26 (b) TempPath length for Demand 1 → 26 (c) TempPath length for Demand 2 → 26

(d) TempPath length for Demand 0 → 27 (e) TempPath length for Demand 0 → 27 (f) TempPath length for Demand 0 → 27

(g) TempPath length for Demand 0 → 28 (h) TempPath length for Demand 0 → 28 (i) TempPath length for Demand 0 → 28
Fig. 10: Evolution of TempPath Lengths During Early Learning Episodes: Illustrates the length of temporary paths (TempPath)
for various demands across the first 50 episodes in RLSR-Routing’s learning process. Despite initial challenges in reaching
destinations within the first 20 episodes, the agent rapidly adapted, ensuring paths without loops and mostly under 20 nodes.
Convergence to efficient routing paths was typically achieved by the 30th episode, highlighting RLSR-Routing’s effectiveness
in learning optimal paths for traffic demands.

testing traffic demands, RLSR-Routing was able to converge from subsection IV-E did not use global Q-table; test groups
around 30 episodes; after that, the temporary paths produced I to IV applied 𝛾 value 0.3, 0.5, 0.7, and 0.9 to update
in each episode are the same as the final paths. global Q-table’s entries. The final paths for each traffic demand
3) Global Q-table and Effects of 𝛾 Value: We present our and the number of episodes RLSR-Routing used to converge
study about whether using global Q-table speeds up algorithm were recorded for further analysis. Table XI to XIV illustrate
convergence. We applied the same network topology (T8) and different test groups’ final paths for given traffic demands.
the same initial links used bandwidth as in subsection IV-E. 4) RL Convergence Speed: Table XV shows for each traffic
The same set of traffic demands were given to RLSR-Routing demand, the number of episodes different experiment groups
and the user preferred path was defined by the same QoS used to reach convergence. Compared with the control group,
weights. The only difference is that RLSR-Routing used a in most cases, test groups used fewer episodes to reach
global Q-table to initialize local Q-table. To study the effect convergence for the same traffic demand. Fig. 11 summarizes
of 𝛾 (importance of long term reward) on applying global Q- the total episode numbers for each group to reach convergence
table, we set up five experiment groups. The control group on 9 traffic demands. All test groups used fewer total episode
 17

TABLE XI: Final Paths by Test Group I (𝛾 = 0.3)

Traffic Demand Final Path
0 → 26 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
1 → 26 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
2 → 26 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
0 → 27 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
1 → 27 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
2 → 27 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
0 → 28 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
1 → 28 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
2 → 28 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28

TABLE XII: Final Paths by Test Group II (𝛾 = 0.5)

Traffic Demand Final Path
0 → 26 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
1 → 26 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
2 → 26 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
0 → 27 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
1 → 27 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
2 → 27 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
0 → 28 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
1 → 28 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
2 → 28 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28

TABLE XIII: Final Paths by Test Group III (𝛾 = 0.7)

Traffic Demand Final Path
0 → 26 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
1 → 26 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
2 → 26 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
0 → 27 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
1 → 27 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
2 → 27 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
0 → 28 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
1 → 28 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
2 → 28 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28

TABLE XIV: Final Paths by Test Group IV (𝛾 = 0.9)

Traffic Demand Final Path
0 → 26 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
1 → 26 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
2 → 26 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 26
0 → 27 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
1 → 27 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
2 → 27 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 27
0 → 28 0 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
1 → 28 1 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28
2 → 28 2 → 4 → 7 → 6 → 10 → 14 → 18 → 19 → 23 → 28

TABLE XV: Episode Number when RL Converged

Demand Control Test I 𝛾 = 0.3 Test II 𝛾 = 0.5 Test III 𝛾 = 0.7 Test IV 𝛾 = 0.9
0 → 26 25 25 25 25 25
1 → 26 27 26 25 21 18
2 → 26 30 28 38 17 17
0 → 27 30 21 24 23 20
1 → 27 27 24 25 21 20
2 → 27 31 25 26 24 25
0 → 28 29 22 25 27 21
1 → 28 30 26 28 29 22
2 → 28 34 34 32 32 24

numbers to find paths for all traffic demands than the control V. C ONCLUSION
group. Test group IV, which used 𝛾 value 0.9, has the fastest We developed a RL-based routing algorithm, RLSR-
convergence speed: it took 192 episodes to find paths for all Routing, which works on SR-enabled SDN. We modified
given traffic demands. SARSA, an on-policy RL algorithm, by aggregating action
selection and setting a dual rewards scheme. To handle po-
tential packet loss or failure to reach the desired destination
 18

more practical setting of traffic engineering problem, such as

a 1000 nodes networks with thousands of traffic demands. In
addition, future experiment should test RLSR-Routing’s ability
to find user preferred path in terms of other QoS parameters,
such as link reliability, instead of just using link utilization
and traffic intensity. Finally, we will modify the whole RL
algorithm component so that it can compute the user preferred
path for a traffic demand matrix in parallel.
R EFERENCES
[1] CISCO Systems, “Cisco Annual Internet Report (2018-2023).” http://
[Link]/binfj, 2020. [Online; accessed 2023-04-01].
[2] J. F. Kurose and K. W. Ross, Computer Networking: A Top-down
Fig. 11: Total Episode Numbers when RL Converged for All Approach. Boston, MA, USA: Pearson, 7th ed., 2017.
Demands [3] J. Moy, “RFC 2328: OSPF Version 2,” 1998.
[4] C. Hedrick, “RFC 1058: Routing Information Protocol,” 1988.
[5] Y. Guo, Z. Wang, X. Yin, X. Shi, and J. Wu, “Traffic Engineering
in SDN/OSPF Hybrid Network,” in 2014 IEEE 22nd International
during the learning process, we added additional steps than Conference on Network Protocols, pp. 563–568, 2014.
the normal function to update Q-tables for failed actions. [6] Y. Wang, Z. Wang, and L. Zhang, “Internet Traffic Engineering without
Full Mesh Overlaying,” in Proceedings IEEE INFOCOM 2001. Confer-
Experiment results indicate that RLSR-Routing can find user ence on Computer Communications. Twentieth Annual Joint Conference
preferred paths for given traffic demand, based on customized of the IEEE Computer and Communications Societies, vol. 1, pp. 565–
QoS considerations. 571, 2001.
[7] M. Karakusa and A. Durresi, “Quality of service (QoS) in software
Our research contribution is listed as follows: defined networking (SDN): A survey,” Journal of Network and Computer
• RLSR-Routing directly applies RL in the path finding Applications, pp. 200–218, 2017.
[8] R. Masoudi and A. Ghaffari, “Software defined networks: A survey,”
process, without a prior knowledge of the network. It Journal of Network and Computer Applications, vol. 67, pp. 1–25, 2016.
does not rely on additional inputs such as pre-defined [9] I. T. Haque and N. Abu-Ghazaleh, “Wireless Software Defined Net-
link weights; or a set of pre-calculated paths between a working: A Survey and Taxonomy,” IEEE Communications Surveys &
Tutorials, vol. 18, no. 4, pp. 2713–2737, 2016.
source and destination pair to find the user preferred path [10] C. Filsfils, S. Previdi, L. Ginsberg, B. Decraene, S. Litkowski, and
of a traffic demand. R. Shakir, “RFC 8402: Segment Routing Architecture,” 2018.
• We further reduced the number of communications re- [11] C. Filsfils, N. K. Nainar, C. Pignataro, J. C. Cardonay, and P. Francois,
“The Segment Routing Architecture,” in 2015 IEEE Global Communi-
quired between SDN controller and network data planes cations Conference (GLOBECOM), p. 106, 2015.
by exploiting Segment Routing, compared with previous [12] S. Previdi, C. Filsfils, B. Decraene, S. Litkowski, M. Horneffer, and
RL-based routing in SDN. R. Shakir, “RFC 7855: Source Packet Routing in Networking (SPRING)
Problem Statement and Requirements,” 2016.
• We ensured that no packet would be routed in a loop
[13] R. Boutaba, M. A. Salahuddin, N. Limam, and S. A. et al., “A
during the RLSR-Routing learning process. comprehensive survey on machine learning for networking: evolution,
• We gave users the ability to customize QoS weights to applications and research opportunities,” Journal of Internet Services
and Applications, 2018.
define what is their preferred path. [14] D. M. C-Velasco, O. M. C. Rendon, and N. L. S. d. Fonseca, “Intel-
• Our proposed framework can reuse previously learned ligent Routing Based on Reinforcement Learning for Software-Defined
knowledge to speed up algorithm convergence. Networking,” IEEE Transactions on Network and Service Management,
pp. 870–881, 2021.
Action selection based on a greedy approach may not find [15] Z. Mammeri, “Reinforcement Learning Based Routing in Networks: Re-
the best path in some cases. Relative medium size networks view and Classification of Approaches,” IEEE Access, vol. 7, pp. 55916–
and small number of traffic demands tested may be the reasons 55950, 2019.
[16] P. Wang and T. Wang, “Adaptive Routing for Sensor Networks using
that we only observed user preferred final paths in experiment Reinforcement Learning,” in The Sixth IEEE International Conference
results. Another issue is that RLSR-Routing does not support on Computer and Information Technology (CIT’06), pp. 219–219, 2006.
finding paths for multiple paths in parallel. When working [17] T. Hu and Y. Fei, “QELAR: A Machine-Learning-Based Adaptive Rout-
ing Protocol for Energy-Efficient and Lifetime-Extended Underwater
on a set of traffic demands, RLSR-Routing only focuses on Sensor Networks,” IEEE Transactions on Mobile Computing, vol. 9,
finding the best path for one traffic demand at a time, without no. 6, pp. 796–809, 2010.
considering the effect of placing traffic on the best path on [18] S. Z. Jafarzadeh and M. H. Y. Moghaddam, “Design of Energy-aware
QoS Routing Algorithm in Wireless Sensor Networks Using Reinforce-
other traffic demands that have not been assigned a path. ment Learning,” in 2014 4th International Conference on Computer and
To provide better demonstrations of paths assignments and Knowledge Engineering (ICCKE), pp. 722–727, 2014.
links utilizations by RLSR-Routing and NR-Routing, we only [19] B. Debowski, P. Spachos, and S. Areibi, “Q-Learning Enhanced Gradient
Based Routing for Balancing Energy Consumption in WSNs,” in IEEE
used a subset of randomly generated demands during the 21st International Workshop on Computer Aided Modelling and Design
comparative study. In practice, RLSR-Routing should assign of Communication Links and Networks (CAMAD), pp. 18–23, 2016.
paths for all traffic demands from the users of the network. [20] G. Santhi, A. Nachiappan, M. Z. Ibrahime, R. Raghunadhane, and M. K.
Favas, “Q-learning based adaptive QoS routing protocol for MANETs,”
As explained earlier, several upgrades can further improve in International Conference on Recent Trends in Information Technology
RLSR-Routing’s performance. For example, a better action (ICRTIT), pp. 1233–1238, 2011.
selection policy, and more flexibility for users to decide how [21] C. Wu, S. Ohzahata, and T. Kato, “Flexible, Portable, and Practicable
Solution for Routing in VANETs: A Fuzzy Constraint Q-Learning
to apply a global Q-table to initiate a local Q-table. For the Approach,” IEEE Transactions on Vehicular Technology, vol. 62, no. 9,
comparative study, we plan to test our RLSR-Routing in a pp. 4251–4263, 2013.
 19

[22] W. Jung, J. Yim, and Y. Ko, “QGeo: Q-Learning-Based Geographic [43] R. E. Ali, B. Erman, E. Baştuğ, and B. Cilli, “Hierarchical Deep Double
Ad Hoc Routing Protocol for Unmanned Robotic Networks,” IEEE Q-Routing,” in IEEE International Conference on Communications
Communications Letters, vol. 21, no. 10, pp. 2258–2261, 2017. (ICC), pp. 1–7, 2020.
[23] C. Wu, T. Yoshinaga, Y. Ji, and Y. Zhang, “Computational Intelligence [44] H. Hasselt, “Double Q-learning,” in Advances in Neural Information
Inspired Data Delivery for Vehicle-to-Roadside Communications,” IEEE Processing Systems (23), 2010.
Transactions on Vehicular Technology, vol. 67, no. 12, pp. 12038–12048,
2018.
[24] T. Hendriks, M. Camelo, and S. Latré, “Q2-Routing: A Qos-aware Q-
Routing algorithm for Wireless Ad Hoc Networks,” in 14th International
Conference on Wireless and Mobile Computing, Networking and Com- Wumian Wang, earned a bachelor’s degree in mi-
munications (WiMob), pp. 108–115, 2018. crobiology & immunology from the University of
[25] F. Li, X. Song, H. Chen, X. Li, and Y. Wang, “Hierarchical Routing British Columbia between 2014 and 2019. Following
for Vehicular Ad Hoc Networks via Reinforcement Learning,” IEEE that, Wang pursued a master’s degree in computer
Transactions on Vehicular Technology, vol. 68, no. 2, pp. 1852–1865, science at The University of Western Ontario from
2019. 2020 to 2022, during which he won a research
[26] C. Perkins, E. Belding-Royer, and S. Das, “RFC 3561: Ad hoc On- scholarship for the year 2021-2022. Alongside his
Demand Distance Vector (AODV) Routing,” 2003. studies, Wang worked as a teaching assistant in a
[27] C. Wu, S. Ohzahata, and T. Kato, “Routing in VANETs: A fuzzy computer science course in 2022.
constraint Q-Learning approach,” in IEEE Global Communications
Conference (GLOBECOM), pp. 195–200, 2012.
[28] M. Nurchis, R. Bruno, M. Conti, and L. Lenzini, “A Self-Adaptive
Routing Paradigm for Wireless Mesh Networks Based on Reinforcement
Learning,” in Proceedings of the 14th ACM international conference
on Modeling, analysis and simulation of wireless and mobile systems Sajal Saha, Ph.D. is an Assistant Professor in the
(MSWiM ’11), pp. 197–204, 2011. Department of Computer Science at the University
[29] M. Boushaba, A. Hafid, A. Belbekkouche, and M. Gendreau, “Rein- of Northern British Columbia, Canada. His research
forcement learning based routing in wireless mesh networks,” Wireless interests are primarily focused on Internet Traffic
Networks, vol. 19, pp. 2079–2091, 2013. Analysis, Cyber-attack Detection, and Data Mining,
[30] K. Tang, C. Li, H. Xiong, J. Zou, and P. Frossard, “Reinforcement with special emphasis on traffic prediction, anomaly
Learning-Based Opportunistic Routing for Live Video Streaming over detection, and association rule mining. He completed
Multi-Hop Wireless Networks,” in IEEE 19th International Workshop his Ph.D. at Western University, Canada, where his
on Multimedia Signal Processing (MMSP), pp. 1–6, 2017. research contributed significantly to efficient traffic
[31] S.-C. Lin, I. F. Akyildiz, P. Wang, and M. Luo, “QoS-Aware Adaptive forecasting and cyber-attack detection. Alongside his
Routing in Multi-layer Hierarchical Software Defined Networks: A academic contributions, he has also gained consid-
Reinforcement Learning Approach,” in IEEE International Conference erable industry experience as a Research Fellow at Juniper Network and
on Services Computing (SCC), pp. 25–33, 2016. Software Engineer at Samsung Electronics.
[32] Z. Yuan, P. Zhou, S. Wang, and X. Zhang, “Research on Routing
Optimization of SDN Network Using Reinforcement Learning Method,”
in 2nd International Conference on Safety Produce Informatization
(IICSPI), pp. 442–445, 2019.
[33] S. Yang, H. Shi, and H. Zhang, “Dynamic Load Balancing of Multiple Dr. Anwar Haque is an Associate Professor and
Controller based on Intelligent Collaboration in SDN,” in International the Undergraduate Chair of the Department of Com-
Conference on Computer Vision, Image and Deep Learning (CVIDL), puter Science at Western University, Canada. Dr.
pp. 354–359, 2020. Haque holds a Ph.D. in Electrical & Computer
[34] S. Kim, J. Son, A. Talukder, and C. S. Hong, “Congestion prevention Engineering and a Master’s degree in Computer Sci-
mechanism based on Q-leaning for efficient routing in SDN,” in Inter- ence, both from the University of Waterloo. Before
national Conference on Information Networking (ICOIN), pp. 124–128, joining Western, he was an Associate Director at
2016. Bell Canada. Dr. Haque’s primary research areas of
[35] D. Tennakoon, S. Karunarathna, and B. Udugama, “Q-learning Approach interest include next-gen communication networks,
for Load-balancing in Software Defined Networks,” in Moratuwa Engi- IoT, cyber-security, and applied AI focusing on
neering Research Conference (MERCon), pp. 1–6, 2018. developing secure and resilient autonomous cyber-
[36] J. Rischke, P. Sossalla, H. Salah, F. H. P. Fitzek, and M. Reisslein, “QR- physical systems (CPS). He has authored/co-authored more than 100 peer-
SDN: Towards Reinforcement Learning States, Actions, and Rewards reviewed research publications in leading journals and conferences, holds
for Direct Flow Routing in Software-Defined Networks,” IEEE Access, several patents/licenses, and supervised more than 100 HQPs. His collabo-
vol. 8, pp. 174773–174791, 2020. rative research grants are valued at more than $15 million. At Western, he
[37] S. Sendra, A. Rego, J. Lloret, J. M. Jimenez, and O. Romero, “Including served as the industry expert-in-residence in the Faculty of Science, a member
artificial intelligence in a routing protocol using Software Defined of Western’s Senate, and on the inaugural advisory committee for the newly
Networks,” in IEEE International Conference on Communications Work- established Bell-Western 5G Research Centre. Additionally, Dr. Haque is the
shops (ICC Workshops), pp. 670–674, 2017. founder and CEO of Bamboo Innovations Inc., a successful tech venture in
[38] C. Wang, L. Zhang, Z. Li, and C. Jiang, “SDCoR: Software Defined the smart systems industry.
Cognitive Routing for Internet of Vehicles,” IEEE Internet of Things
Journal, vol. 5, no. 5, pp. 3513–3520, 2018.
[39] C. Yu, J. Lan, Z. Guo, and Y. Hu, “DROM: Optimizing the Routing in
Software-Defined Networks With Deep Reinforcement Learning,” IEEE
Access, vol. 6, pp. 64533–64539, 2018. Greg Sidebottom received his PhD. degree in Com-
[40] Y. T. et al., “Traffic Engineering in Partially Deployed Segment Routing puter Science from Simon Fraser University. He is
Over IPv6 Network With Deep Reinforcement Learning,” IEEE/ACM a Principal Engineer at Juniper Networks. He is the
Transactions on Networking, vol. 28, no. 4, pp. 1573–1586, 2020. architect of Juniper’s industry pioneering Software
[41] P. Almasan, S. Xiao, X. Cheng, X. Shi, P. B. Ros, and A. C. Aparicio, Defined Network and Network Function Virtualiza-
“ENERO: Efficient Real-Time WAN Routing Optimization with Deep tion products. Greg is currently advancing the state
Reinforcement Learning,” 2021. arXiv:2109.10883v2. of the art in segment routing and peer engineering
[42] Q. Fu, E. Sun, K. Meng, M. Li, and Y. Zhang, “Deep Q-Learning for with Juniper’s Northstar WAN SDN controller.
Routing Schemes in SDN-Based Data Center Networks,” IEEE Access,
vol. 8, pp. 103491–103499, 2020.