Towards Real-Time Routing Optimization with

Deep Reinforcement Learning: Open Challenges

Paul Almasan∗ , José Suárez-Varela∗ , Bo Wu† , Shihan Xiao† , Pere Barlet-Ros∗ , Albert Cabellos-Aparicio∗
∗ Barcelona Neural Networking Center, Universitat Politècnica de Catalunya, Spain
† Network Technology Lab., Huawei Technologies Co.,Ltd.

Abstract—The digital transformation is pushing the existing Recent advances in Deep Reinforcement Learning (DRL)
network technologies towards new horizons, enabling new ap- showcased that this technology is capable to operate efficiently
plications (e.g., vehicular networks). As a result, the networking in complex optimization problems [4], [6]. However, state-
community has seen a noticeable increase in the requirements of
emerging network applications. One main open challenge is the of-the-art DRL-based solutions used in other fields are not
need to accommodate control systems to highly dynamic network directly applicable to modern networks, where it is needed
scenarios. Nowadays, existing network optimization technologies to generalize across scenarios with dynamic changes (e.g.,
do not meet the needed requirements to effectively operate in real topology, traffic, link failures) [7]. One key technology that
time. Some of them are based on hand-crafted heuristics with enables DRL to handle network changes is Graph Neural
limited performance and adaptability, while some technologies
use optimizers which are often too time-consuming. Recent Network (GNN) [8]. GNN is a novel family of neural networks
advances in Deep Reinforcement Learning (DRL) have shown a specifically intended to operate on graph-structured data. Par-
dramatic improvement in decision-making and automated control ticularly, these recent neural networks have a strong relational
problems. Consequently, DRL represents a promising technique bias over graphs, which results in an unprecedented level of
to efficiently solve a variety of relevant network optimization combinatorial generalization over graph-structured data [9]. As
problems, such as online routing. In this paper, we explore the
use of state-of-the-art DRL technologies for real-time routing a result, GNNs achieve outstanding generalization capabilities
optimization and outline some relevant open challenges to achieve over networks [7], [10] and, similarly to other deep learning
production-ready DRL-based solutions. methods, they achieve low execution times – often sub-second
operation in commodity hardware.
I. I NTRODUCTION In this paper, we explore the applicability of state-of-the-
art solutions based on DRL and GNN to modern networking
The digital transformation is driving existing network tech- scenarios with highly dynamic topology changes. Specifically,
nologies towards supporting emergent networking paradigms we design a DRL+GNN architecture for online routing opti-
(e.g., vehicular networks [1], smart cities [2], satellite net- mization, and discuss the open challenges of this technology.
works [3]). These modern applications impose new require- First, we train a GNN-based DRL agent in a single topology,
ments for existing network optimization technologies (e.g., and then we test if the agent is able to operate successfully
adaptability to dynamic scenarios, low latency). In this context, on different topologies never seen before. This allows us to
Machine Learning (ML) seems a good candidate to enable assess the capabilities of this solution to adapt to fundamental
real-time operation and allow efficient optimization across changes in the network topology. Lastly, we discuss the
dynamic systems [4]. In the present paper, we explore the implications and challenges that represents the new paradigm
application of modern ML-based solutions for routing opti- of DRL+GNN.
mization in dynamic network scenarios.
Traditional solutions for routing optimization can be mainly II. R ELATED WORK
differentiated in two different categories: (i) Specific-purpose Finding the optimal routing configuration given an estimated
heuristics and (ii) Computationally-intensive mathematical traffic matrix is a fundamental networking problem, which
solvers, which intrinsically are not adequate for efficient real- is known to be NP-hard [11], [12]. This problem has been
time optimization. Heuristic-based solutions (e.g., shortest largely studied in the past and we outline some of the most
path, load balancing) are often sufficiently fast, but they re- relevant works. In DEFO [12], the authors propose a solution
quire expert knowledge to be adapted to specific optimization that converts high-level optimization goals, indicated by the
problems, and often do not adapt well to dynamic network network operator, into specific routing configurations using
scenarios. Solutions based on general-purpose mathematical constraint programming. In [13], the authors propose to use
optimizers (e.g., CPLEX [5]) are often too time-consuming local search, where they sacrifice space exploration to achieve
to achieve real-time operation. This pushes the networking lower execution times. A more recent work [14] leverages the
community to explore new methods and technologies that column generation algorithm and dynamic programming to
can adapt well to the requirements of emerging network solve Traffic Engineering problems.
applications. Recently, numerous DRL-based solutions have been pro-
978-1-6654-4005-9/21/$31.00 ©2021 IEEE posed to solve network optimization problems. In [15], they

Authorized licensed use limited to: San Francisco State Univ. Downloaded on December 13,2025 at 03:06:53 UTC from IEEE Xplore. Restrictions apply.
propose a generic DRL framework for Traffic Engineering. DRL Agent
Their solution is based on a DRL agent that defines the split - Current network state
- Traffic demand Graph Neural Network
ratios of a flow over a set of paths. In the field of optical - Reward
networks, the work [16] proposes an elaborated representation Control plane
of the network state to help a DRL agent learn to route traffic Data plane
demands. A more recent work [7] proposes a different ap- Old
routing
proach where the authors combine DRL and GNN to optimize
the resource allocation in optical circuit-switched networks. - ACTION:
The recent work from [17] proposes a scalable DRL-based New Routing
New policy for the
solution where they use pinning control theory to select critical routing
current traffic
Network state with all
links in the network and optimize their weights assuming a traffic demands allocated demand
weighted shortest path routing policy (e.g., OSPF and ECMP).
Fig. 1: Schematic representation of the DRL agent in the
III. ROUTING O PTIMIZATION SCENARIO routing optimization scenario.
In this paper we explore the challenges when using DRL-
based solutions for routing optimization in dynamic net-
IV. DRL+GNN ROUTING OPTIMIZATION SOLUTION
work scenarios. Particularly, we consider a network optimiza-
tion scenario based on Software-Defined Networking (SDN), We implement a DRL-based solution for routing optimiza-
where the DRL agent (located in the control plane) has a tion that integrates a GNN. We adapt the DRL+GNN solution
global view of the current network state, and has to optimize from [7] for solving the routing problem addressed in the
the routing configuration considering the current traffic matrix. present paper (Sec. III). While the solution from [7] is tailored
This is a complex optimization problem that has been widely for routing optimization in optical networks, in this work we
studied in the past [12], [14]. design a DRL+GNN agent that is tailored for Traffic Engi-
All the source-destination traffic demands are initially al- neering optimization in IP networks. Our agent implements
located using OSPF routing [18] and the DRL agent must the Proximal Policy Optimization (PPO) algorithm [21], which
find a new routing policy that minimizes the utilization of the is a DRL on-policy algorithm based on actor-critic methods,
most loaded link. This is a common scenario where overlay in difference with the off-policy DQN algorithm used in the
technologies (e.g., Segment Routing [19], LISP [20]) are used previous work. We also implement a simulation environment
in the optimization layer [12]. Traffic demands are defined by based on a fluid model, as the one used in [12].
the tuple {src, dst, bw} where src and dst are the source and The DRL agent’s training process is based on a trial-and-
destination nodes respectively and bw represents the average error process. At each time step, the agent receives a network
traffic volume estimated for this demand. The DRL agent state observation from the simulation environment, including
iterates over all demands and it has to decide the new routing the current link’s utilization. Then, the DRL agent uses a
path for each of them. Considering all the possible src-dst GNN to construct an internal graph representation with the
combinations in the network, there is a total of N*(N-1) topology links. Each link has associated a state represented
traffic demands, where N is the number of nodes (i.e., one by a fixed-size vector with some real values (Sec. IV-A). With
demand per src-dst pair). A DRL episode ends when the this representation, a message passing algorithm runs between
agent has iterated over all traffic demands. Figure 1 shows the graph elements (i.e., the links of the network topology)
a schematic representation of the optimization scenario where according to the topology structure, and updates the link states
the DRL agent interacts with the environment. This figure also [22].
illustrates how the DRL agent changes the routing policy of a The outputs of this algorithm (i.e., the new links states)
traffic demand. are aggregated into a global state, which encodes information
Traffic demands do not expire during the episode. This about the whole network, and this global embedding is then
implies a challenging task for the DRL agent, since it has not processed by a fully-connected NN. Particularly, the GNN
only to identify critical resources on networks (e.g., potential outputs a set of probabilities over the possible actions of the
bottlenecks), but also to plan ahead and optimize considering DRL agent. Finally, the output probability distribution is used
future demands. We summarize below the main features of by the DRL agent to sample the action to perform.
the SDN-based routing optimization scenario considered in
this paper: A. Network Simulation Environment
• The DRL agent makes sequential routing decisions for The network state is defined by features on the links, which
every traffic demand. includes the link capacity and the current utilization. These
• It selects per-demand routing paths in order to minimize features are stored in a fixed-size vector padded with zeros.
the most loaded link. At the beginning of a DRL episode, all demands are allocated
• Once a routing decision is made for a traffic demand, it according to the OSPF routing (i.e., shortest path policy). The
cannot be re-routed, at the demand remains until the end links states change as the DRL re-allocates traffic demands to
of the episode. specific src-dst paths (i.e., sequences of links in the network).

Authorized licensed use limited to: San Francisco State Univ. Downloaded on December 13,2025 at 03:06:53 UTC from IEEE Xplore. Restrictions apply.
 hL6 Execute T times
x1 x2 0 … 0 Message passing Update
hL1
x1 x2 0 … 0
4 5 For all neighbors
hL1 of link hL1
hL1
hL3 M + RNN
hL2 Action
x1 x2 0 … 0
1 . M . . Probability
hL2 ... +
hL4 . . . . Readout
x1 x2 7 … 0 .
x1 x2 7 … 0
.
.
. .
hLK M
2 x1 x2 7 … 0
3 hLK
M + RNN
hL5 ...
For all neighbors
Fig. 2: Action representation in the link states. Nodes 1 and 5 of link hLK
are the source and destination nodes respectively, and node 3
is the middlepoint selected. Fig. 3: Message passing architecture.

C. GNN architecture
The final goal is to minimize the maximum link utilization at
The GNN we implemented is based on the Message Passing
the end of the episode (i.e., when the DRL agent has iterated
Neural Network model [22]. In our case, we consider the
over all the traffic demands).
network links as the graph entities and perform an iterative
message passing process between adjacent links in the topol-
B. Action space ogy. This process consists of combining the state of all links
with those of their neighbours in the graph. Particularly, the
The number of possible routing combinations for each traffic states of connected links are processed by a fully-connected
demand (i.e., src-dst node pairs) results in a high dimensional NN. Then, the results (called messages) are aggregated for
state/action space, even in small networks [23]. This makes the each link using an element-wise summation. Finally, the links’
routing problem complex for the DRL agent, since it should hidden states are individually updated using a Recurrent Neu-
estimate for each action which is the one that will lead to lower ral Network (RNN), whose input is the previous link state and
maximum utilization in the long-term. In other words, which the new aggregated message. This message passing process is
is the optimal routing configuration per traffic demand that performed T times and, at the end of this phase, the resulting
leads to the minimum maximum link utilization. Moreover, to link states are aggregated using an element-wise summation.
exploit the generalization capabilities of GNNs over graphs, The result is then a general graph embedding that is passed
we need to define the action in a way that is invariant to edge through a fully-connected NN, which models the Readout
and node permutation, using only link-level features rather function of the GNN. The output of this latter function is the
than specific identifiers or labels. action probability. Thus, given a traffic demand, this process is
To overcome these problems, we use the middlepoint rout- executed for each action (or possible middlepoint) and the final
ing (MR) model [12] to define the action space. This model action of the DRL agent is sampled over the individual actions’
abstracts the routing paths between nodes as middlepoints, probabilities. Figure 3 represents the internal architecture of
where the paths are pre-configured. In our paper, the pre- the GNN, including the iterative message passing phase and
configured paths between nodes and middlepoints are com- the final readout.
puted using the OSPF routing policy. Given a traffic demand
with an associated src-dst node pair, we have as many different D. DRL Agent operation
actions as number of middlepoints. Particularly, we limited the The DRL agent operates by interacting with the network
options to selecting only one middlepoint. This means that simulation environment (Sec. IV-A), and the learning process
to reach a destination node dst, the traffic demand can only is implemented using PPO [21]. First, we collect experiences
cross at most one middlepoint before reaching the destination from the DRL agent interaction with the environment. Then,
node. In our DRL agent, all the nodes except the source node using the collected experiences, we improve the current policy
can be middlepoint nodes, and src-dst traffic demands can be weights using mini-batch updates. Afterwards, the old experi-
directed through specific middlepoints using Segment Routing, ences are removed and we collect new experiences using the
as in [12]. new policy weights. This process is repeated until convergence
To represent the action, we add an additional feature in the or a pre-defined criteria is reached.
link’s state (see Figure 2). This feature includes the traffic At the beginning of an episode, all traffic demands are
volume of the current traffic demand bw in case the link is allocated using OSPF routing. Then, the DRL agent iterates
within the selected path. Otherwise, the links that are not over all traffic demands and tries to change their current
included in the path have this feature set to zero. Features x1 routing configuration. For each traffic demand, the DRL agent
and x2 from Figure 2 are the link capacity and link utilization evaluates the action of re-allocating the demand on each possi-
respectively. In this example, the traffic to allocate of the ble middlepoint using the GNN. Notice that each middlepoint
current demand is 7. corresponds to an action and the traffic demand cannot be

Authorized licensed use limited to: San Francisco State Univ. Downloaded on December 13,2025 at 03:06:53 UTC from IEEE Xplore. Restrictions apply.
 1.2 1.2
B. Methodology
Max Link Utilization

Max Link Utilization

0.9
1.0 GEANT2 [29] topologies. The traffic demands were generated
0.8 0.9 using a realistic gravity model from [30]. We consider four
0.7 0.8
possible different values for the link capacities for the NSFNet
0.6
0.7
topology and four different ones for the GBN and GEANT2
OSPF DRL+GNN SAP Simulated OSPF DRL+GNN SAP Simulated topologies. Specifically, the NSFNet has 5K, 10K, 15K and
Annealing Annealing
20K as possible link capacities, and GBN and GEANT2 have
(a) Evaluation on GBN (b) Evaluation on GEANT2 25K, 50K, 75K and 100K possible link capacities. We set the
1.0
OSPF
1.0
OSPF
capacity individually on each link to ensure that the maximum
DRL+GNN DRL+GNN
0.8 SAP 0.8 SAP link utilization is between 0.95 and 1.3 when applying the
0.6 0.6 OSPF routing policy.
CDF

CDF

0.4 0.4
We compare the DRL’s agent performance against two base-
Better Better lines. The first one is the Shortest Available Path (SAP). This
0.2 0.2
consists on starting with an empty network, and iterate over the
0.0 0.0
−1.0 −0.8 −0.6 −0.4 −0.2 0.0 0.2 0.4 −1.0 −0.8 −0.6 −0.4 −0.2 0.0 0.2 0.4 demands to allocate them on the shortest path that has more
Relative performance to Relative performance to
Simulated Annealing Simulated Annealing available capacity [23]. The second baseline we implemented
(c) Evaluation on GBN (d) Evaluation on GEANT2
is based on the Simulated Annealing [31] algorithm. Starting
from an initial routing configuration using OSPF, the algorithm
Fig. 4: DRL+GNN evaluation on GBN (a) and GEANT2 (b) performs an iterative process trying to decrease the initial
using 50 TMs. state energy (i.e., minimize the maximum link utilization).
In each step, the algorithm explores all possible next states
split between different middlepoints. Then, the GNN outputs
and it decides whether to move to a new state or stay in
a probability over each action, and the action performed by the
the same using a temperature parameter. This parameter will
agent is sampled from the resulting probability distribution of
guide the algorithm to move to states with lower maximum
all actions. The chosen action is then applied to the network,
link utilization. As more iterations the algorithm performs,
leading to a new state (i.e., new links utilization), a reward,
more chances it will have to approach to the global optimum.
and a flag indicating if the agent finished iterating over all
Therefore, we parameterized the algorithm to perform 4·106
traffic demands. We set the reward to be the difference in the
steps. This makes Simulated Annealing not suitable for real-
maximum link utilization between two steps.
time operation since it’s cost (in time) is too high for finding
V. E XPERIMENTAL R ESULTS good routing configurations. Specifically, Simulated Annealing
A. Evaluation setup spent around 30 minutes and 24 minutes for optimizing the
GEANT2 and GBN topologies respectively.
We implemented the DRL+GNN solution of Section IV
We trained the DRL+GNN agent solely on the 14-node
using Tensorflow [24] and evaluated it on a network simulator
NSFNet topology [27]. During the training process, we used
implemented using the OpenAI Gym framework [25]. All the
100 different Traffic Matrices (TM) and we evaluated the agent
experiments were executed on off-the-shelf hardware without
performance on 25 TMs never seen during training.
any specific hardware accelerator (Ubuntu 20.04.1 LTS with
processor AMD Ryzen 9 3950X 16-Core Processor). The
C. Evaluation
simulation environment is configured by initializing the link
features for each topology with the respective link capacities. In the evaluation experiments, we compare our DRL+GNN
All paths between nodes and middlepoints are pre-computed agent against the baseline solutions. Particularly, we pick the
using OSPF, considering equal weights for all the links. model with more performance during the training phase, and
The traffic demands are initially allocated following the pre- evaluate it in two real-world network topologies: GBN (17
computed paths. Then, the environment sorts the demands nodes) and GEANT2 (24 nodes). We used 50 different TMs
in decreasing order of traffic volume, and the DRL agent to evaluate the performance for each topology.
sequentially re-routes them using MR. In Figures 4a and 4b we can observe the evaluation results
Initial experiments were performed to choose an appropri- on both networks. Particularly, these figures show the maxi-
ate gradient-based optimization algorithm and hyperparameter mum link utilization for the different optimization strategies.
values for the DRL+GNN architecture. We defined the links’ The OSPF label corresponds to the initial network state before
hidden states hl as 20-element vectors (filled initially with optimization (i.e., shortest path routing). Likewise, in Figures
the link features as shown in Figure 2). In every execution 4c and 4d we plot the CDFs of the previous experiments, but
of the GNN, we run T=5 message passing steps, and use in this case in relative values w.r.t. the Simulated Annealing
mini-batches of 50 samples. The optimizer used is the Adam baseline. Note that Simulated Annealing represents a near-
Optimizer [26], with an initial learning rate of 2·10−4 and optimal solution that is not suitable for real-time operation as
following an exponential learning rate decay during training. it has a very high computational cost.

Authorized licensed use limited to: San Francisco State Univ. Downloaded on December 13,2025 at 03:06:53 UTC from IEEE Xplore. Restrictions apply.
 The experimental results indicate that our DRL+GNN agent production network, without requiring any further training or
outperforms the baseline heuristics (OSPF and SAP) on instrumentation of the network.
topologies not seen during the training phase (17 and 24 2) Action space: The definition of the DRL agent’s action
nodes), after being trained in a single 14-node topology. space is fundamental to achieve a good optimization solution.
Specifically, the DRL+GNN agent reduces the maximum link An inadequate action space might explode as the topology
utilization in average by ≈21.7 % and ≈14 % w.r.t. OSPF size grows, having a direct impact in the complexity of the
in the GBN and GEANT2 topologies respectively, while SAP optimization problem to be addressed. Thus, it is important to
only reduces it by ≈6.8 % and ≈10 %. This reveals the ability find good network abstractions that can limit the dimension
of our our DRL+GNN solution to adapt to different network of the action space in order to facilitate the learning process
scenarios – even to new network topologies –, which is a to the DRL agent. In addition, the action space should offer
essential property to operate in highly dynamic networks (e.g., enough flexibility and expressiveness to enable the finding of
traffic changes, link failures, new nodes), such as vehicular good routing configurations that lead to a high optimization
networks [1]. In addition, the DRL+GNN agent was able to performance.
operate efficiently in scenarios with different link capacities In our work we leveraged MR [12] to limit the action space
than in the training scenario. This indicates that the DRL agent dimension (see Sec. IV-B). We consider that MR offers a good
is able to adapt to different network topologies with different level of abstraction to represent routing, but other methods can
link features than those seen during training. be used to define DRL’s agent action space. For each traffic
We also measured the average optimization time that each demand, we have as many actions as possible middlepoints.
baseline spends on optimizing all the traffic demands. For the This means that the maximum number of actions are N-
GBN topology we obtained an average time (in seconds) of 1, where N is the number of nodes (i.e., the action space
≈1476 s for the Simulated Annealing, ≈4 s for the DRL+GNN grows linearly with the topology size). At the same time,
architecture and ≈0.05 s for the SAP heuristic. The optimiza- this abstraction provides enough flexibility to achieve a near-
tion times for the GEANT2 topology are ≈1873 s for the optimal performance, as in [12].
Simulated Annealing, ≈12 s for the DRL+GNN and ≈0.15 s 3) Training cost: The training cost of the DRL agent
for the SAP. Notice that the larger is the topology (in number must be taken into account, especially when it is trained in
of nodes), the more traffic demands need to be optimized. This large topologies. The DRL training process is, by definition,
makes the problem complexity grow with the topology size. extremely sequential. This significantly hinders the paralleliza-
However, DRL is inherently a technology based on NNs whose tion of the learning process. New methods should be explored
operation process can be easily parallelized using commodity to scale DRL solutions to larger optimization scenarios. For
hardware accelerators (e.g., GPUs). In consequence, DRL is example, these methods could be to accelerate the training
key on enabling real-time network optimization. process [32], transfer learning [33], or problem reduction [17].
Besides the challenges mentioned previously, there are some
VI. D ISCUSSION AND O PEN C HALLENGES open challenges that require further attention by the research
community and here we mention some of them. One of
In this section we discuss which are the main challenges the open challenges is that DRL doesn’t offer any kind of
that need to be addressed to enable the operation of DRL on performance bounds. In other words, once a DRL agent is
dynamic networks. We consider that the most relevant aspects trained, there is no way to know exactly which are the
to address are generalization, action space design, and training upper or lower bounds in optimization performance. However,
cost. this problem is also present in many existing heuristic-based
1) Generalization: We argue that generalization is an es- solutions.
sential property for the successful adoption of DRL technolo-
gies in networks with dynamic topologies. In this context, VII. C ONCLUSION
generalization refers to the ability of the DRL agent to adapt to In this paper, we explored a DRL agent based on GNNs
new network scenarios not seen during training (e.g., network for solving a routing optimization problem. To do this, we
topologies, traffic, configurations). With generalization, a DRL trained a DRL agent in a single topology and we evaluated it
agent can be trained with a representative set of network on two topologies never seen before. The experimental results
topologies and configurations, and afterwards be applied to show that the DRL agent is able to operate unseen network
other scenarios never seen before. configurations while still obtaining a high performance. These
In our work, we addressed generalization by designing results indicate that the DRL agent is able to adapt to different
a GNN-based DRL agent. In our experimental results, we network topologies, which is an essential property to operate
have seen that our DRL+GNN agent is able to operate highly dynamic networks. In addition, the DRL agent has
successfully in network topologies never seen before. From a low optimization cost, in the scale of seconds, and it’s
a commercialization standpoint, such “universal” DRL agent performance is comparable to near-optimal iterative solutions.
can be trained in a laboratory and later on be incorporated This makes DRL a key technology to enable modern network
in a product or a network device (e.g., router, load balancer). applications that require real-time efficient operation in highly
The resulting solution would be ready to be deployed in a dynamic network topologies.

Authorized licensed use limited to: San Francisco State Univ. Downloaded on December 13,2025 at 03:06:53 UTC from IEEE Xplore. Restrictions apply.
 ACKNOWLEDGMENT [16] J. Suárez-Varela, A. Mestres, J. Yu, L. Kuang, H. Feng, P. Barlet-Ros,
and A. Cabellos-Aparicio, “Feature engineering for deep reinforcement
This work has received funding from the European Union’s learning based routing,” in ICC 2019-2019 IEEE International Confer-
ence on Communications (ICC). IEEE, 2019, pp. 1–6.
Horizon 2020 research and innovation programme within the [17] P. Sun, J. Lan, J. Li, J. Zhang, Y. Hu, and Z. Guo, “A scalable deep
framework of the NGI-POINTER Project funded under grant reinforcement learning approach for traffic engineering based on link
agreement No 871528. This paper reflects only the author’s control,” IEEE Communications Letters, 2020.
[18] J. Moy, “Rfc2328: Ospf version 2,” 1998.
view; the European Commission is not responsible for any [19] C. Filsfils, N. K. Nainar, C. Pignataro, J. C. Cardona, and P. Francois,
use that may be made of the information it contains. This “The segment routing architecture,” in 2015 IEEE Global Communica-
work was also supported by the Spanish MINECO under tions Conference (GLOBECOM). IEEE, 2015, pp. 1–6.
[20] A. Rodriguez-Natal, M. Portoles-Comeras, V. Ermagan, D. Lewis,
contract TEC2017-90034-C2-1-R (ALLIANCE), the Catalan D. Farinacci, F. Maino, and A. Cabellos-Aparicio, “Lisp: a southbound
Institution for Research and Advanced Studies (ICREA) and sdn protocol?” IEEE Communications Magazine, vol. 53, no. 7, pp.
the Secretariat for Universities and Research of the Ministry 201–207, 2015.
[21] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox-
of Business and Knowledge of the Government of Catalonia imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347,
and the European Social Fund. 2017.
[22] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl,
“Neural message passing for quantum chemistry,” in Proceedings of the
R EFERENCES International Conference on Machine Learning (ICML) - Volume 70,
2017, pp. 1263–1272.
[1] S. Zeadally, R. Hunt, Y.-S. Chen, A. Irwin, and A. Hassan, “Vehicular [23] J. Suárez-Varela, A. Mestres, J. Yu, L. Kuang, H. Feng, A. Cabellos-
ad hoc networks (vanets): status, results, and challenges,” Telecommu- Aparicio, and P. Barlet-Ros, “Routing in optical transport networks with
nication Systems, vol. 50, no. 4, pp. 217–241, 2012. deep reinforcement learning,” IEEE/OSA Journal of Optical Communi-
[2] I. Yaqoob, I. A. T. Hashem, Y. Mehmood, A. Gani, S. Mokhtar, and cations and Networking, vol. 11, no. 11, pp. 547–558, 2019.
S. Guizani, “Enabling communication technologies for smart cities,” [24] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,
IEEE Communications Magazine, vol. 55, no. 1, pp. 112–120, 2017. S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large-
[3] F. Alagoz, O. Korcak, and A. Jamalipour, “Exploring the routing scale machine learning,” in Proceedings of the 12th USENIX Symposium
strategies in next-generation satellite networks,” IEEE Wireless Com- on Operating Systems Design and Implementation (OSDI), 2016, pp.
munications, vol. 14, no. 3, pp. 79–88, 2007. 265–283.
[4] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wier- [25] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schul-
stra, and M. Riedmiller, “Playing atari with deep reinforcement learn- man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint
ing,” arXiv preprint arXiv:1312.5602, 2013. arXiv:1606.01540, 2016.
[5] “IBM CPLEX optimizer,” [Link] [26] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
optimizer. arXiv preprint arXiv:1412.6980, 2014.
[6] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, [27] X. Hei, J. Zhang, B. Bensaou, and C.-C. Cheung, “Wavelength converter
S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel et al., placement in least-load-routing-based optical networks using genetic
“Mastering atari, go, chess and shogi by planning with a learned model,” algorithms,” Journal of Optical Networking, vol. 3, no. 5, pp. 363–378,
Nature, vol. 588, no. 7839, pp. 604–609, 2020. 2004.
[7] P. Almasan, J. Suárez-Varela, A. Badia-Sampera, K. Rusek, P. Barlet- [28] J. Pedro, J. Santos, and J. Pires, “Performance evaluation of integrated
Ros, and A. Cabellos-Aparicio, “Deep reinforcement learning meets otn/dwdm networks with single-stage multiplexing of optical channel
graph neural networks: Exploring a routing optimization use case,” arXiv data units,” in 2011 13th International Conference on Transparent
preprint arXiv:1910.07421, 2019. Optical Networks. IEEE, 2011, pp. 1–4.
[8] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfar- [29] F. Barreto, E. C. Wille, and L. Nacamura Jr, “Fast emergency paths
dini, “The graph neural network model,” IEEE transactions on neural schema to overcome transient link failures in ospf routing,” arXiv
networks, vol. 20, no. 1, pp. 61–80, 2008. preprint arXiv:1204.2465, 2012.
[9] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zam- [30] M. Roughan, “Simplifying the synthesis of internet traffic matrices,”
baldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner ACM SIGCOMM Computer Communication Review, vol. 35, no. 5, pp.
et al., “Relational inductive biases, deep learning, and graph networks,” 93–96, 2005.
arXiv preprint arXiv:1806.01261, 2018. [31] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by
[10] J. Suárez-Varela, S. Carol-Bosch, K. Rusek, P. Almasan, M. Arias, simulated annealing,” science, vol. 220, no. 4598, pp. 671–680, 1983.
P. Barlet-Ros, and A. Cabellos-Aparicio, “Challenging the generaliza- [32] A. Stooke and P. Abbeel, “Accelerated methods for deep reinforcement
tion capabilities of graph neural networks for network modeling,” in learning,” arXiv preprint arXiv:1803.02811, 2018.
Proceedings of ACM SIGCOMM Posters and Demos, 2019, pp. 114– [33] R. Glatt, F. L. Da Silva, and A. H. R. Costa, “Towards knowledge trans-
115. fer in deep reinforcement learning,” in 2016 5th Brazilian Conference
[11] D. Xu, M. Chiang, and J. Rexford, “Link-state routing with hop-by- on Intelligent Systems (BRACIS). IEEE, 2016, pp. 91–96.
hop forwarding can achieve optimal traffic engineering,” IEEE/ACM
Transactions on networking, vol. 19, no. 6, pp. 1717–1730, 2011.
[12] R. Hartert, S. Vissicchio, P. Schaus, O. Bonaventure, C. Filsfils,
T. Telkamp, and P. Francois, “A declarative and expressive approach to
control forwarding paths in carrier-grade networks,” ACM SIGCOMM,
p. 15–28, 2015.
[13] S. Gay, R. Hartert, and S. Vissicchio, “Expect the unexpected: Sub-
second optimization for segment routing,” in IEEE Conference on
Computer Communications (INFOCOM). IEEE, 2017, pp. 1–9.
[14] M. Jadin, F. Aubry, P. Schaus, and O. Bonaventure, “Cg4sr: Near optimal
traffic engineering for segment routing with column generation,” in
IEEE Conference on Computer Communications (INFOCOM), 2019,
pp. 1333–1341.
[15] Z. Xu, J. Tang, J. Meng, W. Zhang, Y. Wang, C. H. Liu, and D. Yang,
“Experience-driven networking: A deep reinforcement learning based
approach,” in IEEE Conference on Computer Communications (INFO-
COM), 2018, pp. 1871–1879.

Authorized licensed use limited to: San Francisco State Univ. Downloaded on December 13,2025 at 03:06:53 UTC from IEEE Xplore. Restrictions apply.