Reinforcement Learning Based Routing in Networks: Review and Classification

of Approaches

Authors

NAME ID
Abdulaziz Abdulrabb 202109135
Abdulla Ahmed Hassan Hurubi 202100883
Jaber salman jathmi 201909664
Reinforcement Learning Based Routing
ABSTRACT Reinforcement learning (RL), which is a class of machine learning, provides a framework by
which a system can learn from its previous interactions with its environment to efficiently select its actions
in the future. RL has been used in a number of application fields, including game playing, robotics and
control, networks, and telecommunications, for building autonomous systems that improve themselves with
experience. It is commonly accepted that RL is suitable for solving optimization problems related to
distributed systems in general and to routing in networks in particular. RL also has reasonable overhead—in
terms of control packets, memory and computation—compared to other optimization techniques used to
solve the same problems. Since the mid-1990s, over sixty protocols have been proposed, with major or
minor contributions in the field of optimal route selection to convey packets in different types of
communication networks under various user QoS requirements. This paper provides a comprehensive
review of literature on the topic. The review is structured in a way that shows how network characteristics
and requirements were gradually considered over time. Classification criteria are proposed to present and
qualitatively compare existing RL-based routing protocols.

I. INTRODUCTION
Machine learning (ML) is a field of computer science
and statistics that encompasses a set of algorithms and
methods that learn from datasets and are capable of
making predictions or helping to make them [1], [2], [3].
Nowadays, ML has a leading role in computerized
societies. It is very likely that almost all future devices
and machines will include ML-based components to
improve their operation management and adapt
themselves to their environment. ML is a powerful tool
to address complex problems. Widely used in image and
speech recognition, robot guidance, autonomous car
guidance, telecommunication, and many other sectors,
ML techniques proved their efficiency. For tasks such as
classification and optimization, ML are known to (often)
produce better results than human beings.
ML techniques are categorized in four classes based on
how learning is carried out [1], [2]: supervised,
unsupervised, semi-supervised, and reinforcement. In
supervised learning—called learning with teacher—
input and output variables are used to learn the mapping
function from input to output; the goal is to approximate
the mapping function in such a way that an output (also
called label) can be accurately predicted from its
associated input. In unsupervised learning—called
learning without teacher— only input is used; the goal is
to model the structure or distribution (e.g. data
clustering) in the data in order to learn specific
characteristics about data. Semi-supervised learning is
similar to supervised one, but not all observations have
labels (outputs). Finally, reinforcement learning is a
technique inspired by the behavioral psychology and it
provides system modeling based on
 Example reward criteria:
Working Mechanism of RTR
The RTR algorithm uses Reinforcement Learning Positive reward for low-energy consumption or
principles (e.g., Q-learning) to dynamically decide reduced delay.
the best routes for data transmission. Below is a Negative reward for high-energy consumption,
step-by-step explanation of its working mechanism: packet loss, or link failure.
Learning and Decision-making:
1. Cluster Formation The RL agent (e.g., a Cluster Head) updates its Q-
Purpose: To group nodes into clusters for efficient values (or policies) based on rewards received.
communication and resource management. Over time, the agent learns the most energy-
Mechanism: efficient and reliable routes for data transmission.
Nodes in the network are grouped into clusters 4. Energy Considerations
based on proximity, energy levels, or connectivity. Minimizing Energy Consumption:
A Cluster Head (CH) is selected for each cluster, RTR prioritizes routes that minimize energy
typically based on parameters such as: consumption by avoiding nodes with low residual
Residual energy. energy.
Node connectivity (number of neighbors). Cluster Heads are rotated periodically to prevent
Proximity to other nodes. energy depletion of a single node.
Ordinary nodes communicate with their respective Balancing Energy Load:
Cluster Head, reducing communication overhead. The algorithm ensures that energy consumption is
2. Node Communication balanced across the network to extend its overall
Intra-cluster Communication: lifetime.
Nodes within a cluster communicate with the Nodes with higher energy levels are preferred for
Cluster Head using short-range communication. cluster formation and routing roles.
This minimizes energy consumption as nodes do 5. Data Routing Process
not directly communicate with the base station or Data routing proceeds in the following steps:
other clusters. Cluster Formation:
Inter-cluster Communication: Nodes are grouped into clusters, and Cluster Heads
Cluster Heads are responsible for forwarding the are selected.
aggregated data to the next hop (another CH or the Intra-cluster Communication:
base station). Sensor nodes send their data to their respective
The routing decision is made dynamically based on Cluster Head.
the Reinforcement Learning model. Inter-cluster Communication:
3. Reinforcement Learning for Routing Cluster Heads use the RL algorithm to decide the
Core Principle: Nodes or Cluster Heads use RL best next hop (another Cluster Head or the base
techniques (e.g., Q-learning) to learn the best station).
routing paths based on feedback from the Base Station Communication:
environment. Data is finally forwarded to the base station,
Steps in RL-based Routing: completing the routing process.
State Representation:
The state is defined by the current condition of the
network, such as:
Residual energy of nodes.
Distance between nodes.
Traffic load on a node or link.
Number of neighboring nodes.
Actions:
Actions represent potential next hops in the routing
process (e.g., selecting a neighboring node or
Cluster Head as the next hop).
Reward Function:
A reward is assigned to each action based on how
beneficial it is for the routing process.
2

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but
republication/redistribution requires IEEE permission. See [Link] for more information.
  Example Applications:
real-world use cases and applications o Remote Patient Monitoring:
Tracking vital signs like heart rate,
1. Wireless Sensor Networks (WSNs) blood pressure, and glucose levels.
Scenario: WSNs are used in applications like o Emergency Alerts: Routing critical
environmental monitoring, disaster management, data immediately to doctors or
and industrial automation. emergency responders.
How RTR is Used:
o Clusters sensors to optimize
communication and reduce energy
consumption.
o Learns the most energy-efficient and
reliable routes to transmit data from
sensors to a central base station.
 Example Applications:
o Agriculture: Monitoring soil moisture,
temperature, and humidity.
o Wildlife Tracking: Routing data from
sensors in remote and energy-
constrained environments.

2. Internet of Things (IoT) Networks

 Scenario: IoT devices often operate in
environments with limited energy and
bandwidth, requiring efficient data routing.
 How RTR is Used:
o Optimizes data transmission between
IoT devices and cloud servers by
dynamically learning the best paths.
o Ensures load balancing and energy
efficiency in smart homes, cities, and
industrial IoT.
 Example Applications:
o Smart Homes: Routing data between
connected devices like thermostats,
security cameras, and lighting systems.
o Smart Cities: Traffic monitoring, waste
management, and energy distribution.


3. Healthcare Monitoring
Systems
 Scenario: Healthcare applications involve
wearable devices and medical sensors that
collect patient data in real time.
 How RTR is Used:
o Clusters wearable devices to reduce
communication overhead and extend
battery life.
o Dynamically routes data to ensure
reliable delivery to healthcare
providers or cloud storage.

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but
republication/redistribution requires IEEE permission. See [Link] for more information.
 Advantages of RTR
1. Energy Efficiency
 Description:
o RTR actively selects routes that Disadvantages of RTR
minimize energy consumption by 1. Computational Complexity
considering the energy levels of  Description:
nodes and the cost of o Learning algorithms, such as Q-
communication. learning or other reinforcement
o Balances energy usage across the learning techniques, require
network, avoiding overuse of specific significant computation to maintain
nodes and extending the overall and update Q-values or policies for
network lifetime. each node.
 Impact: o The computational burden increases
o Particularly beneficial in energy- with the size of the network and the
constrained networks such as number of states and actions.
Wireless Sensor Networks (WSNs),  Impact:
Internet of Things (IoT) devices, and o Resource-constrained devices, such
UAV networks. as sensors or IoT nodes, may
struggle to handle the additional
2. Adaptability computational overhead.
 Description:
o RTR dynamically learns and 2. Learning Delay
updates routing policies based on  Description:
changes in the network, such as node o RTR requires time to "learn" the
failures, topology changes, or traffic optimal routing paths, especially in
congestion. the early stages of deployment when
o This makes RTR highly suitable for RL agents have limited knowledge of
dynamic networks like Mobile Ad the network.
Hoc Networks (MANETs) and o During this learning phase,
Vehicular Ad Hoc Networks suboptimal routing decisions may
(VANETs). lead to increased energy
 Impact: consumption or delays.
o Ensures reliable data delivery even  Impact:
in unpredictable and rapidly o May not be suitable for real-time
changing environments. applications or networks where
immediate optimal performance is
3. Scalability required.
 Description:
o RTR can handle large-scale 3. Communication Overhead
networks due to its decentralized  Description:
learning approach and clustering o RTR involves frequent exchange of
mechanisms. information, such as state updates,
o As the network grows, RTR rewards, or Q-values, between nodes
continues to make routing decisions or clusters.
based on localized information while o As the network scales, this overhead
maintaining global optimization. can grow, consuming bandwidth and
 Impact: increasing communication costs.
o Suitable for large IoT deployments,  Impact:
such as smart cities or industrial o Can reduce efficiency in bandwidth-
IoT. constrained networks or dense
deployments.

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but
republication/redistribution requires IEEE permission. See [Link] for more information.
 Comparison of RTR (Reinforcement Learning-based Routing) with LEACH (Low-Energy Adaptive
Clustering Hierarchy)

o 1. Performance
o Aspect o RTR o LEACH
o RTR dynamically learns
and adjusts routes using o LEACH uses a fixed
feedback from the clustering mechanism,
environment. It can adapt which might not adapt
o Adaptability
to changing network well to dynamic
conditions, such as node changes in the
failures or topology network.
changes.
o Uses Reinforcement o LEACH does not use
Learning (e.g., Q- learning mechanisms.
Learning) to optimize It relies on a
o Learning and
routing paths based on probabilistic cluster
Optimization
rewards, ensuring head selection, which
continuous improvement may not always yield
in performance. optimal results.
o Routes are selected based o Routing decisions are
on factors like energy, static within clusters
o Routing traffic, and node and depend solely on
Decisions availability, leading to the cluster head,
highly optimized which may lead to
communication paths. suboptimal routing.

2. Efficiency

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but
republication/redistribution requires IEEE permission. See [Link] for more information.
 Aspect RTR LEACH
Energy RTR prioritizes paths LEACH rotates cluster
Efficiency with lower energy heads randomly to
consumption and balance energy
balances energy usage consumption, but this
across the network. randomness can lead to
Cluster heads or nodes inefficient energy usage if
with high energy are low-energy nodes become
favored for routing. cluster heads.
Requires additional
computation and LEACH has lower
communication overhead overhead since it uses a
Overhead for reinforcement simpler, probabilistic
learning updates (e.g., Q- cluster-based routing
value updates), which can mechanism.
be resource-intensive.
Handles dense networks
Works efficiently in small
effectively by learning
to moderately sized
Scalability optimal routes
networks but struggles in
in Dense dynamically, but
very dense networks due
Networks computation costs can
to high intra-cluster
increase with network
communication overhead.
size.

3. Scalability
Aspect RTR LEACH
Handling Large RTR can scale to large LEACH scales poorly
Networks networks by in large networks

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but
republication/redistribution requires IEEE permission. See [Link] for more information.
 Aspect RTR LEACH
decentralizing decision-
because random
making and using
cluster head selection
clustering techniques.
can lead to uneven
However, as the network
energy distribution
grows, reinforcement
and communication
learning computations
bottlenecks.
may become slower.
LEACH struggles
RTR excels in dynamic
with dynamic
topologies (e.g., mobile
topologies due to its
Dynamic ad hoc networks) by
fixed clustering
Topologies continuously updating
structure, which may
routes based on real-
become invalid when
time conditions.
nodes move or fail.
LEACH's
Communication communication
overhead increases with overhead is
Communication
the number of nodes, proportional to the
Overhead
especially during the number of clusters
learning phase. and is independent of
network dynamics.

Conclusion
RTR leverages Reinforcement Learning to intelligently route data in energy-constrained
networks. By dynamically learning and adapting to the network state, RTR ensures efficient data
transmission while balancing energy consumption and prolonging the network's lifetime. Its
combination of cluster-based communication and RL-based routing makes it a robust solution for
modern IoT and WSN applications.

References
1. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.
o A foundational book on reinforcement learning, explaining key principles such as Q-learning and
policy optimization, which are critical for routing algorithms like RTR.
2. Heinzelman, W. B., Chandrakasan, A., & Balakrishnan, H. (2000). "Energy-efficient communication
protocol for wireless microsensor networks." Proceedings of the 33rd Annual Hawaii International
Conference on System Sciences, Maui, HI, USA, pp. 1-10.
7

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but
republication/redistribution requires IEEE permission. See [Link] for more information.
 oIntroduced LEACH, a foundational protocol for energy-efficient communication, providing a
basis for comparison with RTR.
3. Raza, S., Javaid, N., Khan, M. A., & Alrajeh, N. (2016). "Q-LEACH: A New Routing Protocol for WSNs."
Procedia Computer Science, vol. 19, pp. 926-931.
o Discusses a reinforcement learning-based improvement to LEACH, focusing on clustering and
energy optimization in wireless sensor networks.
4. Li, H., & Wang, Y. (2019). "A Reinforcement Learning-Based Routing Protocol with QoS Guarantee for
Industrial Internet of Things." IEEE Internet of Things Journal, vol. 6, no. 3, pp. 6242-6252.
o Explores reinforcement learning-based routing for IoT applications, emphasizing energy efficiency
and quality of service.
5. Tan, X., & Jiang, C. (2020). "Multi-Agent Deep Reinforcement Learning for Dynamic Routing Decision-
Making in VANETs." IEEE Transactions on Vehicular Technology, vol. 69, no. 7, pp. 7780-7795.
o Demonstrates the use of reinforcement learning for dynamic routing in vehicular ad hoc networks,
highlighting RTR's adaptability in mobile and dynamic topologies.
6. Kumar, P., & Kumar, S. (2021). "Energy-Efficient and Adaptive Routing Protocol Using Reinforcement
Learning for Wireless Sensor Networks." Wireless Personal Communications, vol. 121, pp. 1-23.
o Details an RL-based routing protocol, focusing on improving energy efficiency and network
lifetime in WSNs.
7. Kaur, J., & Singh, M. (2022). "Reinforcement Learning-Based Routing in Mobile Ad Hoc Networks."
Journal of Communications and Networks, vol. 24, no. 2, pp. 150-164.
o Explores reinforcement learning in MANETs, discussing RTR's scalability and adaptability in
highly dynamic networks.
8. Kurose, J. F., & Ross, K. W. (2021). Computer Networking: A Top-Down Approach (8th ed.). Pearson.
o Provides a comprehensive overview of networking concepts, including routing protocols and their
applications in modern networks.
9. IEEE Communications Society. "Reinforcement Learning Applications in Networking."
o Overview of RL applications for solving networking challenges, including routing and resource
allocation. Accessible via [Link]
10. Reddy, T., & Ramesh, P. (2020). "Reinforcement Learning in IoT: Applications and Case Studies."
SpringerLink.
o Highlights reinforcement learning techniques applied to IoT networks, including routing protocols
and energy optimization.

2169-3536 (c) 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but
republication/redistribution requires IEEE permission. See [Link] for more information.