Title: Deep Q-Learning-Based Handover Optimization Framework for
5G Ultra-Dense Networks
Abstract: In ultra-dense 5G cellular networks, seamless mobility
management remains a critical challenge due to high user density,
dynamic channel conditions, and rapid handover events. Traditional
rule-based handover mechanisms—such as A3 event-based
algorithms—are prone to frequent handovers, increased signaling
overhead, and ping-pong effects. This paper proposes a Deep Q-
Learning (DQL)-based framework for intelligent handover
optimization. The model is trained in a simulated environment
incorporating realistic 5G characteristics such as gNB load, signal
propagation, user mobility, and Xn-based inter-gNB communication.
The agent utilizes a composite reward function that balances
throughput gain, handover stability, and interference management.
Experimental results demonstrate that the proposed DQL method
significantly improves throughput, reduces ping-pong handovers, and
achieves superior load balancing compared to conventional methods.
1. Introduction
The fifth-generation (5G) wireless communication standard
introduces massive improvements in network capacity, latency, and
data rates. One key enabling feature is the deployment of dense small
cells, which significantly improves spectral efficiency. However, ultra-
dense deployments result in an increased number of handovers, which
can lead to service degradation and elevated signaling overhead.
Traditional handover algorithms based on Reference Signal Received
Power (RSRP) thresholds and fixed hysteresis values do not adapt well
to varying environmental conditions and user mobility.
This paper addresses the handover decision problem using Deep
Reinforcement Learning (DRL), specifically Deep Q-Learning (DQL), to
learn optimal policies for handover execution. Unlike deterministic
approaches, DRL enables the agent to make context-aware decisions
by continuously interacting with the environment and optimizing a
cumulative reward function.
2. System Model and Implementation (Python Code Explanation)
2.1 Cellular Environment Simulation – Class: CellularEnvironment
The CellularEnvironment class models a 5G urban network environment
in Python. The constructor __init__ initializes the simulation space,
number of gNBs and UEs, and their properties. Each gNB is assigned a
random position, a transmit power (30–40 dBm), and a frequency.
UEs are initialized with random positions and velocities. This setup
mimics random deployment and realistic user movement in urban
settings.
[Link]({
'id': i,
'position': [Link](2) * grid_size,
'tx_power': 30 + [Link]() * 10,
'frequency': 3500 + (i % 3) * 100,
'capacity': 100,
'load': 0,
'neighbors': []
})
Neighbor relationships between gNBs are defined based on Euclidean
distance. gNBs within 40% of the grid length are assigned as Xn
neighbors.
The _calculate_path_loss method implements the standard 3GPP path
loss model:
path_loss = 128.1 + 37.6 * np.log10(distance_km)
RSRP is computed as transmit power minus path loss plus log-normal
shadow fading.
UEs update their position using _update_ue_positions, simulating
velocity with minor randomness and capping speed at 10 units.
_update_measurements refreshes RSRP values from each gNB per step.
Throughput is calculated using SINR and Shannon’s capacity formula,
penalized by gNB load. Handover logic (_perform_xn_handover) validates
Xn connectivity and gNB capacity before switching. Ping-pong
detection checks recent handover history.
sinr = signal / (interference + noise_floor)
throughput = bandwidth * np.log2(1 + sinr) * load_factor
The step() method advances the environment by one timestep. reset()
reinitializes UE and gNB states. get_interference_level() estimates
normalized interference for reward shaping.
3. Deep Q-Learning Agent – Class: HandoverRL
This class implements the DQL handover agent. If use_dqn=True, it
constructs separate Q-networks for each UE using TensorFlow/Keras.
Neural Network Architecture:
model = Sequential()
[Link](Dense(24, input_dim=input_dim, activation='relu'))
[Link](Dense(24, activation='relu'))
[Link](Dense(self.num_gnbs, activation='linear'))
Inputs are the 12-dimensional state vector consisting of RSRP (5),
velocity (1), load ratios (5), and time since last handover (1). The
output is Q-values for each gNB.
Action Policy: Uses epsilon-greedy selection:
Random gNB with probability epsilon.
Otherwise, gNB with highest Q-value.
Experience Replay: Transitions are stored in a memory buffer (size:
2000). During training, a minibatch is sampled and backpropagation
updates the Q-network weights using:
Q(s,a) = r + gamma * max Q(s')
Reward Function:
R_t = log(1 + TP) - 0.5 * Stability - 0.3 * Interference
TP: throughput after simulated handover
Stability: penalizes frequent switching
Interference: estimated normalized interference on target
frequency
After reward calculation, the environment is restored to its previous
state to preserve episode continuity.
4. Simulation Workflow – Script: [Link]
The main script runs both the traditional and RL-based handover
policies for comparative analysis.
env = CellularEnvironment(num_gnbs=5, num_ues=20, grid_size=1000, episo
de_length=100)
Initializes the environment with specified parameters.
agent = HandoverRL(env, use_dqn=True)
train_rl_agent(env, agent, episodes=100)
Trains the DQL agent across multiple episodes. Each UE observes its
state, selects an action, performs a trial handover, receives a reward,
and updates its Q-network.
[Link](rl_rewards, label='RL-Based Handover')
[Link](traditional_rewards, label='Traditional A3 Handover')
Plots average episode rewards to compare strategies.
5. Results and Discussion
Simulations over 50 episodes show that the DQL agent significantly
outperforms the traditional handover approach in all metrics. The
DQL policy effectively balances gNB load, avoids unnecessary
handovers, and improves network-wide throughput.
6. Conclusion and Future Work
This paper has detailed a complete Deep Q-Learning pipeline for
handover optimization in 5G networks. The code implements all
components from signal modeling to policy learning and evaluation.
Future directions include extending the agent to LSTM-based
prediction, multi-agent RL, and O-RAN deployment.
Keywords: Deep Reinforcement Learning, Handover Optimization, 5G,
Q-Learning, SINR, RSRP, Neural Network, Cellular Simulation,
Python