Noname manuscript No.

(will be inserted by the editor)

Optimizing QoS in Edge Computing through

DRL-Driven Task allocation

Received: date / Accepted: date

Abstract Edge computing stands out as a pivotal technology in enhanc-

ing the performance and efficiency of various applications, including online
gaming, autonomous vehicles, smart cities, and more. This model expands
the reach of cloud paradigm to the periphery of the network, closer to data
sources, thus reducing delay, conserving bandwidth, and enhancing privacy.
However, it introduces significant challenges in resource management and task
offloading due to the diverse and dispersed nature of edge devices. To tackle
these challenges, this paper advocates the use of Deep Reinforcement Learn-
ing (DRL) techniques to handle heterogeneous task in dynamic environment.
DRL’s adaptive and predictive capabilities make it an ideal solution for dynam-
ically managing tasks in an edge environment, optimizing Quality of Service
(QoS) aspects such as delay, energy usage, and resource allocation. We present
a comprehensive framework, explore various DRL algorithms, and simulate
the environment to evaluate the proposed methods’ effectiveness. This study
aims to contribute significantly to the field of edge computing by enhancing
the QoS, thereby supporting the robust development of real-time, distributed
applications.
Keywords Edge Computing, Quality of Service, Task Offloading, Deep
Reinforcement Learning, Resource Management

1 INTRODUCTION

Edge computing marks a significant shift in data processing, advancing the

functionalities of cloud paradigm to the periphery of the network, near to edge
devices like IoT devices and sensors. The edge paradigm encompasses network
devices typically situated within proximity, often just a single hop away from
the sources where data is generated. The proliferation of IoT devices, including

Address(es) of author(s) should be given

2

Fig. 1 Working of Edge Computing architecture

sensors, intelligent Closed Circuit Television (CCTV) cameras, smart gadgets,

and various other connected devices, is experiencing rapid and exponential
growth. This architectural shift is driving the ever-increasing data demands
and real-time processing needs of modern applications such as online gaming,
Healthcare, 5G communications, Industrial IoT, Driverless transportation, VR
technology, and AR technology [1]. As per the findings from the International
Data Corporation (IDC), it is projected that by 2025, an estimated 41.6 bil-
lion interconnected IoT devices will collectively produce an astounding 79.4
zettabytes of data. This anticipated surge in data generation surpasses the ca-
pacity envisioned by the existing centralized cloud infrastructure, which was
not originally engineered to accommodate such an immense volume of in-
formation [2]. Edge computing leverages the functionalities of cloud, offering
resources closer to end devices. Although edge resources are generally smaller
in scale than cloud resources, they serve an essential function in processing
large volumes of data for real-time applications [3]. Edge computing mitigates
latency, enhances privacy, and reduces the bandwidth demand on the core
network by enabling localized data processing.
Figure 1 illustrates the typical architecture of edge computing, highlighting
its role as a mediator between end devices and the cloud. Edge computing
nodes, dispersed geographically, provide storage, processing, networking, and
control services near data sources. This proximity to data origins significantly
improves the Quality of Experience (QoE) by reducing latency and allowing
for cost-effective scaling of services [4].
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 3

1.1 Motivation

Managing resources at the edge layer presents numerous hurdles arising from
the distributed, dynamic, and heterogeneous nature of edge devices [5]. Unlike
the relatively uniform and centralized resources in cloud data centers, edge
nodes are diverse in capabilities and scattered across locations[6]. Managing
these resources effectively is a non-trivial task. Thankfully, AI techniques like
deep reinforcement learning (DRL) have addressed these requirements effec-
tively. By combining the robust perceptual capabilities of deep learning with
reinforcement learning (RL), DRL can derive an optimal policy through dy-
namic interaction with complex, high-dimensional environments, without re-
quiring prior knowledge of environmental dynamics[7]. Consequently, trained
DRL models empower agents to make timely real-time decisions regarding task
offloading[8].
For example, Huang et al. [9] employed a DQN algorithm to determine
Task Offloading and Bandwidth Resource Allocation decisions for individual
mobile terminals, to reduce overall offloading costs. In a related study[10], re-
searchers applied a DDQN algorithm to derive the optimal task offloading and
resource allocation strategy across all end devices, aiming to enhance compu-
tational efficiency. Another approach, as described in[11], utilized a DDPG
algorithm to optimize task offloading and computing resource allocation de-
cisions for multiple user devices, with a focus on minimizing system energy
consumption. Furthermore, in[12], researchers utilized an asynchronous actor-
critic algorithm to optimize task offloading and resource allocation, aiming to
reduce energy consumption while maximizing task completion rates.
This paper solves the task offloading problem using DRL, considering a
realistic dynamic edge environment with heterogeneous end devices. These
end devices execute different types of applications with diverse resource re-
quirements. Edge computing must address factors like latency, computational
complexity, resource utilization, energy consumption, and task priority to pro-
vide effective service. Therefore, our proposed approach integrates task prior-
ity and deadline management alongside factors like latency and energy con-
sumption. Tasks are categorized into three segments to facilitate prioritization:
lower-priority, higher-priority, and critical. Critical tasks encompass security-
sensitive operations that must remain localized on the user device due to their
sensitivity. Each user device must allocate sufficient computational resources
for the execution of critical tasks. Our model ensures that high-priority work-
loads meet their deadlines while critical tasks are effectively handled at the
local level.

1.2 Objective and Contribution

To tackle the aforementioned challenges, this work proposes the use of DRL
techniques for intelligent task offloading in the edge computing environment.
DRL, a subset of AI focused on learning optimal actions through trial and
4

Table 1 Abbreviation Interpretations

Abbreviation Interpretation Abbreviation Interpretation

AI Artificial Intelligence IoT Internet of Things
A2C Advantage Actor Critic IIoT Industrial Internet of Things
CCTV Closed Circuit Television MDP Markov Decision Process
DDPG Deep Deterministic Policy Gradient MEC Mobile Edge Computing
DRL Deep Reinforcement Learning PPO Proximal Policy Optimization
DQN Deep Q-Networks QoE Quality of Experience
DDQN Double Deep Q-Network QoS Quality of Service
DTOS Dependent Task-Offloading Strategy RATO Request Aware Task Offloading
ECS Edge computing server VRNN Variational Recurrent Neural Network
FIFO First-In-First-Out WD Wireless device

error, is well-suited to the dynamic and complex nature of edge environments.

It enables adaptive decision-making for where, when, and how to offload tasks
from devices with limited resources to edge servers or the cloud. The primary
contribution of this research is given below:

– We introduce a framework utilizing Deep Reinforcement Learning (DRL)

for efficient task offloading, significantly reducing latency by adapting dy-
namically to changes in network conditions and device capabilities.
– Our approach holistically enhances Quality of Service (QoS) by address-
ing key aspects like energy efficiency, and resource utilization, ensuring a
comprehensive upgrade in service quality.
– A variety of DRL algorithms, including Deep Q-Networks (DQN), Advan-
tage Actor Critic (A2C), and Proximal Policy Optimization (PPO) are
explored and implemented to provide a robust and versatile solution for
diverse edge computing scenarios.
– We present a comparative analysis of the proposed DRL strategies through
extensive simulations, demonstrating their effectiveness and superiority
over traditional methods in optimizing task offloading decisions.
– The paper contributes a practical implementation guide for DRL in edge
computing, offering insights into algorithm selection, parameter tuning,
and deployment strategies to foster future research and real-world appli-
cations.

Through these contributions, this research proposes a foundational frame-

work for enhancing edge computing services using DRL, thereby supporting
the next generation of real-time applications.

1.3 Paper Organization

Common-used abbreviations in this work are specified in Table 1. The re-

mainder of the paper is organized as follows: In the second section, we delve
into the existing literature concerning similar problems in edge computing,
employing various methodologies. Next, Section 3 outlines the system model
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 5

employed in the proposed work. The optimization problem at hand and its
solution leveraging DRL are expounded upon in Section 4. Subsequently, Sec-
tion 5 delves into extensive experimentation and the resulting findings. Finally,
Section 6 encapsulates the entirety of the work, concluding with avenues for
future research.

2 RELATED WORK

Fog computing plays a vital role in diminishing latency by handling data

at the network’s periphery. A significant portion of the analyzed literature
concentrates on latency enhancement[13, 15–26], indicating the superiority of
fog/edge computing over traditional cloud setups. The architecture of fog/edge
facilitates data processing closer to end devices, effectively minimizing latency
in contrast to the conventional cloud model.
Typically, fog nodes positioned near end devices are directly powered, while
those situated at upper layers rely on fixed batteries, emphasizing the neces-
sity for energy-efficient practices. Employing efficient techniques can diminish
energy usage by enabling fog nodes to activate and deactivate as needed. Many
of the latest reviewed articles worked on energy consumption improvement[13–
15, 18, 19, 21–24].
Athough, very few papers focused on the priority[23] and deadline[16, 17,
24] for real world applications. This paper worked on real-world task offloading
problem by considering QoS factors delay, energy, priority, and deadline. This
section discussed the related research articles in more detail. Chen et al. [13]
objective is to reduce power usage while optimizing the task completion rate
within an acceptable delay. During the training phase, the suggested algorithm
successfully predicts edge server load in real time while integrating competing
offloading goals. Although, they only take into account jobs that are delay-
sensitive and non-divisible.
The issue of dynamic task offloading in MEC enhanced by digital twin
technology is examined by Y. Chen et al. [14]. Techniques known as ”digital
twins” are used to exchange agent training data and environment informa-
tion that is deployed on Internet of Things devices. They aim to balance the
workload among the edge servers and maximize energy efficiency.
Zhang et al. [15] consider the existence of multiple Mobile Edge Computing
(MEC) servers and propose a dynamic task offloading approach grounded in
DRL. Their proposed algorithm exhibits superior performance in terms of
latency, energy efficiency, and cumulative system load across different task
quantities and wireless channel bandwidths.
Utilizing numerous wireless nodes and multiple MEC servers, Ke et al.
[16] deploy an MEC framework that considers task priorities and fluctuating
channel conditions. To reduce overall weighted costs, encompassing both delay
and bandwidth costs over time, they introduce a decentralized multi-agent
algorithm called DeMADRL, based on DRL, optimizing resource allocation
through partial task offloading within their proposed MEC system.
6

A collaborative task offloading algorithm driven by priority, utilizing multi-

agent DRL is proposed by Yang et al. [17]. They go on to create a global state
sharing model based on Variational Recurrent Neural Networks (VRNNs),
which drastically lowers the transmission burden between edge servers. The
suggested algorithm shorten task consumption times and enhances the use of
edge computing resources.
The dependent task model is defined by Bencan Gong et al. [18] rep-
resented in the form of directed acyclic graph. They proposed Dependent
Task-Offloading Strategy (DTOS) derived from DRL to reduce the combined
weighted network service latency and energy utilization. To determine the best
task-offloading choices, DTOS adds a penalty mechanism that is activated
when either the service delay or energy consumption surpasses a predeter-
mined threshold.
Cai et al. [19] have reduced overall system latency and device energy con-
sumption. Their model enhances the stability and convergence speed of the
DRL model by employing a recurrent neural network to extract features from
task and network state information..
The workload on mobile edge computing (MEC) network featuring mul-
tiple agents and accommodating tasks that are time-sensitive and indivisible
are examined by Minxuan Chen et al. [20]. They suggested an algorithm that
minimizes the average delay in a variety of multi-agent environments by us-
ing a model of centralized training and distributed execution. Their proposed
algorithm shows high stability.
To distribute the workload evenly across ECSs and improve resource uti-
lization rates, F. Zhang et al. [21]. introduced the Cooperative Partial Task
Offloading and Resource Allocation (CPTORA) framework. There system pro-
motes collaboration among different local and non-local ECSs, industrial IoT
devices, and cloud servers. Additionally, they proposed an enhanced soft actor-
critic-based CPTORA algorithm (ISAC-CPTORA), which efficiently deter-
mines optimal resource allocation and task offloading strategies for each in-
dustrial IoT device. Their model achieves improved performance through the
utilization of prioritized experience replay.
L. Tan et al. [22] Optimize the energy consumption of the Mobile Edge
Computing system while adhering to a delay limit. To enhance DRL perfor-
mance in MEC, they merge double-DQN and dueling-DQN while incorporating
adaptive parameter space noise.
The Request Aware Task Offloading (RATO) scheme, employing the Deep
Q Network (DQN) and utilizes limited edge server computing resources, was
proposed by Z. Sheng et al. [23]. Its goal is to optimize the cumulative weighted
overhead, which includes the energy consumption and delay performance met-
rics of mobile devices.
Linbo Liao et al. [24] present a MEC system comprising many mobile de-
vices and a server, tasked with computation prioritization. Their proposed
algorithm, double reinforcement learning computation offloading (DRLCO),
collectively determines CPU frequency, transmit power, and offloading deci-
sions for computation offloading. Addressing energy consumption, they specif-
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 7

ically define the power management challenge encountered by users of mobile

devices and present an innovative algorithm rooted in DDPG to address it.
Additionally, to mitigate task delays, they model the resource allocation prob-
lem and suggest a double DQN based algorithm. Their approach achieves a
notable reduction in delays by 48% and energy consumption by 53%.

B. Yamansavascilar et al. [25], implementing the DeepEdge system involves

utilizing the DDQN algorithm. DeepEdge can fulfill diverse task specifications
autonomously, eliminating the need for human intervention, even in situa-
tions where the network is highly loaded with mobile users and applications.
Through experimentation with four distinct applications—infotainment, im-
age rendering, augmented reality, and pervasive health, they test the resilience
of DeepEdge in the network under varied loads.

In a time-varying MEC system accommodating multiple users and servers,

Yu Sun et al. [26] presented a DQN based multi-branch algorithm for resource
distribution and offloading determinations. The suggested algorithm reduces
the discard rate and average task latency.

The comparative analysis of literature papers based on delay, energy con-

sumption, priority, deadline, and applied algorithms is provided in Table 2.

Table 2: QoS based comparison of literature work

Citation

Delay Energy Priority Deadline Applied Algorithm

[13] Yes Yes No No DRL DDTO
[14] No Yes No No DEETO
[15] Yes Yes No No DRLOS
[16] Yes No No Yes DeMADRL
[17] Yes No No Yes PDMA
[18] Yes Yes No No DRL
[19] Yes Yes No No DRL RNN
[20] Yes No No No MADRL
[21] Yes Yes No No SAC
[22] Yes Yes No No DDQN
[23] Yes Yes Yes No DQN
[24] Yes Yes No Yes DDPG DQN
[25] Yes No No No DDQN
[26] Yes No No No BDQN
Proposed Yes Yes Yes Yes DQN A2C PPO
8

3 System Model

3.1 Network Model

The Edge Computing model is conceptualized as a grid-based architecture,

characterized by M edge servers, denoted as {mi | i = 1, 2, . . . , M }, each co-
located with a base station within a defined grid area (x, y). Additionally, N
wireless devices, represented as {nj | j = 1, 2, . . . , N }, are capable of initiating
and offloading computational tasks within this network.
The spatial disposition of this network is outlined by an area A, with di-
mensions (x, y), serving as the operational landscape for the base stations and
wireless devices. Each wireless device, within any given time frame t ∈ T ,
where T = {1, 2, . . . , Tmax }, has the potential to generate a set of tasks Tnj ,
with |Tnj | ≤ N . These tasks, ranging from immersive gaming to critical nav-
igation services, are distinguished by their computational requirements and
urgency, necessitating a dynamic offloading strategy based on their character-
istics.
Mathematically, the decision to offload a task from device nj at time t,
denoted by xnj ,t , can be formulated as:

(
0 if task is processed locally,
xnj ,t =
1 if task is offloaded.

The offloading decision xnj ,t is influenced by several factors, including the

device’s energy state, network bandwidth, and task priority. The network aims
to optimize the processing of tasks T by leveraging the computational re-
sources of edge servers to minimize processing latency and energy consump-
tion while adhering to Quality of Service (QoS) standards, especially for time-
sensitive tasks.
Let τnj ,mi (t) denote the processing time of a task generated by device nj
and processed by server mi at time t, and let ϵnj ,mi (t) represent the corre-
sponding energy consumption. The network’s objective is to minimize the ag-
gregate processing time and energy consumption across all tasks and devices,
expressed as:

X X

min α · τnj ,mi (t) + β · ϵnj ,mi (t) ,
nj ∈N mi ∈M

where α and β are weighting factors that reflect the relative importance of
processing time and energy consumption, respectively.
This model underscores the MEC network’s commitment to efficiently man-
aging the computational demands of a diverse array of wireless devices, thereby
optimizing resource allocation and enhancing overall system performance.
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 9

3.2 Wireless Device Model

Wireless devices (WDs) in the network are characterized by features such as

wireless charging and rechargeable batteries. Each WD, during a given time
frame t, follows a task generation model. The probability of task generation by
a device at any given time is represented as λt and follows a Bernoulli distribu-
tion with parameter λ. In terms of task offloading decisions, each WD decides
whether to process a task locally or offload it to an MEC server. This decision
is denoted by xnj ,t , where xnj ,t ∈ {0, 1}. Here, 0 indicates local processing of
the task, and 1 represents the offloading of the task to an MEC server. These
decisions are made based on various factors, including the device’s current
battery level, network conditions, and the computational requirements of the
task at hand.
Additionally, power consciousness is an inherent factor of work-offloading
decisions in the wireless device model. Each one of these WDs has an onboard
rechargeable battery that acts as its main power supplier. Energy management
is a critical function given the fact that the batteries have finite resources and
would, therefore, enhance user experience by prolonging the operational time
of the devices. Therefore, the determination of the task selection and delega-
tion is also taken care of by the WDs along with the continuous monitoring
of the energy consumption and the remaining battery capacity. WDs dynami-
cally change their offloading policy by adjusting the required QoS parameters
to achieve energy efficiency. The adaptive approach, however, does not only
use existing resources efficiently on the network level but also prolongs the
operational life of WDs, with the system always ready for new features to be
added.

3.3 Edge Computing Modes and Local Computing

In the Mobile Edge Computing (MEC) ecosystem, task computation is bi-

furcated into two predominant paradigms: local computing (ϕloc ) and edge
computing (ϕedge ), each tailored for optimizing task processing based on an
amalgamation of latency, energy efficiency, and computational demands.

3.3.1 Local Computing (ϕloc )

Local computing paradigm entails the direct processing of tasks on the wireless
devices themselves. This mode is mathematically characterized by the process-
ing delay (τproc,nj ) and the energy consumption (ϵloc,nj ) for a task by device
nj at any given time t, formalized as:

Dnj ,t
τproc,nj (t) = ,
ϕ nj · f nj

ϵloc,nj (t) = knj · fn3j · τproc,nj (t),

10

where Dnj ,t signifies the task’s data size, ϕnj denotes the computational
capability of device nj , fnj represents the CPU frequency, and knj is a coeffi-
cient mirroring the energy efficiency of the device.

3.3.2 Edge Computing (ϕedge )

Contrastingly, edge computing embodies the offloading of tasks to MEC servers

for processing. The offloading decision influences both the transmission delay
(τtrans,nj ,mi (t)) and the associated energy consumption (ϵtrans,nj ,mi (t)), de-
fined as:

Dnj ,t
τtrans,nj ,mi (t) = ,
rnj ,mi

ϵtrans,nj ,mi (t) = Pnj · τtrans,nj ,mi (t),

with rnj ,mi depicting the data transmission rate between device nj and
server mi , and Pnj the transmission power of device nj .

3.4 Task Classification and Priority Assignment

Tasks are classified into distinct categories based on urgency, computational

requirements, and security sensitivity. This classification is pivotal for manag-
ing resource allocation and task processing efficiently, adhering to Quality of
Service (QoS) standards.
Let T = {T1 , T2 , . . . , Tn } represent the set of tasks within the MEC net-
work. Each task Ti is characterized by a tuple (ci , di , pi , si ), where:

– ci denotes the computational requirement,

– di indicates the deadline,
– pi signifies the priority level (Critical, Low, High),
– si indicates security sensitivity (Boolean, where True implies Critical).

3.4.1 Priority Levels

Tasks are categorized based on priority levels as follows:

Critical Priority Tasks (pi = Critical): These are security-sensitive tasks, de-
noted by si = True, requiring processing on the local device to ensure data
security. They are not offloaded due to their sensitive nature.

For Ti ∈ T , if si = True, then pi = Critical (1)

Optimizing QoS in Edge Computing through DRL-Driven Task allocation 11

Fig. 2 Working of Queuing model

High Priority Tasks (pi = High): Tasks with stringent deadlines, demanding
urgent processing. These tasks are typically offloaded to edge servers to ensure
timely completion.

For Ti ∈ T , if di is tight and si = False, then pi = High (2)

Low Priority Tasks (pi = Low): Less time-sensitive tasks with greater flexi-
bility in processing. They can be processed locally or offloaded based on the
current availability of resources.

For Ti ∈ T , if di is flexible and si = False, then pi = Low (3)

3.5 Queuing Model

Within the Mobile Edge Computing (MEC) framework, an advanced queuing

model plays a pivotal role in optimizing task processing by efficiently man-
aging tasks according to their priority levels [27]. Each edge server mi , for
i = 1, 2, . . . , M , implements two queues for this purpose: Qm high for high priority
i

tasks, and Qm i
low for low priority tasks as shown in Figure 2. This setup ensures
that the network can effectively prioritize urgent tasks while also accommodat-
ing less critical workloads in a manner that upholds Quality of Service (QoS)
standards.
Tasks entering the MEC network, denoted as Tj = (cj , dj , pj , sj ), are cat-
egorized based on their computational requirements cj , deadlines dj , priority
levels pj (Critical, High, Low), and security sensitivity sj (true for critical
tasks). The priority level pj primarily dictates the queue assignment for each
task:

– High priority tasks (pj = High) are immediately placed in Qmhigh to ensure
i

they are processed with precedence. The operation of Qm high adheres to

i

a First-In-First-Out (FIFO) principle, supplemented by real-time priority

adjustments to accommodate changing network conditions.
12

– Low priority tasks (pj = Low) are assigned to Qm low , where they are pro-
i

cessed as computational resources permit, typically after the high priority

tasks have been addressed. This queue also employs a FIFO approach but
allows for greater flexibility in task scheduling to enhance resource utiliza-
tion.

The queuing model is meticulously designed to balance the processing

of Qm mi
high and Qlow across all edge servers, ensuring that critical tasks are
i

given immediate attention by being processed locally on devices due to their

security-sensitive nature. Through this structured and mathematically sup-
ported model, the MEC network efficiently prioritizes urgent tasks, manages
the execution of less critical tasks, and optimizes overall resource utilization
while maintaining exemplary QoS levels.

3.6 Task Utility and Optimization Objective

The overarching objective within the MEC network is to optimize task pro-
cessing in a manner that balances security, timeliness, and efficiency, while
also minimizing energy consumption and processing delay. This objective is
shaped by the classification of tasks into critical, high, and low priority levels,
each dictating a specific processing strategy to adhere to Quality of Service
(QoS) standards.

3.6.1 Critical Task Processing

Critical tasks, identified by their security-sensitive nature, are mandated to

run locally on the device. This decision is driven by the need to safeguard
sensitive information, ensuring that such tasks are processed in the most se-
cure environment possible. Mathematically, for a critical task Ti with security
sensitivity si = True, the processing location L(Ti ) is determined as follows:
(
Local, if si = True,
L(Ti ) = (4)
Edge, otherwise.

3.6.2 High and Low Priority Task Offloading

High and low priority tasks are primarily delegated to the edge for process-
ing, leveraging the computational capabilities of edge servers. High priority
tasks, in particular, are processed with an emphasis on meeting their strin-
gent deadlines. This involves dynamically managing the network’s resources
to ensure that such tasks are completed within their allotted time frames,
thereby optimizing the overall network performance and user experience.
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 13

3.6.3 Delay and Energy Minimization

The primary goal of the MEC network is to optimize both energy consumption
and processing delay across all tasks, regardless of whether they’re handled
locally or offloaded to the edge. The network employs sophisticated algorithms
and scheduling policies to achieve this, carefully balancing the computational
load across devices and edge servers. For any given task Ti , the network seeks
to minimize the energy E(Ti ) and delay D(Ti ), subject to the constraints
imposed by the task’s priority level and security requirements:
min E(Ti ) + D(Ti ), (5)
s.t. L(Ti ) is determined based on pi and si . (6)
This optimization framework ensures that critical tasks are processed se-
curely on local devices, while high and low priority tasks are efficiently of-
floaded to edge servers. By doing so, the MEC network effectively reduces
energy consumption and processing delay, enhancing the QoS for all network
participants.

4 Proposed DRL Task Offloading Framework

The proposed framework for task offloading in edge computing environments

employs various DRL techniques. These DRL techniques are designed to opti-
mize task offloading decisions to effectively balance latency, energy efficiency,
and resource utilization in dynamic and resource-constrained edge environ-
ments.

4.1 Framework Overview

The task offloading process is represented as a Markov Decision Process (MDP),

characterized by a state space, a set of possible actions, and a reward mecha-
nism [28].

4.1.1 State Space

The state st at time step t encapsulates the current environment’s status and is
defined as a vector containing information about all users and their respective
tasks, as well as the network’s status:
st = [u1 , u2 , . . . , uNU ; bs1 , bs2 , . . . , bsNBS ]

4.1.2 Action Space

In a mobile edge computing (MEC) environment, the action at taken by an

agent at time step t can be defined as:
at ∈ A = {0, 1, . . . , NBS }
14

4.1.3 Reward Function

The reward function R(st , at ) at time step t aims to balance various objectives,
such as minimizing task processing delay, energy consumption, and ensuring
priority tasks meet their deadlines:
R(st , at ) = w1 · D(st , at ) + w2 · E(st , at ) + w3 · P (st , at )

Algorithm 1 DRL-Based Task Offloading in MEC Environments

1: Inputs:
2: U = {u1 , u2 , . . . , uN }: Set of mobile users.
3: BS = {bs1 , bs2 , . . . , bsM }: Set of base stations.
4: T = {τ1 , τ2 , . . . , τK }: Set of tasks.
5: Parameters:
6: fBSj : Computation capacity of base station bsj .
7: B: Available bandwidth.
8: Ptransi : Transmission power of user ui .
9: RSINRij : SINR-based data rate.
10: R(s, a): Reward function.
11: Initialization: Set up the MEC environment with user locations, task characteristics,
and system resources.
12: for each episode do
13: Obtain current state st reflecting the MEC system’s status.
14: for each task τk ∈ T do
15: Determine offloading decision using the DRL agent.
16: Execute the action and measure energy consumption and latency.
17: Calculate immediate reward R(st , at ).
18: Update DRL model parameters based on reward and transition to new state
st+1 .
19: end for
20: if termination condition is met then
21: Break
22: end if
23: end for

4.2 DRL Algorithms for Task Offloading

In the context of MEC, task offloading decisions are critical in determining the
efficiency and effectiveness of computing resources. Deep Reinforcement Learn-
ing (DRL) algorithms provide a dynamic and adaptive approach to optimize
these decisions. Below is a detailed explanation of several DRL algorithms and
their application in task offloading.

4.2.1 Proximal Policy Optimization (PPO)

PPO is a popular policy gradient method used for its stability and reliability. It
is particularly effective in environments with large and complex action spaces,
such as those encountered in task offloading scenarios [29].
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 15

Fig. 3 PPO architecture for task offloading

The objective of PPO is to avoid large updates to the policy, which can lead
to destructive steps. It achieves this through a clipped surrogate objective:

LP P O (θ) = E [min (rt (θ)At , clip(rt (θ), 1 − ϵ, 1 + ϵ)At )]

In task offloading, PPO can dynamically adjust the policy for deciding
whether a task should be computed locally or offloaded to an edge server,
considering the current state of the network, resources, and task requirements.
The PPO architecture for task offloading is shown in fig: 3.

Algorithm 2 PPO for Task Offloading

1: Inputs: Users U , Base Stations BS, Tasks T , System Parameters
2: Initialize: Policy parameters θ, Value function parameters ϕ, Environment State s0
3: while not converged do
4: Collect set of trajectories D = {τi } by executing current policy πθ in the environment
5: Compute advantage estimates Ât using the value function Vϕ
6: Update policy by maximizing the  PPO-Clip objective: 
1 P PT
7: θ ← θ + η∇θ |D| τ ∈D t=0 min rt (θ)Ât , clip(rt (θ), 1 − ϵ, 1 + ϵ)Ât
8: Update value function by P
minimizing MSE loss:
1 P T 2
9: ϕ ← ϕ − η∇ϕ |D|T τ ∈D t=0 (Vϕ (st ) − Rt )
10: end while
11: Output: Optimized policy πθ

4.2.2 Advantage Actor-Critic (A2C)

A2C is an algorithm that utilizes two neural networks: the Actor for policy ap-
proximation and the Critic for value function approximation. This separation
helps in stabilizing the training process [30].
16

The actor updates the policy distribution based on the feedback provided
by the critic:

LA2C (θ) = E [log πθ (at |st )Aπθ (st , at )]

The critic, on the other hand, evaluates the chosen action by calculating
the temporal difference error:

LCritic (ϕ) = E (Vϕ (st ) − Rt )2

 

For task offloading, A2C helps in making informed decisions by considering

both the current policy’s advantage and the value of the states, thus leading
to more efficient resource utilization and reduced latency.

Algorithm 3 A2C for Task Offloading in MEC Environments

1: Inputs: Users U , Base Stations BS, Tasks T , System Parameters
2: Initialize: Actor parameters θ, Critic parameters ϕ, Environment State s0
3: while not converged do
4: Execute actions at according to current policy πθ (a|s)
5: Observe reward rt and new state st+1
6: Compute advantage estimate At = rt + γVϕ (st+1 ) − Vϕ (st )
7: Update critic by minimizing loss: L(ϕ) = (rt + γVϕ (st+1 ) − Vϕ (st ))2
8: Update actor using policy gradient: ∇θ log πθ (at |st )At
9: st ← st+1
10: end while
11: Output: Optimized policy πθ and value function Vϕ

4.2.3 Deep Q-Network (DQN)

DQN stands out as an effective approach for addressing challenges posed by

environments characterized by complex state representations and a finite set
of actions. Leveraging a neural network, it approximates the optimal action-
value function, while integrating methods such as experience replay and target
networks to enhance learning stability. [31].
The loss function for DQN is:

h i
′
LDQN (θ) = E (rt + γ max
′
Qθ − (st+1 , a ) − Qθ (st , at ))
2
a

In the context of MEC, DQN can be used to determine the optimal of-
floading decisions to minimize latency and energy consumption while ensuring
tasks are completed within their deadlines.
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 17

Algorithm 4 DQN for Task Offloading

1: Inputs: Users U , Base Stations BS, Tasks T , System Parameters
2: Initialize: Q-network parameters θ, Target Q-network parameters θ− , Replay buffer D
3: while not converged do
4: Select action at using ϵ-greedy policy derived from Qθ
5: Execute action at and observe reward rt and new state st+1
6: Store transition (st , at , rt , st+1 ) in D
7: Sample random minibatch of transitions (sj , aj , rj , sj+1 ) from D
8: Set yj = rj + γ maxa′ Qθ− (sj+1 , a′ ) for non-terminal sj+1 and yj = rj for terminal
sj+1
9: Perform a gradient descent step on (yj − Qθ (sj , aj ))2 with respect to θ
10: Every C steps reset θ− = θ
11: end while
12: Output: Optimized Q-network Qθ

5 PERFORMANCE EVALUATION

To evaluate the effectiveness of our task offloading strategies in a mobile edge

computing environment, we have set up a detailed simulation using a custom
environment built on Python’s Gym framework. This section describes the
simulation setup, evaluation metrics, and the results of comparing different
models including A2C, PPO, and DQN.

5.1 Simulation Setup

Our simulation environment models a real-world scenario where multiple base

stations and users interact within a specified area. Each base station has a
set of communication channels and computational resources to manage users’
tasks. Users generate tasks with varying computational needs and offload these
tasks to the base stations. The main objective of the simulation is to optimize
task offloading decisions to balance important factors such as latency, energy
consumption, and resource utilization.
The Table 3 outlines the parameters used in our simulation scenario:

Table 3 Parameters used in simulation

Parameter Values
Count of Base Stations 4
Count of Users 50
Area Size 15km × 15km
Base Station Locations Specified Coordinates
User Task Generation Randomized
Channel Frequency Range 24-27 GHz
Bandwidth per Channel 20 MHz
Computation Power Varied per Base Station

The Edge environment orchestrates the interaction between users and base
stations, managing task offloading decisions, channel assignments, and com-
18

munications. The environment steps through a sequence of states, actions, and

rewards, simulating the dynamic nature of the network.

5.2 Performance Comparison

The evaluation of Deep Reinforcement Learning (DRL) algorithms in our mo-

bile edge computing (MEC) environment simulation reveals distinctive learn-
ing behaviors across A2C, PPO, and DQN. As depicted in Figure 4, the mean
training reward serves as a quantifiable measure of each algorithm’s capabil-
ity to optimize task offloading decisions effectively, emphasizing the balance
between latency, energy efficiency, and resource utilization.
PPO consistently demonstrates superior performance, achieving an early
convergence to high reward values, which suggests its proficiency in rapidly
identifying and adhering to efficient offloading strategies. Conversely, A2C
showcases a pattern of gradual improvement interspersed with fluctuations,
indicative of a more exploratory yet effective learning process. DQN, while
exhibiting a steady ascent in reward values, does not reach the performance
peaks of PPO, highlighting a slower adaptation to the optimal offloading policy
within the confines of our simulation parameters.
This comparative analysis underscores the variance in learning dynamics
and efficiency among DRL algorithms, with PPO distinctly outperforming
in the context of task offloading in MEC environments, as evidenced by its
accelerated learning curve and sustained optimal performance throughout the
simulation episodes.

Mean Training Reward

1.2

1.1

1.0
Mean Reward

0.9

0.8

0.7

0.6
PPO
0.5 A2C
DQN
0 250 500 750 1000 1250 1500 1750 2000
Episode

Fig. 4 Mean training reward

Optimizing QoS in Edge Computing through DRL-Driven Task allocation 19

5.3 Delay Analysis

Our comprehensive analysis of task offloading strategies within the MEC sim-
ulation environment also extends to examining the mean processing delay
across different epochs for PPO, A2C, and DQN. The graphical representa-
tion in Figure 5 elucidates the evolution of processing delay as the algorithms
progressively learn and adapt their offloading decisions over time.
From the onset, PPO exhibits a robust reduction in processing delay, un-
derscoring its capability to quickly assimilate and implement efficient offload-
ing strategies that minimize latency. This trend is consistently maintained
throughout the simulation, confirming PPO’s effectiveness in reducing task
processing times.

Mean Delay
PPO
A2C
40
DQN

35
Mean Processing delay (ms)

30

25

20

15

10
0 500 1000 1500 2000 2500 3000 3500 4000
Epoches

Fig. 5 Mean processing delay by epochs

In contrast, A2C demonstrates a more variable trajectory, where initial

delay reductions are observed, followed by periods of stabilization and minor
fluctuations. This pattern suggests a balanced approach towards exploration
and exploitation, ultimately leading to significant, albeit gradually achieved,
improvements in processing delay.
DQN’s performance in delay reduction starts off at a slower pace compared
to PPO, yet exhibits steady progress as the epochs advance. This indicates
DQN’s gradual learning curve and its eventual convergence towards an efficient
offloading policy that effectively reduces the processing delay.
The depicted trends across PPO, A2C, and DQN not only highlight the
distinctive learning behaviors and efficiencies of these DRL algorithms but
also emphasize the critical role of intelligent task offloading in optimizing
latency within MEC environments. PPO’s quick adaptation and sustained
performance in minimizing delay are particularly noteworthy, showcasing its
potential as a preferable choice for MEC scenarios demanding low latency.
20

5.4 Energy Analysis

In our examination of energy efficiency within the MEC environment, the

focus shifts to the average energy consumption over the course of training
epochs for each DRL algorithm: PPO, A2C, and DQN. The corresponding
graph, as illustrated in Figure 6, provides an insightful look into how each
algorithm manages to optimize energy usage while addressing the demands of
task offloading.
A notable observation is the consistent energy efficiency displayed by the
PPO algorithm, which manages to maintain a relatively lower energy con-
sumption throughout the simulation. This indicates PPO’s effective balance
between computational resource usage and the execution of offloading deci-
sions, highlighting its ability to sustain optimal operation with minimal energy
expenditure.

Mean Energy Consumption

PPO
20.0
A2C
DQN
17.5

15.0
Mean Energy (Joule)

12.5

10.0

7.5

5.0

2.5

0 500 1000 1500 2000 2500 3000 3500 4000

Epoches

Fig. 6 Average Energy consumption by epochs

On the other hand, A2C exhibits a slight variance in energy consumption

levels, suggesting a more dynamic adjustment to the offloading strategy that
may lead to periodic increases in energy utilization. This behavior underscores
the exploratory nature of A2C, which potentially navigates through a broader
spectrum of strategies to identify efficient offloading solutions.
DQN, while demonstrating a gradual improvement in energy efficiency as
the epochs progress, displays a higher average energy consumption compared
to PPO. This pattern reflects DQN’s iterative learning process, where initial
strategies may incur higher energy costs, gradually optimizing towards more
energy-efficient offloading decisions.
This energy analysis accentuates the differential impact of DRL algorithms
on energy consumption in task offloading scenarios. PPO emerges as a highly
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 21

energy-efficient approach, underscoring its applicability for energy-sensitive

MEC environments.

5.5 Delay by Priority Analysis

The analysis of task processing delay based on priority levels, as shown in Fig-
ure 7, illustrates the effectiveness of our DRL-based task offloading framework
in handling tasks with varying urgencies. In this context, ”Low” and ”High”
priority tasks were simulated to observe the average processing delay across
different DRL algorithms, including PPO, A2C, and DQN.

Average Task Processing Delay by Task Priority

PPO Delay
A2C Delay
25 DQN Delay
Average Task Processing Delay (ms)

20

15

10

0
Low High
Task Priority

Fig. 7 Average Task Processing Delay by Task Priority

The results indicate that PPO consistently achieves lower delays across
both priority levels, maintaining its lead as the most efficient algorithm for
minimizing task processing time. This is particularly evident with high-priority
tasks, where the immediacy of processing is crucial. A2C and DQN also show
commendable performance, with DQN experiencing slightly higher delays,
suggesting a more cautious approach to offloading decisions. This nuanced
understanding of delay implications across priorities underscores the adap-
tive capabilities of DRL algorithms in prioritizing critical tasks, ensuring that
high-priority tasks are expedited over less urgent ones, thereby optimizing the
overall system responsiveness.

5.6 Energy by Priority Analysis

Similarly, the evaluation of energy consumption in relation to task priority

reveals insights into the operational efficiency of the deployed DRL algorithms
22

(Figure 8). This analysis sheds light on how well each algorithm manages to
conserve energy while processing tasks of different priorities.

Average Energy Consumption by Task Priority

PPO Energy Consumption
A2C Energy Consumption
DQN Energy Consumption
4
Average Energy Consumption (Joule)

0
Low High
Task Priority

Fig. 8 Average Energy Consumption by Task Priority

PPO emerges as the most energy-efficient approach, particularly for high-

priority tasks, indicating its effectiveness in balancing the need for swift pro-
cessing with energy conservation. A2C and DQN follow closely, with all algo-
rithms showing an increase in energy consumption for high-priority tasks. This
increase is expected due to the additional computational resources required to
ensure prompt task processing. Nonetheless, the relatively low energy con-
sumption across all algorithms, even for high-priority tasks, highlights their
potential in optimizing energy usage in edge computing environments, an es-
sential factor for extending the battery life of mobile devices and sustaining
the operation of edge servers.

5.7 Total Delay Analysis

The cumulative analysis of total delay, illustrated in Figure 9, elucidates the

scalability and efficiency of the DRL-based task offloading strategy across a
broad spectrum of tasks. As the count of tasks grows, the framework adeptly
manages to keep the total delay within reasonable bounds, showcasing the
robustness of the DRL algorithms in handling varying workloads without sig-
nificant detriments to service latency.

5.8 Total Energy Consumption

Similarly, Figure 10 delves into the overall energy consumption trends as a

function of the number of tasks. The gradual increase in energy usage, as
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 23

Average Total Delay

PPO Total Delay
15 A2C Total Delay
DQN Total Delay

14
Average Total Delay (ms)

13

12

11

10

0 20 40 60 80 100
Number of Task

Fig. 9 Average total delay

tasks proliferate, underscores the system’s energy scalability. Notably, the DRL
algorithms exhibit a degree of energy efficiency, modulating the energy con-
sumption rate in response to the increasing demand, thereby ensuring that the
system remains energy-efficient even as task volume escalates.

Average Total Energy

PPO Total Energy
A2C Total Energy
4.0
DQN Total Energy
Average Total Energy (Joule)

3.5

3.0

2.5

2.0

1.5

1.0

0 20 40 60 80 100
Number of Task

Fig. 10 Average total Energy

6 CONCLUSION

In our proposed study, we have proposed a Deep Reinforcement Learning

(DRL)-based solution for task offloading, comparing three distinct method-
24

ologies: Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), and

Advance Actor Critic (A2C). We have formulated the mean reward function
by considering various Quality of Service (QoS) factors like Delay, Energy
consumption and task priority that the learning algorithm aims to optimize.
Our results indicate a clear preference for executing high-priority tasks over
low-priority ones. While our experimentation shows that DQN underperforms
in comparison to the other two algorithms, PPO and A2C exhibit highly com-
petitive performances. It’s worth noting that in this paper, we assume each
network component to be static.
In our forthcoming endeavors, we plan to extend our investigation by incor-
porating mobility into the designed fog-cloud architecture. Although we have
endeavored to simulate a realistic environment closely, unforeseen factors may
cause our proposed approach to behave differently in real-world scenarios, such
as encountering unknown delays, navigating high-dimensional spaces, and ex-
ecuting the reward function. Thus, our forthcoming endeavors will focus on
constructing a more realistic environment for performance evaluation.

References

1. Kansal, P., Kumar, M. Verma, O.P. Classification of resource management approaches

in fog/edge paradigm and future research prospects: a systematic review. J Supercomput
78, 13145–13204 (2022). [Link]
2. Kansal P, Kumar D, Kumar M (2020) Introduction to fog data analytics for IoT appli-
cations. A book chapter publish in “Springer Singapore” with ISBN 978–981–15–6044–6.
3. Kassir S, Veciana GD, Wang N, Wang X, Palacharla P (2020) Service placement for real-
time applications: rate-adaptation and load-balancing at the network edge. In: 2020 7th
IEEE Inter-national Conference on Cyber Security and Cloud Computing (CSCloud)/2020
6th IEEE Inter-national Conference on Edge Computing and Scalable Cloud (EdgeCom),
New York, NY, USA, 2020, pp. 207–215. https:// doi. org/ 10. 1109/ CSClo ud- EdgeC
om497 38. 2020. 00044
4. Brogi A, Forti S (2017) QoS-Aware deployment of IoT applications through the Fog.
IEEE Internet Things J 4(5):1185–1192. https:// doi. org/ 10. 1109/ JIOT. 2017. 27014
08
5. Atlam HF, Walters RJ, Wills GB (2018) “Fog computing and the internet of things: a
review” Big Data Cogn. Comput 2(2):10. https:// doi. org/ 10. 3390/ bdcc2 020010
6. Varghese B, Wang N, Nikolopoulos DS, Buyya R. 2017b. Feasibility of fog computing.
(2017). arXiv:1701.05451arXiv: 1701. 05451
7. X. Liu, J. Yu, J. Wang, and Y. Gao, “Resource allocation with edge computing in IoT
networks via machine learning,” IEEE Internet Things J., vol. 7, no. 4, pp. 3415–3426,
Apr. 2020.
8. D. Zeng, L. Gu, S. Pan, J. Cai, and S. Guo, “Resource management at the network edge:
A deep reinforcement learning approach,” IEEE Netw., vol. 33, no. 3, pp. 26–33, May/Jun.
2019
9. L. Huang, X. Feng, C. Zhang, L. Qian, and Y. Wu, “Deep reinforcement learning-based
joint task offloading and bandwidth allocation for multiuser mobile edge computing,”
Digit. Commun. Netw., vol. 5, no. 1, pp. 10–17, Feb. 2019.
10. F. Khoramnejad and M. Erol-Kantarci, “On joint offloading and resource allocation: A.
double deep Q-network approach,” IEEE Trans. Cogn. Commun. Netw., vol. 7, no. 4, pp.
1126–1141, Dec. 2021.
11. Y. Dai, K. Zhang, S. Maharjan, and Y. Zhang, “Edge intelligence for energy-efficient
computation offloading and resource allocation in 5G beyond,” IEEE Trans. Veh. Technol.,
vol. 69, no. 10, pp. 12175–12186, Oct. 2020.
Optimizing QoS in Edge Computing through DRL-Driven Task allocation 25

12. Y. Dai, K. Zhang, S. Maharjan, and Y. Zhang, “Deep reinforcement learning for stochas-
tic computation offloading in digital twin networks,” IEEE Trans. Ind. Informat., vol. 17,
no. 7, pp. 4968–4977, Jul. 2021
13. Chen, Y, Gu, W, Li, K. Dynamic task offloading for Internet of Things in mo-
bile edge computing via deep reinforcement learning. Int J Commun Syst. 2022;e5154.
doi:10.1002/dac.5154
14. Y. Chen, W. Gu, J. Xu, Y. Zhang and G. Min, ”Dynamic task offloading for digi-
tal twin-empowered mobile edge computing via deep reinforcement learning,” in China
Communications, vol. 20, no. 11, pp. 164-175, Nov. 2023, doi: 10.23919/[Link].2022-
0372.202302.
15. Zhang, D., Cao, L., Zhu, H. et al. Task offloading method of edge computing in internet
of vehicles based on deep reinforcement learning. Cluster Comput 25, 1175–1187 (2022).
[Link]
16. Ke, H.; Wang, H.; Sun, H. Multi-Agent Deep Reinforcement Learning-Based Partial
Task Offloading and Resource Allocation in Edge Computing Environment. Electronics
2022, 11, 2394. [Link]
17. J. Yang, Q. Yuan, S. Chen, H. He, X. Jiang and X. Tan, ”Cooperative Task Offloading
for Mobile Edge Computing Based on Multi-Agent Deep Reinforcement Learning,” in
IEEE Transactions on Network and Service Management, vol. 20, no. 3, pp. 3205-3219,
Sept. 2023, doi: 10.1109/TNSM.2023.3240415.
18. Bencan Gong, Xiaowei Jiang, ”Dependent Task-Offloading Strategy Based on
Deep Reinforcement Learning in Mobile Edge Computing”, Wireless Communi-
cations and Mobile Computing, vol. 2023, Article ID 4665067, 12 pages, 2023.
[Link]
19. Cai, J, Fu, H, Liu, Y. Deep reinforcement learning-based multitask hybrid comput-
ing offloading for multiaccess edge computing. Int J Intell Syst. 2022; 37: 6221-6243.
doi:10.1002/int.22841
20. Minxuan Chen, Aihuang Guo, Chunlin Song, Multi-agent deep reinforce-
ment learning for collaborative task offloading in mobile edge computing
networks, Digital Signal Processing, Volume 140, 2023, 104127, ISSN 1051-
2004,[Link]
21. F. Zhang, G. Han, L. Liu, M. Martı́nez-Garcı́a and Y. Peng, ”Deep Reinforcement
Learning Based Cooperative Partial Task Offloading and Resource Allocation for IIoT
Applications,” in IEEE Transactions on Network Science and Engineering, vol. 10, no. 5,
pp. 2991-3006, 1 Sept.-Oct. 2023, doi: 10.1109/TNSE.2022.3167949.
22. L. Tan, Z. Kuang, J. Gao and L. Zhao, ”Energy-Efficient Collaborative Multi-Access
Edge Computing via Deep Reinforcement Learning,” in IEEE Transactions on Industrial
Informatics, vol. 19, no. 6, pp. 7689-7699, June 2023, doi: 10.1109/TII.2022.3213603.
23. Z. Sheng, Y. Mao, J. Wang, H. Nie and J. Huang, ”Request-Aware Task Offloading
in Mobile Edge Computing via Deep Reinforcement Learning,” 2022 Tenth International
Conference on Advanced Cloud and Big Data (CBD), Guilin, China, 2022, pp. 294-299,
doi: 10.1109/CBD58033.2022.00059.
24. Linbo Liao, Yongxuan Lai, Fan Yang, Wenhua Zeng, Online computation offload-
ing with double reinforcement learning algorithm in mobile edge computing, Journal
of Parallel and Distributed Computing, Volume 171, 2023, Pages 28-39, ISSN 0743-
7315,[Link]
25. B. Yamansavascilar, A. C. Baktir, C. Sonmez, A. Ozgovde and C. Ersoy, ”DeepEdge:
A Deep Reinforcement Learning Based Task Orchestrator for Edge Computing,” in IEEE
Transactions on Network Science and Engineering, vol. 10, no. 1, pp. 538-552, 1 Jan.-Feb.
2023, doi: 10.1109/TNSE.2022.3217311.
26. Yu Sun, Qijie He, Joint task offloading and resource allocation for multi-user and multi-
server MEC networks: A deep reinforcement learning approach with multi-branch archi-
tecture, Engineering Applications of Artificial Intelligence, Volume 126, Part A, 2023,
106790, ISSN 0952-1976, [Link]
27. S. Misra and N. Saha, ”Detour: Dynamic Task Offloading in Software-Defined Fog for
IoT Applications,” in IEEE Journal on Selected Areas in Communications, vol. 37, no. 5,
pp. 1159-1166, May 2019, doi: 10.1109/JSAC.2019.2906793.
26

28. K. Arulkumaran, M. P. Deisenroth, M. Brundage and A. A. Bharath, ”Deep Reinforce-

ment Learning: A Brief Survey,” in IEEE Signal Processing Magazine, vol. 34, no. 6, pp.
26-38, Nov. 2017, doi: 10.1109/MSP.2017.2743240.
29. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. (2017). Proximal Policy
Optimization Algorithms. ArXiv. /abs/1707.06347
30. Huang, S., Kanervisto, A., Raffin, A., Wang, W., Ontañón, S., Dossa, R. F. (2022).
A2C is a special case of PPO. ArXiv. /abs/2205.09123
31. Zhang, L., Zhou, W., Xia, J. et al. DQN-based mobile edge computing for
smart Internet of vehicle. EURASIP J. Adv. Signal Process. 2022, 45 (2022).
[Link]

7 Declarations

Funding Not applicable

Conflicts of interest/Competing interests The authors declare that they have

no conflict of interest.

Availability of data and material Not applicable

Code availability Not applicable