[Link]

fi

Fog Computing for Deep Learning with Pipelines

Vainio, Antero
2023-05-01

Vainio , A , Mudvari , A , Kiedanski , D , Tarkoma , S & Tassiulas , L 2023 , Fog Computing

for Deep Learning with Pipelines . in 2023 IEEE 7th International Conference on Fog and
Edge Computing (ICFEC) . , 9 , IEEE Foundation , New York , pp. 64-72 , International
Conference on Fog and Edge Computing , Bangalore , India , 01/05/2023 . [Link]

[Link]
10.1109/ICFEC57925.2023.00017

unspecified
acceptedVersion

Downloaded from Helda, University of Helsinki institutional repository.

This is an electronic reprint of the original article.
This reprint may differ from the original in pagination and typographic detail.
Please cite the original version.
 Fog Computing for Deep Learning with Pipelines
Antero Vainio Akrit Mudvari Diego Kiedanski Sasu Tarkoma Leandros Tassiulas
University of Helsinki Yale University Yale University University of Helsinki Yale University

Abstract—In this article, we introduce a fog system design for Edge computing provides an alternative for centralized data
processing data collected from edge devices, such as mobile, sen- collection, by utilizing compute hardware located at the edge
sor, and extended (mixed, augmented, virtual) reality equipment. of the network [4]. By avoiding the need for communication,
Our system enables the network to provide hardware-accelerated
processors for resource-intensive computations on data gathered edge devices can run computations with a lower latency, and
from remote locations, such as 5G and beyond mobile networks. have the potential to improve users’ privacy. However, the edge
By splitting heavy computations into pipelines, and distributing is limited by the resource-constraints of mobile devices; certain
them among processors in the edge, fog and the cloud, our design computations may be practically infeasible to execute in the
benefits from the processing power of the cloud, while utilizing edge altogether. When developing applications that utilize the
fog devices with a lower network latency.
We implement our design, and use it for distributed training edge, one needs to consider the details of the device, such as
and inference with industry-grade deep learning models for the processor architecture as well as the available memory and
computer vision. We deploy our architecture in infrastructure disk space. [5]
including cloud and edge servers supporting GPU-accelerated Fog computing is another alternative to cloud and edge
computations. We benchmark pipelines in various deployment computing. It complements the two, by scattering compute
settings to study the overhead that they introduce. Our con-
tributions are a new design for wide-area data processing, a resources within the network. Edge devices can utilize the fog
framework that realizes this design and provides means of architecture, by offloading computations to the fog hardware
developing applications that are optimized in terms of infras- instead of directly to the cloud. Compared to cloud computing,
tructure and hardware. These contributions are complemented fog computing provides a lower expected network latency for
with our benchmark results, which reveal the potential causes of offloading. Fog computing has also advantages over edge com-
processing overhead.
Index Terms—fog computing, stream processing systems, deep puting, since the fog devices need not be mobile; static devices
learning have less physical space constraints than mobile devices, and
can include more compute power. Furthermore, static devices
I. I NTRODUCTION do not need to rely on batteries for power and offloading
Mobile devices constitute the largest share of devices con- appropriate computations from mobile devices can save their
nected to the Internet, and according to CISCO forecast, their batteries.
share will keep steadily increasing. Modern Internet-based When it comes to real-world deployment scenarios, the
applications, such as multimedia streaming and environmental primary considerations in fog computing are resource place-
sensing, produce massive amounts of data, and according to ment and allocation. Firstly, in order for the fog devices to
the current trends, they are expected to constitute the majority provide lower network latency than cloud, they need to be
of the network traffic in the future [1]. geographically closer to the users. For instance, in mobile edge
6G networks are envisioned to improve the broadband computing (MEC), this is guaranteed by deploying compute
connectivity in wireless networks drastically [2]. The peak data hardware to the mobile base stations, which the users connect
rate is expected to increase from 20 Gbps in 5G networks, to in order to get access to the Internet [6]. Secondly, due to
to 1 Tbps in 6G, and the end-to-end latency is expected to the highly distributed nature of fog computing, the deployed
reduce from 1 ms closer to .1 ms. These, among other quality hardware must be sized according to the demand; under-
improvements, enable new emerging applications, such as provisioning hardware negatively affects the quality of service,
holographic teleportation, autonomous cyber-physical systems, while over-provisioning makes fog computing economically
intelligent industrial automation, as well as smart infrastructure infeasible.
and environments [3]. While 6G networks will be able to col- In this paper, we investigate fog computing for geo-
lect more data for the purposes of the applications, in doing so, distributed data processing, by using software virtualisation
they also increase the traffic in the backbone network, in case in combination with hardware-acceleration. We present a
all of the data are transmitted to centralized data centers, such lightweight system design for integrating edge, fog and cloud
as cloud platforms. However, as most of the computational devices seamlessly via pipelined execution, and the associ-
resources are currently centralized, collecting the data is the ated programming framework, that enables the application-
primary alternative for computationally demanding tasks, such developer to create processing pipelines by defining each sub-
as data analytics and machine learning. task as a function. We use our framework to implement two
deep learning applications for computer vision, and prototype
our system design with devices, that provide GPU-accelerated
processing for deep learning. We conduct benchmarks to com-
979-8-3503-2288-0/23/$31.00 ©2023 IEEE.
Accepted manuscript, final version available at [Link]
Fig. 1: An overview of data processing in the cloud continuum where next-generation applications like XR and IoT may utilize
5G and other communication methods

pare the performance of the pipelines in various deployment A. Cloud, Edge and Fog Computing
scenarios. The source code for both our system and our
Cloud computing and big data have paved the way of dis-
example applications is open-sourced.
tributed data processing. Since the primary bottleneck for data
Combined our work provides answers to the following
processing tasks results from the data transfer speeds, which
questions:
tend to be orders of magnitude slower than the processing
1) In what conditions can the fog support data processing
speed of the chips, in cloud computing large computational
in the edge-cloud continuum?
tasks are parallelised by partitioning the data among the
2) What are the main causes and symptoms of processing
available servers. This approach can even the gap between
overhead in the fog?
the slow data access speed and the fast data processing speed
3) How does communication overhead involved in fog
of contemporary computer systems.
processing compare to that in cloud processing?
The existing data processing cluster solutions are primarily
The paper is organized as follows. In Section II we discuss
designed for data center environments, and there are no
the background and motivation for our work. In section III,
widely adapted data processing clusters designed for fog
we present the design, implementation and the deployment of
infrastructure. Cloud-based clusters generally take advantage
our architecture, as well as the two deep learning pipelines
of distributed file systems, such as HDFS [7], which rely on
using our framework: model training for image classification,
high-bandwidth and low-latency networks, and may exploit
and inference for object detection. In section IV we show the
a data center network topology designed according to the
results from our system evaluation, where we measured the
needs of a specific data infrastructure [8]. Due to these
runtime performance of the two deep learning pipelines. In
factors, using the Internet as the network for cloud-based data
section V we provide additional discussion about our results.
processing clusters is not possible, or it significantly affects
Finally, we give our final conclusion in section VI.
their performance, and new cluster designs aim to optimize
II. M OTIVATION & R ELATED W ORK the distributed execution of data processing tasks over wide-
An overview of the set of future use cases in networked area networks [9].
applications is illustrated in Figure 1. Here, different end The next generation cellular communication, in the form of
devices, be it environment-sensing IoT, AR/VR headsets, or 5G [10] and beyond, has become an important enabler for the
cell phones, will be utilizing the 5G infrastructure to access edge and the fog. While fog computing previously lacked the
the computing resources in the edge-cloud continuum. infrastructure for deployments, now with the proliferation of
 Master (1)
Intermediate
(∇T )
i+1
task deployer RX pipe Master
(1) (1)
(T ) (∇T )
data source i+1 i+1
task deployer
task executor
data source (T
(1)
i+1
) Worker
node cache
task deployer RX pipe

Worker Master (T
(2)
)
task executor
j+1
node cache
RX pipe task deployer

data source
task executor

node cache

(a) A three-node pipeline. (b) Two two-node pipelines using the same worker node.
(i) (i)
Fig. 2: Sparse architecture for pipelines. The offloaded tasks’ input parameters are Tj and the result values are ∇Tj .

5G infrastructure and drafting of the MEC standards [11], the [16], [17], so the current landscape of machine learning has
monopoly of the cloud is giving way to the edge and the fog. become highly distributed.
However, in this new 5G edge-cloud continuum, the number Our work builds upon the existing solutions for cloud-
of alternatives for deployment settings is higher, and the right based data processing, and edge computing with hardware-
selection of infrastructure may not always be obvious. acceleration, while considering the emerging landscape of 5G
As individual processor are limited by physical constraints, and beyond mobile networks, as well as distributed deep
and Moore’s Law has gradually ceased to improve the per-chip learning. In addition to outlining a practical system design,
compute performance, new ways to improve the performance we set out to develop a ubiquitous programming interface that
of data processing tasks are needed [12]. While using hardware can help data scientists to find the optimal configuration for
dedicated for specific computations used to be the norm before data processing applications, both in terms of the infrastructure
the popularity of general purpose CPUs, this direction has and the hardware.
gained a lot of recent interest, and today most of the cloud
service providers are either manufacturing their own chips, or III. S PARSE : TASK E XECUTION P IPELINES
using customized chips. By using specialized processors, more Unlike in cloud computing, where all of the resources are
computational performance can be gained per watt, however located in a few global regions, the placement and allocation
the gain depends on the task at hand. As with the selection of compute resources is a major challenge in fog computing,
of infrastructure, the proper choice of specialized processors where the end users’ locations continuously change. In fog
is application-specific. paradigm, compute resources are deployed sparsely in the
network to provide maximal coverage area. As a trade-off,
B. Distributed Machine Learning the fog devices cannot have processing power comparable to
As the size of the models used for state-of-the-art machine cloud environments.
learning has increased, their computational requirements have As a solution to improve the resource-constrained fog hard-
increased as well, resulting in a need to distribute their compu- ware utilization, we study networked processing pipelines,
tations. For neural networks with billions of parameters, model that, in line with OpenFog Reference Architecture [18], origi-
training in a feasible time requires cloud-scale infrastructure nate from the data source and progress among the fog devices,
[13]. Model training within a data center is usually distributed until either the task is completed, or the pipeline reaches the
by partitioning the training data and/or model parameters, and cloud. In practice, the end user offloads processing to the
parallelising access to them. However, centralized processing closest fog device, which runs part of the computation, and
requires uploading the training data, which may be subject offloads the rest to another nearby fog device. This makes it
to privacy regulations, to the cloud. Federated learning [14] possible for the fog devices to distribute computations among
avoids this problem, by instead bringing the model to the each other, enabling devices that are not being directly con-
mobile devices for training. While avoiding the need to col- sumed by end users to be utilized by other nearby locations.
lect raw user data, federated training generally convergences By utilizing pipelines, the fog environment becomes more
slower due to having to aggregate model updates by averaging robust to continuously changing demands, and avoids the need
gradients computed from non-IID data. The model size in fed- to physically relocate hardware. As a trade-off, having to
erated learning is also constrained by the mobile device, which partition computer program over the wide area network adds
must be able to compute the model update locally. To combine overhead. In this work, we study the details of task pipeline
edge processing and cloud offloading, authors in [15] suggest execution, identify and explain some causes of overhead, and
splitting neural network vertically, and offloading part of the how they can be avoided. We introduce a novel programming
processing to the cloud. New machine learning methods are framework to realize fog computing. We call our framework
constantly being explored, and even entire data systems have Sparse: Stream Processing Architecture for Resource-Subtle
been designed particularly for distributed machine learning Environments.
 Master Worker x∈X

Xk ⊂ X, yk ⊂ Y li = (Li ◦ Li−1 ◦ · · · ◦ L1 )(x) Master ∇l1 = (LT1 ◦ LT2 ◦ · · · ◦ Li )(∇li )

li = (Li ◦ · · · ◦ L1 )(Xk ) li ∇li
Send input (li , yk ) lj = (Lj ◦ Lj−1 ◦ · · · ◦ Li+1 )(li ) Intermediate ∇li = (LTi+1 ◦ LTi+2 · · · ◦ LTj )(∇lj )
Receive input
ŷ = (LN ◦ · · · ◦ Li+1 )(li )
lj ∇lj

∇li = (LT T
i+1 ◦ · · · ◦ LN )(G(ŷ, y))
ŷ = (LN ◦ LN −1 ◦ · · · ◦ Lj+1 )(lj ) Worker ∇lj = (LTj+1 ◦ LTj+2 ◦ · · · ◦ LTN )(G(y, ŷ))
(∇li , G(ŷ, y)) Send result

Receive result y∈Y

∇l1 = (LT T
1 ◦ · · · ◦ Li )(∇li )

Finish back propagation

Fig. 4: Extended split learning data flow

Fig. 3: Communication flow during model training pipeline. (explained later in more detail) is displayed in Figure 3.
The execution of the offloaded task is awaited without block- A new connection is established for each task, and it is
ing the execution of other tasks. kept open during the entire offloaded processing. The nodes
serialize messages using Python’s Pickle library, which, unlike
in text-based serialization schemes like JSON, instead converts
A. Pipeline Design Python objects directly into byte streams, providing lower
The general design of a basic three-node pipeline in Sparse transformation overhead for large payloads.
is depicted in Figure 2a. A Sparse cluster is formed by nodes The when serving offloading requests, a worker node uses
located either in the cloud, the fog or in the edge. The nodes first-come-first-serve policy for task execution. The task execu-
can create new processing tasks, or execute tasks created by tor is implemented as an abstract Python class, and it provides
other nodes. To execute tasks, a node assumes the role of access to the node functionality, such as the task deployer
a worker. To offload tasks, a node assumes the role of a in case the nodes is also offloading tasks. When using the
master and connects to a nearby worker node. In pipelines framework, an application developer only needs to implement
with three or more nodes, the intermediate nodes are acting the task executor. In our deep learning pipelines, we implement
as both masters and workers simultaneously. A worker node executors for training a model, and for using a trained model
can serve any number of masters, as is depicted in Figure 2b. for inference. These two executors are included in the Sparse
In order to submit new tasks, a master node includes a task library, and should work out-of-the-box for basic deep learning
deployer component. When submitting a new task, the task training and inference.
deployer opens a network connection to one or more workers, While the Sparse framework is mainly designed for creating
and starts sending the task’s input data. The worker node pipelines, there is no application level primitives included for
includes an RX pipe which accepts the connection and receives pipelines, but instead pipelines are created by connecting the
the input data. Once all the needed data are transmitted, nodes with the partitioned data. For dynamic configuration
RX pipe transfers them to the task executor, which runs the of pipelines, the framework can be interfaced with external
computation. In practice a task executor may be a hardware- orchestration tools, like Kubernetes or Docker Swarm.
accelerator. The RX pipe then responds to the master node
with the result of the computation. C. Deep Learning Pipelines
Since nodes can cache some of the data used in consecutive We use our framework to implement two deep learning
computations, such as machine learning model parameters, pipelines: one for training, and another for inference. The
pipelines can reduce the need for communication. This is par- training pipeline takes the split learning [15] approach to
ticularly useful in iterative algorithms, like machine learning partition the used model based on neural layers, so that the
model training. training can be performed collaboratively by nodes, which
B. Implementation do not have enough resources to operate on the full model.
The inference pipeline splits the model similarly, but does not
We implement our framework with Python, and deploy it
include backward propagation.
in containers using Docker. The source code for the imple-
mentation is available in GitHub1 . The implementation uses Figure 4 illustrates an example of a three node train-
an asynchronous programming pattern for concurrency. The ing pipeline, where a neural network, consisting of layers
implementation includes a module for deep learning, using L1 , . . . LN is trained by three nodes, by using a feature vector
PyTorch, which supports general purpose CPUs, as well as x ∈ X and a label y ∈ Y . The model is trained with standard
hardware-accelerators like GPUs and TPUs. forward/backward propagation approach. First a prediction is
The prototype currently implements RPC for communica- computed for x, using the model’s current parameters, in a
tion. The communication flow during model training pipeline sequential fashion. Since the model is split, the computation
for the Intermediate node would happen in the following
1 [Link] fashion:
 (a) Training VGG (b) Inference YOLOv3
Fig. 5: Task completion time measured in the data source. The curve step height corresponds to the used batch size (1 without
batching).

our deep learning pipelines for computer vision, a widely pop-

lj = (Lj ◦ Lj−1 ◦ · · · ◦ Li+1 )(li ) (1) ular task in machine learning. In particular, computer vision
models often involve billions of parameters [13], making it
, including j − i layers of neural network. The input for
challenging to deploy them in resource-constrained devices.
the Master node would be the feature vector x, and the output
We measure the task completion time and the network usage
of the Worker node is the final prediction ŷ. This forward
in the data source, as well as the energy consumption and the
propagation is illustrated on the left side of Equation 1.
hardware utilization in the fog device.
Following the approach in supervised learning, the correct
label y ∈ Y for the prediction is assumed to be known.
In practice this can be achieved by manual labeling, or A. Methodology
by extending the system for Reinforcement Learning (RL) Our testbed consists of a cloud server, and an SoC as
[19]. The correct label is used along with a cost function a cloudlet. As a data source we use a laptop, that runs
G : Y × Y → R, to find adjustments for the model parameters PyTorch data loader. Both the cloud server and the cloudlet
that best minimise the cost function, fixed for x. The most support CUDA for GPU-accelerated processing. The hardware
common technique is to use the chain derivation rule for specifications of the devices are shown in Table I. The es-
the function described in Figure 4. Since Lk is usually not timated peak performances are based on the manufacturer’s
differentiable, the intermediate results, gained during forward specifications. The ping latency was measured before running
propagation, are used to compute the derivative numerically. the benchmarks.
Following similar notations to what is used in [15], we will We evaluate the system with industry grade deep learning
denote the computations involved in backward propagation models for computer vision. For training, we use Visual
simply LTk . During the computation across layers, ∇lk is Geometry Group (VGG) model architecture [20] for image
computed for each layer k = 1 . . . N , and they constitute the classification, and the CIFAR-10 [21] dataset. For inference,
final model update. The back propagation for the Intermediate we use the YOLOv3 [22] object detection model, and the
node takes the gradient of the split layer it sent during the COCO dataset.
forward propagation as an input, calculated as: We ran various benchmarks suites, each utilizing a different
∇li = (LTi+1 ◦ LTi+2 ◦ · · · ◦ LTj )(∇lj ) (2) combination of cloud, edge and fog resources. Two of the
suites split the task, while two process everything in the same
For the Worker node, the calculation of the gradient includes node. Throughout this section, we will use the following
discerning the loss function between the prediction ŷ and the names for different benchmark suites used in the system
true label y. This calculation is denoted as: evaluation:
∇lj = (LTj+1 ◦ LTj+2 ◦ · · · ◦ LTN )(G(y, ŷ)) (3) • Cloud: A two-node pipeline including data source and a
cloud server. The data source offloads entire processing
In our framework, we define each model partition as a to the cloud server, without splitting the task.
separate function to create the pipeline. Forward and back- • Cloudlet: Similar to the cloud suite, except the processing
ward propagations computations use hardware-accelerated task is offloaded to the cloudlet instead.
executors, that cache model parameters in memory during the
entire training. We also batch training data to boost the training
on GPUs. TABLE I: Hardware specifications of the devices used in the
testbed.
IV. P IPELINE B ENCHMARKS
Device Peak Perf. (INT8) Ping latency
To study the processing overhead introduced by our
pipelines, we benchmark the execution of the same task in Cloud 130 TOPS ≈ 9 ms
Cloudlet 22 TOPS ≈ 1 ms
different pipeline configurations. As an example case, we use
 (a) Training VGG in the Hybrid On-Device suite. (b) Inferring with YOLOv3 in the Hybrid On-Device suite.

(c) Training VGG in the Cloudlet suite. (d) Training VGG in the Cloud suite.
Fig. 6: Network usage measured in the data source.

• Hybrid Off-Device A three-node pipeline, where the data result demonstrates that the fog infrastructure can complement
source offloads the processing to the cloudlet and the cloud-based data processing to improve task throughput.
cloud server, which split the task. In contrast to training however, for inference without batch-
• Hybrid On-Device A two-node pipeline, where the data ing, the off-device hybrid deployment achieved the worst
source also processes the first split, and offloads the rest overall throughput. Since batching was the only method that
to a cloudlet. we used to fine-tune the workloads, our results for inference
During benchmarks, we measure the task completion time mainly display the effectiveness of split neural network traver-
in the data source, representing the task throughput, as well as sal. Even though inference in the cloudlet without splitting the
hardware statistics in the cloudlet. Benchmarks are measured model provided the highest throughput, combination of on-
by an additional daemon process running at each cluster node. device processing and edge offloading was faster than cloud
After a task has been processed, the monitored node sends offloading.
a JSON message to the monitor daemon via a Unix socket, The communication overhead of neural network splitting
without waiting for the reply. For more details, see Appendix in the pipelines is displayed for training in Figure 6a and
A for the pseudocode of the benchmarks. The source code for for inference in Figure 6b. The added communication results
the monitoring is also included in Sparse source repository. mainly from the fact that intermediate layers in feed forward
networks tend to be of higher dimension than the feature vec-
B. Results tors. In our experiments, the network load of communicating
Figure 5 shows the CDF of processed samples in training the intermediate layer outputs is roughly four times higher
and inference. Appendix B shows, how the batch size affects than communicating the raw data.
the training time in each, and essentially it shows the batch Figures 6c and 6d show the effect of batch size for commu-
sizes that provided the fastest training for each benchmark. nication overhead during training without task splitting in the
Even though nodes load the model parameters into memory cloudlet and the cloud. For the resource-constrained cloudlet,
before the benchmarks are started, Figure 5 reveals that there using too big or too small batch size reduces the average trans-
is also a slow start involved in our pipelines; in all cases the mission rate significantly. Essentially, there is a period during
first batch is the slowest to compute. While the slow start each batch, when no data is being communicated, suggesting
does not affect our main results, we note that it may be an that the pipelines become compute-bound. In contrast, when
important consideration in practice, and advocates executing using an appropriate batch size, the transmission rate stays
pipelines for longer periods of time. more constant.
For training, by using a three node pipeline between the Figure 6d highlights, how the overall communication over-
cloud and the edge, we were able to increase the batch size head reduces as batch size increases. As the cloud GPU
for the pipeline, enabling the hybrid off-device suite to achieve supports a larger batch size for training the same model as the
the fastest overall training time, slightly beating the cloudlet cloudlet, it avoids a lot of additional communication during
suite. Even though out of all the suites hybrid off-device training, such as gradients which are computed per batch. For
uses the most hardware, and as a result more power, this a massive batch size of 2048, the overall network traffic drops
 (a) 64 batch size (too small) (b) 512 batch size (too big)

(c) 128 batch size (optimal)

Fig. 7: GPU and power usage in the fog device during the Cloudlet training suite.

by almost half in comparison to a more conservative batch acceptable end-to-end latency, although these result also from
size of 256. While this would be ideal in terms of overall data the used model and the dataset, which are not adjusted for an
traffic, it has the biggest potential to congest the backbone XR use scenario.
network, and the fact that with a batch size smaller than 2048,
V. D ISCUSSION
cloud node shows signs of becoming compute-bound makes
cloud offloading seem challenging to optimize without having Our benchmark results demonstrate the difference, that the
to transmit massive amounts of data. Nevertheless, our results selection of infrastructure and hardware make for data process-
demonstrate how proper batching has the potential to reduce ing in the edge-cloud continuum. They highlight the impact of
communication overhead during neural network training. proper hardware utilization to the throughput of deep learning
tasks. In practice, the applications determine the available
Figure 7 displays the GPU usage and the power usage optimization methods, and we noticed how the infrastructure
measured in the fog device during the training with different that performed the best for batch training provided the worst
batch sizes. The measured GPU usage is at maximum for performance for inference without batching.
most of the training, even when using a suboptimal batch Some of the results in our study are explained by the fact
size. However, different batch sizes make a clear difference that we used mainly GPUs, which are designed for computa-
in the device power usage. The base power consumption is tions that can be parallelized, by including a high number of
approximately 10 W, which is the system power use when the cores: as a result of this design choice, the processing speed of
GPU is not being used. When training with too small batches, individual GPU cores is generally lower than cores in a CPU.
the overall power usage increases only slightly, indicating that Without batching, some other means of parallelizing individual
the GPU is not being fully utilized. Similar patterns can be inference tasks are needed to fully utilize GPUs, which can
seen when using too big batches, with the exception that there be seen in our benchmark results.
are small peaks in power usage; these peaks likely result from Task splitting proved to be a viable option to scale opera-
the details of the VGG model architecture. In comparison to tions on feed forward networks, particularly for the purposes
training with a too small batch size, when using the optimal of model training. The freed compute capacity can this way be
batch size, the power usage increases to almost twice as high, used to increase throughput of tasks, at least when tasks can be
but the training lasts for less than a quarter. batched. Even without batching, by starting the processing in
Similar measurements for inference are displayed in Figure the data source, our hybrid on-device pipeline reached nearly
8. Similarly to training, the optimal workload in inference as fast processing as edge offloading, and even outperformed
has the highest power consumption during processing, while cloud offloading.
the overall processing time is smaller. Overall, for inference We notice in Figure 6c, how suboptimal configuration
without batching, all of the benchmark suites show signs of results in a step-like curve in network traffic CDF. Essentially,
hardware underutilization, and none of the benchmarks reach there are periods during which no data is being transmitted,
 (a) Cloudlet (fastest) (b) Hybrid Off-Device (slowest)
Fig. 8: GPU and power usage in the fog device during inference suites.

and instead the device is processing previous task. Combined Interestingly, even though the split processing pipelines that
with the GPU and power measurements, the network usage we studied included more communication than their unsplit
reveals that when using too small or too big a batch size, the counterparts, in some instances, including the optimal config-
task becomes compute-bound, whereas with an optimal batch urations, the resulting overhead was mitigated by the benefits
size, the task is primarily communication-bound. gained in task throughput. Due to these results, we conclude
By implementing pipelines, we have provided benchmark that a smaller overall communication does not guarantee that
results for two real-world use cases utilizing deep learning a task is processed faster, but note that it may be desirable for
for computer vision. Our results show the difference in the other reasons, such as avoiding network congestion.
hardware utilization, when using different infrastructure for Based on our current results, it seems that longer pipelines
processing, leading to a significant difference in the comple- are best utilized for background tasks, that are expensive to
tion time of computations. compute, but do not have strict latency requirements, such
as model training. Unlike training however, inference tasks
VI. C ONCLUSION
have strict latency requirements, making batching an infeasible
With many variables affecting the performance of pipelines, solution in many situations, since it results in additional
it is difficult to draw conclusions about which combination expected waiting time for most of the individual requests.
of infrastructure and hardware would be universally the best. On the other hand, there may not be a need to rely on
While this study shows some cases where the fog is or is not offloading for inference in the first place; as an example, model
useful for enhancing cloud-based data processing, we found compression seeks to reduce the size and the computational
our framework to be useful for prototyping such applications, cost of deep learning models [23]. However, this approach
and helpful in finding these optimal configurations. As such, sacrifices models’ representation capabilities, and as a result
we will keep using and developing the framework, and hope their accuracy.
that it will be helpful for other researchers working in this Overall, we haven’t seen similar studies elsewhere, since
domain as well. most of the pipeline studies we know (e.g. [24]) focus on
To answer our first research question, the benchmark results input pipelines and not end-to-end processing. As we expect
demonstrate, that if the fog hardware can be fully utilized, it the future networked applications to keep becoming more data-
will improve the throughput of heavy workloads. To reach high driven, our results show great promise for using fog computing
hardware utilization, the workloads need to be appropriately to enhance or even replace cloud-based data processing.
sized. For neural network training, the workload size can be
fine-tuned, for instance with the choice of the batch size, as we ACKNOWLEDGMENTS
demonstrated. In practice this is achievable in fog computing, This research was in part supported by the Academy of
where the processing hardware is shared by multiple users, Finland (grant number 345008), and the National Science
making aggregating input data is easier than in on-device edge Foundation CNS AI Institute (grant number 2112562), as well
computing. as the NSF-AoF FAIN project (grant number 2132573).
Pipelines introduce overhead to processing, if the aforemen-
tioned condition cannot be met, in which case the intermediate R EFERENCES
processors create bottlenecks in the pipeline. The bottlenecks
[1] “Cisco annual internet report (2018–2023) white
can turn ideally communication-bound processing pipelines paper,” accessed: 2022-1-21. [Online]. Available: https:
compute-bound. While the most crucial symptom of the bot- //[Link]/c/en/us/solutions/collateral/executive-perspectives/
tlenecks is a drop in task throughput, other symptoms include annual-internet-report/[Link]
[2] N. Rajatheva, I. Atzeni, E. Bjornson, A. Bourdoux, S. Buzzi, J.-B. Dore,
uneven communication flow, and oscillating or lower processor S. Erkucuk, M. Fuentes, K. Guan, and Y. Hu, “White paper on broadband
power consumption during the pipeline execution. connectivity in 6G,” arXiv preprint arXiv:2004.14247.
 [3] I. F. Akyildiz, A. Kak, and S. Nie, “6G and beyond: The future of A PPENDIX A
wireless communications systems,” IEEE access, vol. 8, pp. 133 995– B ENCHMARK PSEUDOCODE
134 030, 2020, publisher: IEEE.
[4] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and
challenges,” IEEE internet of things journal, vol. 3, no. 5, pp. 637–646, Algorithm 1 Data source training completion benchmark
2016, publisher: IEEE.
[5] M. Satyanarayanan, N. Beckmann, G. A. Lewis, and B. Lucia, “The role 1: procedure B ENCHMARK T RAINING(epochs)
of edge offload for hardware-accelerated mobile devices,” GetMobile: 2: monitor [Link] benchmark()
Mobile Computing and Communications, vol. 25, no. 2, pp. 5–13, 2021, 3: for i ← 1 to epochs do
publisher: ACM New York, NY, USA.
[6] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey 4: for X, y in batch do
on mobile edge computing: The communication perspective,” IEEE 5: grad, loss ← await offload training(X, y)
communications surveys & tutorials, vol. 19, no. 4, pp. 2322–2358, 6: monitor [Link] processed(batch size(X))
2017, publisher: IEEE.
[7] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The hadoop 7: end for
distributed file system,” in 2010 IEEE 26th symposium on mass storage 8: end for
systems and technologies (MSST). Ieee, 2010, pp. 1–10. 9: end procedure
[8] A. Roy, H. Zeng, J. Bagga, G. Porter, and A. C. Snoeren, “Inside the
social network’s (datacenter) network,” in Proceedings of the 2015 ACM
Conference on Special Interest Group on Data Communication, 2015,
pp. 123–137. Algorithm 2 Data source inference benchmark
[9] F. Lai, J. You, X. Zhu, H. V. Madhyastha, and M. Chowdhury,
“Sol: Fast Distributed Computation Over Slow Networks,” in 17th 1: procedure B ENCHMARK I NFERENCE
${$USENIX$}$ Symposium on Networked Systems Design and Imple- 2: monitor [Link] benchmark()
mentation (${$NSDI$}$ 20), 2020, pp. 273–288.
[10] A. Ghosh, A. Maeder, M. Baker, and D. Chandramouli, “5g evolution: A
3: for x ∈ samples do
view on 5g cellular technology beyond 3gpp release 15,” IEEE access, 4: await offload inference(x)
vol. 7, pp. 127 639–127 651, 2019. 5: monitor [Link] processed(1)
[11] N. Abbas, Y. Zhang, A. Taherkordi, and T. Skeie, “Mobile edge
computing: A survey,” IEEE Internet of Things Journal, vol. 5, no. 1,
6: end for
pp. 450–465, 2017. 7: end procedure
[12] T. N. Theis and H.-S. P. Wong, “The end of moore’s law: A new begin-
ning for information technology,” Computing in Science & Engineering,
vol. 19, no. 2, pp. 41–50, 2017, publisher: IEEE.
[13] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ran- Algorithm 3 Worker task completion benchmark
zato, A. Senior, P. Tucker, and K. Yang, “Large scale distributed deep
networks,” Advances in neural information processing systems, vol. 25,
1: procedure R ECEIVE TASK(input reader)
2012. 2: if ¬ benchmark started then
[14] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, 3: monitor [Link] benchmark()
“Communication-efficient learning of deep networks from decentralized
data,” in Artificial intelligence and statistics. PMLR, 2017, pp. 1273–
4: end if
1282. 5: input data ← await input [Link]()
[15] O. Gupta and R. Raskar, “Distributed learning of deep neural network 6: await process task(input data)
over multiple agents,” Journal of Network and Computer Applications,
vol. 116, pp. 1–8, 2018, publisher: Elsevier.
7: monitor [Link] processed()
[16] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, 8: end procedure
M. Elibol, Z. Yang, W. Paul, and M. I. Jordan, “Ray: A distributed
framework for emerging ${$ai$}$ applications,” in 13th USENIX Sym-
posium on Operating Systems Design and Implementation (OSDI 18),
2018, pp. 561–577. A PPENDIX B
[17] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, BATCH OPTIMIZATION FOR TRAINING
V. Ivanov, C. Kiddon, J. Konecny, S. Mazzocchi, H. B. McMahan,
and others, “Towards federated learning at scale: System design,” arXiv
preprint arXiv:1902.01046, 2019.
[18] “Ieee standard for adoption of openfog reference architecture for fog
computing,” IEEE Std 1934-2018, pp. 1–176, 2018.
[19] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement
learning: A survey,” Journal of artificial intelligence research, vol. 4,
pp. 237–285, 1996.
[20] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[21] A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features
from tiny images,” 2009. (a) Cloudlet (b) Hybrid On-Device
[22] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look
once: Unified, real-time object detection,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2016, pp. 779–
788.
[23] “tinyml foundation,” accessed: 2022-9-12. [Online]. Available: https:
//[Link]/
[24] M. Kuchnik, A. Klimovic, J. Simsa, V. Smith, and G. Amvrosiadis,
“Plumber: Diagnosing and removing performance bottlenecks in ma-
chine learning data pipelines,” Proceedings of Machine Learning and
Systems, vol. 4, pp. 33–51, 2022. (c) Hybrid Off-Device