Deep Learning With Edge

Computing: A Review
This article provides an overview of applications where deep learning is used at the
network edge. Computer vision, natural language processing, network functions, and
virtual and augmented reality are discussed as example application drivers.
By J IASI C HEN AND X UKAN R AN

ABSTRACT | Deep learning is currently widely used in a vari- I. I N T R O D U C T I O N

ety of applications, including computer vision and natural
Deep learning has recently been highly successful in
language processing. End devices, such as smartphones and
machine learning across a variety of application domains,
Internet-of-Things sensors, are generating data that need to
including computer vision, natural language processing,
be analyzed in real time using deep learning or used to
and big data analysis, among others. For example, deep
train deep learning models. However, deep learning inference
learning methods have consistently outperformed tradi-
and training require substantial computation resources to run
tional methods for object recognition and detection in
quickly. Edge computing, where a fine mesh of compute nodes
the ISLVRC Computer Vision Competition since 2012 [1].
are placed close to end devices, is a viable way to meet However, deep learning’s high accuracy comes at the
the high computation and low-latency requirements of deep
expense of high computational and memory requirements
learning on edge devices and also provides additional bene-
for both the training and inference phases of deep learning.
fits in terms of privacy, bandwidth efficiency, and scalability. Training a deep learning model is space and computation-
This paper aims to provide a comprehensive review of the
ally expensive due to millions of parameters that need to
current state of the art at the intersection of deep learning
be iteratively refined over multiple time periods. Inference
and edge computing. Specifically, it will provide an overview
is computationally expensive due to the potentially high
of applications where deep learning is used at the network
dimensionality of the input data (e.g., a high-resolution
edge, discuss various approaches for quickly executing deep image) and millions of computations that need to be
learning inference across a combination of end devices, edge
performed on the input data. High accuracy and high
servers, and the cloud, and describe the methods for training
resource consumption are defining characteristics of deep
deep learning models across multiple edge devices. It will also learning.
discuss open challenges in terms of systems performance,
To meet the computational requirements of deep learn-
network technologies and management, benchmarks, and pri-
ing, a common approach is to leverage cloud computing.
vacy. The reader will take away the following concepts from
To use cloud resources, data must be moved from the
this paper: understanding scenarios where deep learning at
data source location on the network edge [e.g., from
the network edge can be useful, understanding common tech- smartphones and Internet-of-Things (IoT) sensors] to a
niques for speeding up deep learning inference and performing
centralized location in the cloud. This potential solution
distributed training on edge devices, and understanding recent
of moving the data from the source to the cloud introduces
trends and opportunities. several challenges.
KEYWORDS | Artificial intelligence; edge computing; machine 1) Latency: Real-time inference is critical to many
learning; mobile computing; neural networks. applications. For example, camera frames from an
autonomous vehicle need to be quickly processed to
detect and avoid obstacles or a voice-based-assistive
Manuscript received February 7, 2019; revised April 30, 2019; accepted May 29,
2019. Date of publication July 15, 2019; date of current version August 5, 2019. application needs to quickly parse and understand
This work was supported in part by NSF CNS-1817216. (Corresponding author: the user’s query and return a response. However,
Jiasi Chen.)
The authors are with the Department of Computer Science and Engineering, sending data to the cloud for inference or train-
University of California at Riverside, Riverside, CA USA (e-mail: jiasi@[Link]). ing may incur additional queuing and propaga-
Digital Object Identifier 10.1109/JPROC.2019.2921977 tion delays from the network and cannot satisfy

0018-9219 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See [Link] for more information.

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1655

Chen and Ran: Deep Learning With Edge Computing: A Review

strict end-to-end low-latency requirements needed

for real time, interactive applications; for example,
real experiments have shown that offloading a cam-
era frame to an Amazon Web Services server and
executing a computer vision task take more than
200-ms end-to-end [2].
2) Scalability: Sending data from the sources to the
cloud introduces scalability issues, as network access
to the cloud can become a bottleneck as the num-
ber of connected devices increases. Uploading all
data to the cloud is also inefficient in terms of
network resource utilization, particularly if not all
data from all sources are needed by the deep learn-
ing. Bandwidth-intensive data sources, such as video Fig. 1. Deep learning can execute on edge devices (i.e., end
devices and edge servers) and on cloud data centers.
streams, are particularly a concern.
3) Privacy: Sending data to the cloud risks privacy
concerns from the users who own the data or Pi devices. A second challenge is understanding how the
whose behaviors are captured in the data. Users edge devices should coordinate with other edge devices
may be wary of uploading their sensitive informa- and with the cloud, under heterogeneous processing capa-
tion to the cloud (e.g., faces or speech) and of bilities and dynamic network conditions, to ensure a good
how the cloud or application will use these data. end-to-end application-level performance. Finally, privacy
For example, the recent deployment of cameras remains a challenge, even as edge computing naturally
and other sensors in a smart city environment in improves privacy by keeping data local to the network
New York City incurred serious concerns from pri- edge, as some data often still need to be exchanged
vacy watchdogs [3]. between edge devices and possibly the cloud. Researchers
Edge computing is a viable solution to meet the latency, have proposed various approaches from diverse angles to
scalability, and privacy challenges described earlier in tackle these challenges, ranging from hardware design to
this section. In edge computing, a fine mesh of compute system architecture to theoretical modeling and analy-
resources provides computational abilities close to the end sis. The purpose of this paper is to survey works at
devices [4]. For example, an edge compute node could the confluence of the two major trends of deep learning
be co-located with a cellular base station and an IoT and edge computing, in particular focusing on the soft-
gateway or on a campus network. Edge computing is ware aspects and their unique challenges therein. While
already being deployed by industry; for example, a major excellent surveys exist on deep learning [7] as well as
cellular Internet service provider in the United States and edge computing [8], [9] individually, this paper focuses on
a national fast-food chain have both deployed edge com- works at their intersection.
pute services [5], [6]. To address latency challenges, edge Deep learning on edge devices has similarities to, but
computing’s proximity to data sources on the end devices also differences from, other well-studied areas in the litera-
decreases end-to-end latency and thus enables real-time ture. Compared to cloud computing that can help run com-
services. To address scalability challenges, edge computing putationally expensive machine learning (e.g., machine
enables a hierarchical architecture of end devices, edge learning as a service), edge computing has several advan-
compute nodes, and cloud data centers that can pro- tages, such as lower latency and greater geospatial speci-
vide computing resources and scale with the number of ficity that have been leveraged by researchers [10]. Several
clients, avoiding network bottlenecks at a central location. works have combined edge computing with cloud com-
To address privacy challenges, edge computing enables puting, resulting in hybrid edge-cloud architectures [11].
data to be analyzed close to the source, perhaps by a local Compared to traditional machine learning methods (out-
trusted edge server, thus avoiding traversal of the public side of deep learning), deep learning’s computational
Internet and reducing exposure to privacy and security demands are particularly a challenge, but deep learning’s
attacks. specific internal structure can be exploited to address
While edge computing can provide the latency, scalabil- this challenge (see [12]–[14]). Compared to the growing
ity, and privacy benefits discussed earlier in this section, body of work on deep learning for resource-constrained
several major challenges remain to realize deep learning at devices, edge computing has additional challenges relat-
the edge. One major challenge is accommodating the high ing to shared communication and computation resources
resource requirements of deep learning on less powerful across multiple edge devices.
edge compute resources. Deep learning needs to execute In the rest of this paper, we define the edge devices
on a variety of edge devices, ranging from reasonably that include both end devices (e.g., smartphones or IoT
provisioned edge servers equipped with a GPU, to smart- sensors), as well as edge compute nodes or servers,
phones with mobile processors, to barebones Raspberry as shown in Fig. 1. This paper is organized as follows.

1656 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

Table 1 Common Performance Metrics

In each pass, a randomly selected “mini-batch” of samples

Fig. 2. DNN example of image classification.
is selected and used to update the gradients in the direction
that minimizes the training loss (where the training loss is
defined as the difference between the predictions and the
We first provide a brief background on deep learning
ground truth). One pass through the entire training data
(see Section II). We then describe several application
set is called a training epoch [15].
domains where deep learning on the network edge can be
A key takeaway for the purposes of this work is that
useful (see Section III). In Section IV, we discuss different
there are a large number of parameters in the matrix
architectures and methods to speed up deep learning infer-
multiplications, resulting in many computations being
ence, focusing on device-only execution, always computing
performed and thus the latency issues that we see on
on the edge server, and intermediate alternatives, such as
end devices. A second takeaway is that there are many
offloading, hybrid edge-cloud, and distributed computing
choices (hyperparameters) on how to design the DNN
approaches. We then discuss training deep learning models
models (e.g., the number of parameters per layer, and the
on edge devices, with an emphasis on distributed training
number of layers), which makes the model design more of
across devices and privacy (see Section V). Finally, we fin-
an art than a science. Different DNN design decisions result
ish with open research challenges (see Section VI) and
in tradeoffs between system metrics; for example, a DNN
conclusions (see Section VII).
with higher accuracy likely requires more memory to store
all the model parameters and will have higher latency
II. B A C K G R O U N D , M E A S U R E M E N T S , because of all the matrix multiplications being performed.
AND FRAMEWORKS On the other hand, a DNN model with fewer parameters
A. Background on Deep Learning will likely execute more quickly and use less computa-
Since some of the techniques discussed in this paper tional resources and energy, but it may not have sufficient
rely on the specific internals of deep learning, therefore, accuracy to meet the application’s requirements. Several
we first provide a brief background on deep learning. works exploit these tradeoffs, which will be discussed in
Further details can be found in reference texts (see [7]). Sections IV-B and IV-C.
A deep learning prediction algorithm, also known as a
model, consists of a number of layers, as shown in Fig. 2. B. Measurements of Deep Learning Performance
In deep learning inference, the input data pass through Deep learning can be used to perform both supervised
the layers in sequence, and each layer performs matrix learning and unsupervised learning. The metrics of success
multiplications on the data. The output of a layer is depend on the particular application domain where deep
usually the input to the subsequent layer. After data are learning is being applied. For example, in object detection,
processed by the final layer, the output is either a feature the accuracy may be measured by the mean average preci-
or a classification output. When the model contains many sion (mAP) [1], which measures how well the predicted
layers in sequence, the neural network is known as a object location overlaps with the ground-truth location,
deep neural network (DNN). A special case of DNNs is averaged across multiple categories of objects. In machine
when the matrix multiplications include convolutional fil- translation, the accuracy can be measured by the bilin-
ter operations, which is common in DNNs that are designed gual evaluation understudy score metric [16], which com-
for image and video analysis. Such models are known pares a candidate translation with several ground-truth
as convolutional neural networks (CNNs). There are also reference translations. Other general system performance
DNNs designed especially for time series prediction; these metrics not specific to the application include through-
are called recurrent neural networks (RNNs) [7], which put, latency, and energy. These metrics are summarized
have loops in their layer connections to keep state and in Table 1.
enable predictions on sequential inputs. Designing a good DNN model or selecting the right DNN
In deep learning training, the computation proceeds model for a given application is challenging due to the
in reverse order. Given the ground-truth training labels, large number of hyperparameter decisions. A good under-
multiple passes are made over the layers to optimize the standing of the tradeoffs between the speed, accuracy,
parameters of each layer of matrix multiplications, starting memory, energy, and other system resources can be helpful
from the final layer and ending with the first layer. The for the DNN model designer or the application developer.
algorithm used is typically stochastic gradient descent. These comparative measurements are typically presented

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1657

Chen and Ran: Deep Learning With Edge Computing: A Review

in research papers proposing new models or standalone on-device inference abilities, not training, and achieves
measurement papers [17]. An especially important consid- low latency by compressing a pre-trained DNN model.
eration in the context of edge computing is the testbed Caffe [27]–[29] is another deep learning framework,
that the measurements are conducted on. Machine learn- originally developed by Jia, with the current version,
ing research typically focuses on accuracy metrics, and Caffe2, maintained by Facebook. It seeks to provide an easy
their system performance results are often reported from and straightforward way for deep learning with a focus on
powerful server testbeds equipped with GPUs. For exam- mobile devices, including smartphones and Raspberry Pis.
ple, Huang et al. [17] compared the speed and accuracy PyTorch [30] is another deep learning platform developed
tradeoffs when running on a high-end gaming GPU (Nvidia by Facebook, with its main goal differing from Caffe2 in
Titan X). The YOLO DNN model [18], which is designed which it focuses on the integration of research proto-
for real-time performance, provides timing measurements types to production development. Facebook has recently
on the same server GPU. announced that Caffe2 and PyTorch will be merging.
Specifically targeting mobile devices, Lu et al. [19] pro- GPUs are an important factor in efficient DNN inference
vided the measurements for a number of popular DNN and training. Nvidia provides GPU software libraries to
models on mobile CPUs and GPUs (Nvidia TK1 and TX1). make use of Nvidia GPUs, such as CUDA [31] for general
Ran et al. [20] further explored the accuracy–latency GPU processing and cuDNN [32] which is targeted toward
tradeoffs on mobile devices by measuring how reducing deep learning. While such libraries are useful for training
the dimensionality of the input size reduces the overall DNN models on a desktop server, cuDNN and CUDA are
accuracy and latency. DNN models designed specifically not widely available on current mobile devices such as
for mobile devices, such as MobileNets [21], report system smartphones. To utilize smartphone GPUs, Android devel-
performance in terms of a number of multiply–add opera- opers can currently make use of Tensorflow Lite, which
tions, which could be used to estimate latency characteris- provides experimental GPU capabilities. To experiment
tics and other metrics on different mobile hardware, based with edge devices other than smartphones, researchers can
on the processing capabilities of the hardware. turn to edge-specific development kits, such as the Nvidia
Once the system performance is understood, the applica- Jetson TX2 development kit for experimenting with edge
tion developer can choose the right model. There has also computing (e.g., as used in [33]), with Nvidia-provided
been much recent interest in automated machine learning, SDKs used to program the devices. The Intel Edison kit
which uses artificial intelligence to choose which DNN is another popular platform for experimentation, which
model to run and tune the hyperparameters. For exam- is designed for IoT experiments (e.g., as used in [34]).
ple, Tan et al. [22] and Taylor et al. [23] proposed using Additional hardware-based platforms will be discussed in
reinforcement learning and traditional machine learn- Section IV-A3.
ing, respectively, to choose the right hyperparameters for
mobile devices, which is useful in edge scenarios. III. A P P L I C AT I O N S O F D E E P L E A R N I N G
AT T H E E D G E
C. Frameworks Available for DNN Inference We now describe several example applications where deep
and Training learning on edge devices is useful, and what “real time”
means for each of these applications. Other applications
To experiment with deep learning models, researchers
of deep learning exist alongside the ones described in the
commonly turn to open-source software libraries and
following; here, for brevity, we highlight several applica-
hardware development kits. Several open-source software
tions that are relevant in the edge computing context. The
libraries are publicly available for deep learning inference
common theme across these applications is that they are
and training on end devices and edge servers. Google’s
complex machine learning tasks where deep learning has
TensorFlow [24], released in 2015, is an interface for
been shown to provide good performance and they need
expressing machine learning algorithms and an implemen-
to run in real time and/or have privacy concerns, hence
tation for executing such algorithms on heterogeneous
necessitating inference and/or training on the edge.
distributed systems. Tensorflow’s computation workflow
is modeled as a directed graph and utilizes a placement
algorithm to distribute computation tasks based on the A. Computer Vision
estimated or measured execution time and communica- Since the success of deep learning in the ISLVRC
tion time [25]. The placement algorithm uses a greedy Computer Vision Competition from 2012 onward [1], deep
approach that places a computation task on the node learning has been recognized as the state of the art for
that is expected to complete the computation the soonest. image classification and object detection. Image classifica-
Tensorflow can run on edge devices, such as Raspberry Pi tion and object detection are fundamental computer vision
and smartphones. TensorFlow Lite was proposed in the late tasks that are needed in a number of specific domains,
2017 [26], which is an optimized version of Tensorflow such as video surveillance, object counting, and vehicle
for mobile and embedded devices, with mobile GPU sup- detection. Such data naturally originate from cameras
port added in early 2019. Tensorflow Lite only provides located at the network edge, and there have even been

1658 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

commercial cameras released with built-in deep learning detect wakewords (e.g., “Alexa” or “Hey Siri”). Only if
capabilities [35]. Real-time inference in computer vision the wakeword is detected, then the voice recording is
is typically measured in terms of frame rate [36], which sent to the cloud for further parsing, interpretation, and
could be up to the frame rate of the camera, typically query response. In the case of Apple Siri, the wakeword
30–60 frames/s. Uploading camera data to the cloud also processing uses two on-device DNNs to classify speech into
has privacy concerns, especially if the camera frames con- one of 20 classes (including general speech, silence, and
tain sensitive information, such as people’s faces or private wakeword) [46]. The first DNN is smaller (5 layers with
documents, further motivating computation at the edge. 32 units) and runs on a low-power always-ON processor.
Scalability is a third reason why edge computing is useful If the first DNN’s output is above a threshold, it triggers a
for computer vision tasks, as the uplink bandwidth to a second, more powerful DNN (5 layers with 192 units) on
cloud server may become a bottleneck if there are a large the main processor.
number of cameras uploading large video streams. Wakeword detection methods need to be further mod-
Vigil [37] is one example of an edge-based computer ified to run on even more computationally constrained
vision system. Vigil consists of network of wireless cam- devices, such as a smartwatch or an Arduino. On the Apple
eras that perform processing at edge compute nodes to Watch, a single DNN is used, with a hybrid structure bor-
intelligently select frames for analysis (object detection rowing from the aforementioned two-pass approach. For
or counting), for example, to search for missing people speech processing on an Arduino, researchers from
in surveillance cameras or analyze customer queues in Microsoft optimized an RNN-based wakeword (“Hey Cor-
retail environments. The motivation for edge computing tana”) detection module to fit in 1 kB of memory [47].
in Vigil is twofold: to reduce the bandwidth consumption Overall, while edge computing is currently used for wake-
compared to a naive approach of uploading all frames to word detection on edge devices, latency remains a sig-
the cloud for analysis and for scalability as the number of nificant issue for more complex natural language tasks
cameras increases. (e.g., a professional translator can translate 5× faster
VideoEdge [38] similarly motivates the edge-based than Google Translate with the Pixel Buds earbuds [48]),
video analysis from a scalability standpoint. They use a as well as the need for constant cloud connectivity.
hierarchical architecture of edge and cloud compute nodes
to help with load balancing while maintaining high predic-
tion accuracy (further details are provided in Section IV). C. Network Functions
Commercial devices, such as Amazon DeepLens [35], also Using deep learning for network functions, such
follow an edge-based approach, where image detection is as for intrusion detection [49], [50] and wireless
performed locally in order to reduce latency, and scenes scheduling [51], has been proposed. Such systems,
of interest are only uploaded to the cloud for remote by definition, live on the network edge and need to
viewing if an interesting object is detected, in order to save operate with stringent latency requirements. For example,
bandwidth. an intrusion detection system that actively responds to
a detected attack by blocking malicious packets needs
to perform detection at a line rate to avoid creating a
B. Natural Language Processing bottleneck, e.g., 40 µs [52]. If the intrusion detection
Deep learning has also become popular for natural system operates in the passive mode, however, its
language processing tasks [39], including for speech latency requirements are less strict. A wireless scheduler
synthesis [40], named entity recognition [41] (under- also needs to operate at a line rate in order to make
standing different parts of a sentence), and machine real-time decisions on which packets should be delivered
translation [42] (translating from one language to where.
another). For conversational artificial intelligence, latency In-network caching is another example of a network
on the order of hundreds of milliseconds has been achieved function that can use deep learning at the network edge.
in recent systems [43]. At the intersection of natural lan- In an edge computing scenario, different end devices in the
guage processing and computer vision, there are also visual same geographical region may request the same content
question-and-answer systems [44], where the goal is to many times from a remote server. Caching such contents
pose questions about an image (e.g., “how many zebras at an edge server can significantly reduce the perceived
are in this image?”) and receive natural language answers. response time and network traffic. There are generally two
Latency requirements differ based on how information is approaches to apply deep learning in a caching system: use
presented; for example, conversational replies are prefer- deep learning for content popularity prediction or use deep
ably returned within 10 ms, while a response to a written reinforcement learning to decide a caching policy [53].
Web query can tolerate around 200 ms [45]. Saputra et al. [54], for example, used deep learning to pre-
An example of natural language processing on the edge dict content popularity. To train the deep learning model,
is voice assistants, such as Amazon Alexa or Apple Siri. the cloud collects the content popularity information from
While voice assistants perform some of their processing all of the edge caches. Deep reinforcement learning for
in the cloud, they typically use on-device processing to caching, on the other hand, avoids popularity prediction

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1659

Chen and Ran: Deep Learning With Edge Computing: A Review

and is solely based on reward signals from its actions. E. Virtual Reality and Augmented Reality
Chen et al. [55], for example, trained deep reinforce- In 360◦ virtual reality (VR), deep learning has been
ment learning for caching using the cache hit rate as the proposed to predict the field of view of the user [62]–[64].
reward. These predictions are used to determine which spatial
regions of the 360◦ video to fetch from the content
provider and must be computed in real time to minimize
D. Internet of Things
stalls and maximize the quality-of-experience of the user.
Automatic understanding of IoT sensor data is desired In augmented reality (AR), deep learning can be used to
in several verticals, such as wearables for healthcare, detect objects of interest in the user’s field of view and
smart city, and smart grid. The type of analysis that is apply virtual overlays on top [33], [65].
performed on these data depends on the specific IoT Latency in AR/VR is often measured in terms of the
domain, but deep learning has been shown to be success- “motion-to-photons” delay. This is defined as the end-
ful in several of them. Examples include human activity to-end delay starting from when the user moves her
recognition from wearable sensors [56], pedestrian traffic headset to when the display is updated in response to
in a smart city [57], and electrical load prediction in a her movement. Motion-to-photons’ latency is typically
smart grid [58]. One difference in the IoT context is that required to be on the order of tens to hundreds of
there may be multiple streams of data that need to be milliseconds [66]. Since deep learning is only one possible
fused and processed together, and these data streams part of the AR/VR pipeline (retrieving virtual objects from
typically have space and time correlation that should be memory and rendering them can also consume signifi-
leveraged by the machine learning. DeepSense [56] is cant latency), the motion-to-photons’ latency requirement
one framework geared toward IoT data fusion leveraging is an upper bound on the latency requirement of deep
spatiotemporal relationships. It proposes a general deep learning. The motion-to-photons’ latency requirement also
learning framework that incorporates a hierarchy of CNNs depends on the specific application and the type of user
(to capture multiple sensor modalities) and RNNs (to interaction in that application; Chen et al. provided the
capture temporal correlations) and demonstrates how this latency requirements for different cognitive assistance AR
general framework can be applied to different tasks with applications [67]. Since offloading AR computation to the
multiple sensor inputs: car tracking, human activity recog- cloud can incur latencies on the order of hundreds of mil-
nition, and biometric identification using inertial sensors liseconds, edge computing is needed to provide satisfactory
(gyroscope, accelerometer, and magnetometer). performance, as it is done in Gabriel, a cognitive assistance
Another line of work in the context of IoT deep learn- framework using Google Glass [68].
ing focuses on compressing the deep learning models
to fit onto computationally weak end devices, such as
Arduino or Raspberry Pi, which typically have only kilo- IV. M E T H O D S F O R F A S T I N F E R E N C E
bytes of memory and low-power processors. Bonsai [59] To enable the above-mentioned applications to meet their
does experiments with Arduino Uno, DeepThings [60] latency requirements, different architectures for quickly
experiments with Raspberry Pi 3, and DeepIoT [34] works performing DNN inference have been proposed. In this
with Intel’s IoT platform, the Edison board. More details section, we discuss research centered around three major
on how they shrink the deep learning model to fit in architectures: 1) on-device computation, where DNNs are
memory and run on these lightweight devices are dis- executed on the end device; 2) edge server-based archi-
cussed in Section IV. Other examples of applying deep tectures, where data from the end devices are sent to
learning on IoT scenarios, including agriculture, industry, one or more edge servers for computation; and 3) joint
and smart home, can be found in the excellent survey by computation among end devices, edge servers, and the
Mohammadi et al. [61]. cloud. We also discuss privacy-preserving techniques when
Another motivation for edge computing with IoT devices data are communicated between edge devices and with the
is that the significant privacy concerns when IoT sensors cloud, as in scenarios 1 and 2. Fig. 3 shows the taxonomy of
are placed in public locations; for example, the Hudson these methods, and Fig. 5 shows the examples of different
Yards smart city development in New York City seeks scenarios, which will be discussed in further detail in
to use air quality, noise, and temperature sensors, along the following. Tables 2 and 3 provide a summary of the
with cameras, to provide advertisers with estimates of discussed works.
how many and how long people looked at advertisements,
as well as their sentiment based on facial expressions.
However, this has raised significant warnings from privacy A. On-Device Computation
watchdogs [3]. Thus, while analyzing IoT sensor data in Many research efforts have focused on ways to reduce
real time is not always a requirement, and communication the latency of deep learning when it is executed on
bandwidth requirements from sensors are typically small a resource-constrained device [see Fig. 5(a)]. Such
(unless cameras are involved), privacy is a major concern efforts can have benefits throughout the edge ecosystem,
that motivates IoT processing on the edge. by reducing the latency of the DNN while running on

1660 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

Parameter quantization takes an existing DNN and com-

presses its parameters by changing from floating-point
numbers to low-bit width numbers, thus avoiding costly
floating-point multiplications. Pruning involves removing
the least important parameters (e.g., those that are close
to 0), as shown in Fig. 4. Quantization and pruning
approaches have been considered individually as well
as jointly [71]. Specifically for edge and mobile devices,
DeepIoT [34] presents a pruning method for commonly
used deep learning structures in IoT devices, and the
pruned DNN can be immediately deployed on edge devices
without modification. Lai and Suda [72] provided CMSIS-
NN, a library for ARM Cortex-M processors that maximize
DNN performance through quantization. It also optimizes
data reuse in matrix multiplication to speed up DNN execu-
tion. Han et al. [73] proposed pruning and quantization for
an RNN model, with 10× speedup resulting from pruning
and 2× from quantization. Bhattacharya and Lane [74]
compressed the neural network by sparsifying the fully
connected layers and decomposing the convolutional fil-
ters on wearable devices.
Fig. 3. Taxonomy of DNN inference speedup methods on the edge.
Knowledge distillation involves creating a smaller DNN
that imitates the behavior of a larger, more powerful
DNN [75]. This is done by training the smaller DNN using
the end devices or edge servers. Here, we describe major the output predictions produced from the larger DNN.
efforts in efficient hardware and DNN model design. Essentially, the smaller DNN approximates the function
learned by the larger DNN. Fast exiting [76] is another
1) Model Design: When designing DNN models
technique where not all layers are computed; only the
for resource-constrained devices, machine learning
result from computing the initial layers is used to provide
researchers often focus on designing models with a
approximate classification results.
reduced number of parameters in the DNN model, thus
Several works have explored the combinations of these
reducing memory and execution latency, while aiming
model compression techniques. Adadeep [77] automati-
to preserve high accuracy. There are many techniques
cally chooses between different compression techniques,
for doing so, and we briefly mention several popular
including pruning and the special filter structures bor-
deep learning models for resource-constrained devices
rowed from MobileNet and SqueezeNet, to meet applica-
drawn from computer vision. These models include
tion requirements and satisfy mobile resource constraints.
MobileNets [21], solid-state drive (SSD) [69], YOLO [18],
DeepMon [78] combines quantization with caching of
and SqueezeNet [70], with the state of the art that is
results from intermediate layers on GPUs. The caching
evolving rapidly. MobileNets decomposes the convolution
leverages the insight that an input video does not change
filters into two simpler operations, reducing the number
much between the subsequent frames, so some compu-
of computations needed. SqueezeNet downsamples the
tation results from a previous frame can be reused in
data using special 1 × 1 convolution filters. YOLO and
the current frame, reducing redundant computations and
SSD are both single shot detectors that jointly predict the
speeding up execution.
location and class of the object at the same time, which
is much faster than performing these steps sequentially.
Many of these models, with pre-trained weights, are
available for download on open-source machine learning
platforms such as Tensorflow [24] and Caffe [28] for
fast bootstrapping.

2) Model Compression: Compressing the DNN model

is another way to enable DNNs on edge devices. Such
methods usually seek to compress the existing DNN
models with minimal accuracy loss compared with
the original model. There are several popular model
compression methods: parameter quantization, parameter
pruning, and knowledge distillation. We briefly outline
these approaches in the following. Fig. 4. Pruning a neural network.

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1661

Chen and Ran: Deep Learning With Edge Computing: A Review

Fig. 5. Architectures for deep learning inference with edge computing. (a) On-device computation. (b) Secure two-party communication.
(c) Computing across edge devices with DNN model partitioning. (d) Offloading with model selection. (e) Distributed computing with DNN
model partitioning.

3) Hardware: To speed up inference of deep learn- software development kit (SDK) is designed to utilize its
ing, hardware manufacturers are leveraging existing hard- Snapdragon chips [86]. There are also general libraries
ware such as CPUs and GPUs, as well as producing developed for mobile devices not tied to specific hard-
custom application-specific integrated circuits (ASICs) for ware, such as RSTensorFlow [87], which uses the GPU to
deep learning, such as Google’s tensor processing unit speed up matrix multiplication in deep learning. Software
(TPU) [79]. ShiDianNao [80] is another recently proposed approaches have also been developed to efficiently utilize
custom ASIC, which focuses on efficient memory accesses hardware, e.g., Lane et al. [88] decomposed DNNs and
in order to reduce latency and energy consumption. It is assigning them to heterogeneous local processors (e.g.,
part of the DianNao [81] family of DNN accelerators, but CPU and GPU) to accelerate execution. More detail on
it is geared toward embedded devices, which is useful hardware-accelerated deep learning can be found in the
in the edge computing context. field-programmable gate excellent survey by Sze et al. [89]. Since Sze’s survey
array (FPGA)-based DNN accelerators are another promis- has covered hardware-based DNN accelerations in great
ing approach, as FPGA can provide fast computation while depth, the remainder of this paper mainly focuses on
maintaining re-configurability [82]. These custom ASICs software-based approaches.
and FPGA designs are generally more energy efficient than
the traditional CPUs and GPUs, which are designed for B. Edge Server Computation
flexible support of various workloads at the expense of While the above-mentioned hardware speedup and com-
higher energy consumption. pression techniques can help DNNs run on end devices,
Vendors also provide software tools for application deploying large, powerful DNNs with real-time execution
developers to leverage the accelerations provided by requirement on edge devices is still challenging because
the hardware. Chip manufacturers have developed soft- of resource limitations (e.g., power, computation, and
ware tools to optimize deep learning on the existing memory). Thus, it is natural to consider offloading DNN
chips, such as Intel’s OpenVINO Toolkit to leverage Intel computations from end devices to more powerful entities,
chips, including Intel’s CPUs, GPUs, FPGAs, and vision such as edge servers or the cloud. However, the cloud
processing unit [83], [84]. Nvidia’s EGX platform [85] is is not suitable for edge applications that require short
another recent entrant into this space, with support for response times [8]. Since the edge server is close to users
Nvidia hardware ranging from lightweight Jetson Nanos and can respond quickly to users’ request, it becomes the
to powerful T4 servers. Qualcomm’s Neural Processing first-choice helper.

1662 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

Table 2 Summary of the Selected Works on Fast Deep Learning Inference With Edge Computing

The most straightforward method to utilize the edge 1) Data Preprocessing: When sending data to an
server is to offload all the computation from end devices edge server, data preprocessing is useful to reduce data
to the edge server. In such scenarios, the end devices will redundancy and thus decrease communication time.
send its data to a nearby edge server and receive the corre- Glimpse [91] offloads all DNN computation to a nearby
sponding results after server processing. Wang et al. [90], edge server, but it uses change detection to filter which
for example, always offloaded DNNs to the edge server (an camera frames are offloaded. If no changes are detected,
IoT gateway) to analyze wireless signals. Glimpse will perform frame tracking locally on the end
device. This preprocessing improves system processing

Table 3 Summary of the Selected Works on Privacy-Preserving Inference

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1663

Chen and Ran: Deep Learning With Edge Computing: A Review

ability and makes real-time object recognition on mobile size for the different DNN models. The catalog of different
devices possible. Along similar lines, Liu et al. [92] DNN models can be chosen from the existing popular
built a food recognition system with two preprocessing models (e.g., those discussed in Section IV-A2) or new
steps: first, they discard blurry images, and second, they model variants can be constructed through knowledge
crop the image so that it only contains the objects of distillation or by “mix-and-matching” DNN layers from
interest. Both preprocessing steps are lightweight and multiple models [104]. An example of offloading,
can reduce the amount of offloaded data. We note that combined with model selection where a powerful DNN
while feature extraction is a common preprocessing step is available on the edge server and a weaker DNN is
in computer vision, it does not apply in the context of available on the end device, is shown in Fig. 5(d).
deep learning, because the DNNs themselves serve as the We note that while offloading has long been studied
feature extractors. in the networking literature [112], even in the context of
edge computing [113], DNN offloading can consider the
2) Edge Resource Management: When DNN computa-
additional degree of freedom of not only where to run, but
tions run on edge servers, DNN tasks from multiple end
which DNN model or which portion of the model to run.
devices need to run and be efficiently managed on shared
The decision of whether to offload or not thus depends on
compute resources. Several works have explored this prob-
the size of the data, the hardware capabilities, the DNN
lem space, focusing on the tradeoffs between accuracy,
model to be executed, and the network quality, among
latency, and other performance metrics, such as a number
other factors.
of requests served. VideoStorm [93] was one of the first
works in this space, and profiles these tradeoffs to choose
2) DNN Model Partitioning: A fractional offloading
the right DNN configuration for each request, to meet
approach can also be considered, which leverages the
the accuracy and latency goals. The configuration can
unique structure of DNNs, specifically its layers. In such
also be updated online during the streaming video input,
model partitioning approaches, some layers are computed
as done in Chameleon [94]. VideoEdge [38] additionally
on the device, and some layers are computed by the edge
considers computation that is distributed across a hierar-
server or the cloud, as shown in Fig. 5(c). This is known
chy of edge and cloud servers and how to jointly tune all
as DNN model partitioning. These approaches can poten-
the DNN hyperparameters. Mainstream [95] considers a
tially offer latency reductions by leveraging the compute
similar problem setup of accuracy versus latency tradeoffs
cycles of other edge devices; however, care must also be
on edge servers, but their solution uses transfer learning
taken that the latency of communicating the intermediate
to reduce the computational resources consumed by each
results at the DNN partition point still leads to overall net
request. Transfer learning enables multiple applications to
benefits. The intuition behind model partitioning is that
share the common lower layers of the DNN model and
after the first few layers of the DNN model have been
computes higher layers unique to the specific application,
computed, the size of the intermediate results is relatively
thus reducing the overall amount of computation.
small, making them faster to send over the network to an
edge server than the original raw data [60]. This moti-
C. Computing Across Edge Devices vates the approaches that partition after the initial layers.
Neurosurgeon [13] is one work that intelligently decides
Although the edge server can accelerate DNN process-
where to partition the DNN, layer-wise, while accounting
ing, it is not always necessary to have the edge devices
for network conditions.
executing DNNs on the edge servers—intelligent offload-
In addition to partitioning the DNN by layers, the DNN
ing can be used instead. We next discuss four offload-
can also be partitioned along the input dimension (e.g.,
ing scenarios: 1) binary offloading of DNN computation,
select rows of the input image). Such input-wise parti-
where the decision is whether to offload the entire DNN or
tioning allows fine-grained partitioning, because the input
not; 2) partial offloading of partitioned DNNs, where the
and output data size and the memory footprint of each
decision is what fraction of the DNN computations should
partition can be arbitrarily chosen, instead of the min-
be offloaded; 3) hierarchical architectures where offload-
imum partition size being defined by the discrete DNN
ing is performed across a combination of edge devices,
layer sizes. This is especially important for extremely
edge servers, and cloud; and 4) distributed computing
lightweight devices, such as IoT sensors, which may not
approaches where the DNN computation is distributed
have the necessary memory to hold an entire DNN layer.
across multiple peer devices.
However, input-wise partitioning can result in increased
1) Offloading: Recent approaches, such as data dependence, as computing subsequent DNN lay-
DeepDecision [20], [111] and MCDNN [104], take an ers requires data results from adjacent partitions. Two
optimization-based offloading approach with constraints examples of input-wise partitioning as MoDNN [107] and
such as network latency and bandwidth, device energy, DeepThings [60].
and monetary cost. These decisions are based on the Overall, these partial offloading approaches through
empirical measurements of the tradeoffs between these DNN partitioning are similar in spirit to past, non-DNN
parameters, such as energy, accuracy, latency, and input offloading approaches such as MAUI [112] and

1664 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

Odessa [114], which divide an application into its where the DNN computations can be distributed across
constituent subtasks, and decide which subtasks to execute multiple helper edge devices, as shown in Fig. 5(e). For
where based on energy and/or latency considerations. example, MoDNN [107] and DeepThings [60] distribute
However, a new decision in the deep learning scenario is DNN executions using fine-grained partitioning on
how to decide the constituent subtasks, as the DNN can be lightweight end devices such as Raspberry Pis and
divided layer-wise, input-wise, or possibly in other ways Android smartphones. The DNN partition decision is
yet to be explored. made based on the computation capabilities and/or
memory of the end devices. At runtime, the input data
3) Edge Devices Plus the Cloud: Deep learning computa-
are distributed to helpers according to the load-balancing
tion can be performed not only on edge devices but also
principles, with MoDNN using a MapReduce-like model
on the cloud, as shown in Fig. 5(c). While solely offloading
and DeepThings designing a load-balancing heuristic.
to the cloud can violate the real-time requirements of the
The assignment of data to the helper devices can be
deep learning applications under consideration, judicious
adjusted online to account for dynamic changes in
use of the powerful compute resources in the cloud can
compute resource availability or network conditions.
potentially decrease the total processing time. Different
More formal mechanisms from distributed systems could
from a binary decision of whether to perform computation
also be applied in these scenarios to provide provable
on the edge server or cloud, approaches in this space often
performance guarantees.
consider DNN partitioning, where some layers can execute
in the cloud, edge server, and/or end device.
Li et al. [106] divided the DNN model into two D. Private Inference
parts—the edge server computes the initial layers of the
When data from the end devices traverse the edge net-
DNN model, and the cloud computes the higher layers
work (e.g., from end devices to edge servers, as discussed
of the DNN. The edge server receives the input data,
in Section IV-B), it may contain sensitive information (e.g.,
performs lower layer DNN processing, and then sends
GPS coordinates, camera images, and microphone audio),
the intermediate results to the cloud. The cloud, after
leading to privacy concerns. This is especially important
computing the higher layers, sends back the final results to
in edge computing, where the data are typically sourced
the end devices. Such designs utilize both the edge server
from a limited set of users within a limited geograph-
and the cloud, where the cloud can help with computation-
ical region, making privacy breaches more concerning.
ally heavy requests and increase the edge server’s request
Although edge computing naturally improves privacy by
processing rate while reducing the network traffic between
reducing data transfers through the public Internet to the
the edge server and the cloud. DDNN [109] also distributes
cloud, additional techniques can further enhance privacy
computation across a hierarchy of cloud, edge servers,
between end devices and edge servers and protect from
and end devices, and additionally combines this with the
eavesdroppers. In this section, we discuss two methods of
fast exiting idea (discussed in Section IV-A2), so that the
privacy-preserving inference: adding noise to obfuscate the
computation requests do not always reach the cloud.
data uploaded by end devices to edge servers and secure
A unique characteristic of edge computing is that the
computation using cryptographic techniques.
edge server typically serves users within a limited geo-
graphical area, suggesting that their input data and, thus, 1) Add Noise to Data: Several works have considered
their DNN outputs may be similar. Precog [10] leverages how to obfuscate, or add noise, to the inference sam-
this insight in the case of image recognition and places ples uploaded by end devices to a central machine (e.g.,
smaller, specialized image classification models on the end an edge server) performing inference. Wang et al. [115]
devices, based on what has recently been observed by deployed a smaller DNN locally on the device to extract
other devices served by the same edge server. If on-device features, add noise to the features, and then upload the
classification fails, the query is sent to the edge server features to the cloud for further inference processing by a
that stores all the classification models. Although their more powerful DNN. The DNN on the cloud is pre-trained
evaluation does not use DNNs, they discuss how their clas- with noisy samples so that the noisy inference samples
sification model placement decisions would apply to DNNs. uploaded from the end devices can still be classified with
This approach has similarities to knowledge distillation for high accuracy at test time. The formal notion of privacy
compressed models (see Section IV-A2), in which it uses a used in this paper is differential privacy which, at a high
combination of weaker and stronger classification models, level, guarantees that a machine learning model does not
but it provides a more careful look at what specialized remember details about any specific device’s input data.
models are needed on the end devices in edge scenarios.
2) Secure Computation: Cryptographic techniques can
4) Distributed Computation: The above-mentioned be used to compute the DNN prediction. The goal of secure
approaches mainly consider offloading computation from computation in this setup is to ensure that the end device
end devices to other more powerful devices (e.g., edge receives an inference result without learning anything
servers or the cloud). Another line of work considers about the DNN model, and the edge server processes the
the problem from a distributed computing perspective, data without learning anything about the device’s data.

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1665

Chen and Ran: Deep Learning With Edge Computing: A Review

In other words, an end device and an edge server want to assume that a deep learning model has already been
compute the DNN prediction f (a, b), where a is an input trained offline on a centralized, existing data set. In this
sample (e.g., a camera frame) known only to the end section, we discuss the methods of training deep learning
device and b are the DNN parameters known only to the models with edge computing, primarily with a focus on
edge server. Secure computation enables both the device communication efficiency and privacy.
and the server to compute f (a, b) without knowing the Traditionally, training data produced by end devices
other party’s data. would be sent to the cloud, which would then perform
One method of secure computation is homomorphic the training with its large computational resources and
encryption, in which the communicated data are encrypted finally distribute the trained model back to the edge
and computation can be performed on the encrypted data, devices as needed. However, sending the data to the cloud
as done in CryptoNets [116]. The idea is to approximate can consume large amounts of bandwidth and also has
common computations used in DNNs, such as weighted privacy concerns. Leaving data in situ on the end devices
sum, max pooling, mean pooling, sigmoid function, is useful when privacy is desired and also helps reduce
and rectified linear unit (RELU), by low-degree poly- the network bandwidth requirements. For example, a deep
nomials, which are amenable to homomorphic encryp- learning-based typing prediction model for smartphones
tion. However, a bottleneck of homomorphic encryption may benefit from training data from multiple users, but
tends to be their compute times, which means that individual users may not wish to upload their raw key-
offline preprocessing is needed. CryptoNets also requires stroke data to the cloud; similarly, in an image classifica-
re-training of the DNN because of the approximations tion service, uploading all camera frames from end devices
used. to the cloud would consume large amounts of bandwidth
Multiparty computation is another technique for secure and risk uploading sensitive information.
computation [see Fig. 5(b)]. In secure multiparty compu- Edge-based training borrows from distributed DNN
tation, multiple machines work together and communi- training in data centers. In data centers, training is per-
cate in multiple rounds to jointly compute a result (e.g., formed across multiple workers, with each worker holding
a DNN prediction in our scenario). Different from differ- either a partition of the data set (known as data paral-
ential privacy, secure multiparty computation focuses on lelism) or a partition of the model (known as model par-
the privacy of the intermediate steps in the computation, allelism). While both system designs have been explored,
while differential privacy focuses on the privacy guarantees data parallelism is widely used in practical systems [121]
of the overall constructed model. MiniONN [117] and and is the focus of the remainder of this section. In data
DeepSecure [118] employ secure two-party computation parallelism, each worker computes the gradients of its
and homographic encryption and work with the existing local partition of the data set, which are then collected by
pre-trained DNN models without needing to change the a central parameter server, some aggregate computation
DNN training or structure. However, a bottleneck of secure performed, and the updates sent back to the workers
multiparty computation techniques tends to be their com- [see Fig. 7(a)].
munication complexity. Training on edge devices borrows from the data center
Chameleon [120] and Gazelle [119] are two works setup, where the workers are end devices instead of pow-
that try to choose between the above-mentioned crypto- erful servers in a data center, and the central parameter
graphic techniques (homomorphic encryption and secure server is an edge compute node or server. For example,
multiparty computation) based on their computation and DeepCham [122] consists of a master edge server that
communication tradeoffs. Specifically, Gazelle studies the trains domain-aware object recognition on end devices,
tradeoffs between homomorphic encryption (high com- leveraging the insight that users connected to the same
putation and low communication) and two-party secure edge server may have similar domains (e.g., time of day
computation (low computation and high communica- and physical environment). In an edge scenario, commu-
tion) and chooses the right techniques for the scenario. nication latency, network bandwidth, and the compute
It further accelerates the training process with efficient capabilities of the end device are key considerations of
implementation of the cryptographic primitives. Their training performance.
evaluation compared to CryptoNets, MiniONN, DeepSe- Training deep learning on edge devices typically
cure, and Chameleon using standard image classification involves distributed deep learning training techniques.
data sets suggests low runtime latency and communication This section discusses about the techniques to perform
cost. distributed training on edge devices from the following
perspectives: the frequency and size of training updates,
which both contributes to communication cost (see Sec-
V. T R A I N I N G I N P L A C E O N E D G E tions V-A and V-B, respectively); decentralized information
DEVICES sharing (see Section V-C); and finally, privacy-preserving
Thus far, edge computing and deep learning have mostly DNN training (see Section V-D). A taxonomy of these
been discussed for inference, with goals including low techniques is shown in Fig. 6, and a summary of the works
latency, privacy, and bandwidth savings. These methods discussed in the following is shown in Table 4.

1666 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

using stale information from devices and can suffer from

convergence to poor solutions.
Distributed training algorithms usually focus on how to
make synchronous SGD faster or how to make asynchro-
nous SGD converge to better solutions. In a distributed
setting, communication frequency and data volume are
also important. Elastic averaging [124] reduces the com-
munication costs of synchronous and asynchronous SGD
training methods, by allowing each device to perform more
local training computations and deviate/explore further
from the globally shared solution before synchronizing
Fig. 6. Taxonomy of DNN training in place on edge devices. its updates. This reduces the amount of communication
between the local devices and the edge server. Federated
learning [125] is similar in spirit, but it considers non-ideal
scenarios, such as non-independent and identically dis-
tributed (i.i.d) data distributions (e.g., one device has
A. Frequency of Training Updates
more data samples of a given class than another device).
Communication costs are a major concern for edge Computing more local gradient updates without uploading
devices. Reducing the frequency of communications and the raw training data to the server trades off accuracy
the size of each communication is a key method to for communication cost: doing more computation locally
reduce communication costs. In this section, we discuss lowers the prediction accuracy (due to overfitting to local
the distributed training methods that focus on communi- data sets), but it can also save communication cost, and
cation timing and frequency, while in Section V-B, we dis- vice versa. Wang et al. [126] further explored this issue by
cuss the size of the communicated data. There are two considering some practical concerns with implementation
general methods for synchronizing updates to a central on a real testbed. They proposed a control policy for
edge server: synchronous and asynchronous stochastic gra- deciding how much computation should be performed
dient descent [123]. In synchronous stochastic gradient locally in between global gradient updates and performed
descent (SGD), individual devices update their parameters experiments with Raspberry Pis and laptops.
in lockstep when all the devices have finished computing Tiered architectures and their communication costs
the gradients on their current batch of training data. have also been considered. Gaia [127] studies synchro-
In asynchronous SGD, the devices update their parameters nous SGD in the scenario where devices are geographi-
independently to the central server. Both synchronous SGD cally distributed across a large area. In their test setup,
and asynchronous SGD have their own pros and cons. the clients are servers inside a data center and across
Although synchronous SGD typically converges to better data centers. Because bandwidth constraints are tighter
solutions, it is often slower in practice because of the need across geo-distributed data centers than within a single
to wait for straggler devices in each iteration. Asynchro- data center, gradient updates need to be carefully coordi-
nous SGD, on the other hand, tends to converge faster nated between the workers. Gaia proposes a policy where
than synchronous SGD, but it may update parameters updates are synchronized across different data centers only

Table 4 Summary of the Selected Works on Distributed Training

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1667

Chen and Ran: Deep Learning With Edge Computing: A Review

Fig. 7. Architectures for deep learning training on the edge. (a) Centralized training. (b) Decentralized training.

when the aggregated updates are higher than a given (e.g., wireless and access networks) can be quite con-
threshold. strained. In this section, we review gradient compression
Along with synchronous and asynchronous updates, techniques, which can reduce the size of the updates
distillation is another method that has been applied to communicated to a central server.
reduce communication frequency. Distillation, as discussed There are two general approaches to gradient
in Section IV-A in the context of inference, uses the pre- compression: gradient quantization and gradient
diction outputs of one model to help train another model. sparsification [129]. Gradient quantization approximates
Anil et al. [128] proposed incorporating distillation into the floating-point gradients using low-bit width numbers.
distributed training of DNNs. In their method, each device For example, a 32-bit floating-point numbers can be
trains on a subset of the data and updates its gradients approximated by an 8-bit number, reducing the size by
based on its computed training loss as usual, but it also a factor of 4. Note that gradient quantization is similar
uses the prediction outputs from other devices that are to parameter quantization (see Section IV-A), with the
also simultaneously training to improve training efficacy. difference being whether the quantization is applied to
Since they find that the training is robust to stale predic- the model gradients or the model parameters. Gradient
tion results from other devices, information needs to be sparsification discards unimportant gradient updates
exchanged with other devices less frequently (compared and only communicates updates that exceed a certain
to the gradient sharing methods described earlier in this threshold. Gradient quantization and sparsification
section). In this way, frequent communication of gradients can work together. For example, Lin et al. [129]
is avoided or reduced. Furthermore, distillation can be did gradient sparsification combined with other
combined with distributed SGD and can improve training training tricks such as momentum correction [138]
efficacy even when distributed SGD is not possible due to and warm-up training [139] techniques to speed up
network constraints. training convergence. Hardy et al. [140] performed
Finally, if some devices have poor connectivity and are gradient sparsification and also chose which gradients to
subjected to atypically long latencies, they can hold up communicate based on their staleness.
distributed training. Chen et al. [123] proposed improve-
ments to synchronous SGD to mitigate such straggler C. Decentralized Communication Protocols
effects. Their main idea is to have backup devices that Thus far, we have considered centralized training archi-
are “on call” to compute the gradient updates of any tectures where multiple end devices communicate with an
straggling regular devices. Once the server receives the edge server. Having a central edge compute or server node
gradient updates from a sufficient of devices, the training helps ensure that all devices converge to the same model
process will update the global parameters and move on to parameters. However, communication throughput of a cen-
next iteration, without waiting for the straggler devices, tralized architecture is limited by the bandwidth of the
thereby reducing training latency. central node. To overcome this, a gossip-type algorithm has
been proposed as a method to exchange training informa-
B. Size of Training Updates tion in a decentralized fashion. In gossip-type algorithms,
Along with the frequency of training updates, the size each device computes its own gradient updates based on its
of training updates also contributed to bandwidth usage. training data and then communicates its updates to some
With model sizes on the order of hundreds of megabytes, of the other devices [see Fig. 7(b)]. The overall goal is
and multiple rounds of communication needed, the band- to design a gossiping algorithm that allows the devices to
width demands can be considerable. Bandwidth concerns reach a consensus on a good DNN model. Gossiping can be
are crucial in the edge scenario, where last-mile bandwidth considered as a decentralized version of elastic averaging,

1668 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

where clients are allowed to deviate more significantly sent). Abadi et al. [134] studied a similar problem where
from each other. the privacy loss over the overall model was bounded,
Blot et al. [131] proposed an asynchronous algorithm rather than per parameter as in Shokri, and their method
for gossip-based training of deep learning. Their experi- involves modifying the gradient by clipping, averaging,
ments show faster convergence than the elastic averag- and adding noise before communicating it to the parame-
ing. Jin et al. [132] proposed gossiping SGD based on ter server. Mao et al. [137] combined differential privacy
their study of convergence rates for synchronous and with model partitioning, where the initial layers of the
asynchronous SGD. Their primary concern was scalability, DNN were computed on the device, mixed with noise,
i.e., which SGD methods would be appropriate for a dif- and uploaded to the edge server, in order to obfuscate the
ferent number of clients. They found that asynchronous uploaded training data and preserve privacy.
methods, such as gossiping and elastic averaging, con- Along with modifying the gradients, adding noise to
verged more quickly with a small number of workers the training data has also been considered. Zhang [135]
(32 workers in their simulations), whereas synchronous considered different types of noise that can be added
SGD scaled up better and had higher accuracy when to the input data before training. Rather than using for-
there were more workers (100 workers in their simu- mal notions of differential privacy, they empirically guard
lations). Li et al. [133] developed a distributed system against the adversary discovering statistical properties of
called INCEPTIONN, which combines gradient compres- individual training data samples or aggregate statistics
sion and gossiping. Their gossiping method involves divid- about groups of training samples. This is essentially a
ing devices into different groups, and within each group, preprocessing step for the training data, which can pro-
each device shares some of its gradients with the next vide protection even if the adversary has taken over the
device. The algorithm guarantees that all parts of the parameter server and has access to the model parameters
DNN across all devices will be updated after several or post-processed training data.
iterations. Within each group, the parameters can be
2) Secure Computation: SecureML [136] proposes a
shared either in the traditional centralized way or through
two-server model where end devices communicate their
gossiping.
data to two servers, which then train a neural network
based on the combined data from the end devices with-
D. Private Training out learning anything beyond the DNN parameters. Their
We now shift gears and return to the baseline SGD scheme is based on secure two-party linear and logistic
algorithms (e.g., synchronous SGD) but consider the pri- regression that are fundamental computations in DNN
vacy implications of communicating gradient information. training. A modified softmax function and RELU function
Such techniques can be useful whenever the training are also proposed in order to improve efficiency. Unlike
data collected by end devices are shared with other edge the multiparty computation schemes discussed earlier (see
devices. While in situ training naturally improves privacy Section IV-D), SecureML focuses on DNN training, rather
by eliminating direct sharing of end devices’ collected than inference.
data, gradient information communicated between the
edge devices can still indirectly leak information about VI. O P E N C H A L L E N G E S
the private data [14]. Hence, further privacy-enhancing Many challenges remain in deploying deep learning
techniques are needed. In this section, we will consider on the edge, not only on end devices but also on the
two main classes of privacy-enhancing techniques: adding edge servers and on a combination of end devices, edge
noise to gradient or data transmissions as part of training servers, and the cloud. We next discuss some of the open
and secure computation for training a DNN. challenges.
1) Add Noise to Data or Training Updates: In the fol-
lowing works, the threat model consists of a passive A. Systems Challenges
adversary, such as an end device, which follows the pre- 1) Latency: While several works described in the ear-
scribed training protocol and is not actively malicious, lier sections have focused on reducing inference latency,
but it may attempt to learn about the model or data the current state of the art still results in quite high latency,
from observing others’ communicated data. Shokri and particularly when operating on high-dimensional input
Shmatikov [14] considered the privacy aspects of training data, such as images, and on mobile devices. For example,
a DNN in such a scenario, specifically with respect to even DNN models designed for mobile devices execute at
differential privacy, and they modified the typical policy 1–2 frames/s on modern smartphones [20], [78]. There is
of devices uploading all the gradients to a central server still much work remaining on DNN model compression to
by: 1) selecting only some gradients above a threshold enable deep learning to run on edge devices, particularly
to be transmitted and 2) adding noise to each uploaded on IoT devices that tend to have the most severe resource
gradient. This enables the model to be trained reasonably constraints. Furthermore, while the offloading approaches
accurately while reducing information leakages from the described earlier (see Sections IV-B and IV-C) propose inno-
training updates (intuitively, since fewer of the updates are vative approaches to minimize latency, machine learning

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1669

Chen and Ran: Deep Learning With Edge Computing: A Review

experts are also constantly innovating, leading to new DNN and NFV should manage these types of flows and what
models with ever more parameters and new layer designs. types of QoS guarantees the flows require. How can deep
For example, the DNN partitioning approach may work learning flows be identified, even under encryption? Given
well for standard sequential DNNs, but not as well for other a set of network functions that need to operate on deep
deep learning methods such as RNNs, which have loops in learning flows, how to design an SDN controller to best
their layer structure. Keeping up with new deep learning manage these flows (e.g., by carving out network slices
designs will continue to be a major systems’ challenge. for deep learning traffic)? How should network resources
be shared between competing deep learning flows or
2) Energy: Minimizing the energy consumption of
with other non-deep learning traffic, such as Web or
deep learning is very important for battery-powered
video?
edge devices, such as smartphones. While reducing the
Another direction is using deep learning itself as a
amount of computation implicitly reduces energy con-
network function, such as the network intrusion detection
sumption, understanding the interactions of the deep
and caching applications described in Section III. If deep
learning computations with other battery management
learning is adopted for various network tasks, NFV plat-
mechanisms, such as CPU throttling or sensor hardware
forms need to account for the resource requirements of
optimizations [141], is an important avenue for investi-
deep learning in order, for the network functions, to run
gation. Performing change detection on the input data,
in real time. While fast instantiation and performance of
either in software or hardware [142], can help reduce
NFVs has been investigated [146], deep learning inference
the frequency of deep learning executions and the overall
can be greatly accelerated with GPU access on the edge
energy consumption. Reducing energy consumption of the
server, necessitating GPU support in NFV platforms.
specific hardware chips (e.g., GPUs and TPUs) is already
a key priority for hardware designers, but understanding
their interaction with the rest of the system (e.g., battery C. Management and Scheduling of Edge Compute
management mechanisms and tradeoffs with edge server Resources
compute resources) is needed to reduce overall energy Deep learning is often treated as a black box by
consumption. application developers and network administrators. How-
3) Migration: Migrating edge computing applications ever, deep learning models have many tradeoffs between
between different edge servers can be useful for load bal- latency, accuracy, battery, and so on. While several works
ancing or to accommodate user movement, with the goal described earlier (in Section IV-C) have discussed how to
of minimizing the end-to-end latency of the user. While tune such control knobs to achieve overall good system
edge migration has been studied in the general case, for performance [20], [93], exposing these control knobs in a
example, using VM migration techniques [143] or Docker consistent and unified manner to the application developer
containers [144] or using multipath TCP to speed up the and/or server administrator through a standard specifica-
migration [145], understanding how deep learning appli- tion could be valuable. This would enable developers and
cations should be migrated is still an area of consideration. server administrators without in-depth machine learning
DNN models can be fairly large; for example, a pre-trained knowledge to understand the available knobs and tune
YOLO model [18] is approximately 200 MB, and loading a them to achieve good system performance, especially on
DNN model can take several seconds, in our experience. edge compute nodes with limited resources. Specifying the
What parts of the DNN model should be migrated and application’s needs and the tradeoffs of the DNN model
what parts should be included in the standalone virtual being run can allow the edge server to effectively schedule
image? Can the program state be migrated in the midst the end device requests. Not doing this carefully (e.g.,
of a DNN execution, similar to the DNN partitioning incurring long latency on a video frame analysis request
approaches for offloading (see Section IV-C2)? Addressing from an AR headset) would negate the latency benefits of
these challenges requires system measurements and exper- edge computing.
iments to gain an empirical understanding of the migration A natural question is then how to schedule such
challenges. requests, given knowledge of the tradeoffs and control
knobs. The question is complicated by time dependence
between sequential inputs from an end device (e.g., multi-
B. Relationship to SDN and NFV Technologies ple frames from a camera), which could introduce priority
Recently, network abstractions, such as software-defined into the scheduling policy and thus influence the decisions
networking (SDN), to abstract the data plane from the made by the edge server of which requests to serve when.
control plane, and network function virtualization (NFV), For example, should a new camera frame inference request
to abstract the network functions from the hardware, are from device A receive higher priority than the hundredth
gaining importance and are being adopted by the telecom- frame from device B? Incorporating freshness metrics, such
munications industry. If deep learning grows in popularity as the age of information [147], could allow for more
and these flows containing deep learning data appear on intelligent scheduling decisions by the edge server. While
the edge network, this leads to questions of how SDN this problem has some overlap with task scheduling in

1670 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

cloud data centers, edge computing brings new challenges training process means that the data item can be more
in which the number and variety of requests are likely easily pinpointed as belonging to a small subset of users
less on an edge server serving geo-located end devices, who accessed that edge server. Another concern is data
so statistical multiplexing cannot necessarily be relied on. obfuscation. While data obfuscation techniques have been
New analysis of load balancing and request scheduling studied in cases where there are a large number of users,
mechanisms is needed. Furthermore, the compute resource such as in the cloud, whether such obfuscation can still be
allocations may be coupled with the traffic steering from successful in an edge computing scenario, where more spe-
the end devices to the edge server. Existing work on mainly cialized deep learning models are being used [10], [122],
considers proximity as the primary factor behind traffic or smaller training sets are available due to fewer end
steering decisions [148]. devices connected to each edge server is unclear. Finally,
the definition of differential privacy [150] means that as
there are fewer devices, more noise must be added. This
D. Deep Learning Benchmarks on Edge Devices
is exactly the scenario of edge computing, where a smaller
The state of the art of deep learning is evolving rapidly. set of geo-located end devices communicate with an edge
For researchers and developers wishing to deploy deep server. How much noise must be added to compensate for
learning on edge devices, choosing the right DNN model fewer end devices? Overall, the privacy problems described
is difficult due to lack of apples-to-apples comparison on earlier (see Sections IV-D and V-D) have been studied
the target hardware. Even though new machine learning mainly in the context of general distributed machine learn-
papers contain comparative evaluation with prior existing ing, but their study with regard to edge computing, which
models, the subset of models compared is chosen at the has a smaller set of users and more specialized deep
discretion of the researchers and may not include the learning models, could be valuable.
desired comparisons or hardware platforms. Furthermore,
standalone measurement papers can quickly become out-
VII. C O N C L U S I O N
dated as new DNN models emerge. A public repository
containing apples-to-apples containing benchmark com- This paper reviewed the current state of the art for
parisons between the models on different hardwares could deep learning operating on the network edge. Computer
be of great benefit to the community. This task is made vision, natural language processing, network functions,
slightly easier by the existence of standard data sets in and VR and AR were discussed as example application
certain application domains, such as image classification drivers, with the commonality being the need for real-time
and natural language processing, as well as standard processing of data produced by end devices. Methods for
machine learning platforms such as TensorFlow, Caffe, accelerating deep learning inference across end devices,
and PyTorch. Especially important to edge computing is edge servers, and the cloud were described, which lever-
the comparison on a variety of edge device hardware, age the unique structure of DNN models as well as the
including the simple devices (e.g., Raspberry Pi), smart- geospatial locality of user requests in edge computing.
phones, home gateways, and edge servers. Much of the The tradeoffs between accuracy, latency, and other per-
current work has focused on either on powerful servers formance metrics were found to be important factors in
or on smartphones, but as deep learning and edge com- several works discussed. Training of deep learning models,
puting become prevalent, a comparative understanding of where multiple end devices collaboratively train a DNN
deep learning performance on heterogeneous hardware is model (possibly with the help of an edge server and/or the
needed. cloud) was also discussed, including techniques for further
enhancing privacy.
Many open challenges remain, both in terms of further
E. Privacy performance improvements, as well as privacy, resource
While privacy has been studied generally in the context management, benchmarking, and integration with other
of distributed deep learning, there are several implications networking technologies such as SDN and NFV. These chal-
for edge computing, which merit further investigation. One lenges can be addressed through technological innovations
possible concern is membership attacks. A membership in algorithms, system design, and hardware accelerations.
attack seeks to determine whether a particular item was As the pace of deep learning innovation remains high in
part of the training set used to generate the deep learning the near term, new technical challenges in edge computing
model [149]. This attack gains significance in edge com- may emerge in the future, alongside the existing opportu-
puting, as a successful attack on an edge server’s DNN nities for innovation.

REFERENCES
[1] O. Russakovsky et al., “ImageNet large scale visual [3] D. Jeans. (Mar. 2019). Related’s Hudson Yards: N. Davies, “The case for VM-based cloudlets in
recognition challenge,” Int. J. Comput. Vis., Smart City or Surveillance City? [Online]. mobile computing,” IEEE Pervasive Comput., vol. 8,
vol. 115, no. 3, pp. 211–252, Dec. 2015. Available: no. 4, pp. 14–23, Oct./Dec. 2009.
[2] M. Satyanarayanan, “The emergence of edge [Link] [5] AT&T Multi-Access Edge Computing. [Online].
computing,” Computer, vol. 50, no. 1, pp. 30–39, yards-smart-city-or-surveillance-city/ Available:
2017. [4] M. Satyanarayanan, V. Bahl, R. Caceres, and [Link]

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1671

Chen and Ran: Deep Learning With Edge Computing: A Review

[Link] [32] NVIDIA. Cudnn. [Online]. Available: [53] H. Zhu, Y. Cao, W. Wang, T. Jiang, and S. Jin,
[6] C.-F.-A. T. Blog. Edge Computing at Chick-Fil-A. [Link] “Deep reinforcement learning for mobile edge
[Online]. Available: [33] L. Liu, H. Li, and M. Gruteser, “Edge assisted caching: Review, new features, and open issues,”
[Link] real-time object detection for mobile augmented IEEE Netw., vol. 32, no. 6, pp. 50–57, Nov. 2018.
computing-at-chick-fil-a-7d67242675e2 reality,” in Proc. ACM MobiCom, 2019, pp. 1–16. [54] Y. M. Saputra, D. T. Hoang, D. N. Nguyen,
[7] I. Goodfellow, Y. Bengio, A. Courville, and [34] S. Liu, Y. Lin, Z. Zhou, K. Nan, H. Liu, and J. Du, E. Dutkiewicz, D. Niyato, and D. I. Kim,
Y. Bengio, Deep Learning, vol. 1. Cambridge, MA, “DeepIoT: Compressing deep neural network “Distributed deep learning at the edge: A novel
USA: MIT Press, 2016. structures for sensing systems with a proactive and cooperative caching framework for
[8] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge compressor-critic framework,” in Proc. SenSys, mobile edge networks,” CoRR,
computing: Vision and challenges,” IEEE Internet 2017, pp. 1–4. vol. abs/1812.05374, pp. 1–4, Dec. 2018.
Things J., vol. 3, no. 5, pp. 637–646, Oct. 2016. [35] Amazon. AWS Deeplens. [Online]. Available: [55] C. Zhong, M. C. Gursoy, and S. Velipasalar, “A
[9] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. [Link] deep reinforcement learning-based framework for
Letaief, “A survey on mobile edge computing: The [36] S. Ren, K. He, R. Girshick, and J. Sun, “Faster content caching,” CoRR, vol. abs/1712.08132,
communication perspective,” IEEE Commun. R-Cnn: Towards real-time object detection with pp. 1–6, Dec. 2017.
Surveys Tuts., vol. 19, no. 4, pp. 2322–2358, region proposal networks,” in Proc. Adv. Neural [56] S. Yao, S. Hu, Y. Zhao, A. Zhang, and
4th Quart., 2017. Inf. Process. Syst., 2015, pp. 91–99. T. Abdelzaher, “DeepSense: A unified deep
[10] U. Drolia, K. Guo, and P. Narasimhan, “Precog: [37] T. Zhang, A. Chowdhery, P. V. Bahl, K. Jamieson, learning framework for time-series mobile sensing
Prefetching for image recognition applications at and S. Banerjee, “The design and implementation data processing,” in Proc. 26th Int. Conf. World
the edge,” in Proc. ACM/IEEE Symp. Edge Comput., of a wireless video surveillance system,” in Proc. Wide Web, 2017, pp. 351–360.
2017, pp. 1–17. 21st Annu. Int. Conf. Mobile Comput. Netw., 2015, [57] W. Ouyang and X. Wang, “Joint deep learning for
[11] H. Li, K. Ota, and M. Dong, “Learning IoT in edge: pp. 426–438. pedestrian detection,” in Proc. IEEE Int. Conf.
Deep learning for the Internet of Things with edge [38] C.-C. Hung et al., “VideoEdge: Processing camera Comput. Vis., Dec. 2013, pp. 2056–2063.
computing,” IEEE Netw., vol. 32, no. 1, streams using hierarchical clusters,” in Proc. [58] L. Li, K. Ota, and M. Dong, “When weather
pp. 96–101, Jan./Feb. 2018. IEEE/ACM Symp. Edge Comput. (SEC), Oct. 2018, matters: IoT-based electrical load forecasting for
[12] L. N. Huynh, Y. Lee, and R. K. Balan, “DeepMon: pp. 115–131. smart grid,” IEEE Commun. Mag., vol. 55, no. 10,
Mobile GPU-based deep learning framework for [39] T. Young, D. Hazarika, S. Poria, and E. Cambria, pp. 46–51, Oct. 2017.
continuous vision applications,” in Proc. 15th “Recent trends in deep learning based natural [59] A. Kumar, S. Goyal, and M. Varma,
Annu. Int. Conf. Mobile Syst., Appl., Services, 2017, language processing,” IEEE Comput. Intell. Mag., “Resource-efficient machine learning in 2 KB RAM
pp. 82–95. vol. 13, no. 3, pp. 55–75, Aug. 2018. for the Internet of Things,” in Proc. 34th Int. Conf.
[13] Y. Kang et al., “Neurosurgeon: Collaborative [40] Apple. Deep Learning for Siri’s Voice: On-Device Mach. Learn., vol. 70, 2017, pp. 1935–1944.
intelligence between the cloud and mobile edge,” Deep Mixture Density Networks for Hybrid Unit [60] Z. Zhao, K. M. Barijough, and A. Gerstlauer,
ACM SIGPLAN Notices, vol. 52, no. 4, Selection Synthesis. [Online]. Available: https:// “Deepthings: Distributed adaptive deep learning
pp. 615–629, 2017. [Link]/2017/08/06/siri- inference on resource-constrained iot edge
[14] R. Shokri and V. Shmatikov, “Privacy-preserving [Link] clusters,” IEEE Trans. Comput.-Aided Design Integr.
deep learning,” in Proc. 22nd ACM SIGSAC Conf. [41] G. Lample, M. Ballesteros, S. Subramanian, K. Circuits Syst., vol. 37, no. 11, pp. 2348–2359,
Comput. Commun. Secur., 2015, pp. 1310–1321. Kawakami, and C. Dyer, “Neural architectures for Nov. 2018.
[15] S. Ruder, “An overview of gradient descent named entity recognition,” 2016, [61] M. Mohammadi, A. Al-Fuqaha, S. Sorour, and
optimization algorithms,” CoRR, arXiv:1603.01360. [Online]. Available: M. Guizani, “Deep learning for IoT big data and
vol. abs/1609.04747, pp. 1–14, Sep. 2016. [Link] streaming analytics: A survey,” IEEE Commun.
[16] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, [42] Y. Wu et al., “Google’s neural machine translation Surveys Tuts., vol. 20, no. 4, pp. 2923–2960, 4th
“BLEU: A method for automatic evaluation of system: Bridging the gap between human and Quart., 2018.
machine translation,” in Proc. 40th Annu. Meeting machine translation,” 2016, arXiv:1609.08144. [62] X. Hou, S. Dey, J. Zhang, and M. Budagavi,
Assoc. Comput. Linguistics, 2002, pp. 311–318. [Online]. Available: “Predictive view generation to enable mobile
[17] J. Huang et al., “Speed/accuracy trade-offs for [Link] 360-degree and VR experiences,” in Proc. Morning
modern convolutional object detectors,” in Proc. [43] Google AI Blog. Google Duplex: An AI System for Workshop Virtual Reality Augmented Reality Netw.,
IEEE CVPR, vol. 4, Jul. 2017, pp. 7310–7311. Accomplishing Real-World Tasks Over the Phone. 2018, pp. 20–26.
[18] J. Redmon and A. Farhadi, “YOLO9000: Better, [Online]. Available: [63] Y. Xu et al., “Gaze prediction in dynamic 360
faster, stronger,” in Proc. IEEE CVPR, Jul. 2017, [Link] immersive videos,” in Proc. IEEE Conf. Comput. Vis.
pp. 7263–7271. [Link] Pattern Recognit., Jun. 2018, pp. 5333–5342.
[19] Z. Lu, S. Rallapalli, K. Chan, and T. La Porta, [44] S. Antol et al., “VQA: Visual question answering,” [64] S. Afzal, J. Chen, and K. K. Ramakrishnan,
“Modeling the resource requirements of in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2015, “Characterization of 360-degree videos,” in Proc.
convolutional neural networks on mobile devices,” pp. 2425–2433. ACM SIGCOMM Workshop Virtual Reality
in Proc. ACM Multimedia, 2017, pp. 1663–1671. [45] Pagespeed Insights: Improve Server Response Time, Augmented Reality Netw., 2017, pp. 1–6.
[20] X. Ran, H. Chen, X. Zhu, Z. Liu, and J. Chen, Google. [65] A. Jindal et al. (Jan. 2018). Enabling Full Body AR
“DeepDecision: A mobile deep learning [46] Apple. (2017). Hey Siri: An On-Device With Mask R-CNN2GO. [Online]. Available:
framework for edge video analytics,” in Proc. IEEE DNN-Powered Voice Trigger for Apple’s Personal [Link]
INFOCOM, Apr. 2018, pp. 1421–1429. Assistant. [Online]. Available: https:// with-mask-r-cnn2go/
[21] A. G. Howard et al., “MobileNets: Efficient [Link]/2017/10/01/hey- [66] S. LaValle, Virtual Reality. Cambridge, U.K.:
convolutional neural networks for mobile vision [Link] Cambridge Univ. Press, 2016.
applications,” 2017, arXiv:1704.04861. [Online]. [47] A. Kusupati, M. Singh, K. Bhatia, A. Kumar, P. Jain, [67] Z. Chen et al., “An empirical study of latency in an
Available: [Link] and M. Varma, “FastGRNN: A fast, accurate, stable emerging class of edge computing applications for
[22] M. Tan et al., “MnasNet: Platform-aware neural and tiny kilobyte sized gated recurrent neural wearable cognitive assistance,” in Proc. 2nd
architecture search for mobile,” 2018, network,” in Proc. Adv. Neural Inf. Process. Syst., ACM/IEEE Symp. Edge Comput., Oct. 2017, p. 14.
arXiv:1807.11626. [Online]. Available: 2018, pp. 9017–9028. [68] K. Ha, Z. Chen, W. Hu, W. Richter, P. Pillai, and
[Link] [48] Raymond Wong. Google’s Pixel Buds are no Match M. Satyanarayanan, “Towards wearable cognitive
[23] B. Taylor, V. S. Marco, W. Wolff, Y. Elkhatib, and for Professional Interpreters. [Online]. Available: assistance,” in Proc. ACM MobiSys, 2014,
Z. Wang, “Adaptive deep learning model selection [Link] pp. 68–81.
on embedded systems,” in Proc. LCTES, 2018, buds-real-time-translations-vs-un- [69] W. Liu et al., “SSD: Single shot multibox detector,”
pp. 31–43. interpreter/?europe=true in Proc. Eur. Conf. Comput. Vis. Springer, 2016,
[24] Tensorflow. [Online]. Available: [49] J. Ryan, M.-J. Lin, and R. Miikkulainen, “Intrusion pp. 21–37.
[Link] detection with neural networks,” in Proc. Adv. [70] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf,
[25] M. Abadi et al., “TensorFlow: Large-scale machine Neural Inf. Process. Syst., 1998, pp. 943–949. W. J. Dally, and K. Keutzer, “SqueezeNet:
learning on heterogeneous distributed systems,” [50] Y. Mirsky, T. Doitshman, Y. Elovici, and A. Shabtai, Alexnet-level accuracy with 50x fewer parameters
CoRR, vol. abs/1603.04467, pp. 1–19, Mar. 2016. “Kitsune: An ensemble of autoencoders for online and <0.5 MB model size,” 2016,
[26] Tensorflowlite. [Online]. Available: network intrusion detection,” 2018, arXiv:1602.07360. [Online]. Available:
[Link] arXiv:1802.09089. [Online]. Available: [Link]
[27] Y. Jia et al., “Caffe: Convolutional architecture for [Link] [71] S. Han, H. Mao, and W. J. Dally, “Deep
fast feature embedding,” in Proc. 22nd ACM Int. [51] S. Chinchali et al., “Cellular network traffic compression: Compressing deep neural networks
Conf. Multimedia, 2014, pp. 675–678. scheduling with deep reinforcement learning,” in with pruning, trained quantization and Huffman
[28] Caffe2. [Online]. Available: [Link] Proc. 32nd AAAI Conf. Artif. Intell., 2018, pp. 1–9. coding,” 2015, arXiv:1510.00149. [Online].
[29] Caffe. [Online]. Available: [52] N. Tsikoudis, A. Papadogiannakis, and E. P. Available: [Link]
[Link] Markatos, “LEoNIDS: A low-latency and [72] L. Lai and N. Suda, “Enabling deep learning at the
[30] Pytorch. [Online]. Available: [Link] energy-efficient network-level intrusion detection IoT edge,” in Proc. Int. Conf. Comput.-Aided Design
[31] NVIDIA. Cuda. [Online]. Available: system,” IEEE Trans. Emerg. Topics Comput., vol. 4, (ICCAD), 2018, p. 135.
[Link] no. 1, pp. 142–155, Jan. 2016. [73] S. Han et al., “ESE: Efficient speech recognition

1672 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

 Chen and Ran: Deep Learning With Edge Computing: A Review

engine with sparse LSTM on FPGA,” in Proc. pp. 377–392. P. S. Yu, “Not just privacy: Improving performance
ACM/SIGDA Int. Symp. Field-Program. Gate Arrays [94] J. Jiang, G. Ananthanarayanan, P. Bodik, S. Sen, of private deep learning in mobile cloud,” in Proc.
(FPGA), 2017, pp. 75–84. and I. Stoica, “Chameleon: Scalable adaptation of 24th ACM SIGKDD Int. Conf. Knowl. Discovery
[74] S. Bhattacharya and N. D. Lane, “Sparsification video analytics,” in Proc. Conf. ACM Special Data Mining, 2018, pp. 2407–2416.
and separation of deep learning layers for Interest Group Data Commun., 2018, pp. 253–266. [116] R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter,
constrained resource inference on wearables,” in [95] A. H. Jiang et al., “Mainstream: Dynamic M. Naehrig, and J. Wernsing, “CryptoNets:
Proc. 14th ACM Conf. Embedded Netw. Sensor Syst. stem-sharing for multi-tenant video processing,” Applying neural networks to encrypted data with
CD-ROM (SenSys), 2016, pp. 176–189. in Proc. USENIX Annu. Tech. Conf. (USENIXATC), high throughput and accuracy,” in Proc. Int. Conf.
[75] G. Hinton, O. Vinyals, and J. Dean, “Distilling the 2018, pp. 29–42. Mach. Learn., 2016, pp. 201–210.
knowledge in a neural network,” 2015, [96] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, [117] J. Liu, M. Juuti, Y. Lu, and N. Asokan, “Oblivious
arXiv:1503.02531. [Online]. Available: “Gradient-based learning applied to document neural network predictions via minionn
[Link] recognition,” Proc. IEEE, vol. 86, no. 11, transformations,” in Proc. ACM SIGSAC Conf.
[76] S. Teerapittayanon, B. McDanel, and H. Kung, pp. 2278–2324, Nov. 1998. Comput. Commun. Secur., 2017, pp. 619–631.
“BranchyNet: Fast inference via early exiting from [97] Z. Huang, W. Xu, and K. Yu, “Bidirectional [118] B. D. Rouhani, M. S. Riazi, and F. Koushanfar,
deep neural networks,” in Proc. Int. Conf. Pattern LSTM-CRF models for sequence tagging,” CoRR, “DeepSecure: Scalable provably-secure deep
Recognit., Dec. 2016, pp. 2464–2469. vol. abs/1508.01991, pp. 1–10, Aug. 2015. learning,” in Proc. 55th Annu. Design Autom. Conf.,
[77] S. Yao, Y. Zhao, Z. Aston, L. Su, and T. Abdelzaher, [98] S. Hochreiter and J. Schmidhuber, “Long 2018, p. 2.
“On-demand deep model compression for mobile short-term memory,” Neural Comput., vol. 9, no. 8, [119] C. Juvekar, V. Vaikuntanathan, and
devices: A usage-driven model selection pp. 1735–1780, 1997. A. Chandrakasan, “GAZELLE: A low latency
framework,” in Proc. MobiSys, 2018, pp. 389–400. [99] A. Krizhevsky, I. Sutskever, and G. E. Hinton, framework for secure neural network inference,”
[78] N. Loc Huynh, Y. Lee, and R. K. Balan, “DeepMon: “ImageNet classification with deep convolutional in Proc. 27th USENIX Secur. Symp. (USENIX
Mobile GPU-based deep learning framework for neural networks,” in Proc. Adv. Neural Inf. Process. Security), 2018, pp. 1651–1669.
continuous vision applications,” in Proc. ACM Syst., 2012, pp. 1097–1105. [120] M. S. Riazi, C. Weinert, O. Tkachenko,
MobiSys, 2017, pp. 82–95. [100] K. Simonyan and A. Zisserman, “Very deep E. M. Songhori, T. Schneider, and F. Koushanfar,
[79] Edge TPU. [Online]. Available: convolutional networks for large-scale image “Chameleon: A hybrid secure computation
[Link] recognition,” CoRR, vol. abs/1409.1556, pp. 1–14, framework for machine learning applications,” in
[80] Z. Du et al., “Shidiannao: Shifting vision Sep. 2014. Proc. Asia Conf. Comput. Commun. Secur., 2018,
processing closer to the sensor,” ACM SIGARCH [101] A. Vedaldi and K. Lenc, “MatConvNet: pp. 707–721.
Comput. Archit. News, vol. 43, no. 3, pp. 92–104, Convolutional neural networks for MATLAB,” in [121] J. Dean et al., “Large scale distributed deep
2015. Proc. 23rd ACM Int. Conf. Multimedia (MM), 2015, networks,” in Proc. Adv. Neural Inf. Process. Syst.,
[81] Y. Chen, T. Chen, Z. Xu, N. Sun, and O. Temam, pp. 689–692. 2012, pp. 1223–1231.
“Diannao family: Energy-efficient hardware [102] C. Szegedy et al., “Going deeper with [122] D. Li, T. Salonidis, N. V. Desai, and M. C. Chuah,
accelerators for machine learning,” Commun. convolutions,” CoRR, vol. abs/1409.4842, “DeepCham: Collaborative edge-mediated
ACM, vol. 59, no. 11, pp. 105–112, 2016. pp. 1–12, Sep. 2014. adaptive deep learning for mobile object
[82] K. Ovtcharov, O. Ruwase, J.-Y. Kim, J. Fowers, [103] K. He, X. Zhang, S. Ren, and J. Sun, “Deep recognition,” in Proc. IEEE/ACM Symp. Edge
K. Strauss, and E. S. Chung. Accelerating Deep residual learning for image recognition,” CoRR, Comput. (SEC), 2016, pp. 64–76.
Convolutional Neural Networks Using Specialized vol. abs/1512.03385, pp. 1–12, Dec. 2015. [123] J. Chen, R. Monga, S. Bengio, and R. Jözefowicz,
Hardware. [Online]. Available: [104] S. Han, H. Shen, M. Philipose, S. Agarwal, A. “Revisiting distributed synchronous SGD,” CoRR,
[Link] Wolman, and vol. abs/1604.00981, pp. 1–10, Apr. 2016.
content/uploads/2016/02/[Link] A. Krishnamurthy, “MCDNN: An [124] S. Zhang, A. E. Choromanska, and Y. LeCun,
[83] VPU. [Online]. Available: approximation-based execution framework for “Deep learning with elastic averaging SGD,” in
[Link] deep stream processing under resource Advances in Neural Information Processing Systems,
processing-unit constraints,” in Proc. ACM Mobisys, 2016, C. Cortes, N. D. Lawrence, D. D. Lee,
[84] S. Rivas-Gomez, A. J. Pena, D. Moloney, E. Laure, pp. 123–136. M. Sugiyama, and R. Garnett, Eds. Red Hook, NY,
and S. Markidis, “Exploring the vision processing [105] Y. Taigman, M. Yang, M. A. Ranzato, and L. Wolf, USA: Curran Associates, 2015, pp. 685–693.
unit as co-processor for inference,” in Proc. IEEE “DeepFace: Closing the gap to human-level [125] H. B. McMahan et al., “Communication-efficient
Int. Parallel Distrib. Process. Symp. Workshops performance in face verification,” in Proc. IEEE learning of deep networks from decentralized
(IPDPSW), May 2018, pp. 589–598. Conf. Comput. Vis. Pattern Recognit., Jun. 2014, data,” 2016, arXiv:1602.05629. [Online].
[85] Nvidia. NVIDIA EGX Edge Computing Platform. pp. 1701–1708. Available: [Link]
[Online]. Available: [Link] [106] H. Li, K. Ota, and M. Dong, “Learning IoT in edge: [126] S. Wang et al., “When edge meets learning:
us/data-center/products/egx-edge-computing/ Deep learning for the Internet of Things with edge Adaptive control for resource-constrained
[86] Qualcomm. Qualcomm Neural Processing SDK for computing,” IEEE Netw., vol. 32, no. 1, distributed machine learning,” CoRR,
AI. [Online]. Available: [Link] pp. 96–101, Jan. 2018. vol. abs/1804.05271, pp. 1–20, Feb. 2018.
[Link]/software/qualcomm-neural- [107] J. Mao, X. Chen, K. W. Nixon, C. Krieger, and [127] K. Hsieh et al., “Gaia: Geo-distributed machine
processing-sdk Y. Chen, “MoDNN: Local distributed mobile learning approaching LAN speeds,” in Proc. 14th
[87] M. Alzantot, Y. Wang, Z. Ren, and M. B. computing system for deep neural network,” in USENIX Symp. Netw. Syst. Design Implement.
Srivastava, “RSTensorFlow: GPU enabled Proc. Design, Autom. Test Eur. Conf. Exhibit. (NSDI), 2017, pp. 629–647.
tensorflow for deep learning on commodity (DATE), Mar. 2017, pp. 1396–1401. [128] R. Anil et al., “Large scale distributed neural
android devices,” in Proc. 1st Int. Workshop Deep [108] Arcore Overview. [Online]. Available: network training through online distillation,”
Learn. Mobile Syst. Appl. (EMDL), 2017, pp. 7–12. [Link] CoRR, vol. abs/1804.03235, pp. 1–12, Apr. 2018.
[88] N. D. Lane et al., “DeepX: A software accelerator [109] S. Teerapittayanon, B. McDanel, and H. Kung, [129] Y. Lin, S. Han, H. Mao, Y. Wang, and J. William
for low-power deep learning inference on mobile “Distributed deep neural networks over the cloud, Dally, “Deep gradient compression: Reducing the
devices,” in Proc. 15th ACM/IEEE Int. Conf. Inf. the edge and end devices,” in Proc. IEEE 37th Int. communication bandwidth for distributed
Process. Sensor Netw. (IPSN), 2016, p. 23. Conf. Distrib. Comput. Syst. (ICDCS), Jun. 2017, training,” CoRR, vol. abs/1712.01887, pp. 1–13,
[89] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, pp. 328–339. Feb. 2017.
“Efficient processing of deep neural networks: A [110] S. Teerapittayanon, B. McDanel, and H. T. Kung, [130] C. Hardy, E. L. Merrer, and B. Sericola,
tutorial and survey,” Proc. IEEE, vol. 105, no. 12, “BranchyNet: Fast inference via early exiting from “Distributed deep learning on edge-devices:
pp. 2295–2329, Dec. 2017. deep neural networks,” CoRR, Feasibility via adaptive compression,” CoRR,
[90] X. Wang, X. Wang, and S. Mao, “RF sensing in the vol. abs/1709.01686, pp. 1–7, Sep. 2017. vol. abs/1702.04683, pp. 1–8, Nov. 2017.
Internet of Things: A general deep learning [111] X. Ran, H. Chen, Z. Liu, and J. Chen, “Delivering [131] M. Blot, D. Picard, M. Cord, and N. Thome,
framework,” IEEE Commun. Mag., vol. 56, no. 9, deep learning to mobile devices via offloading,” in “Gossip training for deep learning,” Nov. 2016,
pp. 62–67, Sep. 2018. Proc. ACM SIGCOMM Workshop Virtual Reality arXiv:1611.09726. [Online]. Available:
[91] T. Y.-H. Chen, L. Ravindranath, S. Deng, P. Bahl, Augmented Reality Netw., 2017, pp. 42–47. [Link]
and H. Balakrishnan, “Glimpse: Continuous, [112] E. Cuervo et al., “MAUI: Making smartphones last [132] P. H. Jin, Q. Yuan, F. Iandola, and K. Keutzer, “How
real-time object recognition on mobile devices,” in longer with code offload,” ACM MobiSys, 2010, to scale distributed deep learning?” 2016,
Proc. ACM SenSys, 2015, pp. 155–168. pp. 49–62. arXiv:1611.04581. [Online]. Available:
[92] C. Liu et al., “A new deep learning-based food [113] S. Yi, Z. Hao, Q. Zhang, Q. Zhang, W. Shi, and [Link]
recognition system for dietary assessment on an Q. Li, “Lavea: Latency-aware video analytics on [133] Y. Li et al., “A network-centric hardware/algorithm
edge computing service infrastructure,” IEEE edge computing platform,” in Proc. ACM/IEEE co-design to accelerate distributed training of
Trans. Services Comput., vol. 11, no. 2, Symp. Edge Comput., 2017, p. 15. deep neural networks,” in Proc. 51st Annu.
pp. 249–261, Jan. 2018. [114] M.-R. Ra, A. Sheth, L. Mummert, P. Pillai, IEEE/ACM Int. Symp. Microarchitecture (MICRO),
[93] H. Zhang, G. Ananthanarayanan, P. Bodik, D. Wetherall, and R. Govindan, “Odessa: Enabling Oct. 2018, pp. 175–188.
M. Philipose, P. Bahl, and M. J. Freedman, “Live interactive perception applications on mobile [134] M. Abadi et al., “Deep learning with differential
video analytics at scale with approximation and devices,” ACM MobiSys, 2011, pp. 43–56. privacy,” in Proc. ACM SIGSAC Conf. Comput.
delay-tolerance,” in Proc. USENIX NSDI, 2017, [115] J. Wang, J. Zhang, W. Bao, X. Zhu, B. Cao, and Commun. Secur., Oct. 2016, pp. 308–318.

Vol. 107, No. 8, August 2019 | P ROCEEDINGS OF THE IEEE 1673

 Chen and Ran: Deep Learning With Edge Computing: A Review

[135] T. Zhang, Z. He, and R. B. Lee, “Privacy-preserving Oct. 2017, pp. 1–8. Comput., 2017, p. 10.
machine learning through data obfuscation,” [141] R. LiKamWa, B. Priyantha, M. Philipose, L. Zhong, [146] W. Zhang, J. Hwang, S. Rajagopalan,
Jul. 2018, arXiv:1807.01860. [Online]. Available: and P. Bahl, “Energy characterization and K. Ramakrishnan, and T. Wood, “Flurries:
[Link] optimization of image sensing toward continuous Countless fine-grained nfs for flexible per-flow
[136] P. Mohassel and Y. Zhang, “SecureML: A system mobile vision,” in Proc. 11th Annu. Int. Conf. customization,” in Proc. 12th Int. Conf. Emerg.
for scalable privacy-preserving machine learning,” Mobile Syst., Appl., Services, 2013, pp. 69–82. Netw. Exp. Technol., 2016, pp. 3–17.
in Proc. IEEE Symp. Secur. Privacy (SP), May 2017, [142] S. Naderiparizi, P. Zhang, M. Philipose, [147] S. Kaul, R. Yates, and M. Gruteser, “Real-time
pp. 19–38. B. Priyantha, J. Liu, and D. Ganesan, “Glimpse: A status: How often should one update?” in Proc.
[137] Y. Mao, S. Yi, Q. Li, J. Feng, F. Xu, and S. Zhong, programmable early-discard camera architecture IEEE INFOCOM, Mar. 2012, pp. 2731–2735.
“Learning from differentially private neural for continuous mobile vision,” in Proc. 15th Annu. [148] J. Cho, K. Sundaresan, R. Mahindra,
activations with edge computing,” in Proc. Int. Conf. Mobile Syst., Appl., Services, 2017, J. van der Merwe, and S. Rangarajan, “ACACIA:
IEEE/ACM Symp. Edge Comput. (SEC), Oct. 2018, pp. 292–305. Context-aware edge computing for continuous
pp. 90–102. [143] K. Ha et al., “You can teach elephants to dance: interactive applications over mobile networks,” in
[138] N. Qian, “On the momentum term in gradient Agile VM handoff for edge computing,” in Proc. Proc. 12th Int. Conf. Emerg. Netw. Exp. Technol.,
descent learning algorithms,” Neural Netw., 2nd ACM/IEEE Symp. Edge Comput., 2017, p. 12. 2016, pp. 375–389.
vol. 12, no. 1, pp. 145–151, 1999. [144] L. Ma, S. Yi, and Q. Li, “Efficient service handoff [149] R. Shokri, M. Stronati, C. Song, and V. Shmatikov,
[139] P. Goyal et al., “Accurate, large minibatch SGD: across edge servers via docker container “Membership inference attacks against machine
Training imagenet in 1 hour,” CoRR, migration,” in Proc. 2nd ACM/IEEE Symp. Edge learning models,” in Proc. IEEE Symp. Secur.
vol. abs/1706.02677, pp. 1–12, Jun. 2017. Comput., 2017, p. 11. Privacy (SP), May 2017, pp. 3–18.
[140] C. Hardy, E. L. Merrer, and B. Sericola, [145] L. Chaufournier, P. Sharma, F. Le, E. Nahum, [150] C. Dwork, F. McSherry, K. Nissim, and A. Smith,
“Distributed deep learning on edge-devices: P. Shenoy, and D. Towsley, “Fast transparent “Calibrating noise to sensitivity in private data
Feasibility via adaptive compression,” in Proc. virtual machine migration in distributed edge analysis,” in Proc. Theory Cryptogr. Conf. Springer,
IEEE 16th Int. Symp. Netw. Comput. Appl. (NCA), clouds,” in Proc. 2nd ACM/IEEE Symp. Edge 2006, pp. 265–284.

ABOUT THE AUTHORS

Jiasi Chen received the B.S. degree from Xukan Ran received the B.S. degree in
Columbia University, New York, NY, USA, network engineering from Xidian University,
with internships at AT&T Labs Research, Xi’an, China. He is currently working toward
Florham Park, NJ, USA, and NEC Labs Amer- the Ph.D. degree in computer science at the
ica, Princeton, NJ, USA, and the Ph.D. degree University of California at Riverside, River-
from Princeton University, Princeton, NJ, side, CA, USA. He also studied computer
USA. science at Zhejiang University, Hangzhou,
She is currently an Assistant Professor with China.
the Department of Computer Science and His current research interests include
Engineering, University of California at Riverside, Riverside, CA, edge computing, deep learning, and simultaneous localization and
USA. Her current research interests include edge computing, wire- mapping on mobile devices.
less and mobile systems, and multimedia networking, with a recent Mr. Ran received the Best-in-Session Presentation Award at the
focus on machine learning at the network edge to aid augmented IEEE INFOCOM in 2018.
reality (AR)/virtual reality (VR) applications.
Dr. Chen was a recipient of the Hellman Fellowship and the UCR
Regents Faculty Fellowship.

1674 P ROCEEDINGS OF THE IEEE | Vol. 107, No. 8, August 2019

View publication stats