Multimedia Tools and Applications

[Link]

A deep learning framework with edge computing

for severity level detection of diabetic retinopathy

Ahmed Al-Karawi 1 & Ercan Avşar

2,3

Received: 31 May 2022 / Revised: 23 September 2022 / Accepted: 13 March 2023

# The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023

Abstract
Diabetic retinopathy is one of the major causes of the vision loss worldwide. Its timely
detection is critical for planning an efficient treatment process. Typically, fundus images
are taken for diagnosis of diabetic retinopathy and determining its corresponding severity
level. In this study, a framework that uses a mobile edge device for detecting the severity
level of diabetic retinopathy is proposed. For this purpose, a dataset of fundus images
containing five different diabetic retinopathy severity levels is utilized. The mobile device
is responsible for performing the edge processing operations in which the fundus images
are preprocessed by cropping, unsharp masking, and resizing. The preprocessed images
are then transmitted to a cloud computing platform over the internet. In the cloud server, a
concatenation ensemble deep learning models is trained for detecting the severity level of
the diabetic retinopathy. The ensemble model involves three benchmark convolutional
neural network architectures that are EfficientNetB7, ResNet50, and VGG19. The clas-
sification accuracy achieved using the concatenation ensemble is 96%, which is higher
than those obtained via individual convolutional neural network models. In addition,
contributions of edge computing are shown by calculating the total amount of transmitted
data and the response time from the cloud server. It was observed that for classifying the
entire test set 2984.52 Kb of less data, corresponding to average data size reduction of
85.2%, was transmitted and the response time was reduced by 6.14 seconds when the
preprocessing steps are performed at the edge device.

Keywords Classification . Concatenation ensemble . Deep learning . Diabetic retinopathy . Edge

computing . Fundus

* Ercan Avşar
erca@[Link]

1
Department of Electrical and Electronics Engineering, Çukurova University, 01330 Adana, Turkey
2
National Institute of Aquatic Resources, Technical University of Denmark (DTU Aqua),
9850 Hirtshals, Denmark
3
Department of Computer Engineering, Dokuz Eylül University, 35390 İzmir, Turkey
 Multimedia Tools and Applications

1 Introduction

Diabetic eye diseases are serious problems that may cause vision loss for the patients. In the
recent years, the number of cases related to diabetic eye diseases has an increasing trend and is
expected to reach epidemic proportions worldwide over the next few decades [57]. The most
frequent and insidious microvascular complication of diabetes is diabetic retinopathy (DR) that
may advance asymptomatically until a sudden vision loss occurs. Approximately one-third of
the 285 million people with diabetes mellitus internationally experience symptoms of DR,
which is considered to be the one of the major causes of vision loss worldwide [56]. Diabetic
retinopathy may be observed at different levels of severity in the patients and this severity level
information enables appropriate planning for the treatment. Today’s DR grading system is
based on the collaborative study, namely, Early Treatment Diabetic Retinopathy Study
(ETDRS), that was performed in 1980s [14]. According to this system, five DR classes are
present depending on their level of severity. This classification is based on presence and
density of small new vessel formation in the eye, cotton wool spots, and bleeding into the eye.
The properties of these five classes are summarized in Table 1. The imaging modality named
as fundus photography is capable of generating digital images with which the grading of DR is
possible. Fundus imaging is a useful tool for visualizing possible lesions by taking images
from different angles. Besides, it is associated with a specific low power microscope that is
easy to use and having no serious side effects [37]. Basically, images of the rear of the eye are
taken then and then the images are visually observed. Obviously, this procedure for diagnosis
of DR requires experience and expertise.
With the recent advancements in the area of deep learning, it is now possible to help
clinicians to diagnose in various medical areas such as detection of skin cancer [1], brain
tumors [7, 26], and lung diseases [6, 8]. As one area of deep learning, the convolutional neural
networks (CNN) allow for end-to-end evaluation of the raw medical images by automatically
extracting the features from the images. As a result, many methods based on CNN addressing
the complex problems like image classification, object detection and segmentation have
emerged [16]. Therefore, detection of DR from the fundus images using CNN methods is
possible. This computer-assisted automatic process has potential to reduce the risk of
misdetection of the disease eventually.
The next generation of network technologies let the internet of things (IoT) systems be
deployed in numerous areas. As a result, contemporary smart healthcare frameworks allow for
remote analysis of the healthcare data. Typically, the healthcare data are transmitted through
the internet to the doctors located at a distant clinic. The pipeline for this transmission requires
processing of the healthcare data for two main reasons, which are (i) saving bandwidth in the
network traffic and (ii) helping the doctors to make a more accurate diagnosis [18]. Here, these

Table 1 Basic properties of different diabetic retinopathy grades [14]

DR Grade No DR Grade Name Properties

0 No Visible Disease Healthy vessels and tissue

1 Mild Non-Proliferative DR Growth in the small blood vessels
2 Moderate Non-Proliferative Small bleeds and cotton wool spots in the small blood vessels
DR
3 Severe Non-Proliferative DR Severe level of damage in the blood vessels
4 Proliferative DR New vessel formations in the retina and bleeding at higher level
Multimedia Tools and Applications

points are accomplished by edge computing and cloud computing layers, respectively. Edge
computing involves preprocessing of the raw data before transmitting it to the cloud. Hence, it
reduces the amount of data traffic by operations like filtering and compression. On the other
hand, processing steps requiring heavier computations, such as running the deep learning
algorithms, are performed at the cloud computing layer (Fig. 1). Therefore, the processing
capabilities of the cloud computing hardware is expected to be much higher than edge
computing devices. Accordingly, the devices with limited processing feature like single board
computers or mobile phones may be used as the edge computing hardware. However, the
devices in the cloud computing layer should be able to handle heavy computation tasks, so
they are usually provided as virtual machines (VM) involved in infrastructure as a service
(IaaS) [Link] this study, a framework for detecting the severity level of DR is proposed.
The framework involves operations both at the edge and cloud sides. The edge computing
steps involve cropping, filtering and resizing the fundus images on a mobile phone. For this
purpose, a mobile application is developed which makes it possible to perform these prepro-
cessing steps. The preprocessed images are then transmitted to the cloud server where deep
learning algorithms are executed for detecting the severity level of the DR for each of the
images. The final classification result denoting the severity level of the DR is also made
available to the user through the mobile application. The overall steps of the framework are
illustrated in Fig. 2.
During the development of the study, numerous experiments were carried out with various
combinations of CNN architectures. As a result of these experiments, it was observed that
using three models that are Residual Networks (ResNet50) [19], EfficientNetB7 [49] and
VGG19 [44] achieves better performance. Therefore, these models are selected as the CNN
architectures for classification of the images on the cloud side. In the experiments, the
individual classification performances of these models are calculated, and they are compared
with the performances of the concatenated ensemble models generated from these methods.
Furthermore, the effects of performing edge computing, on the required bandwidth and

Fig. 1 A smart healthcare system configuration

 Multimedia Tools and Applications

Fig. 2 Steps of the proposed framework

response time from the cloud server are computed. As a result, the research questions (RQ)
addressed with this work can be given as:

& RQ1- Is it possible to increase the detection performances of some benchmark models
using ensemble models?
& RQ2- Are the edge computing steps useful in reducing both the network data traffic and
response time while maintaining higher classification accuracy?

The rest of the paper is organized as follows. In Section 2, the previous works about diabetic
retinopathy detection and deep learning applications are provided together with the studies
focusing on edge computing. The details of the proposed frameworks and the experiments
performed are explained in Section 3. Obtained results and their corresponding discussions are
given in Section 4. Finally, the paper is concluded by Section 5.

2 Related work

2.1 Studies based on deep learning

Deep learning has various application areas like computer vision, speech recognition, natural
language processing, and image classification [29]. Deep learning models are based on
artificial neural networks (ANN) that involves hierarchical layers of non-linear processing
stages for learning patterns using the input features.
In the literature, there are many applications of deep learning for computer-aided diagnosis
tasks including detection and classification of DR. One reason of this situation may be the
possibility of applying transfer learning which means transferring the knowledge learned from
one problem to another one. Also, the researchers mainly tend to use large open-source
benchmark CNN architectures in their applications. AlexNet is one of the older architectures
and utilized in recent works through being modified [43] or after preprocessing the input
fundus images [25]. Alternatively, a sequential architecture resembling VGG model was
Multimedia Tools and Applications

proposed and gradually improved by adding features like multi-resolution training and patient-
based analysis [34]. This method achieved an AUC score of 98.2% on Messior-2 dataset [13].
By Khalifa et al., an exhaustive research on transfer learning using six different CNN
benchmark architectures was performed to detect the severity levels of fundus images [22].
Asia Pacific Tele-Ophthalmology Society (APTOS) 2019 dataset [5] was used in the study and
it was reported that AlexNet outputs the highest accuracy as 97.9%.
One other strategy is to replace the fully connected classification top of a CNN model with
another classifier model. For example, Taufiqurrahman et al. generated a hybrid method by
combining convolutional layers of MobileNetV2 model with a support vector machine (SVM)
classifier [50]. In a more comprehensive study, Butt et al. proposed to combine the features
from GoogleNet and ResNet-18 models and perform classification by four classifiers namely,
SVM, random forests (RF), radial basis functions (RBF), and naïve Bayes (NB) [9]. They
showed that SVM performs better than the others when the features are concatenated from
these deep learning models. Alternatively, ensemble of various traditional machine learning
methods was used for this task. An ensemble method based on AdaBoost that utilizes several
hand-crafted features for detection was proposed by Antal et al. [4]. In a more recent study, an
ensemble method based on stacking of five CNN architectures is proposed and it was shown
that the ensemble method improves the detection accuracy [36].
Apart from these modified or combined methods, there exists simpler and effective
methods where transfer learning is directly applied to different architectures [52], ensemble
of candidate detections is determined through image processing methods [3] or entire VGG19
model is trained from scratch after region of interest is extracted in the fundus images [31]. In
addition, improving the performance of machine learning models through clustering methods,
such as fuzzy C-means [46, 48], or nature-inspired metaheuristic algorithms [12] are com-
monly used strategies in different medical applications.
Finally, it is possible to mention the novel deep learning architectures developed particu-
larly for DR detection or grading. Deep learning models typically require large amount of
input data for an efficient training. For some cases where the data is not easy to collect, this
requirement may be an issue. Few-shot learning is an alternative approach to address this issue.
DRNet framework proposed by Murugappan et al. uses an idea based on few-shot classifica-
tion of the DR images [32]. On the other hand, active deep learning model proposed by
Qureshi et al. improves feature space representation of the fundus images by segmentation of
region of interests [38]. This approach eventually allows for more effective classification of the
fundus images compared to its counterparts. Another example of such networks can be the
Hinge Attention Network (HA-Net) which contains multiple attention steps based on
autoencoders [42]. HA-Net model was tested on two different public datasets and shown to
provide relatively effective solution to DR grading problem.

2.2 Studies about edge computing

In the related literature, there are many recent works emphasizing the importance and
usefulness of edge/fog computing for healthcare. For example, a system implementing
cognitive computing on edge for detecting emergency situations of the patients was proposed
by Chen et al. [11]. The system was evaluated by quantizing the user experience and it was
shown that the edge-based healthcare system improves the quality of experience (QoE). In
addition, utilization of mobile devices for edge computing is another widely-used solution in
medical applications. Sohro et al. proposed a framework improving the quality of service
 Multimedia Tools and Applications

(QoS) that enables efficient medical video streaming for remote tele-surgery operations over
mobile edge computing networks [45]. Furthermore, there are some mobile patient monitoring
systems proposed for patient with chronic diseases [17] or elderly people [27], but such works
do not either perform any computing at the edge side or do not quantify the benefits of mobile
edge computing on the metrics like latency, total transmitted data amount, and overall system
performance.

2.3 Research gaps and contributions

As can be seen from the existing studies in the literature, the studies about DR classification
mainly focus on improving the performance of the models using different models and
strategies. On the other hand, the studies about edge computing in healthcare concentrate on
features like patient monitoring, streaming and detection of emergency situations, but they
overlook the contributions of the edge computing to the network traffic.
There are commercially available retinal imaging hardware that can be attached to
smartphones [23, 28]. Two examples of these systems, iExaminer [2] and D-Eye [55], are
shown in Fig. 3. Even though some of these systems may be featuring cloud-based services,
none of them offer advanced image analysis and diagnosis tools [20]. There are some existing
studies that process the fundus images collected from such systems and automatically detect
DR using artificial intelligence [39] or specific deep learning methods [20]. Therefore, a
framework that can process the images taken by these systems on a cloud-based service and
display the results to the users can be useful for such systems.
The current status of networking and computing technologies allows for utilizing the related
methods in a collaborative manner to develop sophisticated systems that can be used with the
retinal imaging hardware. However, there are no research studies that use edge and cloud
computing together to detect the severity levels of DR images. As a result, this work aims to
fill this gap by the proposed novel framework in which the network traffic load is reduced by
edge computing steps and the DR images are classified on a distant edge computer. Further
contributions are made by performing the edge processing on a mobile device and generating
deep learning ensemble models to improve the detection accuracy.

Fig. 3 Commercially available retinal imaging systems attachable to smartphones: (a) iExaminer, and (b) D-Eye
Multimedia Tools and Applications

3 Materials and methods

3.1 The dataset

The fundus image dataset generated by Asia Pacific Tele-Ophthalmology Society (APTOS) is
used in this work [5]. It is a publicly available dataset containing 3662 labelled image files
each of which corresponds to one of the five different severity levels of DR. The images in the
dataset have various resolutions from 433 × 289 pixels to 5184 × 3456 pixels.
The data augmentation steps, rotation, horizontal flip, and vertical flip have been applied
randomly to the images of the specific DR classes with relatively lower number of images. As
a result, the total number of images were increased to 5388. In Table 2, names of the DR
severity levels and distribution of images for these classes are given together with some sample
images. The proportions of the training, validation, and test sets were specified as 72.25%,
12.75%, and 15%, respectively.

3.2 The edge computing steps

The purpose of the steps involved in edge computing is to reduce the total amount of data
transferred to the cloud server and at the same time, preprocess the images. To this end, three
operations, namely, cropping, edge enhancing, and resizing, are performed on the images.
Some of the images contained uninformative dark areas in the outer parts. Therefore,
cropping these parts allows for reducing the image size without any loss of information. Next,
the details like tiny vessels in the images have been made visible by applying unsharp
masking. High-frequency components (i.e. edges) are enhanced by subtracting the blurred
version of the input image from the original image.
The CNN architectures typically have a predefined dimension for the input images. So, in
general, the initial step in a CNN-based classification scheme is to resize the image to the
appropriate dimension. This resizing process is also performed on the mobile edge device and
then the final preprocessed image is transmitted to the cloud side where the CNN architectures
are trained and predictions are performed. The effects of these steps are presented in Fig. 4.
The edge computing steps are performed with an Android application run on a virtual
mobile device that is emulated using Android Studio [54]. A simple graphical user interface
(GUI) enabling features like image selection, preprocessing, and sending the image to the
cloud is designed within the application on the virtual device (Fig. 5). This application is a
prototype for performing the edge processing operations and obtaining the parameters related
with system performance.
In order to underline the contributions of the edge computing, the experiments were
performed with and without edge computing steps and three corresponding metrics, (i)
classification performance, (ii) total amount of transmitted data, and (iii) system response time
are calculated.

3.3 Experiments performed on the cloud side

The preprocessed images by the edge device are transmitted to the cloud server over an
internet connection. At the cloud side, these images are assigned to one of the classes (i.e. the
DR severity levels) in the dataset then the result is transmitted back to edge device and
displayed to the user.
 Multimedia Tools and Applications

Table 2 Dataset information and sample images

Number of
Severity Level Sample Image images in the
dataset

No DR 1410

Mild DR 1288

Moderate DR 874

Severe DR 720

Proliferative DR 1096
Multimedia Tools and Applications

Fig. 4 Preprocessing steps performed on the edge. (a) A sample image from the dataset (819 × 614) (b) Black
regions around the image removed. (c) Unsharp masking applied. (d) Filtered image resized (224 × 224)

The cloud-side operations include training of an ensemble classifier that is based on

concatenation of three benchmark CNN architectures, namely, EfficientNetB7, VGG19,
and ResNet50. The general procedure for training of a concatenation ensemble model
involves inputting the images to the feature extraction layers to different models sepa-
rately. The outputs from these feature extraction steps are then flattened and concatenat-
ed into one single longer vector. Finally, this vector is connected to a dense layer having
a softmax activation function.
The structure of the proposed concatenation ensemble model begins with extraction
of the features using the benchmark models. The model architectures used in the
ensemble are set to have an input image size of 224 × 224. Feature extraction layers
are followed by max pooling and dense layers that allow each model to have a feature
vector with 128 components. Concatenation of these three feature vectors generates an
ensemble feature vector containing 384 components in total. The subsequent classifi-
cation layers include three dense layers and a dropout layer where the final dense layer
has a softmax activation function with five outputs. This model structure is illustrated
in Fig. 6.
 Multimedia Tools and Applications

Fig. 5 The screenshot of the

Android application for edge
processing

4 Results and discussion

The performance of the framework has been evaluated under two main topics. First one is the
effect of generating an ensemble classifier instead of training individual models. The other one
involves analyzing the effects of edge computing steps on the network-related parameters such
as system response time and total transmitted data.

4.1 Performance evaluation metrics

To calculate the performance of the predictions made during the experiments, five-class
confusion matrices were generated for each of the classifications made by the models. Since
this is a multi-class classification problem, it is required to define the true positive (TP), true
negative (TN), false positive (FP), false negative (FN) values for each class separately. These
definitions are illustrated with the help of the generic confusion matrix given in Table 3.
Having defined the confusion matrices, the values for TP, TN, FP, and FN are determined
according to the reference class. For instance, these values are defined for “Moderate” class as:
Multimedia Tools and Applications

Fig. 6 The architecture for the proposed ensemble model. The numbers in parenthesis denote the output shape of
the corresponding layer

TP ¼ c33
TN ¼ c11 þ c12 þ c14 þ c15 þ c21 þ c22 þ c24 þ c25 þ c41 þ c42 þ c44 þ c45 þ c51 þ c52 þ c54 þ c55
FP ¼ c13 þ c23 þ c43 þ c53
FN ¼ c31 þ c32 þ c34 þ c35

These are used for calculating three performance metrics as given below.
TP
precision ¼
TP þ FP
TP
recall ¼
TP þ FN
2  precision  recall
F−score ¼
precision þ recall

On the other hand, the accuracy value is considered as a global metric that is calculated by
dividing the total correct classifications by the total number of samples in the test set.
Therefore, an identical accuracy value is obtained for all the classes. The mathematical
expression for accuracy is defined as:
 Multimedia Tools and Applications

Table 3 The generic confusion matrix

Predicted Label

No DR Mild Moderate Severe Proliferative

Actual Label No DR c11 c12 c13 c14 c15

Mild c21 c22 c23 c24 c25
Moderate c31 c32 c33 c34 c35
Severe c41 c42 c43 c44 c45
Proliferative c51 c52 c53 c54 c55

∑5i¼1 cii
accuracy ¼
∑i¼1 ∑5j¼1 cij
5

4.2 Evaluation of the ensemble model

The training of the models and the classification of the unseen test samples are all performed at the
cloud side of the framework. To understand the effect of generating ensemble models on the
classification performance, five different models were trained using the DR dataset. Three of these
models are individual classifiers based on benchmark CNN architectures that are EfficientNetB7,
ResNet50, and VGG19. The other two models are ensemble models that combines the features
extracted by these three architectures where averaging and concatenation operations are used for

Table 4 Classification performance of the models

EfficientNetB7 ResNet50 VGG19 Average Concatenation

Ensemble Ensemble

a. No DR Train Acc. 0.91 0.90 0.87 0.89 0.97

Test Acc. 0.87 0.84 0.83 0.85 0.96
Precision 0.90 0.77 0.77 0.76 1.00
Recall 0.88 0.85 0.85 0.85 1.00
F-Score 0.89 0.81 0.81 0.79 1.00
b. Mild DR Train Acc. 0.91 0.90 0.87 0.89 0.97
Test Acc. 0.87 0.84 0.83 0.85 0.96
Precision 0.55 0.65 0.65 0.68 0.92
Recall 0.52 0.52 0.52 0.60 0.88
F-Score 0.54 0.58 0.58 0.64 0.90
c. Moderate DR Train Acc. 0.91 0.90 0.87 0.89 0.97
Test Acc. 0.87 0.84 0.83 0.85 0.96
Precision 0.85 0.73 0.73 0.74 1.00
Recall 0.99 0.99 0.99 0.98 0.99
F-Score 0.80 0.88 0.84 0.85 0.99
d. Severe DR Train Acc. 0.91 0.90 0.87 0.89 0.97
Test Acc. 0.87 0.84 0.83 0.85 0.96
Precision 0.80 0.73 0.73 0.73 0.98
Recall 0.74 0.74 0.76 0.76 1.00
F-Score 0.77 0.72 0.72 0.74 0.99
e. Proliferative Train Acc. 0.91 0.90 0.87 0.89 0.97
DR Test Acc. 0.87 0.84 0.83 0.85 0.96
Precision 0.79 0.72 0.75 0.73 0.87
Recall 0.75 0.72 0.71 0.66 0.90
F-Score 0.77 0.74 0.74 0.70 0.88
Multimedia Tools and Applications

Table 5 The confusion matrix for the predictions made by the ensemble model

Predicted Label

No DR Mild Moderate Severe Proliferative

Actual Label No DR 212 0 0 0 0

Mild 0 171 0 0 23
Moderate 0 1 131 0 0
Severe 0 0 0 108 0
Proliferative 0 14 0 2 150

the ensemble process. For the training of these models, no transfer learning was applied. In other
words, all layers of the ensemble model were trained from scratch with random initial weights. The
input image dimensions for the models were set as 224 × 224 and Adam optimizer [24] with a
learning rate of 0.001 was used for updating the model parameters in every training step. An online
data augmentation scheme was applied on the training set during the training of the model. This
augmentation involved the operations of rotation, shifting, flipping, zero components analysis
(ZCA) whitening, and mean normalization. The training and validation sets were divided into 64
batches and an early stopping with a threshold value of 4 epochs was employed. All of the models
were adjusted to have 128 hidden units at the dense layer after the concatenation process. Train and
test steps are performed with these settings and four performance metrics, namely, accuracy,
precision, recall, and f-score are calculated (Table 4) after generating a multi-class confusion matrix
for each model. These performance metrics need to be calculated by referencing to each class
separately because in such multi-class problems, the definitions of the terms true positive, false
positive, true negative, and false negative change for each reference class.
According to the results, concatenation ensemble model outperforms all of the other four
models which is an indicator of the suitability of the model for this problem. To check the
presence of an overfitting of the model, learning curves for training and validation set
accuracies are visually observed against a large gap between these accuracies.

Fig. 7 The learning curve for the ensemble model. Horizontal axis denote the epoch number and the vertical axis
denote the accuracy
 Multimedia Tools and Applications

ResNet50

No DR
VGG16

EfficientNetB7

(a)

ResNet50
Mild

VGG16

EfficientNetB7

(b)

ResNet50
Moderate

VGG16

EfficientNetB7

(c)
Figure 8 Feature maps extracted at different layers of the networks for the images belonging to five severity
levels: (a) No DR, (b) Mild DR, (c) Moderate DR, (d) Severe DR, (e) Proliferative DR

When the test accuracy of the average ensemble is compared with the individual models, it is
higher than ResNet50 and VGG19 and lower than EfficientNetB7. This means that averaging the
model outputs may not improve the results all the time. It can also be understood from the results that
the images of the Mild DR class are most difficult to classify since the lowest precision, recall and f-
score values are obtained for this class with four of the models. The other class with relatively lower
performance is Proliferative DR. Therefore, it may be concluded that the models, in general, cannot
distinguish the samples belonging to these two classes. This can be verified by the detailed results in
Table 5, where the confusion matrix for the test set predictions of the proposed ensemble model is
provided. Furthermore, the corresponding learning curve for the training process of the same model
is given Fig. 7. The learning curve shows the improvement in the training and validation accuracies
for different epochs and observation of this curve helps ensuring a successful training process.
Since the CNN architectures have several convolutional blocks at their layers, they extract
various levels of features belonging to different levels of details in the image. The initial layers
act as usual edge detectors, hence their output typically contains more visual information. At
Multimedia Tools and Applications

ResNet50

Severe
VGG16

EfficientNetB7

(d)

ResNet50
Proliferative

VGG16

EfficientNetB7

(e)

Fig. 8 continued.

the deeper layers of the network, the outputs of the convolution operations become more
abstract, where an image resembling the original image is generated. However, the complex
features belonging to the images of a particular class are detected at these deep layers.
Therefore, it is an expected situation to observe some empty or very smooth images in these
layers as these features may not be involved in all of the images. Some of the feature maps
generated at different layers of processing are illustrated in Fig. 8.

4.3 Effects of edge computing

The benefits of performing the preprocessing steps at the edge device have been evaluated by
calculating the total amount of data transferred to the cloud server and the response time from
the server. It is clear that the preprocessing steps, especially the image resize operation, make a
considerable reduction in the file size of the image. Therefore, these steps are expected to
contribute to the response time from the server because the total transmission time of a low-
sized image will be smaller.
The steps followed for quantification of these effects involve calculation of the average file
size of the images in the entire test set and the average response time from the cloud server to
make prediction on all of the images in the test set. This calculation process is repeated for two
different cases, (i) when the preprocessing is performed at the cloud side and (ii) when the
preprocessing is performed at the edge device. These cases are illustrated in Fig. 9.
The response time (TR) is defined by the total elapsed time between sending the image from
the edge device and the receipt of the class label by the edge device. In other words, TR is the
sum of the total transmission time (TT), the processing time at the cloud (TC), and the time
spent for preprocessing the image at the edge device (TE).
 Multimedia Tools and Applications

Fig. 9 Steps followed for calculating the edge computing performance. (a) Preprocessing performed at the cloud
side (no edge computing) and (b) preprocessing performed by the edge device

TR ¼ TT þ TC þ TE

It is possible to determine TC and TE by checking the total execution duration of the

calculations on the cloud side and edge device, respectively. Therefore, TT can be calculated
by simply subtracting TC + TE from TR. Having the knowledge of TT allows for making
conclusions about the occupied bandwidth. Note that, TE = 0 for the first case where the
preprocessing is performed at the cloud side.
The mean and standard deviation of the file size of the raw images in the test set are
calculated as 3502.73 Kb and 795.17 Kb, respectively. After the preprocessing is made on the
edge device, these values are reduced to 518.22 Kb and 72.81 Kb. In addition, the mean and
standard deviation for TR are calculated for both cases. When the preprocessing is performed at
the cloud side, they are measured to be 19.32 seconds and 3.83 seconds. On the other hand, the
contribution of the edge computing is clearly shown by the mean and standard deviation of the
TR values obtained as 13.18 seconds and 1.85 seconds. Example results on some of the test
Multimedia Tools and Applications

Table 6 Calculated results for the first case (preprocessing performed at the cloud side)

Image no File size (KB) Image Width (Pixels) Image Height (Pixels) TT (sec) TC +TE (sec) TR (sec)

1 4000 2588 1958 8.00 13.70 21.70

2 3405 1050 1050 6.81 11.50 18.31
3 2200 3216 2136 4.40 8.30 12.70
4 2030 2048 1536 4.06 8.10 12.16
5 4132 3388 2588 8.26 14.01 22.27
6 4602 1050 1050 9.20 14.89 24.09

images are provided in Tables 6 and 7 where the “File size” column denote the size of the
image file transmitted to the cloud.
As can be seen from the tables that both the total size of the transmitted images as well as
the total transmission time, TT, are reduced significantly when edge computing is performed.
Even though there is no considerable change in the total processing time (TC + TE), the
improvement in TT allows for a faster response from the cloud server. Besides, transmission of
smaller amount of data through the internet contributes to efficient utilization of the bandwidth.

4.4 Comparison with the existing studies

In this section, performances of the other studies about DR severity level classification are
presented and compared with those obtained with the proposed concatenation ensemble model
(Table 8). There are different datasets of fundus images generated for DR classification and
used in the related literature. The results obtained in this work are compared with not only the
studies utilizing the same dataset, APTOS2019, but also the studies using some other datasets.
The highest accuracy achieved with APTOS2019 dataset is reported by Khalifa et al. [22],
where the data augmentation is applied to both train and test sets and the total amount of images
were increased four times. However, data augmentation is typically applied to decrease the risk
of overfitting, increase the number of images for training, or eliminate the class imbalance
problem by generating new samples for the minority class only [40]. In that work, performances
of six different benchmark CNN architectures were analyzed and AlexNet was shown to
outperform the others. Therefore, only the results associated with AlexNet are presented in
Table 8. There have been other efforts for DR severity level detection using the same dataset.
These include developing a sequential CNN architecture [41], utilization of Inception-v3 model
directly [30], modification of DenseNet121 [10] and Xception [21] architectures, and generat-
ing an ensemble model by means of averaging [51]. Among these studies, the highest
classification accuracy, 96.51%, was achieved by modified DenseNet121 [10]. This score is
slightly higher than the accuracy of the proposed concatenation ensemble model.

Table 7 Calculated results for the second case (preprocessing performed at the edge device)

Image no File size (KB) Image Width (Pixels) Image Height (Pixels) TT (sec) TC +TE (sec) TR (sec)

1 640 224 224 1.28 12.33 13.61

2 520 224 224 1.04 10.64 11.68
3 404 224 224 0.81 7.35 8.16
4 410 224 224 0.82 7.57 8.39
5 553 224 224 1.11 11.71 12.82
6 598 224 224 1.20 11.20 12.40
Table 8 Reported performance values of the other related studies

Study Methodology Image dataset Accuracy Precision Recall F-Score Main Drawbacks
(year) (%) (%) (%) (%)

[22] (2019) AlexNet, ResNet18, SqueezeNet, GoogleNet, APTOS 2019 97.9 96.23 95.42 95.82 Data augmentation applied to samples from all
VGG16, VGG19 classes
[35] (2016) Sequential CNN Kaggle 75 – 30 – Poor results and limited discussion of the results
[41] (2019) Sequential CNN APTOS 2019, DRIVE, 94 87.2 76.6 81.557 Limited reasoning of the model structure and
and STARE discussions
[53] (2018) Inception V3 with transfer learning Kaggle 63.23 – – – Small number of test images
[10] (2020) Modified DenseNet121 APTOS 2019 96.51 86 87 86 No class-specific augmentation. Experiments
with one single model type
[21] (2019) Modified Xception APTOS 2019 83.09 – 88.24 – No data augmentation
[30] (2020) Inception-V3 APTOS 2019 88.5 87.293 100 93.215 No detail about the processing on smartphone
[51] (2020) Ensemble of EfficientNet-B4, APTOS 2019 92.9 – 99 – Limited details about the model structure. High
EfficientNet-B5, and SE-ResNeXt50 computational cost
[47] (2018) Ensemble bagging with decision trees Self generated 97.08 96.15 90.9 93.45 Hand crafted features
[15] (2022) Image enhancement + CNN with residual MESSIOR-2, UWF, 92 85 99 – Limited number of test samples
blocks APTOS 2019
[33] (2021) DenseNet-169 APTOS 2019 90 – – – Limited performance metrics reported
[42] (2021) Hinge attention network APTOS 2019 85.54 85.91 85.54 85.03 No class-specific augmentation. High dimen-
sionality in feature space
[32] (2022) Few-shot classification framework APTOS 2019 98.18 96.47 97.41 96.93 Performance compared with only one study
[9] (2022) GoogleNet and ResNet-18 with transfer APTOS 2019 89.29 89.40 89.30 89.30 High dimensionality in feature space. Small
learning number of classes
Proposed Concatenation ensemble APTOS 2019 96 100 100 100 High computational cost during prediction
method
Multimedia Tools and Applications
Multimedia Tools and Applications

In general, the studies using other datasets cannot achieve higher accuracy, as well. Only the
method based on ensemble bagging of decision tree classifiers has an accuracy score of 97.08%,
which is also higher than the accuracy of the proposed method [47]. However, in terms of
precision, recall, and f-score, none of the proposed methods can achieve 100% performance.
The positive effects of ensemble methods on the overall performance of the classifiers were
shown in [47, 51]. This idea is supported by the results of the classification experiments
performed within the scope of this study. Therefore, this information may constitute the
answer for the first research question given in Section 1.
As for the second research question, there are studies found performing mobile edge
computing for DR severity detection. Hence, this feature of the proposed framework is the
novel contribution of this study. On the other hand, a prototype smartphone app was developed
by Majumder et al. for detection of five stages of DR [30]. In that work, the trained deep
learning models are used for real-time classification of the images without the requirement of
the internet connection. In other words, the computations required for class label assignment are
performed on the mobile device. This system has some drawbacks such as using the light
version of the deep learning model which eventually results in limited performance. In addition,
usage of deeper models requiring more computations may not be possible on a mobile device
with constrained processing capabilities and battery power. Therefore, edge computing featur-
ing an internet connection may be useful and its advantages were presented in Section 4.3.

5 Conclusion

Deep learning and edge computing are two of the emerging technologies together with
the concept of IoT. In this work, a framework utilizing these technologies for detecting
severity level of DR disease is proposed. It mainly consists of two parts that are edge
device and cloud computing. The fundus images for diagnosing DR are used as input to
the edge device where the preprocessing steps on the images are performed. After the
edge processing, the images are transmitted to the cloud server over the internet and their
severity levels are detected using deep learning methods. For this detection procedure, a
concatenation ensemble of three benchmark CNN architectures (EfficientNetB7,
ResNet50, and VGG19) is proposed. The performance of the proposed model is com-
pared with the performances of the individual benchmark models and an average
ensemble model. It was shown that the proposed model outperforms the other models
by correctly classifying the highest number of DR images. Furthermore, the advantages
of edge computing steps are underlined by calculating the total amount of transmitted
data and the response time from the cloud server. These calculations are performed for
two cases (i.e. with and without edge computing) and it was seen that performing the
preprocessing steps on the edge device reduces both the transmitted data amount and the
response time. As a result, the proposed framework contributes to classification accuracy
as well as the speed and bandwidth usage.
Since the majority of the misclassifications are encountered in only two out of five classes,
focusing on correct detection of the samples from these classes is planned as one of the future
works. To overcome this problem, increasing the total number of images in the dataset can be a
solution. Also, other benchmark models with deeper architectures may be useful in distinguish-
ing the features in these images. Finally, development of more user-friendly GUI enabling
detection of more diseases is another future work for a practical usage of the framework.
 Multimedia Tools and Applications

Data availability The datasets analysed during the current study are available in the Kaggle repository, https://
[Link]/c/aptos2019-blindness-detection/data

Declarations

Conflict of interest None.

References

1. Al-Karawi A, Avşar E (2022) Stacked cross validation with deep features: a hybrid method for skin Cancer
detection. Tehnički glasnik 16(1):33–39
2. Allyn W., [Link] Accessed on April 20, 2021.
3. Antal B, Hajdu A (2012) An ensemble-based system for microaneurysm detection and diabetic retinopathy
grading. IEEE Trans Biomed Eng 59(6):1720–1726
4. Antal B, Hajdu A (2014) An ensemble-based system for automatic screening of diabetic retinopathy.
Knowl-Based Syst 60:20–27
5. APTOS, [Link] Accessed on April 20, 2021.
6. Avşar E (2021) Effects of Image Preprocessing on the Performance of Convolutional Neural Networks for
Pneumonia Detection. In: 2021 International Conference on INnovations in Intelligent SysTems and
Applications (INISTA)
7. Avşar E, Salçin K (2019) Detection and classification of brain tumours from MRI images using faster R-
CNN. Tehnički glasnik 13(4):337–342
8. Bhandary A, Prabhu GA, Rajinikanth V, Thanaraj KP, Satapathy SC, Robbins DE, Shasky C, Zhang YD,
Tavares JMRS, Raja NSM (2020) Deep-learning framework to detect lung abnormality – a study with chest
X-ray and lung CT scan images. Pattern Recogn Lett 129:271–278
9. Butt MM, et al (2022) Diabetic Retinopathy Detection from Fundus Images of the Eye Using Hybrid Deep
Learning Features. Diagnostics. 12(7).
10. Chaturvedi SS, et al (2020) Automated diabetic retinopathy grading using deep convolutional neural
network. arXiv preprint arXiv:2004.06334
11. Chen M, Li W, Hao Y, Qian Y, Humar I (2018) Edge cognitive computing based smart healthcare system.
Futur Gener Comput Syst 86:403–411
12. Dayana AM, Emmanuel WRS (2022) An enhanced swarm optimization-based deep neural network for
diabetic retinopathy classification in fundus images. Multimed Tools Appl 81(15):20611–20642
13. Decencière E, Zhang X, Cazuguel G, Lay B, Cochener B, Trone C, Gain P, Ordonez R, Massin P, Erginay
A, Charton B, Klein JC (2014) Feedback on a publicly distributed image database: the Messidor database.
Image Analy Stereol 33(3):231–234
14. Diabetic retinopathy screening: a short guide (2020) Increase effectiveness, maximize benefits and minimize
harm. Copenhagen: WHO Regional Office for Europe; Licence: CC BY-NC-SA 3.0 IGO
15. Fatima et al (2022) A unified technique for entropy enhancement based diabetic retinopathy detection using
hybrid neural network. Comput Biol Med 145:105424
16. Fu Y et al (2020) Deep learning in medical image registration: a review. Phys Med Biol 65(20):20TR01
17. Garibaldi-Beltrán JA, Vazquez-Briseno M (2012) Personal Mobile Health Systems for Supporting Patients
with Chronic Diseases. In: 2012 IEEE ninth electronics, Robotics and Automotive Mechanics Conference
18. Ghoneim A, Muhammad G, Amin SU, Gupta B (2018) Medical image forgery detection for smart
healthcare. IEEE Commun Mag 56(4):33–37
19. He K, et al. (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on
computer vision and pattern recognition
20. Karakaya M, Hacisoftaoglu RE (2020) Comparison of smartphone-based retinal imaging systems for
diabetic retinopathy detection using deep learning. Bmc Bioinform 21:259
21. Kassani SH et al (2019) Diabetic retinopathy classification using a modified xception architecture. In: 2019
IEEE international symposium on signal processing and information technology (ISSPIT). IEEE
22. Khalifa NEM, Loey M, Taha MHN, Mohamed HNET (2019) Deep transfer learning models for medical
diabetic retinopathy detection. Acta informatica medica : AIM : journal of the Society for Medical
Informatics of Bosnia & Herzegovina : casopis Drustva za medicinsku informatiku BiH 27(5):327–332
23. Khanamiri HN, Nakatsuka A, El-Annan J (2017) Smartphone fundus photography. Jove-J Visualized Exp 125
24. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Multimedia Tools and Applications

25. Lam C, Yi D, Guo M, Lindsey T (2018) Automated detection of diabetic retinopathy using deep learning.
AMIA Joint Summ Trans Sci Proc AMIA Joint Summ Trans Sci 2017:147–155
26. Liu Z, et al (2020) Deep learning based brain tumor segmentation: A survey. arXiv preprint arXiv:
2007.09479
27. Lv Z, et al (2010) iCare: a mobile health monitoring system for the elderly. In: 2010 IEEE/ACM Int'l
conference on green computing and Communications & Int'l conference on cyber, physical and social
computing. . IEEE
28. Maamari RN, Keenan JD, Fletcher DA, Margolis TP (2014) A mobile phone-based retinal camera for
portable wide field imaging. Br J Ophthalmol 98(4):438–441
29. Maier A, Syben C, Lasser T, Riess C (2019) A gentle introduction to deep learning in medical image
processing. Z Med Phys 29(2):86–101
30. Majumder S, et al (2020) A deep learning-based smartphone app for real-time detection of five stages of
diabetic retinopathy. in Real-Time Image Processing and Deep Learning 2020. International Society for
Optics and Photonics.
31. Mateen M et al (2019) Fundus image classification using VGG-19 architecture with PCA and SVD.
Symmetry 11(1):1
32. Murugappan M, Prakash NB, Jeya R, Mohanarathinam A, Hemalakshmi GR, Mahmud M (2022) A novel
few-shot classification framework for diabetic retinopathy detection and grading. Measurement 200:111485
33. Mushtaq G, Siddiqui F (2021) Detection of diabetic retinopathy using deep learning methodology. IOP
Conf Series: Materials Sci Eng 1070(1):012049
34. Pires R, Avila S, Wainer J, Valle E, Abramoff MD, Rocha A (2019) A data-driven approach to referable
diabetic retinopathy detection. Artif Intell Med 96:93–106
35. Pratt H, Coenen F, Broadbent DM, Harding SP, Zheng Y (2016) Convolutional neural networks for diabetic
retinopathy. Procedia Comput Sci 90:200–205
36. Qummar S, Khan FG, Shah S, Khan A, Shamshirband S, Rehman ZU, Ahmed Khan I, Jadoon W (2019) A
deep learning ensemble approach for diabetic retinopathy detection. IEEE Access 7:150530–150539
37. Qureshi I, Ma J, Abbas Q (2019) Recent development on detection methods for the diagnosis of diabetic
retinopathy. Symmetry 11(6):749
38. Qureshi I, Ma J, Abbas Q (2021) Diabetic retinopathy detection and stage classification in eye fundus
images using active deep learning. Multimed Tools Appl 80(8):11691–11721
39. Rajalakshmi R, Subashini R, Anjana RM, Mohan V (2018) Automated diabetic retinopathy detection in
smartphone-based fundus photography using artificial intelligence. Eye 32(6):1138–1144
40. Saini M, Susan S (2020) Deep transfer with minority data augmentation for imbalanced breast cancer
dataset. Appl Soft Comput 97:106759
41. Sajana T et al (2019) Classifying diabetic retinopathy using deep learning architecture. Int J Innov Technol
Explor Eng 8(6):1273–1277
42. Shaik NS, Cherukuri TK (2022) Hinge attention network: a joint model for diabetic retinopathy severity
grading. Appl Intell 52:15105–15121
43. Shanthi T, Sabeenian RS (2019) Modified Alexnet architecture for classification of diabetic retinopathy
images. Comput Electr Eng 76:56–64
44. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition.
arXiv preprint arXiv:1409.1556
45. Sodhro AH, Luo Z, Sangaiah AK, Baik SW (2019) Mobile edge computing based QoS optimization in
medical healthcare applications. Int J Inf Manag 45:308–318
46. Soleymanifard M, Hamghalam M (2022) Multi-stage glioma segmentation for tumour grade classification
based on multiscale fuzzy C-means. Multimed Tools Appl 81(6):8451–8470
47. Sreng S, Maneerat N, Hamamoto K, Panjaphongse R (2018) Automated diabetic retinopathy screening
system using hybrid simulated annealing and ensemble bagging classifier. Appl Sci 8(7):1198
48. Subha T, et al (2021) Analysis of Diabetic Retinopathy and Its Causes Using Fuzzy C-Means Model. In:
2021 4th International Conference on Computing and Communications Technologies (ICCCT)
49. Tan M, Le Q (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In interna-
tional conference on machine learning. PMLR
50. Taufiqurrahman S, et al (2020) Diabetic Retinopathy Classification Using A Hybrid and Efficient
MobileNetV2-SVM Model. in 2020 IEEE REGION 10 CONFERENCE (TENCON)
51. Tymchenko B, Marchenko P, Spodarets D (2020) Deep learning approach to diabetic retinopathy detection.
arXiv preprint arXiv:2003.02261
52. Wan S, Liang Y, Zhang Y (2018) Deep convolutional neural networks for diabetic retinopathy detection by
image classification. Comput Electr Eng 72:274–282
53. Wang X, et al (2018) Diabetic retinopathy stage classification using convolutional neural networks. In 2018
IEEE international conference on information reuse and integration (IRI). IEEE.
 Multimedia Tools and Applications

54. Web page, [Link] Accessed 20 Apr 2021

55. Web page, [Link] Accessed 20 Apr 2021
56. World Health Organization, [Link]
impairment. Accessed 20 Apr 2021
57. Zheng Y, He M, Congdon N (2012) The worldwide epidemic of diabetic retinopathy. Indian J Ophthalmol
60(5):428–431

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps
and institutional affiliations.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a
publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted
manuscript version of this article is solely governed by the terms of such publishing agreement and
applicable law.