Deep Learning Approaches for Image Classification

Yanzheng Yu†
College of Engineering and Computer Science, Australian National University, Canberra, 2601
[Link]@[Link]

ABSTRACT tillage. In these traditional machine learning methods, it will firstly

Deep learning models can achieve a higher accuracy result com- select and construct the corresponding features from samples by
pared with traditional machine learning algorithm. It is widely manual. Then, the machine learning process will work to fit these
useful in different areas, especially in images classification area. features to solve the classification tasks. However, these methods
In recent years, because of the improvement of hardware and the have two main problems. One is that all the feature engineering is
discovery of new deep learning network structures, the accuracy done by manually. This requires operator to have a corresponding
and reliability of deep learning model used in image classification and enough background knowledge. Besides this, when facing a big
have been greatly improved. However, in the field of images classi- data sample, the cost of selecting features by manually will increase.
fication with deep learning technology, the reviews of the recent Secondly, the features selected manually are always the shallow
researches are lack. This paper will make a review about the recent features based on statistical methods. Some deeply and contained
researches of images classification based on deep learning. It in- more valid information features cannot be identified by manually
cludes the latest studies to improve the performance about deep selecting. This reduces the accuracy of machine learning methods.
learning. Additionally, the potential problems and challenges on Deep learning methods solve above problems.
deep learning technology and the possible future improvement and Deep learning models are composed of multiple hidden layers.
research direction are analyzed and discussed in the review. Each layer in model has activation function. It is possible for deep
learning model to learn a high-level abstract information from the
CCS CONCEPTS samples automatically according to the hidden layers. In this way,
deep learning methods have a significant advantage on feature
• Computing methodologies → Artificial intelligence; Informa-
processing rather than manual processing. Therefore, in recent
tion extraction.
years, there have been many papers topic about the exploration
KEYWORDS and improvement of deep learning. In image classification task,
there are also many applications of deep learning.
Deep learning, Convolution neural network, Image classification However, comparing with the recently research number and
ACM Reference Format: progress, the reviews about the deep learning methods used im-
Yanzheng Yu†. 2022. Deep Learning Approaches for Image Classification. age classification tasks are lack. As a result, it is difficult for the
In 2022 6th International Conference on Electronic Information Technology beginners in deep learning area to obtain the latest progress. In
and Computer Engineering (EITCE 2022), October 21–23, 2022, Xiamen, China. addition, review is a phased summary of existing research and re-
ACM, New York, NY, USA, 5 pages. [Link]
sults. It is useful to avoid repeated researches and to point out the
direction of next research and development. Therefore, this review
1 INTRODUCTION summarizes the recently development of deep learning methods in
In recent years, deep learning technology using in images classifica- image classification tasks. Firstly, some commonly used data sets for
tion area has a significant improvement. The deep learning model, image classification are introduced. It also points out how to judge
AlexNet, introduced by the research [1], described that the deep whether a dataset is suitable for deep learning process. Secondly,
learning method can have a very good classification performance. the review will give a detailed overview of some recent advances
In automatic driving, monitoring system, attitude detection and in the image classification field base on deep learning model. Fi-
other fields, image classification based on deep learning has also nally, some remaining challenges are listed, and some directions
achieved a high performance. about possible future research and solutions in deep learning image
In the early researches, machine learning methods are used for classification area are shown.
image classification tasks. According to the study [2], SVM method
can be used in multilabel image classification. Yang et al. [3] used
decision tree for image classification using hyperspectral images 2 BACKGROUND
of plots of different tillage and received a 0.89 accuracy on classify This review focuses on the recent research about doing image
classification with deep learning models. Image classification is one
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed of the applications using machine learning technology. The image
for profit or commercial advantage and that copies bear this notice and the full citation classification task is that giving an input image and the trained
on the first page. Copyrights for components of this work owned by others than ACM machine learning model can predict it which class it belongs to.
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a A good machine learning model have high accuracy, and some
fee. Request permissions from permissions@[Link]. models’ predictions are better than human predictions. There are
EITCE 2022, October 21–23, 2022, Xiamen, China many image classification applications including medical image
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9714-8/22/10. . . $15.00 recognition, monitoring system, automatic driving and other areas
[Link] involving computer vision and classification. A large number of

1494
EITCE 2022, October 21–23, 2022, Xiamen, China Yanzheng Yu

Table 1: Confusion matrix

Positive Prediction value Negative Prediction value

Positive Truth value True positive - TP False negative - FN
Negative Truth value False positive - FP True negative - TN

researches have shown that deep learning algorithms can be in a

high performance on image classification task.

2.1 Dataset
Deep learning algorithms cannot be trained without data sets. By
feeding a large number of samples from dataset into the training
model, the model can learn and fit the dataset. Finally, the trained
model can predict new and unknown samples. It is a high cost for
researchers to collect and label data from scratch. Therefore, it is
necessary to be aware of the recent publicly datasets.
2.1.1 Patch_camelyon. Patch_camelyon dataset [4] is an open
source dataset of medical images. It contains more than three hun-
dred thousand numbers of color images with 96x96 pixel about the
histopathologic scans of lymph node sections. This dataset can be Figure 1: An example of ROC curve
used as the input to train a model to predict whether given Iymph
node sections image contains metastatic tissue.
2.1.2 SI-SCORE.. The SI-Score dataset [5] is a multi-classification The evaluation about precision can know the rate about positive
image dataset. In addition to different objects in different categories, samples in the positive prediction given by model.
the dataset also contains different states of the same object or dif- 3. Recall:
ferent backgrounds with same object. For example, some samples 𝑇𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 =
are generated by fixing the background and then scaling, rotating 𝑇𝑃 + 𝐹𝑁
and flipping one object. Additionally, some objects are placed in Recall evaluation can give a proportion which describes the
different backgrounds. Based on these properties, the SI-Score can sample correct predicted to be positive within all positive samples.
be worked not only to train deep learning models, but also used to
2.2.2 ROC Curve and AUC.. In dichotomous classification prob-
test the robustness about the trained models.
lems, the model will have a threshold value. The predicted results
2.1.3 Quick Draw Dataset. Quick Draw Dataset [6] contains 50 greater than this threshold value will be classify to true, less than
million bitmaps drew by players. There are 345 categories. The the threshold value will be predicted to be false. According to the
image is in 28x28 pixels. This dataset can be used to train the model confusion matrix, TPR, which represents True Positive Rate, can
to identify painting. be defined by the formula:
𝑇𝑃
2.2 Evaluation index 𝑇 𝑃𝑅 =
𝑇𝑃 + 𝐹𝑁
2.2.1 Accuracy, Precision and Recall. Confusion matrix is a matrix
FPR, False Positive Rate, can be defined as:
used to display the relationship between samples truth value and
predicted value, see Table 1. 𝐹𝑃
𝐹 𝑃𝑅 =
According to the table 1, different model evaluation indexes can 𝐹𝑃 + 𝑇 𝑁
be defined as follows: By defining FPR value as the X-axis and TPR as the Y-axis, one
1. Accuracy: ROC space can be obtained. The ROC curve of the model can be
𝑇𝑃 +𝑇𝑁 draw by plotting the (FPR, TPR) coordinates under each threshold
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = values. Figure 1 displays a ROC curve.
𝑇𝑃 + 𝑇 𝑁 + 𝐹𝑃 + 𝐹𝑁
According to ROC curve, the relationship between TPR and FPR
By observing the formula of accuracy, it can be concluded that
of the model with different thresholds can be obtained intuitively.
the accuracy can make an evaluation of the global accuracy about
A high-performance model is required to keep a low FPR and a
the model. However, this evaluation is limited and it is not possible
high TPR. The AUC value can be collected by computing the area
to obtain the specific evaluation when the model tries to only make
under the ROC curve. If model’s AUC value is high, it indicates
predictions on positive or negative samples.
that a good classification performance can be obtained by setting
2. Precision:
𝑇𝑃 an appropriate threshold value on this model. When AUC is equal
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = to 1, that means this model is a perfect classifier.
𝑇𝑃 + 𝐹𝑃

1495
Deep Learning Approaches for Image Classification EITCE 2022, October 21–23, 2022, Xiamen, China

Figure 2: The process of MICLe

3 RECENT RESEARCH 3.2 Disease diagnosis with Deep CNN of image

3.1 Image Classification with on classification
Self-Supervised field This paper [8] is focus on how to use CNN neural network model
on chest X-ray dataset to make a classification on pneumonia. The
According to the research [7], they studied the effectiveness of the
paper list some methods used previous in medical image classifica-
pre-training process base on the self-learning method in medical
tion area and find that DNN, especially CNN, can make decision
image classification area. In the paper, two experiments had been
with a high accuracy like humans.
designed. One is do classification on dermatology with digital image.
Authors pointed that a transfer learning model, InceptionV3
The second is to do classification on a multi-label chest X-ray image.
model, can hold a well performance on a small X-rays dataset about
The research introduces a novel method, called MICLe, as one
pneumonia image. Also, another research’s experiments prove a
of the self-supervise components. In this method, the input can be
new model, structure-capsule network, works well on small image
the images about one patient’s pathology with different angle of
dataset. By the limitation of the research on InceptionV3 model
view. The whole self-supervised process in this research is that do
and structure-capsule network, this paper designed some new ex-
self-supervised learning with the input of unlabeled images first.
periments, which are: Compare performance with three different
Secondly, unlabeled medical images as the input, do an additional
ML model: SVM classifier with ORB, VGG16 model used on trans-
self-supervised learning. In this step, if one medical condition has
fer learning and the model of InceptionV3, and structure-capsule
multiple images, MICLe method will be used. Finally, use labeled
network. How the influence on small X-rays dataset learning pro-
medical images to make fine-tuning.
cess with the avoid over-fitting methods: make augmentation on
The specific process about MICLe method is shown as figure 2.
dataset, adjust the complexity of network, and make fine-tuned on
In this algorithm, if only one image for the medical condition, it
convolutional layer.
will use data augmentation in standard way to obtain two views of
VGG16 model is a deep layer with 16 weight layers [9]. Structure-
this image. If it exists multiple images about the medical condition,
capsule network is a model which contains multiple special capsules
an image pair can be created directly.
made by a group of neurons [10]. The model of InceptionV3 is
The conclusions of the research are detailed as bellow:
shown as Table 2 [11].
1. The performance is better if doing a pre-training with a self-
supervise model on unlabeled medical images rather than only
doing pre-training with ImageNet.
2. The novel method MICLe improve the performance of the 3.3 Multi-scale relational network to improve
self-supervise model. image classification
3. A more detail information according to the experiments: the
top-1 accuracy on dermatology classification has 6.7% improvement This paper [12] is focus on meta-learning area. Meta-learning is
and the mean AUC on chest X-ray classification has 1% improve- the tech to point machine learn to learn. And the authors done a
ment, compared with only do pre-training with ImageNet dataset. research, to find how to learn from a small image dataset quickly
4. Big self-supervised models have a higher robustness and gen- by using meta-learning. This research provides a new algorithm,
eralization which is meta-learning with model-independent property. And base
on this algorithm, a multi-scale relational network was come up.
In this network, according to ideal about meta-SGD, the learning
process will combine learning rate and model parameters together.

1496
EITCE 2022, October 21–23, 2022, Xiamen, China Yanzheng Yu

Figure 3: Three different inception modules. Left is a, middle one is b and the right one is c.

Table 2: InceptionV3 structure dataset but not ideal on a bigger one, minilmageNet dataset. The sec-
ond is that, in the network, partial gradient information is dropped.
type filter size/stride or remarks This may lose some information.
conv layer 3 × 3/2
3 × 3/1 3.4 CNN skills on image classification
conv padded layer 3 × 3/1 This paper [13] has done a research about a collection of changes
pool layer 3 × 3/2 about CNN, which improve models’ accuracy but no computing
conv layer 3 × 3/1 complex increasing, on image classification area. These changes are
3 × 3/2 summarized from previous papers by other authors. The changes
3 × 3/1 work places include: model structure, data pre-process, loss function
3 x Inception modules see figure 3, a and learning rate.
5 x Inception modules see figure 3, b In this paper, authors used ablation studying method to find the
2 x Inception modules see figure 3, c influence about these changes and proved that all these changes
pool layer 8×8 can individually improve the learning performance.
linear logits Authors also proved that all these changes work together can
softmax classifier significantly improve the performance about CNN models.
This paper also shows that these changes can be worked on other
network and dataset. According to experiments, the changes give
an improvement on transfer learning on object detection area and
semantic segmentation area.

In addition, MAML algorithm will be used to optimize the mod-

els’ parameters. During meta-validation and meta-testing, inner 4 CHALLENGES
gradient iteration will not be used. Although image classification task based on deep learning models
The paper ued MAML and MetaSGD experiments to make a has achieved good performance, there are still some challenges.
comparation. In the experiments, the model structure is 4 CNN These challenges may be some of the limitations waiting to be
layer modules and followed by a full connect layer. Each CNN layer resolved, or some points waiting for further experimental to verify.
module contains 64 numbers of 1 size filters, one batch normalize According to the research [7], they pointed that although self-
layer, one layer of modified linear element and a 2-size max pool supervised learning has been shown can perform a high accuracy
layer. The full connect layer has nodes number 64. Loss function is result in the medical image classification field, the limitations of
used cross-entropy. self-supervised learning in this area are still unknown. Therefore, it
According to the experiments, multi-scale relational network is required to study the limitation by running self-supervise learn-
can give model a better performance on generalizability and im- ing with a huge unlabeled dataset. Additionally, another research
prove the accuracy on benchmark set. Compared with MAML, this direction is to study how to make a transfer from one image type
network avoids the works on fine-tuning. and task to another.
The paper also points out some limitations of the model. One In these researches, deep learning models are all trained based on
is that, the network performs well on a smaller dataset Omniglot given datasets. However, in real world, the input data distribution

1497
Deep Learning Approaches for Image Classification EITCE 2022, October 21–23, 2022, Xiamen, China

may be huge different from the training data distribution. It may systems, 25.
even exist data which are not contained in the training dataset, [2] Li, X., Wang, L., & Sung, E. 2004, October. Multilabel SVM active learning for
image classification. In 2004 International Conference on Image Processing, 2004.
and have completely different feature distributions. For example, ICIP’04. (Vol. 4, pp. 2207-2210). IEEE.
in the research [7] of medical image classification, the models are [3] Yang, C. C., Prasher, S. O., Enright, P., Madramootoo, C., Burgess, M., Goel, P. K.,
& Callum, I. 2003. Application of decision tree technology for image classification
trained with the images of medical conditions which are occurred using remote sensing data. Agricultural Systems, 76(3), 1101-1117.
and recorded. However, when a new medical condition is happened, [4] Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., & Welling, M. 2018, September.
the trained deep learning models may cannot recognize and give a Rotation equivariant CNNs for digital pathology. In International Conference
on Medical image computing and computer-assisted intervention (pp. 210-218).
wrong prediction. This may take a serious influence. How to im- Springer, Cham.
prove the robustness and generalization of deep learning models for [5] Djolonga, J., Yung, J., Tschannen, M., Romijnders, R., Beyer, L., Kolesnikov, A.,
data with different feature distributions is an important challenge ... & Lucic, M. 2021. On robustness and transferability of convolutional neural
networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and
in deep learning researches. Pattern Recognition (pp. 16458-16468).
[6] Ha, D., & Eck, D. 2017. A neural representation of sketch drawings. arXiv preprint
5 CONCLUSION arXiv:1704.03477.
[7] Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., ... & Norouzi,
Deep learning has a good performance in the image classification M. 2021. Big self-supervised models advance medical image classification. In
Proceedings of the IEEE/CVF International Conference on Computer Vision (pp.
tasks. However, as the model structure of deep learning becomes 3478-3488).
more complex and deeper, the requirement for large amount of data [8] Yadav, S. S., & Jadhav, S. M. 2019. Deep convolutional neural network based
for training process is significantly increasing. Some researches medical image classification for disease diagnosis. Journal of Big Data, 6(1), 1-18.
[9] Simonyan, K., & Zisserman, A. 2014. Very deep convolutional networks for
mentioned in this review can solve a part of requirement about big large-scale image recognition. arXiv preprint arXiv:1409.1556.
data in deep learning models. However, this approach still has some [10] Sabour, S., Frosst, N., & Hinton, G. E. 2017. Dynamic routing between capsules.
shortcomings. In the future, the main research direction could focus Advances in neural information processing systems, 30.
[11] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. 2016. Rethinking the
on the requirements of big data in deep learning. The challenges inception architecture for computer vision. In Proceedings of the IEEE conference
faced by the deep learning technology listed in this review can also on computer vision and pattern recognition (pp. 2818-2826).
[12] Zheng, W., Liu, X., & Yin, L. 2021. Research on image classification method based
be the future research directions. on improved multi-scale relational network. PeerJ Computer Science, 7, e613.
[13] He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., & Li, M. 2019. Bag of tricks for
REFERENCES image classification with convolutional neural networks. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 558-567).
[1] Krizhevsky, A., Sutskever, I., & Hinton, G. E. 2012. Imagenet classification with
deep convolutional neural networks. Advances in neural information processing

1498