Handwritten Digit Recognition Using Machine Learning
Handwritten Digit Recognition Using Machine Learning
1
NIRASE FATHIMA ABUBACKER, 2MAFAS RAHEEM
1
Assoc. Professor, School of Computing, Asia Pacific University of Technology & Innovation, Malaysia.
2
Senior Lecturer, School of Computing, Asia Pacific University of Technology & Innovation, Malaysia.
E-mail: [Link]@[Link], 2raheem@[Link]
ABSTRACT
The handwritten digit recognition system is a popular research topic, and much research has been done
throughout the years. The implementation of this system will be beneficial for many sectors in today’s world.
Various types of algorithms can be used to develop a solution for this system. However, the accuracy of the
results plays an important role in determining the best solution for the handwritten digit recognition system.
In this project, selected machine learning and deep learning algorithms were used to build models to find the
most suitable model with the best possible accuracy. According to the results, the CNN model performed
better than the other models with an accuracy of 99.25% and 0.99 for each Precision, Recall and F1 Score
compared to all the other models.
Keywords: Digit Recognition, Handwritten, Recognition Model; Machine Learning; Deep Learning
2172
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
There is much research that has been done previously accuracy of handwritten digit recognition. The two
but achieving 100% accuracy seems to be almost authors have used the MNIST dataset for a
impossible. The problem context behind this comprehensive data analysis on the topic. The results
challenge is that even 1% inaccuracy can lead to were tabulated by parameters of the confusion matrix
many wrong interpretations in recognizing the digits. and precision, Recall and F1 score were calculated
Therefore, this paper aims to study and analyse the using that table for all three models built using the
various methods present in building the models and algorithms. As for the observations, KNN relatively
developing the handwritten digit recognition system. had less accuracy compared to SVM and CNN.
Whereas CNN was crowned as the algorithm with
the highest accuracy of 99.4% for the training data
2. LITERATURE REVIEW and 98.4% for the test data. It was also discussed
that; the number of epochs has impacted the accuracy
The research about recognition started with of CNN where more the number of epochs the higher
optical character recognition (OCR) as physicist the accuracy achieved.
Emanuel Goldberg invented a machine that could
convert characters into telegraph code and in the In 2020, research on handwritten digit
1920’s he went a step further by creating the first recognition was done by Savita Ahlawat and Amit
electronic document retrieval system. Later, OCR Choudhary suggested that a combination of
technology proliferated, and businesses started to convolutional neural network (CNN) and support
rely on it to convert data from paper documents [2]. vector machine (SVM) achieved a higher accuracy
Since then, the research on recognition has become of 99.28% for the training data compared to SVM
in-depth and the understanding of algorithms has alone. The proposal stated that handwritten digit
become better. The most suitable research papers are recognition involves automatic feature generation
reviewed in this section as data analytics methods using CNN and the SVM is implemented in
have technologically improved a lot. These research predicting the output. Various parameters such as
papers cover the latest information and results that gamma and degree and decision function are altered
can be helpful for this project. and go through several stages of testing. Finally, the
maximum accuracy was achieved when the gamma
Reference [3] studied the effectiveness of is 0.1, degree 5 and the decision function with one-
some machine learning algorithms in handwritten on-one [5]. Implementation of hybrid algorithms is
digit recognition systems. The paper presented new for the handwritten digit recognition field and
Multilayer Perceptron (MLP), SVM, Naïve Bayes, still in the early stages.
Bayes Net, Random Forest, J48 and Random Tree as
the approaches used for offline handwritten digit The research was conducted on handwritten
recognition. Simulations were done based on digit recognition, particularly for baking systems
accuracy, time consumption and different errors. using the CNN method. The research is about using
Based on these parameters, MLP has been found to CNN to develop an automatic banking deposit
have the highest accuracy with 90.37% overall number recognition system for the cash deposit
compared to other machine learning algorithms. process at the bank counter. The authors stated that
However, this RESEARCH WAS DONE AS AN CNN is the best algorithm to implement for their
INITIAL ATTEMPT WITHOUT USING ANY proposed idea as it has the highest accuracy of all the
standard classification techniques. This research algorithms present [6]. Similarly, another research
focused only on machine learning algorithms and did was done using the CNN approach to analyze the
not attempt to analyse algorithms from other deep learning algorithm in proving the effectiveness
domains of data analytics. of for handwritten digit recognition system. As
shown in Table 1, CNN achieved its maximum
Reference [4] proposed an automatic accuracy of 99.87% in the MNIST dataset hence it is
handwritten digit recognition on document images proved that CNN is a great approach from deep
using SVM, Artificial Neural Network (ANN) and learning algorithm for handwritten digit recognition
CNN. It resulted that, CNN achieved the highest system by the authors [7].
accuracy with 71% performance compared to ANN
and SVM models. Feature extraction and classification are
machine learning technology's two primary
Unlike the other research works, the K- functions. The CNN design does away with the
Nearest Neighbour (KNN) algorithm was analysed necessity for a separate feature extraction method by
along with SVM, and CNN was analysed to study the
2173
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
combining classification and feature extraction understanding to identify crucial factors for optimal
techniques into a single model [12]. SVM [13] is a handwritten digit recognition. The MNIST dataset,
sophisticated classification method with strong consisting of 60,000 grayscale images of single
generalization capabilities that are based on the digits, underwent data understanding, cleaning, and
principles of structural risk reduction and statistical exploration. Noise reduction techniques were
learning theory through manual feature extraction. applied to enhance image quality. Selected
algorithms were trained and tested with appropriate
Table 1: Comparison of models parameters. Evaluation criteria included accuracy
Algorithms CNN MLP SVM percentage and precision score. The models'
deployment was emphasized for practical use, with
a focus on organizing and presenting knowledge
Accuracy 71.00 [4] 90.37 [3] 98.35 [5] gained for production purposes. The data analysis
(%) 99.87 [7] 39.00 [4] process involved collecting, modelling, and
97.83 [14] analyzing data for decision-making. Data
exploration examined the MNIST dataset's
Parameters GPU: Execution Gamma = structure, revealing 10 rows of digits (0-9) and 785
CPU = time = 0.1
columns in both training and test sets. Data cleaning
30:1 2:32 min
Degree =
confirmed the absence of missing values or outliers.
15000 Number 5 Normalization was performed to scale pixel values
iterations of epochs (0-255) to a range of 0 to 1 and reshape transformed
= 30 C = 1.0, pixel arrays into a (28,28,1) matrix for improved
Lr = 0.01, deep learning model implementation.
Penalty =
Decay = 12, 3.1 Data Visualization
1e-6,
max_iter
Momentu = 1000,
m = 0.9
tol = 1e-4
2174
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
validation data is used for cross-validation of the metric visualization. It is fitted with 15 epochs and
model’s accuracy and to see how well the model is a batch size of 112, completing training in 20
generalized for the data other than training data. seconds due to its simple architecture. The highest
training accuracy achieved is 97.55%, and
validation accuracy is 97.17%. Training loss
4. MODELLING & ANALYSIS gradually decreases, while validation loss varies
4.1 Convolutional Neural Network (CNN) across epochs.
A CNN model was constructed using the
LeNet-5 architecture. The model featured two 4.3 Support Vector Machine (SVM)
Conv2D layers with 32 and 64 filters, respectively. The SVM model-building process began
The kernel size was set to 5x5 for the first layer and with both linear and non-linear models using default
reduced to 3x3 for the second layer. Rectified linear hyperparameters for comparison. The linear SVM
unit (ReLU) activation, same padding, and model yielded an accuracy of approximately
MaxPool2D with a (2,2) pool size and strides were 90.56%, as shown in the confusion matrix. The non-
applied. A Dropout layer with a 25% dropout rate linear SVM model, employing the 'rbf' kernel with
was included. The dense layer had a value of 255, default gamma and C values, demonstrated an
representing the maximum pixel value in the training increased accuracy of 94.39%.
dataset. The final layer was a softmax classifier
related to cross-entropy loss. The RMSProp Considering the higher accuracy of the non-
optimizer with default values for learning rate, rho, linear model, hyperparameter tuning was performed
epsilon, and decay was employed for effective and specifically for it. The optimization focused on
faster model convergence. Learning rate annealing finding the optimal C and gamma values. C controls
was implemented using ReduceLROnPlateau to error, while gamma influences the curvature in the
monitor and adjust the learning rate during training non-linear SVM model. A 5-fold cross-validation
epochs, contributing to achieving high accuracy. The with grid search was employed for this purpose,
training utilized CUDA to enhance speed, with 15 taking nearly 40-50 minutes. The results indicated
epochs and a batch size of 112. The training process that at higher gamma values (0.01), the model
took approximately 40 minutes, revealing that the exhibited overfitting, with 100% training accuracy
training accuracy consistently surpassed the but less than 80% test accuracy. The analysis
validation accuracy. The training and validation revealed that a gamma value of 0.001, combined
losses exhibited minimal differences. The model with C=15, provided the highest test accuracy
achieved the highest validation accuracy of 99.32% around 94% while avoiding overfitting.
and a training accuracy of 99.5%. The final hyperparameters selected were C
= 10 and gamma = 0.001, and the SVM model was
fitted accordingly.
4.2 Multilayer Perceptron (MLP)
The Multilayer Perceptron (MLP) model
has a simple architecture with four sequential layers 5. RESULTS AND DISCUSSION
in the Keras model. The first layer is a flattened
layer that converts a 2D image matrix into a 1D A model’s performance is evaluated based
vector to suit MLP's requirement for 1D input. The on different metrics such as accuracy, loss, error
second layer is a dense layer with 128 neurons and value and precision score. The different evaluation
ReLU activation, facilitating quick convergence. metrics will allow data analysts to understand a
The third hidden layer mirrors the second in terms model’s performance based on its strengths and
of units and activation. The fourth layer is the output weaknesses. For this project, three models, CNN,
layer with softmax activation, representing the MLP and SVM will be evaluated based on certain
network's guess for each digit. To control metrics and a decision will be made to choose the
underfitting or overfitting, a kernel_regularizer best algorithm for handwritten digit recognition.
parameter of 0.002 is applied, like dropout in the
CNN model, with 10 units in this layer.
5.1 CNN
The model summary provides an overview The Convolutional Neural Network (CNN)
of the layers, their order, output shapes, parameters model achieved an accuracy of approximately
per layer, and the total model parameters. The 99.31% with a loss of 0.0326 in the validation test.
optimizer used is the same as in the CNN model. This high accuracy indicates the model's strong
The MLP model is compiled with defined predictive capabilities.
parameters, including a TensorBoard callback for
2175
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
2176
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
5.3 SVM
The Figure 10 presents a classification
report depicting the accuracy results of an SVM
model. This model was constructed using optimal
hyperparameters, specifically C=10 and
gamma=0.001, resulting in a maximum accuracy of
95.31%. Diagram 56 illustrates precision scores
ranging from 0.92 to 0.98 for different classes, while
recall scores fall within the range of 0.93 to 0.98.
Additionally, the f1 scores span from 0.95 to 0.98.
2177
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
These scores collectively indicate a high level of Table 3, which likely summarizes previous research
performance for the model, with values considered on handwritten digit recognition using CNN models.
to be quite good. The accuracies reported in these studies range from
98% to 99.87%. However, it notes that these studies
may lack clear evaluation steps, implying that the
methodologies used to assess model performance
might not have been sufficiently rigorous or
transparent. The CNN model developed in the
current project achieves a notably higher accuracy. It
comes close to the highest recorded accuracy of
99.31% with only a marginal difference of 0.06%.
This implies that the model developed in this project
performs exceptionally well in recognizing
handwritten digits.
2178
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
2179
Journal of Theoretical and Applied Information Technology
15th March 2024. Vol.102. No 5
© Little Lion Scientific
2180