Oral Cancer Detection Using Deep Learning

Renuka Chavva, Jeba Priya S and Dr. Mathu

Computer Science and Engineering, Karunya Institute of Technology and Science, Coimbatore, India

E-mail : rchavva15@[Link], jebapriya@[Link], mathu@[Link]

2024 International Conference on Science Technology Engineering and Management (ICSTEM) | 979-8-3503-7691-3/24/$31.00 ©2024 IEEE | DOI: 10.1109/ICSTEM61137.2024.10561172

Abstract- Use of deep learning methods, and more especially annotated these photographs after collecting them from
the DenseNet architecture, for the purpose of oral cancer clinical settings. With a wide range of oral disorders
detection is the primary goal of this research. Oral cancer, included in the dataset, the model may learn the unique
thrush, lichens, hairy tongue, leukoplakia, and healthy characteristics of each diseases, improving its ability to
tongue photos are all part of the oral conditions dataset. The
dataset includes oral images from both healthy individuals
diagnose [Link] data preparation, model building,
and those with illness. We added new classification layers training, validation, and assessment phases make up the
after fine-tuning a pre-trained DenseNet169 model that had project's essential parts. To improve the model's
been trained on ImageNet using transfer learning. Many generalizability and resilience, data preparation includes
methods of data augmentation were used to improve model arranging the dataset, scaling photos, and using
resilience. A performance comparison using the LeNet augmentation methods. To make training samples more
model was done. The DenseNet based model demonstrated diverse and avoid overfitting, augmentation methods such
exceptional performance, attaining 94.08% accuracy, horizontal flipping, zooming, and rotating are used.
94.16% precision, 94.7% recall, and 94.7% F1 score. By Because of its track record of success in picture
comparison, the LeNet model performed less, with an F1
score of 63.01%, recall of 64.03%, accuracy of 64.02%, and
classification tasks, the DenseNet architecture is selected
precision of 64.06%. Medical image analysis relies on data as the main model for model development. We start with
preparation, model selection, and assessment methods this the DenseNet model, which has already been trained on
study shows that deep learning might be useful for detecting the ImageNet dataset. The model is trained using the oral
oral cancer. Improving the model's performance for real- cancer dataset and then adjusted using transfer learning to
world deployment in oral cancer detection might be achieved fit the unique features of oral pathology pictures. On top
by more tuning and testing. of the pre-trained model, we add unique dense layers to
boost classification performance by capturing higher-level
Keywords—Oral cancer, CNN, Deep learning image analysis,
DenseNet, early detection, classification.
information. Using a predetermined ratio, the dataset is
separated into training and validation sets for the training
and validation procedure. During training, the parameters
I. INTRODUCTION of the model are adjusted using the training set, and to
avoid overfitting, its performance is monitored using the
About 377,713 new instances of oral cancer were validation set. Using a backpropagation-based
recorded worldwide in 2020, making it a major public optimization approach called Adam optimizer, the model
health problem the globe over. The five-year survival rate learns to minimize a specified loss function called
for oral cancer is still poor, partly because the disease is categorical cross-entropy during training. In assessing the
often only detected in its late stages, even though there model’s performance metrics like Accuracy, Precision,
have been improvements in treatment options. To Recall and F1-score are taken into consideration. If you
improve patient outcomes and decrease death rates, early want to see how well the model does across all classes,
identification of oral cancer is critical. Medical image you can build a confusion matrix. To further evaluate the
analysis has recently seen the emergence of deep learning model's discriminatory power, area under the curve
algorithms as a potential tool for automated and accurate (AUC) scores and receiver operating characteristic (ROC)
diagnostics. An oral cancer detection system using curves may be used. These can be applied to various oral
convolutional neural networks (CNNs), with an emphasis conditions. We also investigate an alternate model
on the DenseNet architecture, is the goal of this deep architecture, LeNet, to compare with the DenseNet based
learning-based research. For jobs with limited training model. The LeNet architecture is not described in depth,
data, including medical image analysis, DenseNet short but it is used to measure how well the DenseNet model
for densely connected convolutional networks is a good performs. We may learn about the efficacy of various
fit because to its economical parameter usage and feature architectures for oral cancer detection tasks by comparing
reusability. the results of these two models. In sum, this initiative is
an all-out assault on the urgent need to diagnose and treat
This study's dataset includes pictures of healthy tongues, oral cancer patients as soon as possible by using deep
hairy tongues, leukoplakia, oral lichens, oral cancer, and learning methods. Oral cancer detection might be made
oral thrush, among other oral disorders. To guarantee much more accurate and efficient with the use of
authenticity and dependability, healthcare experts

Authorized licensed use limited to: St. Joseph's College of Engineering & Tech - Palai. Downloaded on September 09,2025 at 04:17:32 UTC from IEEE Xplore. Restrictions apply.
 convolutional neural networks and large-scale picture conventional insulin administration methods [9]. Lastly,
databases. This, in turn, could improve patient outcomes Zambonin et al. (2022) explored the application of
and decrease death rates. MALDI-TOF/MS in identifying cancer biomarkers from
non-invasive human saliva and urine samples, suggesting
[Link] WORKS its potential utility in early cancer detection and
personalized treatment approaches [10].Chang et al. [11]
Existing research showcases a diverse array of non- proposed a predictive strategy for oral cancer prediction
invasive techniques applied across various disciplines, based on clinicopathologic and genomic indicators. They
highlighting significant advancements in fields such as utilized a mix of feature selection and machine learning
agriculture, medicine, and diagnostics. Ndlovu et al. methodologies. Through their study, it was found that the
(2022) conducted a comprehensive review focusing on hybrid model of Relief F-GA-ANFIS, incorporating
non-invasive methods to detect adulteration in dried selected features such as drink, invasion, and p63,
powdered horticultural products, emphasizing the critical achieved the best accuracy with an accuracy of 93.81
need for ensuring the safety and quality of food products percent and an AUC of 0.90. This underscores the
[1]. In a separate study, Ma et al. (2022) demonstrated the significance of integrating clinicopathologic and genetic
efficacy of near infrared spectroscopy in non-invasively indicators to enhance prognosis [Link] et al.
detecting both the anti-inflammatory bioactivity and [12] explored new prognosis tools within the TNM
essential chemical indicators of a commercial oral staging system for oral cancer. Utilizing Multi-Layer
solution, underscoring the reliability and efficiency of Perceptron (MLP) and Gaussian Mixture Model (GMM)
spectroscopic analysis for pharmaceutical applications classifiers, they compared the classification accuracy of
[2].Moreover, Mulani et al. (2022) proposed a pioneering TNM characteristics such as tumor size, number of
machine learning-based approach for estimating blood positive regional nodes, and distant metastasis. Their
glucose levels without invasive procedures, addressing study, which evaluated 75 individuals with oral cancer,
the challenges associated with traditional invasive revealed that GMM outperformed MLP, achieving an
methods and providing a promising avenue for painless average accuracy of 94.18 percent across all stages.
glucose monitoring [3]. Qu et al. (2022) introduced a Additionally, the researchers investigated the performance
novel and cost-effective technique utilizing thermal of Extreme Learning Machines (ELM) as a post-classifier,
imaging coupled with machine learning algorithms for comparing its performance to both GMM and [Link] the
non-invasive pneumonia diagnosis and treatment context of oral cancer detection, our ongoing research
monitoring, offering potential improvements in patient leverages state-of-the-art DenseNet and LeNet
care and disease management [4].In the realm of architectures to analyze oral imaging data non-invasively.
personalized medicine, De Olazarra et al. (2022) explored By harnessing deep learning techniques, we aim to
non-invasive genotyping methods using saliva samples, enhance the accuracy and efficiency of early oral cancer
highlighting the rapid and accurate genotyping detection, thereby facilitating timely intervention and
capabilities enabled by recombinase polymerase improving patient outcomes. Through this
amplification and giant magnetoresistive nanosensors, multidisciplinary approach, we contribute to the
thus paving the way for tailored treatment strategies [5]. advancement of non-invasive diagnostic methods and
Additionally, Nowak et al. (2022) elucidated the their translation into clinical practice, ultimately
principles of sonodynamic therapy as a non-invasive benefiting individuals worldwide.
modality for targeted cancer treatment, showcasing the
potential of sound waves in selectively eliminating tumor [Link] SYSTEM
cells while minimizing collateral damage to healthy
tissues [6].Furthermore, Liu et al. (2022) investigated the While the current deep learning approach for oral cancer
utility of surface-enhanced Raman scattering (SERS) for diagnosis offers notable advantages, it also presents
non-invasive medical diagnostics and wearable flexible significant drawbacks. One primary limitation lies in the
sensing, emphasizing the high sensitivity and specificity necessity for a large and diverse dataset to train the deep
of SERS in detecting biomolecules, thereby offering learning model effectively. Compiling such a dataset,
promising prospects for point-of-care testing and particularly with high-quality photos or samples of
continuous health monitoring [7]. Similarly, Bordbar et al. various stages and types of oral cancer, can be both costly
(2022) developed a microfluidic-based colorimetric and time-consuming, posing challenges, especially for
sensor array for non-invasive COVID-19 detection rare or advanced instances of the disease. Furthermore,
through urine metabolites, presenting a rapid and deep learning models are renowned for their resource-
convenient screening tool for disease surveillance and intensive nature and high computational costs. The
control [8].Moreover, Sabbagh et al. (2022) provided substantial investment of time and money required for
insights into the latest advancements in polymeric non- training may render the approach unfeasible for smaller
invasive insulin delivery devices, discussing the potential clinics and healthcare institutions, particularly if it
benefits in enhancing patient compliance and treatment necessitates sophisticated hardware for implementation.
outcomes, thus addressing the challenges associated with Another critical concern is the inherent lack of

Authorized licensed use limited to: St. Joseph's College of Engineering & Tech - Palai. Downloaded on September 09,2025 at 04:17:32 UTC from IEEE Xplore. Restrictions apply.
 interpretability in deep learning models. Despite their [Link] ARCHITECTURE
remarkable accuracy in identifying oral cancer,
understanding the rationale behind their predictions can
be challenging. This opacity may lead healthcare
providers to hesitate in relying solely on a black-box
approach for critical decision-making, eroding trust and
confidence in the system. Moreover, the success of a deep
learning model hinges significantly on the quality and
diversity of the training dataset. Skewed or inaccurately
represented data can lead to performance degradation,
resulting in false positives or false negatives.
Consequently, meticulous validation and curation of the
training dataset are imperative but entail arduous and
time-intensive efforts. Additionally, deep learning models
are limited to the tasks they were trained on and lack
adaptability to unforeseen circumstances. As new types of
oral cancer emerge and medical research advances, the
risk of these models becoming outdated or inadequate
increases, necessitating regular retraining and updates to
maintain efficacy. In summary, while deep learning-based
Fig. 1. System Architecture
oral cancer detection offers high accuracy and automation
potential, it is constrained by computational demands,
interpretability issues, susceptibility to biased datasets, [Link]
and limited adaptability to novel cases. These challenges
underscore the need for ongoing refinement and 1. Data Collection and Preprocessing Module:
improvement in the development and utilization of deep Images depicting a variety of oral ailments, including
learning models for oral cancer diagnosis. thrush, lichens, hairy tongue, leukoplakia, and healthy
tongue, were gathered for the oral cancer detection
[Link] SYSTEM project's data collection. Healthcare specialists
painstakingly annotated these photographs from clinical
contexts to guarantee their authenticity and dependability.
Using convolutional neural networks(CNNs), and more
Data was efficiently categorized upon collection into
especially the DenseNet architecture, the suggested
directories according to each oral condition, allowing for
system is able to identify oral cancer. As a first step, the
better data administration. Data preparation relied heavily
system gathers and prepares a comprehensive dataset that
on standardization to guarantee dataset homogeneity. The
includes pictures of healthy, hairy, leukoplakia, lichens,
images were adjusted to a standard size of 180x180
oral cancer, thrush, and other oral disorders. The variety
pixels, which is often used in picture classification tasks
of the dataset is increased by using augmentation methods
based on deep learning. To avoid overfitting and boost
such horizontal flipping, zooming, and rotating. The
dataset variety, augmentation methods such horizontal
feature extraction process is based on the pre-trained
flip, zoom, and rotation were used.
DenseNet model, which was fine-tuned using the oral
cancer dataset. Classification custom dense layers are
added, and then the dataset is trained and validated.
Model performance is assessed using measures including
F1-score, recall, accuracy, and precision. To further
evaluate performance and efficacy, we compare it to an
alternate model architecture called Lenet. In order to
improve patient outcomes and healthcare efficiency, the
suggested system is designed to provide an automated and
accurate tool for early identification of oral cancer.

Fig 2: Dataset for oral cancer detection

Authorized licensed use limited to: St. Joseph's College of Engineering & Tech - Palai. Downloaded on September 09,2025 at 04:17:32 UTC from IEEE Xplore. Restrictions apply.
 2. Model Development and Training Module
(DenseNet):
The DenseNet architecture, which is well-known for its
efficiency and efficacy in image classification tasks, was
the primary focus of the model development phase for
oral cancer detection. The feature extraction architecture
was based on a DenseNet169 model that has already been
trained on the ImageNet dataset. To improve
classification performance and capture higher-level
characteristics, custom dense layers were superimposed
on top of the pre-trained model.A set of loss functions
(categorical cross-entropy) and metrics (accuracy)
tailored to multi-class classification were included in the
DenseNet model's build during training. To improve the
model's ability to generalize, data augmentation methods Fig 4 Model Loss of LeNet
were performed during training on the enhanced training
dataset. A validation set was used to continually assess 4. Integration and Deployment Module:
the model's performance during training, and the Model After training was complete, the models were prepared to
Check point callback was used to preserve the best- be deployed into an application or system. To facilitate
performing model for future usage. use and interoperability with preexisting systems,
integration included packaging the trained models with all
required dependencies and APIs. You may put the models
on cloud platforms for scalability and accessibility,
incorporate them into mobile apps, or provide them via
online APIs.

To effectively categorize photos or data points indicative

of the existence of oral cancer, a deep learning model is
first constructed and trained using datasets related to the
disease. After the model is trained, it must be
incorporated into an application or user interface that is
easy to use, which may include creating a web or mobile
application.

[Link] AND DISCUSSION

Fig 3 Model Loss of DenseNet
The main result of the experiment shows how well the
3. Model Development and Training Module (LeNet): DenseNet based model can detect oral diseases, such as
The training of our second deep learning model—LeNet oral cancer, with exceptional accuracy. The model shows
architecture—takes up our focus in the third module. that it is good at dividing up oral photos into several
LeNet was selected because of its historical significance groups, with a validation dataset accuracy of 94.08%.
in the development of neural networks and its simplicity. This impressive precision demonstrates how deep
In order to enable a comparison between the DenseNet learning methods, and the DenseNet architecture in
and LeNet models, LeNet model is trained using the same particular, are capable of evluating intricate visual input
oral cancer dataset. As with the DenseNet module, we and producing accurate diagnostic forecasts. In addition,
carefully assess the LeNet model's performance, optimize the model's dependability and robustness in differentiating
training parameters, and adjust it. The purpose of this distinct oral diseases are reinforced by its excellent
module is to provide some understanding on the relative accuracy, recall, and F1-score values, which provide
advantages and disadvantages of various neural network useful insights for healthcare professionals in clinical
architectures for the detection of oral cancer. decision-making.

Authorized licensed use limited to: St. Joseph's College of Engineering & Tech - Palai. Downloaded on September 09,2025 at 04:17:32 UTC from IEEE Xplore. Restrictions apply.
 DenseNet169, demonstrating its greater capacity to
identify the various forms of oral cancer. These results
highlight how crucial it is to use a cutting-edge deep
learning model, such as DenseNet169, for such crucial
medical image categorization tasks. Due to its improved
performance, the DenseNet169 algorithm has the
potential to significantly improve healthcare outcomes for
cancer patients and their caregivers, as well as speed up
the detection of oral cancer and reduce diagnostic
ambiguity. In the end, it helps healthcare in general by
ensuring that medical personnel can rely on accurate and
consistent automated classification.

Fig 5 Model Accuracy of DenseNet

An evaluation of the DenseNet model and a comparison

with an as-yet-unnamed "LeNet" architecture are both
part of the project. The comparison shows a clear
difference in performance, with LeNet attaining a far
lower accuracy of 60.89% on the validation dataset, even
if there are little information supplied regarding the
setting of the LeNet model. This comparison shows that
DenseNet design is better at detecting oral cancer than
other architectures. DenseNet is able to capture complex
patterns and characteristics in mouth photos more
efficiently. To get the best results, it is crucial to use deep
learning architectures that are designed for certain Fig 7 Confusion Matrix for actual and predicted values –
medical imaging applications. DenseNet

Fig 8 Confusion Matrix for actual and predicted values of

Fig 6 Model Accuracy of LeNet LeNet

TABLE 1: Performance metrics To graphically display the model's categorization

Model Accuracy Precision Recall F1 performance over various oral circumstances, the project
Score utilizes confusion matrices (Figure 7 and figure 8). For
DenseNet 94.08%. 94.16% 94.7% 94.07% each class, these matrices show how well the model
LeNet 64.02% 64.06% 64.03% 63.01% classified true positives, false positives, true negatives,
and false negatives. Looking at the confusion matrix, we
The outcomes clearly show that when it comes to can see that the model does a great job of distinguishing
classifying oral cancer disease, the DenseNet169 between distinct oral diseases with little room for error in
algorithm outperforms the LeNet method. Higher its classifications. This thorough examination provides
accuracy, precision, recall, and F1 score were attained by further evidence of the model's dependability and efficacy

Authorized licensed use limited to: St. Joseph's College of Engineering & Tech - Palai. Downloaded on September 09,2025 at 04:17:32 UTC from IEEE Xplore. Restrictions apply.
 in clinical settings, opening the door to its possible improve the deep learning system's accuracy and enable
incorporation into actual healthcare facilities. more individualized therapy suggestions and diagnoses.
These avenues collectively hold promise for advancing
[Link] the field of oral cancer detection and improving patient
outcomes.
Finally, the experiment demonstrates that DenseNet
architecture, a kind of deep learning, is successful in REFERENCES
detecting oral cancer. In comparison to the LeNet model,
[1] Ndlovu, P. F., Magwaza, L. S., Tesfay, S. Z., & Mphahlele, R.
DenseNet achieves better accuracy (94.08%), recall, and R. (2022). Destructive and rapid non-invasive methods used to
F1 scores. This highlights the significance of choosing detect adulteration of dried powdered horticultural products: A
deep learning architectures that are specialized for review. Food Research International, 157, 111198.
medical imaging applications. A major step forward in [2] Chen, Y., Wu, Y., Geng, Y., Liu, X., Xiao, L., Xu, D., & Ma, H.
(2022). using near-infrared spectroscopy to non-invasively
healthcare has been achieved with the project's determine the anti-inflammatory bioactivity and important
demonstration of deployment readiness, which provides molecular markers of the brand-name oral lanqin solution. 27(9)
scalable solutions for automated diagnosis of oral cancer. Molecules, 2955.
Improving diagnostic skills and patient outcomes in oral [3] Jadhav, M. M., Mulani, A. O., and Seth, M. (2022). A Non-
Invasive Device-Based Painless Machine Learning Method for
cancer detection may be the focus of future study, which Estimating Blood Glucose Level. Internet of Things (IoT),
may also include further improving the DenseNet model Artificial Intelligence, and Smart Materials for Energy
and investigating new approaches. Applications, 83-100.
[4] Qu, Y., Xu, R. X., Fan, H., and Meng, Y. (2022). Machine
learning combined with low-cost thermal imaging for non-
[Link] WORK invasive pneumonia diagnosis and treatment monitoring.
Physics & Technology of Infrared, 123, 104201.
In future work, the integration of multi-modal data [5] Wang, S. X., Cortade, D. L., and de Olazarra, A. S. (2022).
From saliva to SNP: gigantic magnetoresistive nanosensors and
presents a promising avenue for enhancing the recombinase polymerase amplification provide non-invasive,
effectiveness of oral cancer detection using DenseNet and point-of-care genotyping for precision medical applications.
LeNet. By incorporating additional data modalities such 2131-2144 in Lab on a Chip, 22(11).
as patient demographics, histopathological images, and [6] Price, R. J., Breza, V. R., Nowak, K. M., and Schwartz, M. R.
(2022). Sonodynamic therapy: Quick advancements and fresh
genetic markers, the system's accuracy and robustness chances for using sound to kill tumor cells without invasive
could be significantly improved. Furthermore, exploring methods. Letters on Cancer, 532, 215592.
advanced data augmentation techniques like generative [7] Yu, J., Shan, K., Shang, X., Guo, J., Liu, G., Mu, Z., & Liang,
adversarial networks (GANs) or domain adaptation X. (2022). One possible approach for wearable flexible sensing
and point-of-care testing non-invasive medical diagnosis is
methods could address challenges posed by limited surface-enhanced Raman scattering. Chemistry Frontiers 10,
annotated data, thereby enhancing model generalization. 1060322.
Additionally, investigating transfer learning strategies by [8] Sharghi, H., Javid, M., Bordbar, M. M., Samadinia, H., Sheini,
fine-tuning pre-trained models on larger datasets related A., Aboonajmi, J., & Bagheri, H. (2022). Using a microfluidic-
based colorimetric sensor array sensitive to urine metabolites,
to medical imaging or cancer detection tasks could non-invasive COVID-19 detection is possible. Acta
facilitate the extraction of more meaningful features Microchimica, 189(9), 315.
relevant to oral cancer detection. Ensemble learning [9] Sabbagh, F., Muhamad, I. I., Niazmand, R., Dikshit, P. K., &
approaches offer another avenue for future exploration, Kim, B. S. (2022). Recent progress in polymeric non-invasive
insulin delivery. International Journal of Biological
where combining predictions from multiple models Macromolecules, 203, 222-243.
trained with different architectures or variations of [10] Zambonin, C., & Aresta, A. (2022). MALDI-TOF/MS analysis
DenseNet and LeNet could potentially amplify the of non-invasive human urine and saliva samples for the
system's predictive power and reliability. Future research identification of new cancer biomarkers. Molecules, 27(6),
1925.
could be focused on improving the deep learning system [11] S. W. Chang, S. Abdul-Kareem, A. F. Merican, and R. B. Zain,
for oral cancer diagnosis by combining various data "Oral cancer prognosis based on clinicopathologic and genomic
sources and utilizing cutting-edge deep learning methods. markers using a hybrid of feature selection and machine
Adding more imaging modalities, such as oral scans, to learning methods," in BMC Bioinformatics, vol. 14, May 31,
2013, Art. no. 170. doi: 10.1186/1471-2105-14-170
the analysis of pictures of the oral cavity is one potential [12] Rajaguru, Harikumar and Prabhakar, Sunil "Performance
strategy. This combination of information may offer a Comparison of Oral Cancer Classification with Gaussian
more thorough understanding of the malignancy and its Mixture Measures and Multi Layer Perceptron," in Proceedings
traits. Additionally, by utilizing pre-trained models and of the International Conference on Advances in Computing,
Communications and Informatics (ICACCI), 2017, pp. 212-219.
merging many classifiers, investigating the use of doi: 10.1007/978-981-10-4220-1_23.
ensemble learning and transfer learning techniques can
enhance the system's performance. Moreover, exploring
the possibility of integrating genetic and clinical data can

Authorized licensed use limited to: St. Joseph's College of Engineering & Tech - Palai. Downloaded on September 09,2025 at 04:17:32 UTC from IEEE Xplore. Restrictions apply.