ESP32 CAM MODULE BASED HAND GESTURE IDENTIFICATION

FOR MUTE PEOPLE

Surendar S1, Rallapalli Saiprasanna 2, T. Arun Prasath3, S. Kalimuthu Kumar 4
1,2,3,4 Dept. of BME, Kalasalingam Academy of Research and Education, Krishnankoil, India

9920020014@[Link] 1, 9920020020@[Link].in2, [Link]@[Link].in3,

[Link]@[Link].in4

ABSTRACT This kind of approach has significantly improved the

Communication is a way to identify the capacity of the speaking mute peoples.
particular needs and necessary. In verbal
Gesture based communication has huge impact
physically challenged persons require enhanced
between the speaking mute peoples. But the clarity
communication medium to identify their
and gesture identification between speaking note
personal thoughts. Gesture is a best medium to
and ordinary human beings faces enormous
convey the message of speaking mute peoples.
difficulties for the clarity of the meticulous
Each and every action and verb has determined
information transformation. So the system is used to
under the individual gesture. This gesture based
recognize the particular gesture is a necessary
speaking has more useful only between the
operation to modify the system into more simple
physically challenged persons. This gesture based
with the society. Image processing is a vital
communication has not much effective among the
approach to improve the concert of the automation
entire society. Because of this proposed system
and identification. In real-time image processing
has generated a common communication
techniques, the system has continually acquired the
medium with the aid of ESP32 Cam module. This
various parameters of the current data and the
module is used to capture the action in an
information has enhanced and separated with the aid
eminent manner. The enhanced image processing
of various filters, segmentation and classification.
and machine learning operation is used to modify
the system into complete communication Machine learning is a method or ideology to classify
platform. This system has classified the various the particular gesture in an eminent manner. In
gestures and identifies the corresponding action research more number of machine learning
in an effective manner. This proposed system has techniques is available for the particular operation
improved the life-style of the speaking mute and survey helps to identify the preferable machine
personalities with the aid of efficient and highly learning algorithm for the effective gesture
accurate gesture identification strategies. identification. This composed and compact
technique is used to hang out the speaking mute
Key words: ESP32 Cam module based image
peoples with the society. This vast literature survey
classification, Real-time image processing,
has detailed with the upcoming sessions.
Gesture identification and classification.

1. INTRODUCTION 2. LITERATURE SURVEY

Ming jin et al discovered this gesture detection and
Verbally physically challenged persons are facing
identification has most useful for the numerous
numerous difficulties in their day to day life.
number of applications and reduces the difficulties
Communication is a common difficulty facing every
for the human regular life. This proposed machine
speaking mute person. Gesture is a common and
has able to achieve easily understand the operation
popularized medium of communication inter and
and nature of applications. This proposed system has
intra speaking mute persons. Gesture identification
classified into various stages data acquisition, pre-
is a process to identify the particular action and its
processing, segmentation, feature extraction and
corresponding reaction. Image processing is a better
classification, where the various algorithms at each
way to identify the gesture identification and
stage are elaborated and their merits compared.
classification. This classified image has decoded the
particular communicating message. Gesture is a Mohimul islam et al suggested a broad research idea
sign which is made upon with the aid of human body such as Human Computer Interaction (HCI) is a
parts either human hands or other indicating indices. broad research and it can easily interaction between

1
human and computer or machine. This system This module can easily interface with the any kind
recognizes the American Sign Languages based on of system in a multi-dimensional manner. This
the gesture in an effective accuracy. device acquired high definition and high quality
input data in a great frame rate. This device requires
Sakshi et al described hand gestures are the basic low power and comfort with the any kind of
inferences for the particular foundation and which is microcontroller devices.
a visual form of communication. This system
introduces CNN based gesture based sign language 3.2 Input data acquisition
detection. This model has a compact representation
that achieves better classification accuracy with a This proposed system has acquired the information
fewer number of model parameters over the other of the various gestures in an accurate manner. Every
existing architectures of CNN. number consist a unique gesture denotation and the
different kind of information has restored the
Meyback et al designed a optimal linear filter and it proposed system for the purpose of compare and
is named as kalman. This filter normalize the classify the results. Each and every input data has
discrete values of the input image and it modifies the converted in the form of scorecard based on the
image into high accurate and sharpness. Baris et al nodes and connectivity of the input image. This
introduced the new-fangled CNN based medical score has most useful at the time of image
image data segmentation in an efficient manner. It classification in an accurate manner. The various
continually acquired the various features of the input gesture notations have shown in fig.2.
image samples. Sinaga et al delivered an
unsupervised image classification based on the K-
means clustering. Pattern recognition and machine
learning has achieved with the aid of unsupervised
clustering algorithms. This pattern recognition has Fig.2 Gesture Input data
classified with the aid of necessary number of
clusters with the prior. This vast literature survey 3.3 Image Filter
aids to architects the functional blocks in an eminent
manner. Image Filter is a method or pre-processing strategies
to improve the information of the raw input image
3. MATERIALS AND METHODS samples. After the Raw input data fed into the
convoluted bi-lateral filter, the output data improves
This gesture based verbal identification identifies the sharpness of the edges of the input data and it can
the words and communicational signals usage has easily avoids the noise and unwanted signals from
designed and implemented with the aid of building the input data.
blocks as mentioned in fig.1.
3.4 Image Segmentation
Image segmentation is a process that separates
foreground and background information of the input
data. This foreground image consists of fully wanted
information of the proposed system and the
background image consists of unwanted information
of the proposed system. This image segmentation
has applied with the way of following procedure
such as thresholding, image region identification
and image grouping based on the high intensity. This
system is used to extracts the objects form the raw
input data. The process of the image segmentation
Fig.1 Block diagram of Gesture Identification has shown in fig. 3.

3.1 ESP32 Cam module

ESP32 Cam module is a device which acquire the
continuous information of the gesture in real-time.

2
 mute peoples. The flow of the data transformation
has implicated with the flow diagram ass given fig.5.

Fig.3 Image segmentation

3.5 Image Classification

Image classification is an idea to categorizing the
different kind of data and it is grouped and labeled
based on the pixel information. This classification
can also occur in a method of 2-dimensional and
multi-dimensional image classification. The number
of classes and the classification process has purely
based on the visual content of the input image.
Pattern recognition and image clustering are the two
common categories of the classification algorithms. Fig.5 Flow Diagram of the proposed system.
In digital image classification uses the spectral
information represented by the digital numbers in 4. RESULTS AND DISCUSSION
one or more spectral bands. In fig.4 represents the
image classification in a digital manner. This proposed system has integrated with the aid of
ESP32 cam module and mixed with the real-time
image processing techniques. After the enhanced
image acquisition, the image has fed into the
kalmans’ filter algorithm to enhance the sharpness
and clear boundaries of the object. After the effective
filtration of the input data, the input image has
segmented with the aid of loaded CNN algorithm.
Fig.4. Image Classification This algorithm segments the foreground and
background information in an effective manner. This
3.6 Proposed system segmented image has fulfilled only with the
information of gesture information. Using K-means
In this proposed system has initially originated with
clustering classification algorithm, the segmented
the aid of ESP32 cam module for the purpose of
image has classified into different kind of classes of
gesture information from the speaking mute peoples.
the speaking mute persons needs. The various needs
After the confront acquisition of the input image and
are classified such as Hello, water need, Food, Help,
the image fed into the Kalmann’s filter and this filter
Yes or No options. In table-1 explicates the various
removes the unwanted signals. The output data
gesture type and its corresponding classes.
consists of normalized information of the gesture
and renormalize the background information. This Table-1 Gesture Type and conveying message
normalized data is fed into the CNN based image
segmentation algorithm. This algorithm segments [Link] Input Image Classifica Conveyin
the input image into foreground and background tion g
input image samples in the form of enhanced algorith message
manner. This CNN has addressed the appropriate m
input gesture and validated into individual scores for
the each and every gestures. Based on the gestures 1. Zero K-means Hello
score values and this input image have classified the
objects with the aid of K-means clustering
algorithm. After the successful classification of the
gesture and the tail ended voice module is used to
deliver the particular information from the speaking

3
 2. One K-Means Water

3. Two K-means Food

4. Three K-means Help

5. Four K-Means Yes

6. Five K-Means No

After the successful gesture type and its

corresponding information has loaded into the
programmed with the features. The various results
and its information have shown in fig.6. This figure
has deliberated the lessons and effectiveness of the
proposed system.

Fig.6 Real-time outputs of Gesture model and

information

This efficient gesture identification for speaking

mute peoples has successfully implemented and
demonstrated with the aid of wide explanation has
mentioned above. The real-time hardware
implementation of the proposed system has
portrayed in fig.7.

4
 3. Sharma, S. and Singh, S., 2021. Vision-based
hand gesture recognition using deep learning
for the interpretation of sign language. Expert
Systems with Applications, 182, p.115657.
4. Kulkarni, V.S. and Lokhande, S.D., 2010.
Appearance based recognition of american sign
language using gesture
segmentation. International Journal on
Computer Science and Engineering, 2(03),
pp.560-565.
Fig.7 Hardware implementation of hand 5. Ariesta, Meita Chandra, Fanny Wiryana, and
gesture-based speaking system Gede Putra Kusuma. "A Survey of Hand
Gesture Recognition Methods in Sign Language
5. CONCLUSION Recognition." Pertanika Journal of Science &
Technology 26, no. 4 (2018).
This gesture identified proposed system has 6. Liang, R.H. and Ouhyoung, M., 1998, April. A
implemented successfully in a real-time manner. real-time continuous gesture recognition system
This system comprises the information of gesture in for sign language. In Proceedings third IEEE
an accurate and effective manner. This eminent international conference on automatic face and
result discussion has deliberated the performance gesture recognition (pp. 558-567). IEEE.
and efficiency of the gesture identification system 7. Maybeck, P.S., 1990. The Kalman filter: An
compared than previous algorithms. Kalmanns’ introduction to concepts. In Autonomous robot
filter is used to augment the system in conditions of vehicles (pp. 194-204). New York, NY:
blurring and poor weather condition and this system Springer New York.
has modified into independent weather and light 8. Kayalibay, Baris, Grady Jensen, and Patrick van
forecasting. CNN image segmentation is used to der Smagt. "CNN-based segmentation of
separates the gesture value from the clumsy medical imaging data." arXiv preprint
backgrounds in a valuable manner. K-means arXiv:1701.03056 (2017).
algorithm is used to classify the various information 9. Dolz, Jose, Karthik Gopinath, Jing Yuan, Herve
of the gesture and various communication messages Lombaert, Christian Desrosiers, and Ismail Ben
in both mono-model and multi-model data Ayed. "HyperDense-Net: a hyper-densely
modeling. This system improves the performance connected CNN for multi-modal image
with the aid of combined previously mentioned segmentation." IEEE transactions on medical
algorithms. This system creates high impacts in the imaging 38, no. 5 (2018): 1116-1126.
life of speaking mute persons. It modifies the hardest 10. Sinaga, Kristina P., and Miin-Shen Yang.
life style of speaking mute peoples into lighter. In "Unsupervised K-means clustering
future, the accuracy of the projected system has algorithm." IEEE access 8 (2020): 80716-
enhanced with the aid of various machine learning 80727.
algorithms. 11. Bradley, Paul S., Kristin P. Bennett, and Ayhan
Demiriz. "Constrained k-means
REFERENCE clustering." Microsoft Research, Redmond 20,
no. 0 (2000)
1. Cheok, Ming Jin, Zaid Omar, and Mohamed 12. Bora, Jyotishman, Saine Dehingia, Abhijit
Hisham Jaward. "A review of hand gesture and Boruah, Anuraag Anuj Chetia, and Dikhit
sign language recognition Gogoi. "Real-time assamese sign language
techniques." International Journal of Machine recognition using mediapipe and deep
Learning and Cybernetics 10 (2019): 131-153. learning." Procedia Computer Science 218
2. Islam, Md Mohiminul, Sarah Siddiqua, and (2023): 1384-1393.
Jawata Afnan. "Real time hand gesture 13. Sreemathy, R., M. P. Turuk, S. Chaudhary, K.
recognition using different algorithms based on Lavate, A. Ushire, and S. Khurana. "Continuous
American sign language." In 2017 IEEE word level sign language recognition using an
international conference on imaging, vision & expert system based on machine
pattern recognition (icIVPR), pp. 1-6. IEEE, learning." International Journal of Cognitive
2017. Computing in Engineering 4 (2023): 170-178.

5
14. Das, Soumen, Saroj Kr Biswas, and Biswajit
Purkayastha. "A deep sign language recognition
system for Indian sign language." Neural
Computing and Applications 35, no. 2 (2023):
1469-1481.
15. Shin, Jungpil, Abu Saleh Musa Miah, Md Al
Mehedi Hasan, Koki Hirooka, Kota Suzuki,
Hyoun-Sup Lee, and Si-Woong Jang. "Korean
sign language recognition using transformer-
based deep neural network." Applied
Sciences 13, no. 5 (2023): 3029.