2023 International Conferences on Data Science, Agents, and Artificial Intelligence

Sign Language to Text Conversion using RNN-LSTM

Kothai Ganesan, Anna Anbumozhi ,
Amutha Seviappan,
Department of CSE (AIML), Department of Data Science and
2023 International Conference on Data Science, Agents & Artificial Intelligence (ICDSAAI) | 979-8-3503-4891-0/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICDSAAI59313.2023.10452617

Computer Science and Engineering,

Business systems,
KPR Institute of Engineering and
Kalasalingam Academy of Research SRM Institute of Science and
Technology
and Education, Technology,
Coimbatore, India
Virudhunagar, India. Chennai, India.
emailtokothaiganesan@[Link]
amutha@[Link] aamozhi84@[Link]

A. Sameer Reddy B. Vamsi Krishna [Link] Reddy

Computer Science and Engineering, Computer Science and Engineering, Computer Science and Engineering,
Kalasalingam Academy of Research Kalasalingam Academy of Research Kalasalingam Academy of Research
and Education, and Education, and Education,
Virudhunagar, India. Virudhunagar, India. Virudhunagar ,India.
9920004323@[Link] 9920004324@[Link]
9920004303@[Link]

I. INTRODUCTION

Abstract—The speech-impaired community only uses sign Sign language is one of the greatest and most widely used
language; the rest of society interacts verbally. Our research
languages for the dumb and deaf, and it is used to enhance
intends to fill this communication gap by proposing a state-
communication. Since the majority of regular people don't
of-the-art method for comprehending both static and
moving signs in Indian Sign Language and translating them know sign language, we created an initiative to address this
into text. Data is collected, pre-processed, and hand issue. Recently, a lot of novel methods have been
recognition is finished using media pipe holistic before being developed for developing sign language recognition.
categorized into suitable speech output. Because LSTM [1]We used the LSTM (long short-term memory)
networks can develop long-term dependencies, they were algorithm in the context of deep learning to create this
investigated and employed for the classification of gesture project. The benefit of using this LSTM is that it provides
data. The constructed model exhibited a 100% accuracy rate greater precision with sequential data. The Indian sign
while categorizing 26 motions, highlighting the usefulness of
language was the focus of this endeavor. Two- handed
LSTM-based neural networks for sign language translation.
sign language issued by Indians to communicate. The
For those who are deaf or dumb, translating sign language
into text allows them to interact with each other or with French and British sign languages inspired the
people in the general public by using hand gestures. development of the letters used in Indian sign language.
According to a survey we conducted on sign language Because the majority of signs in AmericanSign Language
comprehension, mostpeople are unable to identify the hand are single-handed and therefore simpler, the majority of
gestures and mode ofcommunication used by sign language recent study has concentrated on expanding our
users. After hearing about all the difficulties, they have understanding of this language. [2] The useful standard
communicating with regular people, we developed this library for American Sign Language is another enticing
initiative on sign languagetranscription. The community of
aspect. Indian sign language recognition algorithms are
deaf and mute people will benefit from this initiative because
more complex because they must account for the more
it explains the meaningof each hand gesture and uses hand
gestures to communicate various signs. In this project, hand frequent usage of both hands in Indian sign language
gestures were recognized using a pipeline method called compared to American Sign Language. The recognition of
media pipe, and training and testing were carried out using Indian sign language has lately been the focus of numerous
deep learning techniques. Sign language was then converted initiatives. The system's capacity to recognize the various
to text format. Indian Sign Language alphabetswill aid in cutting down
Keywords—pipeline,Mediapipe,deeplearning on background noise and ensuring error-free output. In
this project, we have used
[3] LSTM long-short-term memory deep learning model
because it will have a high accuracy rate for sequential
data. A deep learning design built on recurrent neural
networks is the LSTM. Instead of using feed- forward
neural networks, this operates using feedback

979-8-3503-4891-0/23/$31.00 ©2023 IEEE

Authorized licensed use limited to: Access provided by VTU consortium. Downloaded on April 01,2026 at 10:03:03 UTC from IEEE Xplore. Restrictions apply.
 connections. In addition to calculating a single data point, sets of non-independent parts. The structure of sign
it can also calculate a complete data sequence. LSTM language, which is different from regular penmanship,
refers to the RNN concealed layer modification. RNN results in communication obstacles between hearing
weights and data can be kept for a very longtime with individuals and deaf and mute persons. As a
LSTM. The media pipe-based hand motion recognition consequence, they primarily use visual means of
component of this undertaking Because it draws lines or communication. If there is a standard interface that
focuses on the hands and facial features, the media pipe transforms sign language to text, the movements can be
pipeline technique was chosen for our research because it easily understood by others. A vision-based interface
increases the precision of gesture recognition. People system has been developed as a consequence, allowing
today don't want to converse with a friend who is dumb or D&M people to communicate even when they don't
mute in American Sign Language. Ifone's hands or digits speak the same language. The goal is to create a user-
are hurt, communicating with others will be difficult friendly, computer-human interface that can recognize
because American Sign Language requires the use of sign language. Just a few of the sign languages that are
hands. There are some issues with this sign language, such frequently used today include ASL, BSL, and Indian
as when a man wishes to communicate with his wife and SignLanguage.
everything was going well until she was involved in an
accident and broke her wrist, rendering communication II. LSTM ALGORITHM
with him impossible for a considerable amount of time. LSTM a type of RNN that can understand long-term
Numerous sign language words are included in the dependencies can be used to address this inefficiency.
American Sign Language Dictionary [4]. People are likely LSTM is a recurrent network design that can train to
to be perplexed by these terms, which could lead to bridge time gaps of more than 1000 steps without
misunderstandings that could be problematic. It is impairing short time lag performance when given noisy,
extremely challenging to comprehend when a discussion incompressible input sequences. Areal-time translation
and another item are mixed up. A handful of speech- system for sign language was developed using LSTM-
impaired people in India use an assortment of sign based Sensor data that can be classified using neural
language gestures known as the Indian sign language. networks by correlating sign language motions to the
Static and dynamic kinds of ISL motions are both possible. appropriate spoken labels. This is possible because they
If the palms do not move while making the sign, it is have the ability to acquire long-term associations.
referred to as a static gesture. The bulk of ISL alphabet
motions is static indicators. While being performed, III. LITERATURE SURVEY
dynamic gestures, which make up the majority of ISL Sign language is used by the deaf and mute
gestures, entail hand motions. communities to communicate. The community of
hearing-impaired people creates sign language on its
A technology that translates Indian Sign Language into own. Both manual and non-manual signs are used in
voice could successfully close the communication gap, sign language communication; manual signs use the
allowing speech-impaired people to communicate with the fingers, hands, and arms while non- manual signals use
general public. Thiswould make it easier for them to adjust the face, head, and irises. [6] A structured language
their living and join the society. For the device to be able having morphology, grammar, syntax, and
to translate ISL, it would need to be able to detect both pronunciation is sign language. A full-fledged natural
static and dynamic motions. Standard neural networks can language, sign language employs a number of
distinguish between static and dynamic gestures, but they expression modalities to facilitate ordinary
cannot classify static motions. communication. Software that recognizes sign
language transforms human-to- human interaction into
Dynamic gestures create sequential data because the
human- computer interaction for communication. By
readings taken at each time point depend on thereadings
providing a quick and precise method of translating text
taken earlier in the process. Standard neural networks
or speech, sign language recognition technology aims
cannot be used to classify sequential data because they
to make it easier for hearing people to "dialogue
need every measurement to be independent from the other
converse" with deaf people. No single sign language is
readings. Recurrent neural networks (RNNs) are
utilized by all hearing-impaired people. However, there
connection is the models that can handle sequential input
are geographic differences in sign languages, just like
one element at a time while only transferring information
there are in spoken languages. If a person is deaf or
to the necessary stages of the sequence. [5]They can
dumb, they cannot communicate with a normal person
therefore simulate input and/or output that consist of
who can speak and hear well unless he normal person
knows sign language. The same issues arise when a deaf
and dumb person attempts to communicate with a
person who is blind or normally sighted. Thus, sensor-
based and vision-based approaches are the two basic
methods employed in sign language recognition.

Authorized licensed use limited to: Access provided by VTU consortium. Downloaded on April 01,2026 at 10:03:03 UTC from IEEE Xplore. Restrictions apply.
PCA and LDA, which stand for Principle Component collected data. The closest match between each movement and
Analysis and Linear Discriminant Analysis, have been the database can be used to translate the gesture into sign
used in earlier projects or research by other people. language. By comparing the input with a database, Abhijith
PCA is used for BhaskaranK et al. published a similar method for translating
preprocessing the data, and LDA is used for hand gesture gestures [10]. This system represented each gesture in state
recognition. In other initiatives, they have also utilized space as opposed to merely storing the data, which was
hardware tools like gloves that are useful for the maindistinction.
communicating with deaf or dumb people. Some of them
have also created a project that scans the entire body and IV. WORK PLAN
converts sign language to text or voice based on motion
and motions [7]. Both appearance-based and 3D model-
based gesture representations were listed. We also Our Work Plan includes Real Time Sign Language
showed the two categories of hand gesture recognition Detection using sequences,
applications in the papers from the previous few years.
1. Identify global key points Separate models are
Hand signals can be divided into various groups, such as
integrated into the Media pipe Holistic pipeline for
stance and gesture, dynamic and static, or a combination.
This essay focuses on a review of the hand gesture the position, face, and hand components. They are all
literature and explores the advantages anddisadvantages tailored to each individual domain.
of using these methods in different situations. The
translation of sign language motions into voice is
accomplished using vision-based systems, which
employ image processing and computer vision
techniques. The sign language translation is finished by
determining the best match by comparing the attributes
of the input photographs with the database. J. Singha and
K. Das developed a method for categorizing imaged at a
of the Indian Sign Language using Eigen value weighted
Euclidean distance. [8] Manual feature filtering,
trimming, and categorizing was all part of this
procedure. Deo, Rangnesh, and Trivedi proposed a plan
for utilizing the 2019 World Congress for Advancement
of Technology in Bangalore, India. Markov models are
applied for the recognition of changing hand
movements. The study additionally recommends a CNN
hybrid mode, which provides greater detection accuracy
than traditional [Link] are a number of issues with
the practical deployment of these vision-based
approaches, even though they might translate sign
language effectively. The biggest disadvantage of this
approach is that the quality of the translation is
influenced by the brightness and picture resolution. As a
result, it is unexpected in situations when we lack the V. RNN-LSTM METHODOLOGY
ability to influence these features, such as in the actual In the first step of this project, the data has been collected
world. However, adding functions Sensor- Based: and preprocessed using holistic features which will focus
Alternatively, sensors can be directly applied to thehand on the hand gestures to detect the sign and to convert it.
to detect the orientation of the hand while sign language
gestures are made. Each sign language move has a
unique set of sensor values to go with it. The motions are
then separated into relevant texts and speeches using the
sensor values. A gesture recognition glove based on
charge- transfer touch sensors was demonstrated by
Abhishek, Qubeley, and Ho. The hand movements allow
the capacitive sensors to be selectively activated using a
binary detection method [8]. As a consequence, a
finger's location when it's straight, it's binary 1, and
when it's twisted, its binary 0. This method could not
distinguish between dynamic motions; it could only
classify static movements that matched letters and
numbers. Is a challenging issue, and by using
photographs during training, the model's accuracy was
drastically reduced. You're able to move quickly. Heera,
Murthy, Sravanti, and Salvi also put up an MPU-6050
module-based sensor-based approach for the gathering
of motion data. Computer storage is used to keep the Fig (1.1): sign language
3

Authorized licensed use limited to: Access provided by VTU consortium. Downloaded on April 01,2026 at 10:03:03 UTC from IEEE Xplore. Restrictions apply.
Open CV is a python library that is need for capture
video or computer vision pictures. Fig (1.1) shows the
sign language. Media pipeline which is frame work to
recognize the hand gestures with more accurate also it
recognize the facial and pose expressions etc. Fig (1.2) Fig (1.4) Media Pipe Recognize the image
shows the Feature Extraction.

Fig (1.4) Media Pipe Recognize the image

Fig (1.3) and Fig (1.4) shows the images of how the media pipe
will recognize the hand gestures. Data must be trained and tested.
The learned long short-term memory deep learning model is
evaluated and validated using a confusion matrix. The deep
learning model is made up of a SoftMax categorization layer, a
dropout layer, and layering long and short-term memory The input
layer for the network is the deep learning model layer, which has
50 LSTM units. A collection of 50*11 2D data from a3D sensor
is received as input by this layer. One hidden layer, implemented
as a layer with 100 nodes that are densely connected, constitutes
Fig (1.2): Feature Extraction the complete network. A dot product ofthe linear activation
function and the dense layer is then applied. After that, the input
value and weight matrix are dot products, and the dense layer is
subjected to a linear activation function.

VI. Results and Discussion

Fig (1.5) Sequential Model Result

Fig (1.5) shows the confusion matrix after performing these
all models the accuracy has been evaluated for this project.
Fig (1.3) Media Pipe Recognize the hand Gestures

Fig (1.6) Accuracy

4

Authorized licensed use limited to: Access provided by VTU consortium. Downloaded on April 01,2026 at 10:03:03 UTC from IEEE Xplore. Restrictions apply.
 with verbal impairments can communicate using sign language.
The distance between these two groups could be bridged fairly
rapidly if they could communicate via text messages. Deep
learning/computer vision models, conductor- based sign
language devices, and human translators are a few methods that
can be used to link conversation in these circumstances. This
research, however, shows that the latter strategy is the most
effective.

VIII. REFERENCES

1. Abraham, Ebey, Akshatha Nayak, and Ashna Iqbal. "Real- time

translation of Indian sign
languageusingLSTM."2019globalconference for
advancementin technology (GCAT). IEEE, 2019.

2. Xiao,Qinkun,etal."Multi-information spatial–temporal LSTM

fusion continuous sign language neural machine translation."
IEEE Access 8 (2020): 216718-216728.

3. Jayadeep,Gautham,etal."Mudra:convolutional neural network

based Indian sign language translator for banks." 2020 4th
InternationalConferenceonIntelligent Computing and Control
Systems (ICICCS). IEEE, 2020.

4. Basnin, Nanziba, Lutfun Nahar, and Mohammad Shahadat

Hossain. "An integrated CNN-LSTM model for Bangla
lexical sign language recognition." Proceedings of
International Conference on Trends in Computational and
Cognitive Engineering: Proceedings of TCCE 2020.
Fig(1.7) Output of Sign language Singapore: Springer Singapore, 2020.

The sign language recognition using this LSTM Our deep 5. Sincan, Ozge Mercanoglu, Anil Osman Tur,and Hacer Yalim
learning algorithm has provided an accuracy of roughly Keles. "Isolated signlanguage recognition with multi-scale
100%. This model's primary objective is to concentrate on features using LSTM." 2019 27th signal processing and
hand gestures and poses to correctly predict the text. The communications applications conference (SIU). IEEE, 2019.
same is described in the model description that was just
provided. To put it simply, we will train the model to predict 6. [Link], Biao, Shiliang Huang, and Zhongfu Ye. "Application
sign language using an LSTM deep learning model while of tensor train decomposition in S2VT model for sign
using the webcam of our system with the aid of an open CV language recognition."IEEEAccess9(2021):35646- 35653.
to identify hand gestures. As demonstrated, the precision is
close to 100% shows accuracy in Fig (1.6). The Fig (1.7) 7. Goel, Pragati, et al. "Real-Time Sign Language to Text and
shows the output of sign Language. Speech Translation and Hand Gesture Recognition using the
LSTM Model."2022 3rd International Conference on Issues
and Challenges in Intelligent Computing Techniques (ICICT).
VII. CONCLUSION IEEE, 2022.
The goal of this research was to create a powerful deep-
8. Chaikaew, Anusorn, Kritsana Somkuan, and Thidalak Yuyen.
learning model for textual transcription, understanding, and "Thai sign language recognition: an application of deep neural
recognition of Indian sign language. The study was founded network."2021 joint international conference on digital arts,
on background data indicating a considerable lack of media and technology with ECTI northern section conference
interaction between the mute and deaf group and the general on electrical, electronics, computer and telecommunication
public. Speech-impaired individuals use sign language, but engineering. IEEE, 2021.
other people frequently speak; as a result, these two groups
9. Mittal, Anshul, et al. "A modified LSTM model for continuous
of people communicate using different languages.
sign language recognition using leap motion."IEEE Sensors
Background: People with speech disabilities cannot learn Journal 19.16 (2019): 7056-7063.
spoken languages, and most people find it challenging to
learn sign language. It should be mentioned that different 10. Sonare, Babita, et al. "Video-based sign language translation
racial and geographic groups have distinct sign language systemusing machine learning."2021 2nd International
dialects. that not all people Conference for Emerging Technology (INCET).IEEE, 2021.

11. Wadhawan, Ankita, and Parteek Kumar. "Deep learning

based sign language recognition system for static signs."Neural
5

Authorized licensed use limited to: Access provided by VTU consortium. Downloaded on April 01,2026 at 10:03:03 UTC from IEEE Xplore. Restrictions apply.
 computing and applications 32 (2020):7957-7968.

12. Mallery,[Link] language among North American

Indians compared with that among other peoples and
[Link] de Gruyter GMbH & Co
KG,2019.

13. Rastgoo,razieh,Kourosh Kiani and Sergio

Escalera.”Sign language recognition: “A deep survey
Expert systems with applications 164(2021):113794.

14. Nautiyal Neha et al.”A dynamic gesture recognition

system for mute person.”Proceedings of the second
international conference on information Management
and machine Intelligence ICIMMI [Link]
Singapore, 2021.

Authorized licensed use limited to: Access provided by VTU consortium. Downloaded on April 01,2026 at 10:03:03 UTC from IEEE Xplore. Restrictions apply.