AMERICAN SIGN LANGUAGE

RECOGNITION

TEAM MEMBERS :
PRAVEEN KARUPPASAMY S - 2022510301
KARTHIKEYAN U - 2022510311
SRINIVASAN S - 2022510305

1. Introduction or speech. Today Computer Vision and Deep

Learning have gained a lot of popularity and many
ABSTRACT: State of the Art (SOTA) models can be built. Using
Deep Learning algorithms and Image Processing we
Sign Language is mainly used by deaf (hard hearing) can able to classify these hand gestures and able to
and dumb people to exchange information between produce corresponding text. An example of “0”
their own community and with other people. It is a numerical in sign language notion to “0” text or
language where people use their hand gestures to speech.
communicate as they can’t speak or hear. Sign
Language Recognition (SLR) deals with recognizing
the hand gestures acquisition and continues till text
or speech is generated for corresponding hand
gestures. Here hand gestures for sign language can
be classified as static and dynamic. However, static
hand gesture recognition is simpler than dynamic
hand gesture recognition, but both recognition is
important to the human community. We can use
Deep Learning Computer Vision to recognize the
hand gestures by building Deep Neural Network
architectures (Convolution Neural Network Sign Language Hand Gestures
Architectures) where the model will learn to
recognize the hand gestures images over an epoch. In Deep Learning Convolution Neural Networks
Once the model successfully recognizes the gesture (CNN) is the most popular neural network algorithm
the corresponding English text is generated and then which is a widely used algorithm for Image/Video
text can be converted to speech. This model will be tasks. For Convolution Neural Networks (CNN) we
more efficient and hence communicate for the deaf have advanced architectures like LeNET-5, and
(hard hearing) and dump people will be easier. In this MobileNetV2 where we can use these architectures
paper, we will discuss how Sign Language to achieve the State of the Art (SOTA). We can use
Recognition is done using deep all these architectures and combine them using
neural network ensemble techniques. By this, we can
Learning.
achieve an almost 100% accurate model which will
Index words: Hand Gestures; Sign Language recognize the hand gestures. This model will be
Recognition; Convolution Neural Networks; deployed in web frameworks like Django or a
Computer Vision; Text to Speech. standalone application or embedded devices where
300 different types of Sign Language used around the hand gestures are recognized in the live camera
globally today. In India, there are only about 250 and then converting them to text. This system will
certified sign language interpreters for a deaf help deaf and dumb people to communicate easily.
population of around 7 million. This would be a
problem to teach sign language to the deaf and dumb
people as there is a limited number of sign language
interpreter’s exits today. Sign Language
Recognition is an attempt to recognize these hand
gestures and convert them to the corresponding text

1
 significantly different appearances for different
signers and different viewpoints. This work focuses
on the creation of a static sign language translator by
using a Convolutional Neural Network. We created
a lightweight network that can be used with
embedded devices/standalone applications/web
applications having fewer resources.

Objectives

The main objectives of this project are to contribute

to the field of automatic sign language recognition
and translation to text or speech. In our project, we
focus on static sign language hand gestures. This
work focused on recognizing the hand gestures 10
Convolution Neural Networks
digits (0-9) using Deep Neural Networks (DNN).
Motivation We created a convolution neural networks classifier
that can classify the hand gestures into English
The various advantages of building a Sign Language alphabets and digits. We have trained the neural
Recognition system includes: network under different configurations and
Sign Language hand gestures to architectures like LeNet-5
text/speech translation systems or dialog MobileNetV2 , and our own architecture. We used
systems which are used in specific public the horizontal voting ensemble technique to achieve
domains such as airports, post offices, or the maximum accuracy of the model. We have also
hospitals. created a web application using Django Rest
Sign Language Recognition (SLR) can Frameworks to test our results from a live camera.
help to translate the video to text or speech
enables inter-communication between LITERATURE REVIEW
normal and deaf people.
Real-time sign language fingerspelling
Problem Statement recognition using convolutional neural networks
from depth map.
Sign language uses lots of gestures so that it looks
like movement language which consists of a series This works focuses on static fingerspelling in
of hands and arms motions. For different countries, American Sign Language 0 method for
there are different sign languages and hand gestures. implementing a sign language to text/voice
Also, it is noted that some unknown words are conversion system without using handheld gloves
translated by simply showing gestures for each and sensors, by capturing the gesture continuously
alphabet in the word. In addition, sign language also and converting them to voice.
includes specific gestures to each alphabet in the In this method, only a few images were captured for
English dictionary and for each number between 0 recognition. The design of a communication aid for
and 9. Based on these sign languages are made up the physically challenged.
of two groups, namely static gesture, and dynamic
gesture. The static gesture is used for alphabet and Design of a communication aid for physically
number representation, whereas the dynamic challenged.
gesture is used for specific concepts. Dynamic also
includes words, sentences, etc. The static gesture The system was developed under the MATLAB
consists of hand gestures, whereas the latter includes environment. It consists of mainly two phases via
motion of hands, head, or both. Sign language is a training phase and the testing phase. In the training
visual language and consists of 3 major components, phase, the author used feed-forward neural networks.
such as finger-spelling, word-level sign vocabulary, The problem here is MATLAB is not that efficient
and non-manual features. Finger-spelling is used to and also integrating the concurrent attributes as a
spell words letter by letter and convey the message whole is difficult.
whereas the latter is keyword-based. But the design American Sign Language Interpreter System for
of a sign language translator is quite challenging Deaf and Dumb Individuals.
despite many research efforts during the last few
decades. Also, even the same signs have

2
The numerical 0, 1, 4, couldn’t be recognized due to Sample Image without Pre-processing
the occlusion problem. They have used only a
limited number of images. The steps we have taken for image Pre-processing
are:
IMPLEMENTATION
Read Images.
Dataset Resize or reshape all the images to the same
✓ Remove noise.
We have used multiple datasets and trained multiple
All the image pixels arrays are converted to
models to achieve good accuracy.
0 to 255 by dividing the image array by 255.
ASL Alphabet

The data is a collection of images of the alphabet

from the American Sign Language, separated into 29
folders that represent the various classes.

The training dataset consists of 87000 images which

are 200x200 pixels. There are 29 classes of which 26
are English alphabets A-Z and the rest 3 classes are
SPACE, DELETE, and, NOTHING. These 3 classes
are very important and helpful in real-time
applications.

Sign Language Gesture Images Dataset

The dataset consists of 37 different hand sign

gestures which include A-Z alphabet gestures, 0-9
number gestures, and also a gesture for space which Pre- processed image
means how the deaf (hard hearing) and dumb people Convolution Neural Networks (CNN)
represent space between two letters or two words
while communicating. Computer Vision is a field of Artificial Intelligence
that focuses on problems related to images and
Each gesture has 1500 images which are 50x50 videos. CNN combined with Computer vision is
pixels, so altogether there are 37 gestures which capable of performing complex problems.
means there 55,500 images for all gestures.
Convolutional Neural Network (CNN) is well suited
for this dataset for model training purposes and
gesture prediction.

Data Pre-processing
An image is nothing more than a 2-dimensional array
of numbers or pixels which are ranging from 0 to
255. Typically, 0 means black, and 255 means white.
Image is defined by mathematical function f(x, y)
where ‘x’ represents horizontal and ‘y’ represents
vertical in a coordinate plane. The value of f(x, y) at
any point is giving the pixel value at that point of an
image.

Image Pre-processing is the use of algorithms to

perform operations on images. It is important to
Preprocess the images before sending the images for
model training. For example, all the images should
have the same size of 200x200 pixels. If not, the
model cannot be trained.

3
 Working of CNN
The Convolution Neural Networks has two main The size of the output matrix decreases as we keep
phases namely feature extraction and classification. on applying the filters.
A series of convolution and pooling operations are
performed to extract the features of the image. Size of new matrix = (Size of old matrix — filter
size) +1

A fully connected layer in the convolution neural Convolution

networks will serve as a classifier. In the last layer,
the probability of the class will be predicted. The
main steps involved in convolution neural networks
are:

1. Convolution
2. Pooling
3. Flatten
4. Full connection

Convolution
Feature Extraction
Convolution is nothing but a filter applied to an
image to extract the features from it. We will use Pooling
different filters to extract features like edges, After the convolution operation, the pooling layer
highlighted patterns in an image. The filters will be will be applied. The pooling layer is used to reduce
randomly generated. the size of the image. There are two types of
What this convolution does is, creates a filter of pooling:
some size says 3x3 which is the default size. After 1. Max Pooling
creating the filter, it starts performing the 2. Average Pooling
elementwise multiplication starting from the top left
corner of the image to the bottom right of the image. Max pooling
The obtained results will be extracted feature.
Max pooling is nothing but selecting the maximum pixel
value from the matrix.

4
 Max Pooling Full Connection

This method is helpful to extract the features with A fully connected layer is simply a feed-forward
high importance or which are highlighted in the neural network. All the operations will be
image. performed and prediction is obtained. Based on the
ground truth the loss will be calculated and weights
Average pooling are updated using gradient descent backpropagation
Unlike Max pooling, the average pooling will take algorithm.
average values of the pixel Convolution Neural Network (CNN)
. Architectures

LeNet-5

Average Pooling

In most cases, max pooling is used because its

performance is much better than average pooling.

Flatten

Flatten
LeNet-5 Implementation
The obtained resultant matrix will be in
multidimensional. Flattening is converting the data The LeNet-5 [2] architecture consists of two pairs
into a 1dimensional array for inputting the layer to of convolutional and average pooling layers,
the next layer. We flatten the convolution layers to followed by a flattening convolutional layer, then
create a single feature vector. two fully connected layers, and finally a Softback
classifier.
Full Connection
MobileNetV2

5
 ensemble techniques for Neural Networks like
Stacked generalization [8], Ensemble learning via
MobileNetV2 Implementation negative correlation [9] and, Probabilistic Modelling
MobileNetV2 [3] is a convolutional neural network with Neural Networks [10] [11]. We have
architecture that performs well on mobile devices. implemented the Horizontal Voting Ensemble
The architecture of MobileNetV2 contains the fully method to improve the performance of neural
convolution layer with 32 filters, followed by 19 networks.
residual bottleneck layers. This network is Horizontal voting is an ensemble technique for
lightweight and efficient Own Architecture neural networks where we train several sub-models
and make predictions using these sub-models. For
the final predictions, we make predictions from all
the sub-models and see which class has got
maximum votes. The final prediction will be the
class that has the maximum votes. For this, we have
used 3 models that are an odd number of sub-
models to avoid an even number of votes for two
classes in worst cases.

Let model be the set of neural network models

being trained on the training set T(xi,yi), such that
m ∈ model. Let yhat be the predictions obtained by
all the models on the test set T’(xi’,yi’).

Let ‘array’ be the function for converting lists to arrays

Algorithm 1: Horizontal Voting

Input: models, test set T’(xi’,yi’), and empty yhat list

Output: predictions – final prediction obtained

1. Step 1: Obtain the predictions of each model

2. for each i in range(xi)
Own Architecture Implementation 3. for each m in model, do
yhat[i] ← predict(x[i])
In our own architecture, we have implemented 3 Calculate highest number of
convolution layers followed by batch normalization votes for ith test data and append
and max pooling, followed by global average yhat[i] ← highest voted class
pooling with dense layer and batch normalization, end for 4. end for
and a final dense layer for classification. 5. Step 2: Convert list into an array
6. yhat ← array(yhat)
Proposed Model
7. Step 3: return yhat
We have trained 3 models with 2 different datasets
to perform well on unseen datasets. We have trained
LeNet-5, MobineNetV2 and, our own architectures. For MobileNetV2 model we have taken Adam
We have not taken the best model out of 3 we have optimizer with learning rate = 0.001, eta_1=0.9,
taken all 3 models and made a final model that will beta_2=0.999, and epsilon=1e-07
perform an ensemble of these 3 models. For all the models while training we have used
Neural Network Ensemble Horizontal Voting ReduceLROnPlateau callback with factor = 0.2,
patience = 2, min_lr = 0.001.
In Machine Learning we have an ensemble
technique where we train multiple sub-models and By using this horizontal ensemble technique, we have
average them. Random Forest algorithm is an achieved 99.8% accuracy.
example where it uses multiple Decision tree
algorithms. Similarly, we can perform ensemble for
Neural Networks [4] as well. There are a lot of

6
 We have deployed all the models in Django Web images, making it well-suited for complex visual
Frameworks and built a simple frontend to accept patterns.
the image from users and send the response.

We have also built an API that will pop up the live

camera and detects the hand gestures and then
converts them to the corresponding English
alphabets.

ELU Model:

The ELU Model employs a Convolutional Neural

Network (CNN) architecture, featuring Exponential
Linear Unit (ELU) activation functions. The model's
structure includes convolutional layers for feature
extraction, max-pooling layers for spatial
downsampling, and fully connected layers for
classification. This design is tailored to capture intricate
spatial features present in ASL gestures.

EXPERIMENTAL RESULTS

Training graphs

We have trained all the models for around 10-15

epochs with a batch size of 32.

VGG MODEL:
We have used Django Rest Frameworks as a
backend for our project. This is the sample JSON
The VGG Model is inspired by the VGGNet
response that is sent to the frontend when a user
architecture, renowned for its simplicity and
inputs an Image.
effectiveness. It comprises multiple convolutional
layers characterized by small filter sizes. This model
focuses on learning hierarchical features in ASL sign

7
 COMPARISON TABLE:
Metrics Convoluti Visual Exponenti
on Neural Geomet al linear
Network ry unit
(CNN) Group (ELU)
(VGG)
Accuracy 98.61 % 97.77 % 97.77 %

Linear 0.8958 0.8521 0.8889

Regressi
on
Logistic 0.8287 0.8012 0.8091
Regressi
on
Precision 0.8761 0.8214 0.8931

Recall 0.8724 0.8278 0.8915

Predicted sign 5

CONCLUSION
In conclusion, we were successfully able to develop
a practical and meaningful system that can able to
understand sign language and translate that to the
corresponding text. There are still many shortages of
our system like this system can detect 0-9 digits and
A-Z alphabets hand gestures but doesn’t cover body
gestures and other dynamic gestures. We are sure
and it can be improved and optimized in the future.

REFERENCES
[1] Brill R. 1986. The Conference of
Model Prediction on Live Camera Prediction: Educational Administrators Serving the Deaf: A
“0” History. Washington, DC: Gallaudet University
Press.

Y. Lecun, L. Bottou, Y. Bengio and P. Haffner,

"Gradient-based learning applied to document
recognition," in Proceedings of the IEEE, vol. 86,
no. 11, pp. 2278-2324, Nov. 1998, doi:
10.1109/5.726791.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov

and L. Chen, "MobileNetV2: Inverted Residuals
and Linear Bottlenecks," 2018 IEEE/CVF
Conference on Computer Vision and Pattern
Recognition, 2018, pp. 4510-4520, doi:
10.1109/CVPR.2018.00474.

L. K. Hansen and P. Salamon, "Neural network

ensembles," in IEEE Transactions on Pattern
Model Prediction on Live Camera Prediction:
Analysis and Machine Intelligence, vol. 12, no. 10,
“1” pp. 993-1001, Oct. 1990, doi: 10.1109/34.58871.

8
Kang, Byeongkeun, Subarna Tripathi, and Truong
Q. Nguyen. ”Real- time sign language
fingerspelling recognition using convolutional
neural networks from depth map.” arXiv preprint
arXiv: 1509.03001 (2015).

Suganya, R., and T. Meeradevi. ”Design of a

communication aid for phys- ically challenged.” In
Electronics and Communication Systems (ICECS),
2015 2nd International Conference on, pp. 818-
822. IEEE, 2015.

Sruthi Upendran, Thamizharasi. A,” American

Sign Language Interpreter System for Deaf and
Dumb Individuals”, 2014 International Conference
on Control, Instrumentation, Communication and
Computa.

David H. Wolpert, Stacked generalization, Neural

Networks, Volume 5, Issue 2, 1992, Pages 241-
259, ISSN 0893-6080,
[Link]

Y. Liu, X. Yao, Ensemble learning via negative

correlation, Neural Networks, Volume 12, Issue
10,1999, Pages 1399-1404, ISSN 0893-6080,
[Link]

MacKay D.J.C. (1995) Developments in

Probabilistic Modelling with Neural Networks —
Ensemble Learning. In: Kappen B., Gielen S. (eds)
Neural Networks: Artificial Intelligence and
Industrial Applications. Springer, London.
[Link]

Polikar R. (2012) Ensemble Learning. In: Zhang

C., Ma Y. (eds) Ensemble Machine Learning.

Springer, Boston, MA. [Link]

4419-9326-7_1