BHARATI VIDYAPEETH DEEMED TO BE UNIVERSITY
DEPARTMENT OF ENGINEERING AND TECHNOLOGY, NAVI MUMBAI
CAMPUS
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
HAND GESTURE RECOGNITION
PROJECT MEMBERS:
SANIKA KHANKALE
BHASKAR CHAURASIA
ROHIT MAJUMDER
ADITYA TYAGI
GUIDE:
PROF. DATTA DESHMUKH
INTRODUCTION
Human beings have, since the beginning of time, been described as a
social animal. As a social being, one of the principal aspects of our life
is communication. Social interaction or simply communication has
always been regarded as one of the major aspects of living a happy life.
For an individual to live a normal lifestyle, communication is necessary
and is required for almost all of our daily tasks. But there is a not so
blessed segment of society which faces hearing and vocal disabilities. A
hearing-impaired individual is one who either can’t hear at all or is able
to hear sounds which are above a certain frequency. An individual with
the inability to speak due to any reason whatsoever is considered a
mute or silent person.
So here we have built a project which will help the disabled person to
communicate better by using “sign language”.
INDIAN SIGN LANGUAGE
•History
● ISL uses both hands similar to
British Sign Language and is
similar to International Sign
Language.
● ISL alphabets derived from British
Sign
• Language and French Sign
Language alphabets.
● Unlike its american counterpart
which uses one hand, uses both
hands to represent alphabets.
American Sign Indian Sign
Language Language
LITERATURE REVIEW
Sr. No. Paper Title[Year of Technology Advantages Limitations
publication] used
1 HSRS using KNN [2011] KNN High Accuracy Very Inefficient
2 ISLRS using CNN [2021] CNN Accuracy Requires Huge
And Efficiency Training Data
3 ISLRS using SURF with SURF based Very Robust Requires Huge
SVM and CNN [2022] SVM and CNN Training Data
PROBLEM STATEMENT
Despite the richness and effectiveness of sign language as a mode
of communication for the hearing-impaired community, the lack of
widespread understanding and interpretation by non-signers
poses significant barriers in effective communication.
The primary problem addressed by this project revolves around
the development of an accurate, adaptable, and real-time hand
sign language recognition system capable of overcoming the
challenges posed by variations in gestures, environmental factors,
and different signing styles.
The system aims to bridge the communication gap between signers
and non-signers, thereby fostering inclusivity, enhancing
accessibility, and enabling effective communication across diverse
settings.
OBJECTIVES
The primary objective of this project is to explore the efficacy and potential
advantages of employing Graph Convolutional Networks (GCNs) in contrast to
traditional Convolutional Neural Networks (CNNs) for hand sign language
recognition.
Hand gesture recognition technology uses sensors or web camera to
read and interpret hand movements as commands.
The objective of hand gesture recognition for mute and deaf people is
to provide an alternative means of communication for those who are
unable to speak or hear.
This technology allows drivers and passengers to interact with the
vehicle, usually to control the infotainment system without touching any
EXISTING SYSTEM
1. Indian Sign Language Recognition System using SURF(Speeded-Up Robust
Features) with SVM and CNN [2022]
By: Uma Shanker Tiwary
Shagun Katoch
Varsha Singh
2. Sign Language Recognition : High Performance Deep Learning Approach Applied
To Multiple Sign Languages[2022]
By: Nabil Benaya
El Zaar
El allati
LEARNING
Frame Extraction
Skin Segmentation
Feature Extraction
Training and Testing
SKIN SEGMENTATION
Initial Approaches
● Training on skin segmentation dataset
Tried machine learning models like SVM, random forests on the skin
segmentation dataset from
[Link]
Very bad dataset, after training on around 2,00,000 points, skin segmentation
of hand images gave back almost black image(i.e. almost no skin detection)
● HSV model and constraints on values of H and S
Convert Image from RGB to HSV model and retain pixels satisfying 25<H<230 and
25<S<230
This implementation wasn’t much effective and the authors in the report had
used it along with motion segmentation which made their segmentation slightly
better.
SKIN SEGMENTATION
Final Approach
In this approach, we transform the image from RGB space to YIQ and YUV space.
From U and V, we get theta=tan-1(V/U). In the original approach, the author
classified skin pixels as those with 30<I<100 and 105o<theta<150o .
Since those parameters weren’t working that good for us, we somewhat tweaked
the
parameters and it performed much better than the previous two approaches.
BAG OF VISUAL WORDS
Bag of Words approach
In BoW approach for text
classification,a document is
represented as a
bag(multiset) of its words.
In Bag of Visual Words ,we
use the BOW approach for
image classification,
whereby every image is
treated as a document.
So now “words” need to be
defined for the image
also.
Result obtained For Bag of
Visual Words
We took 25 images per alphabet from 3 person each for training and 25
images per alphabet from another person for testing.
So training over 1950 images, we tested for 650 images and obtained
the following results :-
Train Set Size Test Set Size Correctly Classified Accuracy
1950 650 220 33.84%
Result obtained For Bag of
Visual Words
Observations
● Similar looking alphabets misclassified amongst each other
● One of the persons among the 3 persons was left handed and gave
laterally inverted images for many alphabets.
PROPOSED SYSTEM
Indian Sign Language Recognition System using GCNs(Graphical
Convolutional Networks)
An end-to-end deep neural network, hand gesture graph
convolutional network, is presented in which the convolution is
conducted only on linked skeleton joints.
Since the training dataset is relatively small, this work proposes
expanding the coordinate dimensionality so as to let models learn
more semantic features.
The proposed method is validated on two challenging datasets,
and the experimental results show that it outperforms the state-
of-the-art methods.
Furthermore, it is relatively lightweight in practice for hand
skeleton-based gesture recognition.
HAND GESTURE GRAPH
CONVNET
CONCLUSION
The exploration of Graph Convolutional Networks (GCNs) and Convolutional Neural
Networks (CNNs) for hand sign recognition has provided valuable insights into their
respective strengths and suitability for this specific task.
Throughout this study, both architectures were evaluated in the context of their
applicability to hand sign recognition, considering factors such as model
performance, data representation, and computational efficiency.
CNNs, known for their efficacy in extracting spatial features from images, have
traditionally been the go-to architecture for image-based tasks like sign language
recognition. Their ability to capture spatial hierarchies within pixel data remains a
robust approach, especially when working with image-centric datasets.
On the other hand, Graph Convolutional Networks (GCNs) showcase promising
potential in scenarios where the data naturally lends itself to a graph structure. For
hand sign recognition, if hand poses can be effectively modeled as graphs GCNs have
shown the ability to capture intricate spatial relationships between these key points,
potentially offering advantages in understanding complex hand gestures and their
spatial dependencies.
REFERENC
ES
[Link]://[Link]/~cipolla/lectures/PartIB
/old/[Link]
[Link]://[Link]/shackenberg/Minimal-Bag-of-Visual-Words-Image-
Classifier/blob/master/[Link]
[Link]://[Link]/wiki/YIQ
[Link]://[Link]/wiki/YUV
[Link]://[Link]/proj2011/ChenSenguptaSundaram-
SignLanguageGestureRecognitionWithUnsupervisedFeatureLearni
[Link]
[Link]://[Link]/wiki/Bag-of-words_model
[Link] V. Tavari, P. A. V. D.,Indian sign language recognition
based on histograms of oriented gradient,International Journal
of Computer Science and Information Technologies 5, 3 (2014),
3657-3660
THANK YOU