Hand Sign Recognition

Submitted in partial fulfillment of the requirements of

Third Year

in
B.E.(Artificial Intelligence and Data Science)

By

Kunjir Kartik Gajanan (53)

Chavan Kalpesh Sahebrao (65)
Koli Pankaj Shriram (66)
Tajne Vighnesh Sunil (67)

Supervisor
Prof. Mrs. Shubhangi Katke

Department of Artificial Intelligence and

Data Science
DATTA MEGHE COLLEGE OF ENGINEERING, AIROLI,

NAVI MUMBAI - 400 708

University of Mumbai

(AY 2025-26)
 CERTIFICATE

This is to certify that the Mini Project 2A entitled “Hand Sign Recognition” is a

bonafide work of Kartik Kunjir (53), Kalpesh Chavan (65), Pankaj Koli (66),

Vighnesh Tajne (67) Submitted to the University of Mumbai in partial fulfillment

of the requirement for the award of the degree of “Bachelor of Engineering” in

“Artificial Intelligence and Data Science”.

Prof. Mrs. Shubhangi Katke

Supervisor

Dr. S. M. Patil Dr. P. A. Dode

Head of Department Principal
 Mini Project Approval

This Mini Project entitled “Hand Sign Recognition” by Kartik Kunjir (53), Kalpesh Chavan (65),

Pankaj Koli (66), Vighnesh Tajne (67) is approved for the degree of Bachelor of Engineering in

Artificial Intelligence and Data Science.

Examiners

1………………………………………
(Internal Examiner Name & Sign)

2…………………………………………
(External Examiner name & Sign)

Date:

Place:
DECLARATION

I declare that this project represents my ideas in my own words without plagiarism and
wherever others' ideas or words have been included, I have adequately cited and referenced the
original sources. I also declare that I have adhered to all principles of academic honesty and
integrity and have not misrepresented or fabricated or falsified any idea/data/fact/source in my
project work. I promise to maintain minimum 75% attendance, as per the University of Mumbai
norms. I understand that any violation of the above will be cause for disciplinary action by the
Institute.

Yours Faithfully

1. Kartik Gajanan Kunjir

2. Kalpesh Sahebrao Chavan

3. Pankaj Shriram Koli

4. Vighnesh Sunil Tajne

(Date & Signature of Students)

Contents
Abstract ii

Acknowledgments iii

List of Abbreviations iv

List of Figures v

Introduction
1. 1
1.1 Introduction
1.2 Problem Statement & Objectives
1.3 Organization of the Report
2 Literature Survey 4

2.1 Survey of Existing System

3 System Design 6
3.1 Architecture/ Framework
3.2 Algorithm and Process Design
3.3 Software Stack
4. Implementation and Results 12
4.1 System Implementation
4.2 Results

5. Conclusion and Future Scope 17

References 18

i
Abstract
This project presents a real-time Hand Sign Recognition System designed to facilitate inclusive
communication between sign language users and non-signers. Utilizing a lightweight CNN
optimized for edge computing, the system achieves high-accuracy ASL detection on both Web
and Android platforms, converting hand gestures into stable text strings. To support global
accessibility, an integrated AI agent translates this output into multiple languages and provides
Text-to-Speech feedback. Crucially, the system enables reverse communication through a "Text-
to-Sign" module, which dynamically synthesizes sign language videos from textual input. By
combining computer vision with advanced API integrations, this solution offers a
comprehensive, bidirectional tool to eliminate language barriers for the hearing-impaired
community.

ii
Acknowledgement
We would like to express our sincere gratitude to Mrs. Shubhangi Katke, our
project supervisor, for her invaluable guidance throughout this project. We
also thank the Department of Artificial Intelligence and Data Science at Datta
Meghe College of Engineering for providing us with this opportunity to
explore the e-commerce website. Special thanks go to our peers and families
for their continuous support.

iii
List of Abbreviations

• ASL: American Sign Language

• BSL: British Sign Language

• CNN: Convolutional Neural Network

• CPU: Central Processing Unit

• CSLR: Continuous Sign Language Recognition

• CSS: Cascading Style Sheets

• CVPR: Conference on Computer Vision and Pattern Recognition

• DHH: Deaf and Hard-of-Hearing

• GANs: Generative Adversarial Networks

• GCN: Graph Convolutional Network

• HTML: HyperText Markup Language

• ISL: Indian Sign Language

• LSTM: Long Short-Term Memory

• NLP: Natural Language Processing

• ONNX: Open Neural Network Exchange

• PSNR: Peak Signal-to-Noise Ratio

• SLAvatar: Sign Language Avatar

• SLR: Sign Language Recognition

• SSIM: Structural Similarity Index Measure

• TFLite: TensorFlow Lite

• TPAMI: Transactions on Pattern Analysis and Machine Intelligence

• TTS: Text to Speech

iv
List of Figures

Figure 1: System Architecture

Figure 2: Software Stack

Figure 3: Login and Register

Figure 4: Single Hand Sign Recognition 1 (Sign A)

Figure 5: Single Hand Sign Recognition 2 (Sign C)

Figure 6: Text to sign generator (Generating text sign video from existing dataset)

Figure 7: Continuous hand signs recognition

v
 1. INTRODUCTION

1.1 Introduction

The Hand Sign Recognition is developed to bridge the communication gap between sign
language users and individuals who are not familiar with it. This application uses advanced
machine learning and computer vision techniques to recognize hand gestures and convert
them into text or speech in real time. It also supports two-way interaction by displaying
corresponding signs for spoken or written words, enabling smooth and natural conversations.
Designed with an easy-to-use interface and clear visuals, the system ensures accessibility for
users of all ages and abilities. This technology aims to assist hearing-impaired individuals, their
families, educators, interpreters, and healthcare professionals in improving daily
communication and fostering inclusivity in society.

Recent advancements in deep learning have significantly improved the accuracy and
efficiency of sign language recognition systems. For instance, transfer learning using Inflated
3D Convolutional Neural Networks (I3D) enables effective extraction of spatio-temporal
features from video sequences, enhancing gesture recognition performance in real-time
applications[1].

➢ The application is designed to reduce the communication barrier between sign language
users and people unfamiliar with it, making conversations smoother and more natural in
daily life.

➢ It can recognize sign language gestures and convert them into text or speech, and also
display signs for spoken or written words, ensuring two-way communication.

➢ Useful for hearing-impaired individuals, their family members, teachers, interpreters,

healthcare workers, and anyone interested in learning sign language.

➢ The system leverages advanced deep learning models such as 3D CNNs, Transformers,
and Graph Neural Networks to improve gesture recognition accuracy and robustness
across different environments[1][4][7].

➢ Designed with a simple, user-friendly interface and clear visuals so people of all ages and
abilities can use it without difficulty.

➢ Incorporates continuous learning and scalable approaches such as zero-shot learning to

recognize new gestures without retraining the entire system[6].

➢ Utilizes large-scale datasets and benchmark models to ensure high performance,

accuracy, and real-world usability[3].

1
1.2 Problem Statement

This project addresses the communication gap between the hearing-impaired and the general
public by developing an AI-powered, bidirectional Web and Android system that translates ASL
into multilingual text and speech, and converts text back into sign-language videos for seamless
two-way interaction.

2
1.3 Organisation of the report

This report is divided into several sections:

● The Introduction outlines the problem and objectives.

● The Literature Survey reviews existing solutions for teacher allocation.

● The System Design section provides details on the architecture, framework, algorithm,
and process design.

● The Implementation Results showcase the output of the system.

● The report concludes with Future Scope and References.

3
 2. LITERATURE SURVEY

2.1 Survey of existing system

1) Title: Deep Learning for Sign Language Recognition: Exploring VGG16 and ResNet50
Capabilities
➢ Authors: Sharma, J., Gill, K.S., Kumar, M., Rawat, R.
➢ Year: 2024
➢ Journal: Applied Intelligence and Computing
➢ Important Findings: Compared the performance of VGG16 and ResNet50
architectures for sign language gesture recognition, achieving accuracy rates of 99.92%
and 99.95%, respectively, highlighting the effectiveness of deep learning models in sign
language recognition tasks.
Additionally, the study found that ResNet50 performed slightly better due to its deeper
architecture and residual connections, which help in avoiding vanishing gradient
problems and improving feature learning. The models demonstrated strong
generalization capability across different gesture variations and lighting conditions.
Furthermore, the use of transfer learning significantly reduced training time while
maintaining high accuracy, making these models suitable for real-time sign language
recognition applications.

2) Title: Deep Learning Approach for Combined Indian Sign Language Recognition and Video
Generation Model
➢ Authors: Waghmare, P.P., Deshpande, A.M., Dubewar, S., Dhaybar, T.
➢ Year: 2024
➢ Journal: International Journal of Intelligent Systems and Applications in Engineering
➢ Important Findings: Introduced a system combining CNN and GANs for Indian Sign
Language recognition and video generation. Achieved 98% accuracy in sign recognition
and high-quality video generation with PSNR of 31.14 dB and SSIM of 0.9916,
enhancing communication accessibility.
The integration of Generative Adversarial Networks (GANs) enabled realistic and
smooth video synthesis of sign gestures, improving visual clarity for users. The model
demonstrated efficient performance in both recognition and generation tasks, making it
suitable for real-time communication systems. Additionally, the approach supports
bidirectional interaction by not only recognizing signs but also generating corresponding
gesture videos from text, thus improving usability for both hearing and non-hearing
individuals.

3) Title: Sign Language Recognition Using CNN–LSTM Architecture for Dynamic Gesture
Classification
➢ Authors: Koller, O., Camgoz, N.C., Ney, H., Bowden, R.
➢ Year: 2019
➢ Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence

4
 ➢ Important Findings: Proposed a hybrid deep learning model combining CNN for
spatial feature extraction and LSTM for temporal sequence modeling in continuous sign
language recognition. The system effectively captured dynamic gestures and achieved
over 95% accuracy on benchmark datasets, highlighting the importance of temporal
modeling in real-time recognition systems.
The study demonstrated that integrating spatial and temporal features significantly
improves recognition performance for continuous and complex sign sequences. The
LSTM component effectively handled variations in gesture speed and sequence length,
making the model robust in real-world scenarios. Additionally, the approach reduced
misclassification in similar-looking gestures by considering contextual motion
information, thereby enhancing overall system reliability.

4) Title: Real-Time American Sign Language Recognition Using MediaPipe and Deep Learning
➢ Authors: Rastgoo, R., Kiani, K., Escalera, S.
➢ Year: 2021
➢ Journal: Sensors
➢ Important Findings: Developed a real-time sign language recognition system using
MediaPipe for hand landmark detection and deep learning classifiers like CNN and MLP.
The model achieved 98% accuracy on static ASL alphabets while maintaining low
computational complexity, making it suitable for real-time and embedded applications.
The use of MediaPipe enabled efficient extraction of hand keypoints, reducing the need
for heavy image preprocessing and improving processing speed. The system
demonstrated robustness under varying lighting conditions and backgrounds due to its
reliance on landmark-based features. Additionally, the lightweight architecture allows
deployment on devices with limited resources such as smartphones and edge devices,
making it practical for everyday use.

5
 3. SYSTEM DESIGN
3.1 Architecture/framework
1. Image Upload Mode

Goal: Recognize sign language from a single uploaded image.

• Upload Image:
→ Select an image containing a hand gesture.
→ The system supports common image formats such as JPG, PNG, and JPEG.
→ Image preprocessing like resizing and normalization is applied before processing.

• Upload Photo
→ Provide the input image to the system.
→ The uploaded image is passed to the trained deep learning model.
→ Background noise is minimized using preprocessing techniques.

• Predict
→ Run the model to recognize the sign.
→ A CNN-based model extracts spatial features from the image.
→ The model compares features with trained gesture datasets for classification.

• View Prediction & Confidence

→ Display detected sign with accuracy score.
→ The output includes the predicted label along with confidence percentage.
→ This helps users understand the reliability of the prediction result.

2. Real-time Webcam Mode

Goal: Detect sign language gestures live using a camera.

• Real-time Recognition
→ Activate live gesture detection.
→ The system continuously captures frames from the webcam.
→ Each frame is processed instantly for gesture recognition.

• Start
→ Begin capturing video from the camera.
→ Initializes the camera module and deep learning pipeline.
→ Ensures smooth frame acquisition for real-time processing.

• Allow Camera
→ Grant permission to access webcam.
→ User permission is required to ensure privacy and security.
→ The system only accesses the camera during active sessions.

6
 • Show Hand Signs
→ Perform gestures in front of camera.
→ The system detects hand regions using computer vision techniques.
→ It tracks movement and extracts key features from the hand.

• Predictions Stabilize
→ Ensure consistent detection over frames.
→ Multiple frame analysis reduces noise and improves accuracy.
→ Temporal smoothing techniques ensure stable predictions.

• Stop
→ End the live recognition process.
→ Stops video capture and releases camera resources.
→ Saves system resources and ensures safe termination of the session.

3. Text to Sign Video

Goal: Convert text input into sign language video output.

• Text to Sign
→ Convert written text into sign language.
→ The system processes text using NLP techniques.
→ It maps words to corresponding sign language gestures.

• Enter Text
→ Input the message to be converted.
→ Users can enter simple or complex sentences.
→ Text is preprocessed to remove unnecessary characters.

• Watch Predictions
→ Preview generated sign output.
→ Displays intermediate results before final video generation.
→ Helps users verify correctness before proceeding.

• Generate
→ Create the sign language video.
→ Uses deep learning models like GANs for realistic video generation.
→ Combines individual gestures into a continuous sequence.

• Download Video
→ Save the generated video file.
→ Users can download the video in standard formats like MP4.
→ Enables offline usage and easy sharing of generated content.

7
4. Translation

Goal: Translate text between different languages..

• Enter Text
→ Provide text for translation.
→ Users can input text in various supported languages.
→ The system validates input before processing.

• Select Languages
→ Choose source and target languages.
→ Supports multiple language pairs for flexibility.
→ Language selection ensures accurate translation output.

• Translate
→ Convert text into selected language.
→ Uses machine translation models for accurate conversion.
→ The translated text can be further used for sign generation.

Figure 1:- System Architecture

8
3.2 Algorithm and Process Design

An hand sign recognition (HSR) system is designed to bridge the communication gap between
hearing-impaired individuals and the general public by enabling real-time translation of sign
language into text or speech and vice versa. The system captures hand gestures through a camera,
processes the visual data, and uses deep learning techniques to recognize and interpret signs. It
ensures smooth, accurate, and user-friendly communication with continuous validation for real-time
performance.

1. Video Frame Capture

➢ The system captures real-time video input using a webcam or mobile camera and
divides it into multiple frames for further processing.
➢ Continuous frame capture ensures smooth tracking of hand movements.
➢ A higher frame rate helps in capturing fast and dynamic gestures accurately.
➢ Frames are stored temporarily in a buffer for sequential processing.

2. Pre-processing and Region Detection

➢ Each frame undergoes preprocessing to improve quality by reducing noise, adjusting
lighting, and enhancing contrast. Hand and facial regions are detected using tools
like MediaPipe and OpenCV to focus on relevant areas.
➢ Background removal techniques help isolate the hand region more effectively.
➢ Image normalization ensures consistency across different lighting conditions.
➢ Region detection reduces unnecessary data, improving processing speed.

3. Feature Extraction (Landmark Detection)

➢ Important landmarks such as finger joints, wrist positions, and facial key points are
extracted. These landmarks convert visual input into structured numerical data
representing gesture shape, position, and orientation.
➢ Landmark coordinates provide precise information about hand structure and
movement.
➢ This step reduces the complexity of raw image data into meaningful features.
➢ It improves recognition accuracy by focusing only on relevant gesture details.

4. Gesture Recognition using Deep Learning

➢ The extracted features are passed into a trained deep learning model (e.g.,
MobileNetV2 / CNN-based architecture). The model analyzes spatial and temporal
patterns to accurately classify the gesture into a specific sign, word, or sentence.
➢ Deep learning models learn complex patterns from large gesture datasets.

5. Output Translation (Bidirectional System)

➢ The recognized gesture is converted into:
➢ Text output displayed on the screen
➢ Speech output using text-to-speech
Additionally, the system can convert text input into sign language animations/videos,
enabling two-way communication.
➢ Text-to-speech modules convert recognized text into audible output..
➢ Sign generation modules create visual gestures from input text.
➢ This enables effective communication between deaf and non-deaf user

9
6. Real-Time Validation and User Interaction
➢ The system continuously validates predictions to improve accuracy and provides a
smooth, interactive experience through a simple and user-friendly interface.
➢ Prediction smoothing techniques reduce sudden fluctuations in output
➢ User feedback helps in improving system performance over time
➢ The interface is designed to be simple, responsive, and easy to use

10
3.2 Software Stack

The project uses Python for backend and model development, with HTML, CSS, and JavaScript
for an interactive frontend interface.
OpenCV and TensorFlow/PyTorch handle image processing and deep learning, while Flask and
tools like Jupyter, VS Code, and Docker support integration and deployment.

Figure 2:- Software Stack

11
 4. IMPLEMENTATION AND RESULTS
4.1 System Implementation

The implementation of a Hand Sign Recognition (HSR) system involves setting up a camera
feed for real-time gesture capture, applying advanced computer vision techniques for landmark
extraction, and using deep learning models for accurate translation. Key features include a real-
time display of the recognized text, support for various signer hand shapes and movements,
and a responsive interface for user interaction. This process facilitates inclusive communication
between the Deaf/Hard-of-Hearing community and the general public.

Frontend Implementation

The frontend implementation involves designing a responsive and intuitive user

interface using HTML, CSS, and JavaScript. The display typically includes a live video
feed window captured via a webcam, a separate text output box to show the real-time
translation of the recognized signs, and control buttons (like 'Start/Stop Recognition').
The layout is optimized to work across various devices. JavaScript is crucial for
handling the video stream, managing the communication with the vision processing
backend, and dynamically updating the translated text on the screen.

Data Processing
Data Processing for an HSR system is essential for preparing raw video data and
converting it into a usable format for the deep learning model. It involves :

➢ Video Segmentation
Continuous video is divided into frames for [Link] helps in processing each gesture
step-by-step in real time.
Frame extraction allows the system to analyze motion in a structured manner.
It helps in identifying the start and end of each gesture clearly.
This step is important for handling continuous sign language recognition

➢ Image Pre-processing
Frames are enhanced by noise reduction, resizing, and lighting [Link] improves
image quality and ensures consistent input for the model.
Techniques like filtering and smoothing remove unwanted distortions.
Image resizing ensures uniform input size for deep learning models.
Proper preprocessing improves overall recognition accuracy significantly.

➢ Feature Extraction
Key landmarks (hand and face points) are extracted using OpenCV/MediaPipe.
These landmarks represent the structure and movement of gestures.
Feature extraction reduces raw image data into meaningful information.
It captures both spatial position and movement of hand gestures.
This step plays a crucial role in improving model efficiency and speed.

12
➢ Normalization
Landmark coordinates are scaled to maintain [Link] makes the system
independent of distance, hand size, and position.
Normalization ensures uniformity across different users and environments.
It reduces variability caused by camera angle and user distance.
This helps the model generalize better and improves prediction accuracy.

13
4.2 Results

Figure 3:- Login and Register: The image displays the login and registration interface of the
SignSight AI system with fields for username and password. It features a simple, modern Design
with an “Access Platform” button for secure and easy system access.

Figure 4:- Single Hand Sign Recognition 1 (Sign A): The system detects and classifies a hand
gesture as the letter “A” with a confidence score of 83.6% in real [Link] interface shows the live
input, prediction results and provides options for interaction such as capture, testing, and speech
output.

14
.

Figure 5:- Single Hand Sign Recognition 2 (Sign C): The system detects and classifies a hand
gesture as the letter “C” with a confidence score of 74.8% in real [Link] interface shows the live
input, prediction results, and provides options for interaction such as capture, testing, and speech
output.

Figure 6:- Text to sign generator (Generating text sign video from existing dataset): The
image shows the Text-to-Sign Generator converting input text into corresponding sign language video
using a pre-existing dataset. It displays the generated gesture sequence visually, allowing users to
understand signs for the entered text.

15
Figure 7:Continuous Hand Signs Recognition: The image shows a live recognition system detecting
continuous hand gestures and identifyin the sign “C” with 100% confidence. It provides real-time
feedback along with controls for editing, clearing, and converting the recognized signs into speech.

16
 5. CONCLUSION AND FUTURE SCOPE
Conclusion

The sign language application is a powerful tool that helps eliminate communication barriers
between hearing-impaired individuals and those who do not understand sign language. Through
real-time gesture recognition and translation, it enables smooth, accurate, and inclusive
interactions in various social, educational, and professional settings. Its user-friendly design and
multi-language support make it accessible to people of all ages and backgrounds. Beyond
communication, the application also promotes awareness and learning of sign language,
encouraging more people to engage with it.

Future Scope

While the current system has successfully met its initial objectives, there are several areas for
enhancement and expansion:
.
1. Non-Manual Feature Integration: Enhance recognition accuracy by incorporating and
classifying non-manual cues, such as facial expressions, eye gaze, and body posture,
which are crucial for conveying grammar and emotion in sign language.
2. Multi-Language and Diverse Signer Models: Expand the system’s utility by training it on
various global sign languages (e.g., ASL, ISL, BSL) and increasing robustness to handle
diverse individual signing styles and speeds.
3. Mobile and Embedded Deployment: Optimize the algorithms and model size for
deployment on resource-constrained devices like smartphones, low-cost cameras, and
smart glasses for on-the-go portability.
4. Gamified Learning Platform: Utilize the system's recognition capability to create
interactive applications that provide real-time feedback for non-signers learning sign
language, acting as a personal virtual instructor.
5. Context-Aware Translation: Introduce Natural Language Processing (NLP) techniques to
infer context and translate recognized signs into grammatically correct and contextually
appropriate sentences, not just a sequence of words

17
References

1. R. Töngi, “Application of Transfer Learning to Sign Language Recognition using an Inflated 3D

Deep Convolutional Neural Network,” arXiv preprint arXiv:2103.05111, 2021.

2. Y. C. Bilge, R. G. Cinbis, and N. Ikizler-Cinbis, “Towards Zero-shot Sign Language

Recognition,” arXiv preprint arXiv:2201.05914, 2022.

3. O. M. Sincan, J. C. S. Jacques Junior, S. Escalera, and H. Y. Keles, “ChaLearn LAP Large Scale
Signer Independent Isolated Sign Language Recognition Challenge: Design, Results and Future
Research,” arXiv preprint arXiv:2105.05066, 2021.

4. H. Hu, W. Zhao, W. Zhou, Y. Wang, and H. Li, “SignBERT: Pre-Training of Hand-Model-Aware

Representation for Sign Language Recognition,” arXiv preprint arXiv:2110.05382, 2021.

5. Huang, J., Zhou, W., Zhao, H., Chen, X., Li, J., Li, H., & Wang, Q. (2022). TSM: Temporal
Segment Networks for Action Recognition in Sign Language Videos. IEEE Transactions on
Pattern Analysis and Machine Intelligence (TPAMI), 44(11), 7724–7739.

6. Pu, J., Cheng, K., Huang, H., He, F., & Li, C. (2022). Multi-Granularity Representation Learning for
Continuous Sign Language Recognition. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 8414-8424.

7. Shi, C., Tang, J., Chen, T., & Wu, Y. (2021). Skeleton-Based Sign Language Recognition with
GCN and Multi-Head Attention. Neurocomputing, 456, 280–292.

18