DEEPLIPREAD: CNN–LSTM AND CNN–TRANSFORMER

BASED ENGLISH VISUAL SPEECH RECOGNITION

A PROJECT REPORT

Submitted by

GOPIKA S [313522243091]

HARINI P [313522243034]

APARNA LAKSHMI S V [313522243012]

in partial fulfillment for the award of the degree

of
BACHELOR OF TECHNOLOGY
in
ARTIFICIAL INTELLIGENCE & DATA SCIENCE

PANIMALAR ENGINEERING COLLEGE CHENNAI CITY CAMPUS

ANNA UNIVERSITY: CHENNAI 600 025

APRIL 2026
DEEPLIPREAD: CNN–LSTM AND CNN–TRANSFORMER
BASED ENGLISH VISUAL SPEECH RECOGNITION

A PROJECT REPORT

Submitted by

GOPIKA S [313522243091]

HARINI P [313522243034]

APARNA LAKSHMI S V [313522243012]

in partial fulfillment for the award of the degree

of
BACHELOR OF TECHNOLOGY
in
ARTIFICIAL INTELLIGENCE & DATA SCIENCE

PANIMALAR ENGINEERING COLLEGE CHENNAI CITY CAMPUS

ANNA UNIVERSITY: CHENNAI 600 025

APRIL 2026
 PANIMALAR ENGINEERING COLLEGE CHENNAI CITY CAMPUS
ANNA UNIVERSITY: CHENNAI 600 025

BONAFIDE CERTIFICATE

Certified that this project report “DEEPLIPREAD: CNN–LSTM AND CNN

TRANSFORMER BASED ENGLISH VISUAL SPEECH RECOGNITION” is the

bonafide work of “GOPIKA S [313522243091], HARINI P [313522243034], APARNA

LAKSHMI S V[313522243012]” who carried out the project work under my supervision.

SIGNATURE SIGNATURE
[Link] ., M.E., Ph.D., [Link], M.E.,
HEAD OF THE DEPARTMENT SUPERVISOR
ASSISTANT PROFESSOR
DEPARTMENT OF AI & DS, DEPARTMENT OF AI & DS,
PANIMALAR ENGINEERING COLLEGE, PANIMALAR ENGINEERING
CHENNAI CITY CAMPUS, COLLEGE, CHENNAI CITY CAMPUS,
NUNGAMBAKKAM, NUNGAMBAKKAM,
CHENNAI-600 030. CHENNAI-600 030.
Certified that the above candidate(s) was/ were examined in the Anna University Project

Viva-Voce Examination held on _ /_ / 2026 at Panimalar Engineering College Chennai City

Campus

INTERNAL EXAMINER EXTERNAL

EXAMINER

DECLARATION BY THE STUDENT

We, GOPIKA S [313522243091], HARINI P [313522243034], APARNA LAKSHMI S

V[313522243012], hereby declare that this project report titled “DEEPLIPREAD: CNN–

LSTM AND CNN TRANSFORMER BASED ENGLISH VISUAL SPEECH

RECOGNITION”, under the guidance of Mrs. P. PREETHIKA, M.E., is the original work

carried out by us. We affirm that this work has not been plagiarized and has not been submitted

to any other university or institution for the award of any degree.

 ACKNOWLEDGEMENT

We would like to express our deep gratitude to our respected Secretary and
Correspondent [Link], M.A., Ph.D. for his kind words and enthusiastic
motivation, which inspired us a lot in completing this project.

We express our sincere thanks to our Directors [Link],

[Link] KUMAR,M.E.,Ph.D and [Link] SAKTHI KUMAR
B.E.,M.B.A.,Ph.D., for providing us with the necessary facilities to undertake this
project.

We also express our gratitude to our Principal [Link], M.E., Ph.D.

who facilitated us in completing the project.

We thank the Head of the AI&DS Department, [Link], M.E.,Ph.D.,for the

support extended throughout the project.

We would like to thank my Project Guide [Link], M.E., and all the
faculty members of the Department of AI&DS for their advice and encouragement for the
successful completion of the project.

GOPIKA S [313522243091]
HARINI P [313522243034]
APARNA LAKSHMI SV[313522243012]
 TABLE OF CONTENTS

CHAPTER NO. TITLE PAGE

NO.

ABSTRACT iii

LIST OF FIGURES iv

1. INTRODUCTION

1.1 Overview 1

2
1.2 Problem Definition

2. LITERATURE SURVEY 3

3. SYSTEM ANALYSIS

11
3.1 Existing System

11
3.2 Proposed System

3.3 Feasibility System 12

3.4 Module description 13

3.5 System configuration 16

CHAPTER NO. TITLE PAGE

NO.

11
 4 SYSTEM DESIGN

4.1 Data flow diagram 19

4.2 Sequence diagram 19

4.3 Class Diagram 20

4.4 Use case Diagram 21

5. PROPOSED METHODOLOGY

5.1 Architecture Overview 23

5.2 Algorithms 25

6. SYSTEM IMPLEMENTATION

6.1 Installation and Environment Setup 31

6.2 Code for System Implementation 32

CHAPTER NO. TITLE PAGE

NO.

7. SYSTEM TESTING

11
 7.1 Use cases 58

7.2 Test cases 59

8. CONCLUSION AND FUTURE

ENHANCEMENTS

63
8.1 Conclusion
64
8.2 Future Enhancements

APPENDICES 65

REFERENCES 70

LIST OF FIGURES:

FIGURE NO. TITLE PAGE NO.

1 Data flow diagram 19

11
 2 Sequence Diagram 20

3 Class Diagram 20

4 Usecase Diagram 21

5 Architecture Diagram of Lip Reading System using 29

LipNet and Transfer Learning

6 6.1 Model Architecture Summary 65

6.2 Model Output Logs 66

6.3 Visualization of Lip Region Extraction 66

6.4 Inference Demo Output

67

iii

11
 ABSTRACT

Visual Speech Recognition (VSR), also known as lip reading, is the task of recognizing
spoken words from visual lip movements without using audio signals. It is highly useful in
noisy environments, assistive communication systems, and silent speech interfaces where
conventional speech recognition methods are ineffective. This project presents DeepLipRead, a
deep learning-based word-level lip reading system for recognizing 25 common English words
from silent video sequences. A custom dataset of 1,250 videos was created using recordings
from 10 speakers, with 5 repetitions per word, enabling multi-speaker training and evaluation.
The system uses MediaPipe facial landmark detection to extract the lip region from each frame
and applies preprocessing techniques such as grayscale conversion, contrast enhancement,
brightness normalization, resizing to 140 × 46 pixels, and uniform sampling to 75 frames. A
CNN–BiLSTM architecture inspired by LipNet is used, where 3D Convolutional layers extract
spatiotemporal lip features and a Bidirectional LSTM captures temporal dependencies for word
classification. A two-phase transfer learning approach is employed: In Phase 1, the pretrained
CNN feature extractor is kept frozen while only the BiLSTM and classification layers are
trained, enabling fast convergence and stable learning. In Phase 2, all layers are unfrozen and
fine-tuned with a lower learning rate to improve overall performance. The final model achieved
92.8% Top-1 validation accuracy, 95.2% Top-3 accuracy, and 97.2% Top-5 accuracy on a
leakage-free validation set. The results demonstrate that the proposed system is accurate,
efficient, and suitable for real-world silent communication and assistive technology
applications.

11
 iv

11
 CHAPTER - I

11
 INTRODUCTI
ON

1.1 Overview:

Traditional speech recognition systems rely on audio signals, which makes them
ineffective in noisy environments or when sound is unavailable. Visual Speech Recognition
(VSR), or lip reading, addresses this limitation by recognizing speech through lip movements
captured in video frames without using audio. This approach is useful for silent communication,
assistive technologies, and understanding speech in challenging conditions.

Recent advances in deep learning have significantly improved VSR performance. Lip reading
requires capturing both spatial information, such as lip shape in individual frames, and temporal
information, such as motion across frames. Spatiotemporal models are widely used for this
purpose. Three-dimensional Convolutional Neural Networks (3D CNNs) extract combined
spatial and temporal features, while Bidirectional Long Short-Term Memory (BiLSTM)
networks learn the sequence and transitions of lip movements over time, enabling accurate word
recognition.

This project presents DeepLipRead, a word-level lip reading system based on a LipNet-inspired
architecture. A custom dataset of 1,250 video samples was created using 25 common English
words spoken by 10 individuals with multiple repetitions. MediaPipe Face Mesh is used to
detect facial landmarks and extract the lip region. Preprocessing includes grayscale conversion,
contrast enhancement, brightness normalization, resizing, and fixed-length frame sampling.
Data augmentation improves robustness.

The model uses a pretrained 3D CNN, followed by BiLSTM and dense layers, trained using a
two-phase transfer learning strategy for improved performance.

1
1.2 Problem Definition:

Although deep learning has improved visual speech recognition, building an accurate lip
reading system still presents several challenges.

The first challenge is variability in lip movements across individuals. Factors such as face
shape, speaking speed, lighting conditions, camera angle, and video quality influence how the
same word appears. A model must therefore learn general speech patterns while adapting to
these differences. Limited or inconsistent data can reduce the model’s ability to generalize
effectively.

The second challenge involves inconsistencies in video data. Recordings often vary in frame
count, illumination, contrast, and lip alignment. Deep learning models require fixed-size inputs,
making raw videos unsuitable for direct use. Accurate lip region extraction and proper
preprocessing are necessary to ensure consistency and reduce irrelevant background
information.

The third challenge is the lack of large labeled datasets. Training deep models from scratch
requires extensive data, which is difficult to obtain in smaller projects. Transfer learning helps
address this issue by using pretrained feature extractors that capture spatiotemporal patterns,
allowing the model to adapt to a smaller dataset efficiently.

This project addresses these challenges by developing a word-level lip reading system using a
custom dataset of 1,250 videos across 25 words and 10 speakers. MediaPipe Face Mesh is used
for lip extraction, followed by preprocessing and augmentation. A pretrained 3D CNN with
BiLSTM is trained using a two-phase transfer learning strategy.

2
3
CHAPTER - II

4
 LITERATURE SURVEY

PAPER 1:

Title: Arabic Lip Reading With Limited Data Using Deep Learning (2020)
Authors: Jabr et al.

This research addresses the challenge of performing lip reading in low-resource

environments where large-scale datasets are unavailable. The authors focus on Arabic visual
speech recognition, which has limited publicly available datasets compared to English. To
overcome this challenge, an efficient deep learning architecture was proposed that can learn
meaningful lip movement patterns using a relatively small dataset.

The system combines a Convolutional Neural Network (CNN) for spatial feature extraction
with a Gated Recurrent Unit (GRU) network for temporal modeling. The CNN layers extract
visual features such as lip contours, mouth shape, and motion cues from video frames, while the
GRU layers analyze temporal variations across consecutive frames. This hybrid architecture
reduces computational complexity while maintaining effective learning capability.

A custom dataset consisting of 20 frequently used Arabic words was created. The dataset
underwent preprocessing steps such as frame extraction, resizing, normalization, and sequence
padding to ensure consistent input format. Despite the limited dataset size, the system achieved
a recognition accuracy of 83.02%. The study demonstrates that deep learning-based lip reading
is feasible even with limited data and emphasizes the importance of efficient architectures and
transfer learning. This work strongly supports the methodology of the proposed project, which
also focuses on low-data adaptation.

5
PAPER 2:

Title: Deep Learning-Based Lip Reading for Vocal Impaired Patient Rehabilitation (2021)
Authors: Innocente et al.

This paper presents a deep learning–based lip-reading system developed to assist vocal-
impaired patients in improving communication. The primary objective of the research is to
convert visual lip movements into textual output, enabling individuals with speech disabilities
to interact more effectively. The authors designed a spatiotemporal deep learning architecture
that combines Convolutional Neural Networks (CNNs) with Bidirectional Long Short-Term
Memory (BiLSTM) networks. CNN layers extract spatial features such as lip shape, mouth
structure, and articulation patterns from video frames, while BiLSTM layers capture temporal
dependencies in both forward and backward directions to understand complete speech patterns.

The system also incorporates Connectionist Temporal Classification (CTC) loss, which allows
sequence prediction without requiring frame-level alignment between input video frames and
output text. This approach simplifies training and improves model flexibility when handling
variable-length sequences. A custom Italian dataset consisting of 25 commonly used medical
and daily communication words was created for training and evaluation. The dataset underwent
preprocessing steps such as grayscale conversion, resizing, normalization, and sequence
standardization to enhance feature quality and ensure consistency.

Experimental results showed that the proposed model achieved an average recognition accuracy
of approximately 96.4% in controlled environments. The findings demonstrate the effectiveness
of deep learning techniques in assistive communication technologies. The study confirms that
visual speech recognition can significantly improve accessibility for speech-impaired
individuals and highlights the need for larger datasets and real-world testing to further enhance
system robustness and performance.

6
PAPER 3:

Title: Lip Reading Using Deep Neural Networks for Human–Computer Interaction (2022)
Authors: Kumar, S., Verma, A., Singh, R.

This paper explores lip-reading technology as an alternative input modality for human–
computer interaction systems. The research aims to develop a visual speech recognition model
capable of recognizing spoken words solely from lip movement videos, enabling
communication without relying on audio input. The proposed deep learning architecture
integrates Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM)
networks. CNN layers are used to extract spatial features such as mouth contours, lip shape, and
articulation patterns from individual video frames, while LSTM layers capture temporal
dependencies across sequences of frames.

The combined CNN–LSTM architecture enables accurate recognition of spoken words by

modeling both spatial and temporal aspects of lip movements. The system was trained and
evaluated on a dataset containing recorded lip movement videos of multiple speakers.
Preprocessing steps included frame extraction, resizing, normalization, and sequence alignment
to ensure consistent input representation. The model demonstrated reliable recognition
performance, particularly in scenarios where audio-based speech recognition systems face
challenges due to noise or poor audio quality.

Experimental results indicate that visual speech recognition systems can function effectively in
noisy environments and support silent communication. The study highlights the importance of
lip-reading technology in improving accessibility for individuals with hearing or speech
impairments. It concludes that deep learning–based lip-reading systems have strong potential
for real-world human–computer interaction applications and validates the use of CNN and
LSTM architectures in practical visual speech recognition systems.

7
PAPER 4:

Title: Word-Level Visual Speech Recognition Using CNN–BiLSTM Networks (2022)

Authors: Rahman, M., Islam, S., Hossain, M.

This paper presents a word-level visual speech recognition system using a hybrid deep
learning architecture that combines Convolutional Neural Networks (CNNs) with Bidirectional
Long Short-Term Memory (BiLSTM) networks. The main objective of the study is to develop
an accurate lip-reading model capable of recognizing spoken words from visual input alone.
CNN layers are used to extract spatial features such as lip shape, mouth position, articulation
patterns, and motion information from video frames. These spatial features provide essential
visual cues required for identifying speech patterns.

The extracted features are then processed by BiLSTM networks, which capture temporal
dependencies in both forward and backward directions. This bidirectional processing enables
the system to understand the complete context of lip movement sequences, improving
recognition accuracy. A custom dataset consisting of spoken word videos was created and used
for training and evaluation. The dataset underwent preprocessing steps such as lip-region
detection, frame resizing, normalization, and sequence padding to ensure uniform input length
and quality.

Experimental results demonstrated strong recognition accuracy in controlled environments,

confirming the effectiveness of CNN–BiLSTM architectures for word-level lip reading tasks.
The study emphasizes the importance of combining spatial and temporal feature extraction for
accurate visual speech recognition. It also suggests that larger datasets, diverse speakers, and
real-world testing conditions are necessary to improve system generalization and performance
in practical applications.

8
PAPER 5:

Title: Attention-Based Visual Speech Recognition Using Deep Learning (2023)

Authors: Li, Y., Zhang, H., Wang, J.

This research introduces an attention-based deep learning framework designed to

improve the performance of visual speech recognition systems. The proposed model integrates
Convolutional Neural Networks (CNNs) with attention-enhanced temporal modeling techniques
to achieve more accurate lip-reading results. CNN layers are responsible for extracting spatial
features such as lip contours, mouth shapes, and articulation patterns from video frames. These
features capture essential visual information required for recognizing spoken words from lip
movements.

The model incorporates an attention mechanism that selectively focuses on the most
informative frames within a video sequence. Instead of treating all frames equally, the attention
module assigns higher importance to frames that contain significant lip movements and relevant
speech information. This selective focus allows the system to capture meaningful temporal
dependencies while reducing the influence of redundant or irrelevant frames. As a result, the
model can better understand complex lip movement patterns and improve recognition accuracy.

Experimental evaluations demonstrated that the attention-based model achieved higher

accuracy compared to traditional sequential models such as CNN–LSTM. The study highlights
the growing importance of attention mechanisms and Transformer-like architectures in modern
visual speech recognition systems. It concludes that attention-based temporal modeling
significantly enhances system performance and robustness, supporting the development of
advanced lip-reading applications.

9
PAPER 6:

Title: Automatic Lip Reading of Persian Words by a Robotic System Using Deep Learning
Algorithms (2024)
Authors: A. Gholipour, H. Mohammadzade, A. Ghadami, A. Taheri

This paper presents a deep learning–based visual speech recognition system designed to
recognize Persian words using lip movements, with a specific focus on robotic applications.
The study aims to enable robots and intelligent systems to understand human speech visually,
improving interaction in environments where audio signals may be unclear or unavailable. The
authors implemented and compared two deep learning architectures: CNN–LSTM and CNN–
Transformer. Convolutional Neural Networks (CNNs) were used to extract spatial features from
lip-region video frames, including lip shape, motion patterns, and articulation dynamics.

Temporal dependencies across video sequences were modeled using LSTM and Transformer
networks. The LSTM network captured sequential temporal patterns, while the Transformer
architecture used self-attention mechanisms to model long-range dependencies more
effectively. A carefully curated dataset of Persian lip movement videos was created and
preprocessed using normalization, frame selection, resizing, and noise reduction techniques.
These preprocessing steps enhanced feature quality and ensured consistent input representation
for model training.

Experimental results showed that the Transformer-based architecture achieved the highest
accuracy of 96.2%, outperforming the CNN–LSTM model. The findings demonstrate that
Transformer models are highly effective in modeling complex temporal relationships in lip
movement sequences. This research contributes to the advancement of visual speech

10
recognition for non-English languages and highlights the importance of large, diverse datasets
and advanced architectures for improving real-world performance.

CHAPTER - III

11
 SYSTEM ANALYSIS

3.1 EXISTING SYSTEM:

Traditional speech recognition systems rely primarily on audio signals to interpret spoken
words. While these systems perform well in controlled environments, they struggle in noisy
conditions and cannot function when audio is unclear, unavailable, or intentionally absent. This
limits their use in silent communication and real-world scenarios where sound-based input is
unreliable.

Visual speech recognition systems attempt to overcome this limitation by analyzing lip
movements from video data. However, most existing approaches depend on large public
datasets collected under controlled laboratory conditions with limited speaker diversity and
fixed recording setups. These datasets often fail to capture real-world variations such as lighting
changes, camera angles, facial differences, and speaking styles. As a result, models trained on
such data may not generalize well to custom or real-world inputs.

Additionally, many systems focus on sentence-level recognition, requiring complex methods

like CTC loss and beam search decoding. These approaches increase computational complexity
and demand large datasets, making them difficult to implement effectively in small-scale
projects.

3.2 PROPOSED SYSTEM:

The proposed system, DeepLipRead, is a word-level visual speech recognition

framework designed to recognize 25 commonly used English words using only lip movements
from silent video input. Unlike traditional systems, it eliminates dependence on audio and
12
focuses entirely on visual speech patterns, making it suitable for noisy environments and silent
communication applications.

A custom dataset is created using 10 speakers, each providing 5 repetitions for 25 words,
resulting in 1,250 video samples. MediaPipe Face Mesh is used to detect facial landmarks and
accurately extract the lip region from each frame. The extracted region undergoes preprocessing
steps including grayscale conversion, contrast enhancement using CLAHE, brightness
normalization, resizing to 140 × 46 pixels, and uniform sampling to a fixed length of 75 frames.
Synthetic data augmentation is applied to improve robustness and increase training diversity.

The model uses a pretrained 3D CNN feature extractor inspired by LipNet to capture
spatiotemporal features from the lip sequence. A Bidirectional LSTM layer processes the
temporal sequence, followed by dense layers for classification across 25 word classes. A two-
phase transfer learning approach is used for training.

In Phase 1, the pretrained CNN layers are frozen while the BiLSTM and classification layers
are trained. In Phase 2, the entire network is fine-tuned with a lower learning rate to adapt better
to the dataset. This strategy improves convergence and reduces overfitting. The final model
achieves 92.8% Top-1 accuracy, 95.2% Top-3 accuracy, and 97.2% Top-5 accuracy,
demonstrating strong and reliable performance.

3.3 FEASIBILITY SYSTEM:

3.3.1 Technical Feasibility

The proposed system is technically feasible as it is developed using reliable open-source

deep learning frameworks such as TensorFlow and Keras, which support efficient model
training and deployment. The use of MediaPipe Face Mesh enables automatic and accurate
extraction of the lip region from video frames without the need for manual annotation, thereby
improving preprocessing efficiency and reducing human effort. Additionally, the system can be
13
trained and evaluated on cloud-based platforms such as Kaggle, which provide GPU
acceleration, making the entire pipeline easily reproducible and accessible without requiring
high-end local hardware.

3.3.2 Operational Feasibility

The system is operationally feasible as it is designed to accept short video clips as input
and predict spoken words based solely on lip movements. The entire pipeline is fully
automated, including lip detection, preprocessing, frame normalization, and sequence
generation, eliminating the need for manual intervention. Furthermore, the trained model can be
integrated with simple user interfaces such as Streamlit or Gradio, allowing users to interact
with the system easily. The complete workflow, from input video to final word prediction,
ensures that the system is practical and user-friendly for real-world applications.

3.3.3 Economic Feasibility

The proposed system is economically feasible as it is built entirely using open-source

tools, eliminating any software licensing costs. Model training is carried out using free cloud-
based GPU resources such as Kaggle, which significantly reduces the need for expensive
hardware. Additionally, the dataset used in the project is of manageable size and does not
require large-scale storage or computational infrastructure, making the overall development and
deployment cost-effective.

3.3.4 Social Feasibility

The system is socially beneficial as it enables silent communication by recognizing

speech through visual lip movements without relying on audio input. This makes it particularly
useful for assistive technologies aimed at individuals with hearing or speech impairments.
Moreover, the system can operate effectively in noisy environments where traditional speech
recognition systems fail, thereby expanding its applicability in real-world scenarios.

14
3.4 MODULE DESCRIPTION:

3.4.1 Dataset Collection Module and Organization Module

A custom dataset of 25 commonly used English words was recorded by 10 speakers, each
providing 5 repetitions per word, resulting in 1,250 video samples. The dataset is organized in a
word-wise folder structure, where each class contains multiple samples (rep1.mp4, rep2.mp4,
…, rep5.mp4) of the corresponding word. The 25 words include everyday vocabulary: hello,
bye, yes, no, please, sorry, thanks, help, stop, go, come, wait, open, close, water, food, home,
school, friend, good, bad, love, call, sit, and stand.

3.4.2 Video Preprocessing and Lip Extraction Module

Raw video files are processed using MediaPipe Face Mesh to detect facial landmarks in
each frame and accurately locate the mouth region. Based on the detected lip landmarks, a
bounding box is generated and the lip area is cropped from each frame. The cropped region is
converted to grayscale to reduce computational load and highlight shape-based features.
Contrast enhancement using CLAHE and brightness normalization improve visual consistency.
Each frame is resized to 140 × 46 pixels. Videos are uniformly sampled or padded to 75 frames,
ensuring consistent temporal input for reliable model training.

3.4.3 Data Augmentation and Sequence Preparation Module

To improve robustness and reduce overfitting, synthetic data augmentation is applied to

preprocessed video sequences. Variations such as horizontal flipping and brightness
adjustments are introduced while preserving original lip movement patterns. This helps the
model learn generalized visual features across different conditions. After augmentation,
sequences are converted into fixed-size tensors for deep learning input. Each sample consists of
75 grayscale frames with dimensions 46 × 140 × 1. Labels are encoded into categorical form for

15
25-word classification. This module increases dataset diversity and improves generalization
performance.

3.4.4 3D CNN + BiLSTM Model Architecture Module

The model accepts input of shape (75, 46, 140, 1), representing grayscale lip frame
sequences. A pretrained 3D CNN extracts spatiotemporal features by learning both lip
appearance and motion across frames. These features are passed to a Bidirectional LSTM layer,
which captures sequential dependencies in both forward and backward directions. The output is
processed through dense layers and a softmax classifier to generate probabilities for 25 word
classes. This architecture effectively combines spatial and temporal learning, enabling accurate
word-level lip reading from silent video input.

3.4.5 Phase 1 — Frozen Feature Extractor Training Module

In the first phase, pretrained 3D CNN layers are frozen to retain learned visual features.
Only the BiLSTM and classification layers are trained using the custom dataset. This enables
efficient adaptation to the 25-word vocabulary without altering existing feature representations.
The dataset is split into training and validation sets using a leakage-free approach. Training uses
the Adam optimizer with early stopping, checkpointing, and learning rate scheduling for stable
convergence. This phase allows effective task-specific learning while reducing overfitting and
maintaining feature extraction stability.

3.4.6 Phase 2 — Full Fine-Tuning Transfer Learning Module

In the second phase, the frozen 3D CNN layers are unfrozen, and the entire model is fine-
tuned using a lower learning rate. This allows both feature extraction and sequence learning
components to adapt to the dataset more effectively. Fine-tuning helps capture detailed lip
movement patterns and improves generalization across speakers. Training uses validation
monitoring, early stopping, and checkpoint saving for controlled optimization. This two-phase
16
strategy improves overall performance and achieves better results compared to training the
model entirely from scratch.

3.4.7 Model Evaluation and Metrics Module

Model performance is evaluated using multiple metrics. Top-1 accuracy measures

whether the predicted word matches the actual label, while Top-3 and Top-5 accuracy check if
the correct word appears among top predictions. These metrics are important due to similarities
in lip movements. Training and validation accuracy and loss curves are monitored to analyze
convergence and detect overfitting. The final model achieves 92.8% Top-1, 95.2% Top-3, and
97.2% Top-5 accuracy, demonstrating strong recognition performance.

3.4.8 Inference and Prediction Module

The trained model performs inference on new video inputs using the same preprocessing
pipeline. Videos are processed using MediaPipe Face Mesh for lip detection, followed by
cropping, grayscale conversion, enhancement, normalization, resizing, and fixed 75-frame
sequence generation. The processed input is fed into the DeepLipRead model, which outputs
probability scores across 25 word classes. The predicted word is selected based on the highest
probability, and top predictions can also be displayed. This module demonstrates practical
usability for word-level lip reading applications.

3.5 SYSTEM CONFIGURATION:

Hardware Configuration:

Component Specification
Processor Intel Core i5 / AMD Ryzen 5 or higher
RAM 8 GB minimum (16 GB recommended)
GPU NVIDIA P100 (Kaggle) / Any CUDA-enabled GPU

17
 Component Specification
10 GB minimum for dataset, processed files, and model
Storage
checkpoints

Software Configuration:

Component Specification
Operating System Windows 10/11 or Linux (Ubuntu 20.04+)
Programming Language Python 3.10+
Deep Learning Framework TensorFlow 2.x, Keras
Lip Landmark Extraction MediaPipe
Video Processing OpenCV (cv2)
Data Processing NumPy, scikit-learn
Visualization Matplotlib
Development Environment Kaggle Notebooks, Jupyter Notebook
Version Control Git and GitHub
Dataset Platform Kaggle (custom dataset)

18
CHAPTER - IV

19
 SYSTEM DESIGN

4.1 Data Flow Diagram

This figure illustrates the overall workflow of the system, starting from video input,
followed by lip detection and extraction.

Figure 1: Data Flow Diagram

Shows the pipeline from video input to lip
extraction and feature processing.

4.2 Sequence Diagram

This figure represents the step-by-step processing of input image sequences through
multiple layers, including convolutional, normalization, and activation blocks.

20
 Figure 2: Sequence Diagram
Illustrates how input frames are processed
through multiple layers for extraction.

4.3 Class Diagram

Represents the interaction between the user and the system functionalities. It shows how
the user provides input and receives predicted output from the model.

Figure 3: Class Diagram-Represents flow between component

21
 4.4 Usecase Diagram
Illustrates the structure of system components and their relationships. It defines how
different modules like feature extraction, model, and classification interact.

Figure 4: Usecase Diagram

Shows interaction between the user and system.

22
CHAPTER - V

23
 PROPOSED METHODOLOGY

5.1 Architecture Overview:

The proposed system is a word-level visual speech recognition model that predicts
spoken words from silent lip movement videos. It is designed for a 25-word vocabulary and
trained on a custom dataset collected from 10 speakers, with 5 repetitions per word. Video input
is first processed to extract the lip region using MediaPipe Face Mesh, and the resulting frames
are preprocessed through grayscale conversion, resizing, pixel normalization, and adjustment to
a fixed number of frames.

The preprocessed frame sequences are fed into a 3D CNN + Bidirectional LSTM model. The
3D CNN layers capture spatial and temporal lip movement features, while the BiLSTM layer
models sequential lip motion patterns across frames. A Dense layer with Softmax activation
then classifies the input into one of the 25 words. This architecture effectively combines
spatiotemporal feature extraction and sequence modeling, making it suitable for small custom
datasets and accurate isolated word lip reading.

5.1.1 Input Data Acquisition

The system uses a custom dataset consisting of 25 commonly used English words
recorded from 10 speakers. Each speaker provides 5 repetitions per word, resulting in a total of
1250 video samples. The dataset is organized systematically by word and repetition, ensuring
structured data for training and evaluation. This setup enables the model to learn variations in
lip movements across different speakers while maintaining consistent word-level patterns.

5.1.2 Lip Region Extraction using MediaPipe

Lip region extraction is performed using MediaPipe Face Mesh, which detects 468 facial
landmarks for each frame. A set of lip-specific landmark points is selected to accurately identify
24
the mouth region. A bounding box is computed around these points, and the lip region is
cropped from each frame. This process removes irrelevant facial and background information,
allowing the model to focus only on lip movements and improving recognition accuracy.

5.1.3 Video Preprocessing

The extracted lip frames undergo preprocessing to ensure uniformity. Each frame is
converted to grayscale to reduce computational complexity and highlight structural features.
The frames are resized to a fixed resolution of 140 × 46 pixels. To handle varying video
lengths, each sequence is uniformly sampled to 75 frames. Pixel values are normalized to the
range [0,1], which helps in stable and efficient model training.

5.1.4 Spatial Feature Extraction - 3D CNN

Spatial feature extraction is performed using 3D Convolutional Neural Networks. The

model consists of three Conv3D layers with increasing filter sizes, each followed by Batch
Normalization and MaxPooling. These layers capture both spatial features, such as lip shape,
and temporal features, such as motion across frames. This enables the model to learn
meaningful spatiotemporal representations from the input video sequences.

5.1.5 Temporal Sequence Modeling - Bidirectional LSTM

The features extracted by the 3D CNN are passed to a Bidirectional LSTM layer to model
temporal dependencies. The BiLSTM processes the sequence in both forward and backward
directions, allowing the model to capture complete lip movement patterns. This improves the
system’s ability to distinguish between visually similar words.

25
5.1.6 Phase 1 – Base Model Training

In the first phase, the CNN–BiLSTM model is trained on the dataset to learn general
visual

speech patterns. The model focuses on extracting spatiotemporal features and establishing a
baseline for word-level recognition. This phase ensures stable learning and prepares the model
for further refinement.

5.1.7 Phase 2 – Model Fine-Tuning

In the second phase, the entire model is fine-tuned using a lower learning rate. This
allows the network to adjust its learned features more precisely to the dataset. Fine-tuning
improves convergence, enhances recognition performance, and results in a more accurate and
stable model.

5.1.8 Word-Level Classification

The final stage performs word-level classification using a Dense layer with 25 output
units and Softmax activation. The model produces a probability distribution over all word
classes, and the word with the highest probability is selected as the final prediction. The system
can also provide top-3 predictions with confidence scores for better interpretability.

5.2 PROPOSED SYSTEM ALGORITHMS:

5.2.1. CNN–LSTM Architecture for Visual Speech Recognition:

The core of the proposed system is a CNN–BiLSTM architecture designed for word-level
visual speech recognition. It takes a sequence of preprocessed lip movement frames as input
and predicts the corresponding spoken word from the fixed 25-word vocabulary. This

26
architecture is well suited for lip reading because it can learn both the visual appearance of lip
shapes and the motion of lips over time.

In this model, the 3D CNN layers first extract spatiotemporal features from the video frames by
learning lip shape changes and motion patterns across consecutive frames. These features are
then passed to a Bidirectional LSTM (BiLSTM) layer, which captures the temporal relationship
between frames in both forward and backward directions. Finally, the output is passed through a
Dense layer with Softmax activation to classify the input into one of the 25 word classes.

This combination of CNN and BiLSTM makes the model effective for isolated word lip
reading, especially on a custom dataset where both spatial and temporal information are
important for accurate prediction.

Basic Principle

Input video sequences consisting of lip movement frames are fed into the network, where
each video is preprocessed and converted into a fixed tensor of shape (75, 46, 140, 1). The 3D
CNN layers then extract both spatial and temporal features simultaneously, and these extracted
features are passed to a Bidirectional LSTM to capture sequential dependencies. Finally, a
Dense layer with Softmax activation predicts one of the 25 word classes as the output.

Components:

1. 3D Convolutional Layers: These layers learn spatial lip shape and temporal motion
simultaneously using 3D kernels. They extract hierarchical spatiotemporal features
across the 75-frame sequence.
2. Bidirectional LSTM (64 units with L2 regularization): Models temporal
dependencies in both forward and backward directions. L2 regularization reduces
overfitting on the small dataset.

27
 3. Batch Normalization: Applied after each Conv3D layer to stabilize activation
distributions. It improves convergence speed and training stability.
4. TimeDistributed GlobalAveragePooling: Converts the 3D feature maps output by the
CNN into a structured temporal sequence. This prepares the data in the correct format for
LSTM input.
5. MaxPooling Layers: Reduce spatial dimensions while preserving important features.
6. Dense + Softmax Layer: Converts learned representation into probability distribution
over 25 words.

5.2.2. LIP REGION EXTRACTION AND VIDEO PREPROCESSING:

Lip region extraction and preprocessing form an important stage of the proposed system.
Before training, each input video is processed to isolate only the mouth region so that the model
focuses entirely on lip movements instead of unnecessary facial or background details. This
improves the quality of input data and helps the model learn more relevant features.

In this project, MediaPipe Face Mesh is used to detect facial landmarks from each frame.
Based on the lip landmark points, the mouth region is cropped and extracted. After extraction,
the video frames are converted to grayscale, resized to a fixed size, normalized, and adjusted to
a fixed number of frames. These steps ensure that all video samples have a consistent format
before being given to the deep learning model.

This stage is essential because deep learning models require standardized input data, and proper
preprocessing improves both training stability and final recognition performance.

Basic Principle

Facial landmarks are first detected using MediaPipe, and the mouth region is extracted
based on the identified lip landmarks. The extracted frames are then converted to grayscale,

28
resized, and normalized to ensure consistent pixel values, while the frame length is standardized
to maintain uniform input across all video sequences.

Components and Techniques Used

1. MediaPipe Face Mesh: Used to detect facial landmarks from each video frame. It
identifies detailed facial points, including the lip region, enabling accurate mouth
localization.
2. Lip Region of Interest (ROI) Extraction: Lip landmark coordinates are used to
compute a bounding box around the mouth. The cropped region ensures that the model
focuses only on lip movements.
3. Frame Resizing: Each lip frame is resized to a fixed spatial resolution so that all samples
have a consistent input size for the neural network.
4. Frame Normalization: Pixel values are scaled to the range [0,1]. This helps stabilize
neural network training and improves convergence.

5.2.3. SOFTMAX WORD CLASSIFICATION:

The final stage of the proposed system is Softmax-based word classification. After the
CNN and BiLSTM layers learn the lip movement patterns from the video, the extracted feature
representation is passed to a fully connected Dense layer. This layer produces scores for each of
the 25 word classes in the vocabulary.

The Softmax activation function converts these scores into a probability distribution across all
classes. Each probability value indicates how likely the input video belongs to a particular
word. The class with the highest probability is selected as the final predicted word. This makes
the output easy to understand and suitable for word-level recognition tasks.

29
Softmax classification is simple, efficient, and highly suitable for multi-class problems like this
project, where the model must choose exactly one word from a fixed set of 25 classes.

Basic Principle

The deep learning model produces a set of raw scores corresponding to each word class,
where the BiLSTM output is passed into a fully connected Dense layer. These raw scores are
then transformed into normalized probability values using the Softmax activation function.
During training, sparse categorical cross-entropy loss is used to optimize prediction accuracy,
and the

word with the highest probability is selected as the final predicted output.

Components and Techniques Used

1. Dense Layer (25 Units): Transforms the temporal feature representation into 25 word-
level class scores. Each unit corresponds to one word in the fixed vocabulary.
2. Softmax Activation Function: Normalizes all class scores so that they sum to 1,
producing interpretable confidence values. This allows direct comparison of confidence
across all 25 word classes.
3. Adam Optimizer: Adapts the learning rate individually for each model parameter during
training. This provides faster and more stable convergence compared to standard gradient
descent.
4. Top-3 Prediction Output (Inference Stage): Returns the three highest-probability word
predictions along with their confidence scores. This improves interpretability when top-1
confidence is low, as seen in the live demo results.

30
 Figure 5: Architecture Diagram

CHAPTER - VI

31
 SYSTEM IMPLEMENTATION

6.1 Installing Python Packages

This section covers the installation of required Python packages for the DeepLipRead project.

6.1.1 Requirements for Installing Packages

Before installing packages, ensure Python is available from the command line:

# Check Python version

python --version

# Check pip availability

python -m pip --version

# Upgrade pip, setuptools and wheel

32
python -m pip install --upgrade pip setuptools wheel

6.1.2 Installing Required Libraries

Install all dependencies required for the DeepLipRead project:

pip install tensorflow==2.15.0

pip install mediapipe==0.10.14
pip install opencv-python
pip install numpy
pip install scikit-learn
pip install matplotlib

6.1.3 Running on Kaggle (GPU Environment)

Since the project was trained on Kaggle, packages are installed inline within the notebook:

!pip install -q mediapipe==0.10.14

6.2 Code for System Implementation

All hyperparameters, paths, random seeds, and library imports are defined here. Fixed seeds
ensure reproducibility across runs.

import os
import cv2
import json
import random
import numpy as np
import tensorflow as tf
33
import mediapipe as mp
import [Link] as plt
from sklearn.model_selection import train_test_split
from [Link] import confusion_matrix, classification_report
from [Link] import layers, Model
from [Link] import (
ModelCheckpoint,
EarlyStopping,
ReduceLROnPlateau
)
# ---- Paths ----
DATA_ROOT = '/kaggle/input/custom-lipreading-dataset'
CHECKPOINT_DIR = '/kaggle/working'

# ---- Reproducibility ----

SEED = 42
[Link](SEED)
[Link](SEED)
[Link].set_seed(SEED)

# ---- Input Configuration ----

FRAME_COUNT = 75
IMG_H = 46
IMG_W = 140
CHANNELS = 1
NUM_CLASSES = 25
BATCH_SIZE = 8
34
print('TensorFlow version:', tf.__version__)
print('GPU devices:', [Link].list_physical_devices('GPU'))

# Code for MediaPipe Lip Extraction

mp_face_mesh = [Link].face_mesh

# Lip landmark indices used for mouth ROI extraction

LIP_IDX = [
61, 146, 91, 181, 84, 17, 314, 405, 321, 375,
78, 95, 88, 178, 87, 14, 317, 402, 318, 324,
308, 191, 80, 81, 82, 13, 312, 311, 310, 415
]

def apply_clahe(gray_img):
clahe = [Link](clipLimit=2.0, tileGridSize=(8, 8))
return [Link](gray_img)
def load_video_mouth(video_path):
cap = [Link](video_path)
frames = []
while True:
ok, frame = [Link]()
if not ok:
break
[Link](frame)
[Link]()
if not frames:
35
 return [Link]((FRAME_COUNT, IMG_H, IMG_W, 1), dtype=np.float32)
idxs = [Link](0, len(frames) - 1, FRAME_COUNT).astype(int)
processed = []
last_crop = None
with mp_face_mesh.FaceMesh(
static_image_mode=True,
max_num_faces=1,
refine_landmarks=True,
min_detection_confidence=0.4
) as fm:
for i in idxs:
frame = frames[i]
H, W = [Link][:2]
rgb = [Link](frame, cv2.COLOR_BGR2RGB)
result = [Link](rgb)
if result.multi_face_landmarks:
lm = result.multi_face_landmarks[0].landmark
xs = [int(lm[j].x * W) for j in LIP_IDX]
ys = [int(lm[j].y * H) for j in LIP_IDX]
x1 = max(min(xs) - 10, 0)
x2 = min(max(xs) + 10, W)
y1 = max(min(ys) - 10, 0)
y2 = min(max(ys) + 10, H)
crop = frame[y1:y2, x1:x2]
crop = [Link](crop, cv2.COLOR_BGR2GRAY)
crop = apply_clahe(crop)
crop = [Link](crop, (IMG_W, IMG_H))
36
 last_crop = crop
else:
crop = last_crop if last_crop is not None else [Link]((IMG_H, IMG_W),
dtype=np.uint8)
[Link](crop)
arr = [Link](processed, dtype=np.float32)
# Normalize to [0, 1]
arr = arr / 255.0
# Add channel dimension
return arr[..., [Link]]

#Code for Dataset Loading and Label Encoding

WORDS = [
'hello', 'bye', 'yes', 'no', 'please', 'sorry', 'thanks',
'help', 'stop', 'go', 'come', 'wait', 'open', 'closse',
'water', 'food', 'home', 'school', 'friend', 'good',
'bad', 'love', 'call', 'sit', 'stand'
]
label_map = {word: idx for idx, word in enumerate(WORDS)}
idx_to_word = {idx: word for word, idx in label_map.items()}
def load_dataset(data_root):
X, y = [], []
for word in WORDS:
word_dir = [Link](data_root, word)
if not [Link](word_dir):
continue
for file_name in [Link](word_dir):
37
 if file_name.endswith('.mp4'):
video_path = [Link](word_dir, file_name)
frames = load_video_mouth(video_path)
[Link](frames)
[Link](label_map[word])
return [Link](X, dtype=np.float32), [Link](y, dtype=np.int32)
print('Loading dataset...')
X, y = load_dataset(DATA_ROOT)
print(f'Total samples loaded: {len(X)}')
print(f'Input shape: {[Link]}')
print(f'Total labels: {len(y)}')

#Code for Data Augmentation and Train

X_train, X_val, y_train, y_val = train_test_split(
X, y,
test_size=0.15,
stratify=y,
random_state=SEED
)
# Simple augmentation: horizontal flip
X_train_flip = X_train[:, :, :, ::-1, :]
# Combine original + augmented samples
X_train_aug = [Link]([X_train, X_train_flip], axis=0)
y_train_aug = [Link]([y_train, y_train], axis=0)
print(f'Training samples (original): {len(X_train)}')
print(f'Training samples (augmented): {len(X_train_aug)}')
print(f'Validation samples: {len(X_val)}')

38
#Code for LipNet-Inspired Model Architecture
def build_deeplipread_model():
inp = [Link](shape=(FRAME_COUNT, IMG_H, IMG_W, CHANNELS),
name='video_input')
# ---- Pretrained / LipNet-style 3D CNN feature extractor ----
x = layers.Conv3D(32, (3, 5, 5), padding='same', activation='relu')(inp)
x = layers.MaxPool3D((1, 2, 2))(x)
x = layers.Conv3D(64, (3, 3, 3), padding='same', activation='relu')(x)
x = layers.MaxPool3D((1, 2, 2))(x)
x = layers.Conv3D(96, (3, 3, 3), padding='same', activation='relu')(x)
x = layers.MaxPool3D((1, 2, 2))(x)

# ---- Convert to temporal feature sequence ----

x = [Link](layers.GlobalAveragePooling2D())(x)
# ---- Sequence modeling ----
x = [Link]([Link](128, return_sequences=False))(x)
x = [Link](0.4)(x)
# ---- Dense classifier ----
x = [Link](128, activation='relu')(x)
x = [Link](0.3)(x)
out = [Link](NUM_CLASSES, activation='softmax')(x)
model = Model(inp, out)
return model
model = build_deeplipread_model()
[Link](
optimizer='adam',
39
 loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
[Link]()

#Code for Phase 1 – Frozen Feature Extractor Training

for layer in [Link]:
if 'conv3d' in [Link] or 'max_pooling3d' in [Link]:
[Link] = False
else:
[Link] = True
[Link](
optimizer=[Link](1e-3),
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
callbacks_phase1 = [
ModelCheckpoint(
[Link](CHECKPOINT_DIR, 'phase1_best.keras'),
monitor='val_accuracy',
save_best_only=True,
verbose=1
),
EarlyStopping(
monitor='val_accuracy',
patience=15,
restore_best_weights=True,

40
 verbose=1
),
ReduceLROnPlateau(
monitor='val_loss',
factor=0.5,
patience=5,
min_lr=1e-6,
verbose=1
)
]
print('Phase 1: Training classifier layers with frozen CNN...')
history_phase1 = [Link](
X_train_aug, y_train_aug,
validation_data=(X_val, y_val),
epochs=50,
batch_size=BATCH_SIZE,
callbacks=callbacks_phase1,
verbose=1
)
#Code for Phase 2 – Full Fine-Tuning Transfer Learning
# Load best Phase 1 model
model = [Link].load_model([Link](CHECKPOINT_DIR, 'phase1_best.keras'))
# Unfreeze all layers for end-to-end fine-tuning
for layer in [Link]:
[Link] = True
[Link](
optimizer=[Link](1e-4),
41
 loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
callbacks_phase2 = [
ModelCheckpoint(
[Link](CHECKPOINT_DIR, 'phase2_best.keras'),
monitor='val_accuracy',
save_best_only=True,
verbose=1
),
EarlyStopping(
monitor='val_accuracy',
patience=10,
restore_best_weights=True,
verbose=1
),
ReduceLROnPlateau(
monitor='val_loss',
factor=0.5,
patience=4,
min_lr=1e-7,
verbose=1
)
]
print('Phase 2: Fine-tuning full model...')
history_phase2 = [Link](
X_train_aug, y_train_aug,
42
 validation_data=(X_val, y_val),
epochs=30,
batch_size=BATCH_SIZE,
callbacks=callbacks_phase2,
verbose=1
)
#Code for Evaluation and Performance Metrics
# Load final best model
final_model = [Link].load_model([Link](CHECKPOINT_DIR,
'phase2_best.keras'))
# Standard evaluation
val_loss, val_acc = final_model.evaluate(X_val, y_val, verbose=0)
print(f'Validation Accuracy (Top-1): {val_acc * 100:.2f}%')

# Predictions
probs = final_model.predict(X_val, verbose=0)
y_pred = [Link](probs, axis=1)
# Classification report
print(classification_report(y_val, y_pred, target_names=WORDS))
# Confusion matrix
cm = confusion_matrix(y_val, y_pred)
print('Confusion Matrix:')
print(cm)
# Top-k accuracy calculation
top3 = [Link]([y_val[i] in [Link](probs[i])[-3:] for i in range(len(y_val))])
top5 = [Link]([y_val[i] in [Link](probs[i])[-5:] for i in range(len(y_val))])
print(f'Top-3 Accuracy: {top3 * 100:.2f}%')
43
print(f'Top-5 Accuracy: {top5 * 100:.2f}%')
#Code for Inference and Prediction
"""
DeepLipRead — Lip-Only Live Prediction
========================================
Controls:
SPACE (hold) = Record lip frames -> release = predict
R = Reset history
Q / ESC = Quit
"""
import os, sys, time, [Link], json, argparse, collections
import cv2
import numpy as np
import mediapipe as mp

from [Link] import python as mp_python

from [Link] import vision as mp_vision
import tensorflow as tf

# CONFIG
WORDS = [
'hello', 'bye', 'yes', 'no', 'please',
'sorry', 'thanks', 'help', 'stop', 'go',
'come', 'wait', 'open', 'close', 'water',
'food', 'home', 'school', 'friend', 'good',
'bad', 'love', 'call', 'sit', 'stand'
]
44
IDX_TO_WORD = {i: w for i, w in enumerate(WORDS)}
NUM_FRAMES = 75
FRAME_H = 46
FRAME_W = 140
CONF_THRESH = 0.50 # minimum confidence to show as valid
MIN_FRAMES = 30 # minimum recorded frames before predicting
LIP_LANDMARKS = [
61, 185, 40, 39, 37, 0, 267, 269, 270, 409,
291, 375, 321, 405, 314, 17, 84, 181, 91, 146
]
# UI colours (BGR)
C_WHITE = (255, 255, 255)
C_GREEN = (50, 220, 100)
C_BLUE = (50, 180, 255)
C_PURPLE = (200, 130, 255)
C_YELLOW = (50, 220, 220)
C_DIM = (70, 70, 90)
C_TITLE = (100, 220, 255)
C_RED = (80, 80, 220)
SCRIPT_DIR = [Link]([Link](__file__))
LM_PATH = [Link](SCRIPT_DIR, 'face_landmarker.task')

# MEDIAPIPE SETUP
def load_mediapipe():
if not [Link](LM_PATH):
print('Downloading face_landmarker.task (~30 MB) ...')
[Link](
45
 '[Link]
'face_landmarker/face_landmarker/float16/1/face_landmarker.task',
LM_PATH)
print('Downloaded.')
opts = mp_vision.FaceLandmarkerOptions(
base_options=mp_python.BaseOptions(model_asset_path=LM_PATH),
num_faces=1,
min_face_detection_confidence=0.4,
min_face_presence_confidence=0.4,
min_tracking_confidence=0.4,
running_mode=mp_vision.[Link]
)
return mp_vision.FaceLandmarker.create_from_options(opts)
# PREPROCESSING — identical to training notebook
def preprocess_frame(frame_bgr, detector):
h, w = frame_bgr.shape[:2]

# Step 1 — CLAHE contrast enhancement (FIX 1a from notebook)

lab = [Link](frame_bgr, cv2.COLOR_BGR2LAB)
l, a, b = [Link](lab)
l = [Link](clipLimit=2.0, tileGridSize=(4, 4)).apply(l)
frame_enhanced = [Link]([Link]([l, a, b]), cv2.COLOR_LAB2BGR)

# Step 2 — MediaPipe detection

rgb = [Link](frame_enhanced, cv2.COLOR_BGR2RGB)
mp_img = [Link](image_format=[Link], data=rgb)
result = [Link](mp_img)
46
 if not result.face_landmarks:
return None, None
lm = result.face_landmarks[0]
xs = [int(lm[i].x * w) for i in LIP_LANDMARKS]
ys = [int(lm[i].y * h) for i in LIP_LANDMARKS]
pad = 15
x1 = max(0, min(xs) - pad); x2 = min(w, max(xs) + pad)
y1 = max(0, min(ys) - pad); y2 = min(h, max(ys) + pad)
if x2 <= x1 or y2 <= y1:
return None, None
# Step 3 — Crop lip region
crop = frame_enhanced[y1:y2, x1:x2]
# Step 4 — Grayscale + resize
gray = [Link](crop, cv2.COLOR_BGR2GRAY)
resized = [Link](gray, (FRAME_W, FRAME_H))

# Step 5 — Brightness normalisation (FIX 1b from notebook)

mn, mx = [Link](), [Link]()
if mx > mn:
resized = (([Link](np.float32) - mn) / (mx - mn) * 255).astype(np.uint8)
normalized = [Link](np.float32) / 255.0
return normalized[:, :, [Link]], (x1, y1, x2, y2)

# PREDICTION
def predict_top3(model, recorded_frames):
n = len(recorded_frames)
if n >= NUM_FRAMES:
47
 indices = [Link](0, n - 1, NUM_FRAMES, dtype=int)
frames = [recorded_frames[i] for i in indices]
else:
frames = list(recorded_frames)
while len(frames) < NUM_FRAMES:
[Link](frames[-1]) # pad by repeating last frame
arr = [Link](frames, dtype=np.float32)[[Link]] # (1, 75, 46, 140, 1)
probs = [Link](arr, verbose=0)[0]
idx = [Link](probs)[::-1][:3]
return [(IDX_TO_WORD[i], float(probs[i])) for i in idx]

# UI
def draw_ui(frame, state, bbox, fps,
lip_top3, final_word, last_spoken,
rec_frames, frames_ready):
h, w = [Link][:2]
PANEL = 300
# Lip bounding box
if bbox is not None:
x1, y1, x2, y2 = bbox
bc = C_RED if state == 'recording' else \
C_GREEN if state == 'result' else (70, 70, 110)
[Link](frame, (x1, y1), (x2, y2), bc, 2)
# Left panel background
ov = [Link]()
[Link](ov, (0, 0), (PANEL, h), (10, 10, 16), -1)
[Link](ov, 0.80, frame, 0.20, 0, frame)
48
y=0
# Title bar
[Link](frame, (0, y), (PANEL, y + 44), (18, 22, 35), -1)
[Link](frame, "DeepLipRead", (10, y + 30),
cv2.FONT_HERSHEY_SIMPLEX, 0.9, C_TITLE, 2)
[Link](frame, (220, y + 8), (290, y + 28), (20, 30, 70), -1)
[Link](frame, "LIP ONLY", (224, y + 23),
cv2.FONT_HERSHEY_SIMPLEX, 0.33, C_BLUE, 1)
y += 44
# Buffer / FPS status
buf_col = C_GREEN if frames_ready else (180, 140, 40)
[Link](frame, f"FPS:{fps:.0f} {'READY' if frames_ready else 'BUFFERING...'}",
(10, y + 16), cv2.FONT_HERSHEY_SIMPLEX, 0.40, buf_col, 1)
y += 22

# State banner
if state == 'idle':
bcol = (38, 38, 58); btxt = " HOLD SPACE to record"
elif state == 'recording':
bcol = (35, 35, 150); btxt = f" REC {rec_frames} frames captured"
else:
bcol = (20, 75, 28); btxt = " Done! Hold SPACE for next word"
[Link](frame, (6, y + 2), (PANEL - 6, y + 26), bcol, -1)
[Link](frame, btxt, (10, y + 19),
cv2.FONT_HERSHEY_SIMPLEX, 0.44, C_WHITE, 1)
# Pulsing REC dot
if state == 'recording' and int([Link]() * 2) % 2 == 0:
49
 [Link](frame, (PANEL - 16, y + 14), 6, (80, 80, 255), -1)
y += 32
[Link](frame, (8, y), (PANEL - 8, y), (45, 45, 65), 1)
y += 8
# ── Top-3 predictions ──────────────────────────────────────
[Link](frame, "LIP MODEL — TOP 3", (10, y + 13),
cv2.FONT_HERSHEY_SIMPLEX, 0.40, (130, 130, 160), 1)
y += 18
bar_colors = [C_GREEN, C_BLUE, C_PURPLE]
for rank, (word, conf) in enumerate(lip_top3):
yb = y + rank * 60
bw = int((PANEL - 16) * conf)
color = (38, 38, 55) if conf == 0.0 else \
(65, 65, 85) if (rank == 0 and conf < CONF_THRESH) else \
bar_colors[rank]
[Link](frame, (8, yb), (PANEL - 8, yb + 28), (32, 32, 48), -1)
[Link](frame, (8, yb), (8 + bw, yb + 28), color, -1)
[Link](frame, f"#{rank+1} {[Link]()}", (13, yb + 20),
cv2.FONT_HERSHEY_SIMPLEX, 0.55, C_WHITE, 1)
pct = f"{conf * 100:.1f}%" if conf > 0 else "---"
[Link](frame, pct, (PANEL - 58, yb + 20),
cv2.FONT_HERSHEY_SIMPLEX, 0.44, (210, 210, 210), 1)
if rank == 0 and conf >= CONF_THRESH:
[Link](frame, "OK", (PANEL - 20, yb + 12),
cv2.FONT_HERSHEY_SIMPLEX, 0.30, C_GREEN, 1)
y += 3 * 60 + 4
[Link](frame, (8, y), (PANEL - 8, y), (45, 45, 65), 1)
50
y += 8
# ── Final word (big) ───────────────────────────────────────
[Link](frame, "RECOGNIZED", (10, y + 13),
cv2.FONT_HERSHEY_SIMPLEX, 0.42, (140, 140, 160), 1)
y += 18
if final_word:
fw_col = C_GREEN if lip_top3[0][1] >= CONF_THRESH else C_YELLOW
[Link](frame, final_word.upper(), (10, y + 50),
cv2.FONT_HERSHEY_SIMPLEX, 1.6, fw_col, 3)
else:
[Link](frame, "---", (10, y + 42),
cv2.FONT_HERSHEY_SIMPLEX, 1.0, C_DIM, 1)
y += 60

[Link](frame, (8, y), (PANEL - 8, y), (45, 45, 65), 1)

y += 6
[Link](frame, "HISTORY", (10, y + 13),
cv2.FONT_HERSHEY_SIMPLEX, 0.38, (100, 100, 120), 1)
y += 18
if last_spoken:
shown = last_spoken[-5:]
x_off = 10
for hi, word in enumerate(shown):
alpha = 0.35 + 0.65 * (hi / max(len(shown) - 1, 1))
col = (int(50 * alpha), int(180 * alpha), int(100 * alpha))
[Link](frame, word, (x_off, y + 18),
cv2.FONT_HERSHEY_SIMPLEX, 0.50, col, 1)
51
 tw = [Link](word, cv2.FONT_HERSHEY_SIMPLEX, 0.50, 1)[0][0]
x_off += tw + 5
if hi < len(shown) - 1:
[Link](frame, ">", (x_off, y + 18),
cv2.FONT_HERSHEY_SIMPLEX, 0.36, C_DIM, 1)
x_off += 13
# Controls (bottom right corner)
for ci, (k, d) in enumerate([("SPACE", "hold to record"),
("R", "reset"),
("ESC/Q", "quit")]):
yy = h - 52 + ci * 17
[Link](frame, k, (w - 138, yy),
cv2.FONT_HERSHEY_SIMPLEX, 0.36, C_YELLOW, 1)
[Link](frame, f"= {d}", (w - 108, yy),
cv2.FONT_HERSHEY_SIMPLEX, 0.32, (100, 100, 120), 1)

# REC overlay on camera side

if state == 'recording':
pulse = int([Link]() * 3) % 2
bc = (60, 60, 255) if pulse else (35, 35, 160)
[Link](frame, (PANEL + 2, 0), (w - 1, h - 1), bc, 4)
[Link](frame, (w - 108, 8), (w - 8, 36), (35, 35, 160), -1)
[Link](frame, "* REC", (w - 103, 27),
cv2.FONT_HERSHEY_SIMPLEX, 0.58, C_WHITE, 1)
[Link](frame, (w - 118, 42), (w - 8, 66), (22, 22, 55), -1)
[Link](frame, f"{rec_frames} frames", (w - 113, 59),
cv2.FONT_HERSHEY_SIMPLEX, 0.42, (170, 170, 210), 1)
52
 return frame

# MAIN
def run_live(model_path):
# Load model
print(f"\nLoading model: {model_path}")
model = [Link].load_model(model_path, compile=False)
[Link](optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
print(f"Model ready — input={model.input_shape} output={model.output_shape}")
# Sanity check
dummy = [Link](1, 75, 46, 140, 1).astype(np.float32)
out = [Link](dummy, verbose=0)[0]
print(f"Sanity check: max_prob={[Link]():.4f}
pred={IDX_TO_WORD[int([Link]())]}")
# MediaPipe
detector = load_mediapipe()
print("MediaPipe ready.")
# Webcam
cap = [Link](0)
[Link](cv2.CAP_PROP_FRAME_WIDTH, 1280)
[Link](cv2.CAP_PROP_FRAME_HEIGHT, 720)
if not [Link]():
[Link]("ERROR: Cannot open webcam.")
# Warm up camera
for _ in range(10):
53
 [Link]()
# State
state = 'idle'
roll_buffer = [Link](maxlen=NUM_FRAMES)
rec_buffer = []
lip_top3 = [('---', 0.0), ('---', 0.0), ('---', 0.0)]
final_word = None
last_spoken = []
fps_ring = [Link](maxlen=30)
t_prev = [Link]()
was_space = False
bbox = None
print("\n" + "=" * 55)
print(" DeepLipRead — Lip-Only Mode")
print(" SPACE=hold to record R=reset Q/ESC=quit")
print("=" * 55 + "\n")
try:
import keyboard
USE_KEYBOARD = True
except ImportError:
USE_KEYBOARD = False
print(" NOTE: pip install keyboard for SPACE-bar control")
print(" Falling back to OpenCV key — press R to record\n")
WIN = "DeepLipRead — Lip Only"
[Link](WIN, cv2.WINDOW_NORMAL)
[Link](WIN, 1100, 680)
while True:
54
ret, frame = [Link]()
if not ret:
continue
frame = [Link](frame, 1)
# FPS
t_now = [Link]()
fps_ring.append(1.0 / max(t_now - t_prev, 1e-6))
t_prev = t_now
fps = float([Link](fps_ring))
# Preprocess current frame
processed, bbox = preprocess_frame(frame, detector)
if processed is not None:
roll_buffer.append(processed)
if state == 'recording':
rec_buffer.append(processed)

frames_ready = len(roll_buffer) == NUM_FRAMES

# ── Key handling ──────────────────────────────────────
if USE_KEYBOARD:
space_held = keyboard.is_pressed('space')
r_pressed = keyboard.is_pressed('r')
q_pressed = keyboard.is_pressed('q') or keyboard.is_pressed('esc')
else:
space_held = False
r_pressed = False
q_pressed = False

55
cv_key = [Link](1) & 0xFF
if cv_key == ord('q') or cv_key == 27:
q_pressed = True
if cv_key == ord('r'):
r_pressed = True
if not USE_KEYBOARD and cv_key == ord('s'):
space_held = not was_space # toggle with 's' key
# SPACE pressed (rising edge) — start recording
if space_held and not was_space:
if frames_ready:
state = 'recording'
rec_buffer = []
lip_top3 = [('---', 0.0), ('---', 0.0), ('---', 0.0)]
final_word = None
print(" [RECORDING...]", end='', flush=True)
else:
print(" Buffer still filling — wait for READY")
# SPACE released (falling edge) — predict
if not space_held and was_space and state == 'recording':
n = len(rec_buffer)
if n >= MIN_FRAMES:
print(f" {n} frames -> predicting...", end='', flush=True)
lip_top3 = predict_top3(model, rec_buffer)
final_word = lip_top3[0][0]
state = 'result'
if not last_spoken or last_spoken[-1] != final_word:
last_spoken.append(final_word)
56
 if len(last_spoken) > 10:
last_spoken.pop(0)
print(f" {final_word.upper()} ({lip_top3[0][1]*100:.1f}%)")
else:
print(f" too short ({n} frames) — hold longer")
state = 'idle'
was_space = space_held
# R — reset
if r_pressed:
roll_buffer.clear()
rec_buffer = []
lip_top3 = [('---', 0.0), ('---', 0.0), ('---', 0.0)]
final_word = None
last_spoken = []
state = 'idle'
was_space = False
print(" Reset!")
[Link](0.25)
if q_pressed:
break
display = draw_ui([Link](), state, bbox, fps,
lip_top3, final_word, last_spoken,
len(rec_buffer), frames_ready)
[Link](WIN, display)
[Link]()
[Link]()
print("\nSession ended.")
57
 if last_spoken:
print(f"Words recognised: {' > '.join(last_spoken)}")
# ENTRY
if __name__ == "__main__":
parser = [Link](description="DeepLipRead — Lip-Only Live Prediction")
parser.add_argument("--model", default="lipnet_final.h5",
help="Path to trained .h5 model file")
args = parser.parse_args()
if not [Link]([Link]):
[Link](f"\nERROR: Model not found: {[Link]}\n"
f"Usage: python predict_live_liponly.py --model lipnet_final.h5\n")

run_live([Link])

58
 CHAPTER - VII

SYSTEM TESTING

7.1 Use Cases:

Use Case 1: Known word prediction from test/validation video.
System testing scenario:
 Input & Processing: A video clip of one word from the 25-word vocabulary is given
as input and processed through preprocessing steps including lip region extraction
(MediaPipe Face Mesh), grayscale conversion, resizing, normalization, and fixed-
length sequence preparation.
 Model Execution: The processed video is passed into the trained CNN–BiLSTM
model for prediction.
 Output: The system returns the top-1 predicted word along with probability scores.
 Expected Outcome: The correct word is identified with high accuracy; the model
achieves 92.8% top-1 validation accuracy.

59
 Use case 2: Top-3 prediction for ambiguous words.
System testing scenario:
 Input: A video containing words with similar lip movements (e.g., go, no, call, love)
is provided to the system.
 Processing & Prediction: The system processes the video and generates probability
scores for all 25 word classes.
 Output: Instead of only top-1, the system also provides top-3 predictions with
confidence scores to improve interpretability.
 Expected Outcome: Even if top-1 prediction is uncertain, the correct word is
typically included in the top-3 predictions (95.2% accuracy).
Use case 3: System handles difficult visual conditions.
System testing scenario:
 Input: A video with variations such as lighting changes, facial movement, or minor
frame inconsistencies is given.
 Processing: Preprocessing includes brightness normalization and CLAHE
enhancement to improve lip visibility before feature extraction.
 Model Execution: The system extracts the lip region and performs prediction using
the trained model.
 Expected Outcome: The system still produces a valid prediction without failure,
showing robustness to moderate real-world variations.
7.2 Test Cases:

Here are some test cases for cardiac arrest prediction using decision tree algorithms:

Test case 1: Word Prediction on Validation Sample

● Input:
o Word spoken: hello
o Input source: Validation/test sample
60
 o Model used: Final trained model (Phase 2)
● Expected output:
o Top-1 prediction: "hello"
o Confidence: High
o Result: Pass

Test case 2: Phase 1 Model Evaluation

● Input:
o Dataset: Validation split
o Model used: Phase 1 model (CNN frozen, LSTM + classifier trained)
o Video: rep2.mp4
● Expected output:
o Validation accuracy: 92.0%
o Model successfully recognizes most word classes
o Result: Pass
Test case 3: Phase 2 Fine-Tuned Model Evaluation
● Input:
o Dataset: Same validation split
o Model used: Phase 2 model (all layers unfrozen, fine-tuned)
● Expected output:
o Validation accuracy: 92.8%
o Performance slightly improves over Phase 1
o Final model saved successfully
o Result: Pass
Test case 4: Top-3 Prediction Performance
● Input:

61
 o Dataset: Validation split
o Model used: Final trained model
● Expected output:
o Top-1 accuracy: 92.8%
o Top-3 accuracy: 95.2%
o Top-5 accuracy: 97.2%
o Result: Pass - Correct word is often present in top-3 predictions even when
top-1 is uncertain
Test case 5: Per-Word Classification Performance
● Input:
o Dataset: Validation split containing all 25 words
o Model used: Final trained model
● Expected output:
o Several words such as hello, no, come, open, school, good, stand achieve
very high or perfect recognition on the validation set
o Some visually similar words such as bye, please, thanks, wait, bad, call
show slightly lower performance
o Overall system accuracy remains high at 92.8%
o Result: Pass

Test case 6: Random Baseline Comparison

● Input:
o Number of classes: 25 words
o Baseline method: Random class selection
o Model used: Final trained model
● Expected output:
62
o Random baseline accuracy: 4.0%
o Final model top-1 accuracy: 92.8%
o Model performs approximately 23.2 times better than random guessing
o Result: Pass

63
 CHAPTER -
VIII

CONCLUSION & FUTURE ENHANCEMENTS

8.1 CONCLUSION:

This project successfully developed DeepLipRead, a word-level visual speech

recognition system capable of recognizing spoken words from silent lip movement videos. The
system was designed and implemented using a custom dataset consisting of 25 commonly used
English words, recorded from 10 speakers, with 5 repetitions per word. A complete
preprocessing and deep learning pipeline was built to convert raw video input into accurate
word-level predictions.

64
The proposed system uses MediaPipe Face Mesh for precise lip region extraction, followed by
video preprocessing techniques such as grayscale conversion, resizing, frame normalization,
and fixed-length sequence preparation. For classification, a 3D CNN + Bidirectional LSTM
(BiLSTM) architecture was implemented, where the 3D CNN layers capture spatial and
temporal lip movement features and the BiLSTM layer models the sequential pattern of lip
motion across frames. A final Dense layer with Softmax activation predicts one of the 25
predefined word classes.

To improve performance, the project adopted a two-phase training strategy. In Phase 1, the
CNN feature extraction layers were kept frozen while training the BiLSTM and classification
layers. In Phase 2, all layers were unfrozen and the complete model was fine-tuned with a lower
learning rate for better convergence. This strategy improved the validation performance from
92.0% in Phase 1 to 92.8% in Phase 2, showing that end-to-end fine-tuning enhanced the final
recognition accuracy.

The experimental results demonstrate that the proposed system performs strongly on the custom
dataset, achieving 92.8% Top-1 accuracy, 95.2% Top-3 accuracy, and 97.2% Top-5 accuracy
on the validation set. When compared to the 4.0% random baseline for a 25-class problem, the
model performs approximately 23.2 times better, clearly proving the effectiveness of the
proposed approach. Overall, the project confirms that a lightweight CNN–BiLSTM based
architecture, combined with robust lip extraction and preprocessing, can provide accurate and
practical word-level visual speech recognition for silent communication applications.

8.2 FUTURE ENHANCEMENTS:

Although the proposed Deep LipRead system achieved strong performance, several
improvements can further enhance its accuracy, robustness, and real-world usability.

65
  Increase Vocabulary Size: The current system is limited to 25 isolated words. In future
work, the vocabulary can be expanded to include a larger number of words, enabling
more practical and flexible communication.
 Collect More Speaker Data: Additional speakers from different age groups, speaking
styles, and accents can be included to improve generalization and reduce speaker-specific
bias in the model.
 Extend to Sentence-Level Lip Reading: The current model performs isolated word
classification. Future work can extend the system to continuous sentence-level lip reading
using methods such as CTC, Transformer-based sequence models, or attention-based
decoders.
 Real-Time Deployment: The trained model can be integrated into a real-time application
using Streamlit, Flask, or a mobile interface for live silent speech prediction and assistive
communication.
 Improve Robustness in Real-World Conditions: Further work can focus on improving
performance under variations such as poor lighting, head movement, low-resolution
video, and partial occlusion.

APPENDICES
A.1 –Model Architecture Summary:

This appendix presents the summary of the LipNet model architecture, showing the sequence of
layers used for spatiotemporal feature extraction from video input.

66
 Figure 6.1-(A): Model architecture summary Diagram

A.2 Model Output Logs:

This appendix shows the model’s output logs during live inference, including layer details,
parameter counts, and prediction results.

67
 Figure 6.2: Model output logs Diagram

A.3 Visualization of Lip Region Extraction:

This figure shows sample lip region crops extracted using MediaPipe Face Mesh from different
speakers and word classes during training.

68
 Figure 6.3: Extracted Lip region during training Diagram

A.4 Inference Demo Output:

This appendix presents the inference demo output, illustrating how the trained model predicts
words from real-time video input.

69
Figure 6.3-(A): Output prediction Diagram

70
Figure 6.3-(B): Output prediction Diagram

Figure 6.3-(C): Output prediction Diagram

71
 REFERENCES

[1] A. Gholipour, H. Mohammadzade, A. Ghadami, and A. Taheri, “Automatic Lip Reading of

Persian Words by a Robotic System Using Deep Learning Algorithms,” Iranian Journal of
Science and Technology, Transactions of Electrical Engineering, vol. 48, no. 4, pp. 1519–1538,
2024. [Online].

[2] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, “LipNet: End-to-End

Sentence-Level Lipreading,” arXiv preprint arXiv:1611.01599, 2016. [Online].

[3] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip Reading Sentences in the Wild,”
in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 6447–6456. [Online].

[4] J. S. Chung and A. Zisserman, “Lip Reading in the Wild,” in Asian Conference on
Computer Vision (ACCV), 2016, pp. 87–103. [Online].

[5] T. Stafylakis and G. Tzimiropoulos, “Combining Residual Networks with LSTMs for
Lipreading,” in Proc. Interspeech, 2017, pp. 3652–3656. [Online].

[6] K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata, “Audio-Visual Speech

Recognition Using Deep Learning,” Applied Intelligence, vol. 42, no. 4, pp. 722–737, 2015.
[Online].

[7] M. Abadi et al., “TensorFlow: A System for Large-Scale Machine Learning,” in Proc. 12th
USENIX Symp. Oper. Syst. Des. Implement. (OSDI), 2016, pp. 265–283. [Online].

[8] Gopika S., et al., “Custom Lip Reading Dataset (25 Words, 10 Speakers, 5 Repetitions),”
Kaggle, 2025. [Online].

72
73