VISVESVARAYA TECHNOLOGICAL UNIVERSITY
BELAGAVI, KARNATAKA – 590018
2022-2023
Project
Presentation on
“MACHINE LEARNING BASED SPEECH AID FOR
SILENT COMMUNICATION”
Presented By:
LAKSHMANA 1SK19EC016
ANAND 1SK20EC400
KIRAN KUMAR S 1SK20EC409
MAHESH S 1SK20EC410
Under the guidance of:
Dr. N Sathisha
Assistant Professor
Department of Electronics and Communication Engineering
Government Sri Krishnarajendra Silver Jubilee Technological Institute
K. R. Circle, Bangalore - 560001
Contents
Introduction Hardware Setup
Literature Review Results
Aim & Objectives Limitations and Future works
Methodology Conclusion
Dataset References
Pattern Recognition
Flow chart
Hardware and Software Requirements
INTRODUCTION
1. Effective communication is essential for humans to express their emotions, convey their
opinions and ideas to others. The main objective of communication is to ensure that the
receiver comprehends the message that the speaker intends to convey.
2. Individuals with certain medical conditions such as paralysis, stroke, Amyotrophic Lateral
Sclerosis (ALS) and cerebral palsy lose their ability to speak and move their bodies.
3. Currently, there are conventional speech interfaces available in the market to assist these
individuals. However, these devices are often expensive and complex to use. Some of
these devices even require surgical implantation, adding to the complexity and cost.
4. It is essential to explore and develop advanced technologies that can assist these
individuals in communicating effectively without the need for invasive procedures or
Cont….
Prerequisites and important definitions for Silent
Communication:
Silent speech: Silent speech is a minimally or internally
articulating words without producing sounds or moving the
mouth.
Electromyograph (EMG) signals: Electromyography
signals are electrical signals generated by muscle
contractions.
sEMG signals :EMG signals collected from surface
electrodes into real speech.
Submental Triangle is a key location for EMG-based SSI
systems as it contains the mylohyoid muscle, which is an
important muscle involved in speech production.
Cont….
Silent speech
EMG Signals
Processing and
Pattern recognition
Computer Algorithms
Speech / Text
Literature Survey
SL NO TITLE OF THE PAPER AUTHOR AND YEAR OF OBSERVATION AND OUTCOMES
PUBLICATION
01 Parallel Inception CNN Jinghan Wu, Tao Zhao, Deep learning architecture named parallel
approach for Facials EMG Yakun Zhang inception convolutional neural network
based Silent Speech 2021 (PICCN) for up-to-date feature extraction.
Recognition Accuracy of 89% was achieved by this
proposed system.
02 A wearable graphene strain Dafydd Ravenscroft, Project aimed at designing a wearable
gauge sensor with haptic Ioanniks prattis, Tharun strain sensor, twisted and coiled polymer
feedback for silent Kandukarti and a graphene-based actuator and
communication 2021 developed a dataset for classification.
Haptic feedback was implemented to
inform the user or the listener on successful
translation.
Literature Survey
SL NO TITLE OF THE PAPER AUTHOR AND YEAR OBSERVATION AND OUTCOMES
OF PUBLICATION
03 A review on silent speech Jose E, Gonzalez Lopez This review focuses on providing new
interface for speech restoration 2020 alternative and augmentative
communication methods for the persons
with severe speech disorders.
04 Silent Speech Recognition by Andrzej [Link], Piotr This paper also describes the experimental
Surface Electromyography Pruchnicki, Przemysław setup and evaluation of the system using a
Plaskota, Piotr dataset of sEMG signals recorded from
Staroniewicz, Stefan eight healthy individuals performing silent
Brachmański and Maciej speech tasks. The results showed that the
Walczyński system achieved high accuracy in
recognizing silent speech commands, with
an average recognition rate of 91.5%.
Aim and Objectives
Aim:
To design ML based speech recognition aid for silent communication for disabled and
paralyzed people.
Objectives
To design a low-cost speech aid system, compare to existing system.
Use the available CSV data to train multiple ML models and compare each ML model
accuracy values to identify best EMG classifies and show result on web interface.
Electromyography signals are traced from the sensor (muscle bio amp candy sensor/
myoware muscle sensor).
To build hardware that can be interfaced with software and design a real-time system that
Methodology
Part A
1. Collect EMG CSV data from the internet.
2. Splitting available CSV data into Training, testing and validation datasets
3. Training ML model on dataset (which is Split into train and test datasets) ML model is trained 7
prominent classification algorithms such as, Support vector machine, K neighbor, Decision tree,
linear discriminant, quadratic discriminant and Naïve bayes.
4. Evaluate the performance of different machine learning algorithms and comparing their testing and
training accuracy of each model on split dataset.
5. Evaluating the performance by calculating the accuracy, precision and recall ability, and also the
F1 score for training and testing the model on all ML algorithms.
Cont…
6. Determining the best model for pattern recognition, based on the accuracy of individual
model.
7. For all these building a web application which shows the result on web interface
8. Showing the Step 4 result on streamlit based web application for all trained ML models.
9. Building the Functional switch button on Streamlit web interface so as to interface with the
hardware that will be built so as to predict the signals that occurs in real time.
Cont…
Part B
1. Design and implement a hardware system for the speech recognition process
2. Designing a simple system using muscle bioamp candy sensor that detects the muscle activity
and then assigning a different threshold value for letter A, E, I, O, U and Blank threshold
values.
3. Noting down the threshold values implementing signal processing and feature extraction using
models like CNN for real time signals and saving the Realtime CSV data to SD, so this will be
helpful to integrate with hardware built in Part A.
4. Making new dataset, training and testing new data set and testing the model on already trained
ML Models and test for real-time accuracy on all Machine learning Models.
DATASET
EMG data was collected from the internet.
The dataset contains EMG signals for the letters A, E, I,
O, U, and B, with a total of 1021 instances and 3000
attribute values.
To split the dataset into training, testing, and validation
sets, we used the scikit-learn library in Python.
Dataset randomly divided the data into 70% training,
20% testing, and 10% validation sets. This ensures that
the model is trained on a sufficiently large dataset while
also being tested and validated on new and unseen data.
Pattern Recognition
Algorithms Divisions
Linear Discriminant Linear Line
Quadratic Discriminant Quadratic Line
Native Bayes Independent Predictors
Tree Split into large sets
K Neighbors Closest data point
Ensemble Combining models
Support vector Machine Line with boundary
13
Flow Chart: Part A
Silent Speech EMG Split Dataset to Train
Train ML models
CSV Data and Test
Designing a web
interface to show result
Determining the best Evaluate Each Model
and should be
ML algorithm Performance
interfaced with part B
work
Flow Chart: Part B
Building a hardware Detect the signals Testing Sensor for
using Bio using sensor for muscle particular values and
amp/Myoware sensor Contractions at region observe threshold for
for Silent Speech of Submental triangle particular values
Using and testing Part Pattern recognition of
B work with Part A work Developing new csv EMG signals at same
and check for real time and testing this with threshold values using
working of the device. trained ML model models like CNN
Hardware/Software Requirements
Software requirements
1. Arduino IDE
2. Python+Google Colab
Hardware requirements
3. Arduino Mega 2560.
4. Bread board power supply stick 5v/3.3v.
5. Muscle Bioamp Candy Sensor
6. Myoware muscle sensor development kit.
7. 16 pin LCD
8. SD card.
9. Electrodes
10. Led's.
Hardware Setup
Results
ML Model train
Results
ML Model Train Test
Results
Model Evaluation
Results
Model Evaluation
Results
Model Evaluation
Results
Model Evaluation
Results
Building WEB interface
Results
Building WEB interface
Limitations and Future work
Limitations:
Sensor Limitations: The Muscle Bioamp candy sensor used in the project is only suitable
for hard muscle contractions and is not helpful in detecting speech signals. This limits the
ability of the system to accurately detect and recognize speech signals.
Cost of Sensor: The cost of the Myoware muscle sensor is high, which can be a limitation
for implementing the system in real-world scenarios.
Limitations and Future work
Future work:
Integration with a better sensor
Real-time speech recognition
Integration with other modalities
Improved feature extraction techniques: The current system uses CNN for feature
extraction. Future work can involve exploring other feature extraction techniques that can
improve the accuracy and performance of the system.
Development of a user-friendly interface
Conclusion
Our methodology involved collecting EMG data for the letters A, E, I, O, U, and B, and
splitting this data into training, testing, and validation datasets.
We trained seven different prominent classification algorithms, including Support Vector
Machine, K-Nearest Neighbors, Decision Tree, Linear Discriminant, Quadratic
Discriminant, and Naïve Bayes, on the split dataset.
After training the machine learning models, we evaluated their performance by calculating
accuracy, precision, recall, and F1 scores for both training and testing datasets.
We determined the best model for pattern recognition based on the accuracy of individual
models.
Conclusion
We then built a web application that shows the results on a web interface, which can be used to
switch the hardware on and off to take new real time sensor values.
In addition to the machine learning work, we designed and implemented a hardware system for the
speech recognition process.
However, we faced some limitations in our project. The muscle bioamp candy sensor we used was
not suitable for detecting speech signals and was only helpful for detecting hard muscle
contractions.
In conclusion, our project has demonstrated the potential of machine learning algorithms in
recognizing speech using EMG signals. Our results showed that the Random Forest Machine model
was the best performing model with an training accuracy (100%), and testing accuracy (79.41%).
References
[1]. Jinghan Wu, Tao Zhao, Yakun Zhang, “parallel inception CNN approach for Facial Semg BASED
Silent Speech Recognition”.
[2]. Dafydd Ravenscroft, Ioanniks prattis, Tharun Kandukarti “A wearable graphene strain gauge
sensor with haptic feedback for silent communication”.
[3]. Alaskr “convolutional neural network application in biomedical signals” (2018).
[4]. Arslan “Observations on the characteristics of EMG signals recorded at the different depths”2020.
[5] Benaroya, E-L., Chollet, G., Denby, B., Dreyfus, G., Stone, M., Development of a Silent Speech
Interface Driven by Ultrasound and Optical Images of the Tongue and Lips, Speech Communication
(2009)
[6]. D. E. King. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research, 2009
THANK YOU