International Journal on Advanced Electrical and Computer Engineering

Archives available at [Link]

International Journal on Advanced Electrical and Computer Engineering

ISSN: 2349-9338
Volume 14 Issue 01, 2025

Gesture and voice-based personal computer control system

Mr. Deepak Bhonde1, Mr. Khilesh Mongse2, Mr. Limesh Naikwar3, Mr. Nitesh Dwivedi4 ,Mr.
Om Mahulkar5

UG Student, Department of Computer Engineering, SCET, Nagpur, Maharashtra, India

1deepakbhonde8@gmail, 7558278392, 2mongasekhilesh@[Link], 9766553941,
3niteshdubeya@[Link], 9834264191, 4limeshnaikwar07@[Link], 9322672300,
5mahulkarom37@[Link], 7666071108

Peer Review Information Abstract

Submission: 07 Feb 2025 The rapid advancement of human-computer interaction

Revision: 16 Mar 2025 technologies has led to the development of more intuitive and
Acceptance: 18 April 2025 accessible control systems. This project introduces a gesture and
voice-based control system that enables users to interact with their
Keywords computers using hand gestures and voice commands, enhancing
convenience in situations where traditional input methods are
Gesture Control impractical. It utilizes Media Pipe Hands for real-time gesture
Voice Recognition recognition via a webcam and Speech Recognition in Python for
Human-Computer Interaction processing voice commands, mapping them to actions such as
opening applications, adjusting volume, taking screenshots, and
closing programs. A PyQt5 graphical user interface (GUI) allows
users to manage the system effortlessly. The results demonstrate
accurate recognition, enabling seamless computer control and
highlighting the potential of combining gesture and voice
recognition for a more natural, hands-free computing experience.
This technology has significant implications for accessibility,
providing an alternative input method for individuals with physical
disabilities. Future improvements may include expanding the
command library, enhancing accuracy, and integrating the system
with smart home devices or virtual assistants, showcasing the
transformative potential of gesture and voice-based interactions in
human-computer interaction.

Introduction spoken commands, making computing more

With the rapid advancement of technology, accessible, efficient, and seamless.
human-computer interaction has evolved A Gesture and Voice-Based Personal Computer
significantly, moving beyond traditional input Control System is designed to enhance user
devices such as keyboards and mice. The rise of experience by recognizing hand gestures and
artificial intelligence (AI), machine learning, and voice instructions to execute various tasks. This
computer vision has enabled the development of technology is particularly useful in scenarios
more natural and intuitive interaction methods, where hands-free operation is essential, such as
such as gesture and voice-based control systems. for people with disabilities, professionals in
These systems allow users to communicate with fields like healthcare and engineering, and users
their computers using hand movements and seeking a futuristic, touch-free computing
experience. With the integration of gesture

© 2025 The Authors. Published by MRI INDIA.

This is an open access article under the CC BY-NC-ND license ([Link]
Gesture and voice-based personal computer control system

recognition through cameras and voice while voice recognition can be affected by
processing through microphones, such a system background noise and speech variations.
can be used to open applications, control media, However, advancements in AI, sensor technology,
navigate documents, and perform system-level and machine learning algorithms continue to
actions without requiring any physical contact. improve the efficiency and reliability of these
systems.
The Role of Gesture and Voice Recognition In the future, gesture and voice-based computing
Gesture recognition is a technology that could become the primary mode of interaction,
interprets human hand and body movements eliminating the need for physical peripherals
using computer vision techniques. It typically altogether. With further developments in
involves tracking hand positions, detecting wearable devices, AR/VR, and brain-computer
finger movements, and identifying predefined interfaces, users will be able to seamlessly
gestures to perform corresponding commands. interact with computers using only their natural
This can be achieved through hardware such as movements and speech, paving the way for a
webcams, infrared sensors, and depth cameras truly touch-free digital world.
or software-based solutions using AI-powered
image processing. By analysing a user's hand LITERATURE SURVEY
movements, the system can trigger various The evolution of computer technology has
operations like scrolling through a document, heralded a paradigm shift in human-computer
switching between applications, or adjusting interaction, epitomized by the advent of
system settings. Multimodal Interaction Systems. This innovative
Similarly, voice recognition employs speech approach seamlessly amalgamates hand gesture
processing and natural language understanding recognition and voice recognition, forging a
(NLU) to recognize and interpret spoken dynamic interface that redefines user
commands. Modern voice recognition systems engagement. Leveraging the power of low-
leverage deep learning models to convert speech resolution webcams and OpenCV, the system
into text and match it with predefined commands. empowers users to navigate their digital realms
This allows users to perform actions like opening effortlessly through intuitive gestures. From
software, searching for files, dictating text, and precise cursor manipulation to seamless clicking
controlling smart home devices using only their and dragging, users wield a newfound agency
voice. The integration of gesture and voice over their computing experience.
recognition creates a multi-modal interaction VoiceGesture Fusion (VGF) is like teaching
system, providing an alternative and efficient computers to understand both our voices and
method of controlling a computer. hand movements so we can control them better.
One of the key benefits of a gesture and voice- In our day-to-day activities, wireless gadgets are
based control system is its ability to improve increasingly prevalent, with the computer mouse
accessibility for individuals with physical being a significant advancement in human-
disabilities who may struggle with traditional computer interaction. Even now, Bluetooth and
input devices. By enabling hands-free operation, wireless mice remain popular tools. However,
it also enhances productivity for professionals it's important to note that these wireless
working in fields where direct computer devices still require hardware, such as batteries
interaction is impractical, such as surgeons, for power and a dongle for connecting to the
factory workers, and mechanics. computer. This study looks at how VGF works,
Additionally, such systems offer a more natural the problems it faces, and what might be
and immersive user experience, making improved in the future. By combining how we
computing more intuitive. They can be talk and move, VGF helps us interact with devices
integrated into smart homes, virtual reality (VR), like phones or computers in easier ways. It's
and gaming to create a more engaging useful for things like games, virtual reality
environment. Moreover, with the growing use of Hand gesture mouse control has garnered
AI-driven assistants like Alexa, Siri, and Google significant attention due to its versatile
Assistant, voice-based control is becoming a applications and seamless integration with
standard feature in personal computing, machines through human-computer interaction.
extending beyond mobile devices to desktop and While traditional visual hand motion detection
laptop environments. systems are limited by lighting conditions and
Despite its advantages, the implementation of complex backgrounds, advancements in
gesture and voice-based systems faces computer vision and machine learning are
challenges such as accuracy, environmental driving the demand for enhanced human-
noise interference, and hardware limitations. machine interaction. The proposed methodology
Gesture recognition may struggle with poor offers a simple yet effective solution for rapid
lighting conditions or varying hand positions, manual tracking, overcoming the complexities of

191
 International Journal on Advanced Electrical and Computer Engineering

the past. This system not only tracks hand Voice Input:
movements and detects gestures but also • Captured via a microphone and processed
addresses issues like motion blur using speech recognition libraries such as
Elderly people face unique challenges when Google Speech API, CMU Sphinx, Deep
using conventional computer interfaces. Speech, or Microsoft Azure Speech.
Therefore, there is an essential need to model a • The system continuously listens for user
system for such people for easy accessing of commands while filtering out background
modern computer technologies. This paper noise.
presents an inventive solution, the "Gesture and
Voice Controlled Virtual Mouse" designed to 2. Data Pre-processing
improve the digital interaction experience for Raw input data is noisy and must be pre-
older individuals facing challenges with processed before being analyzed.
traditional computer input peripherals.
Employing advanced technology, this project Gesture Pre-processing:
establishes an intuitive interface using natural • Background removal to isolate hand
gestures and vocal commands. Gesture movements using techniques like
recognition employs cutting-edge computer background subtraction.
vision and machine learning models, for accurate • Image filtering and smoothing (Gaussian
interpretation of automatic hand gestures blur) to reduce noise.
extraction which is implemented using Media • Normalization and resizing to ensure
Pipe framework on top of pybind11 along with consistency across different lighting
OpenCV. conditions.
• Hand tracking and segmentation using
METHODOLOGY AND WORKFLOW OpenCV and MediaPipe Hand Tracking.
A Gesture and Voice-Based Personal Computer
Control System is an advanced human-computer Voice Pre-processing:
interaction mechanism that enables users to • Noise reduction and filtering to enhance
operate their computers using hand gestures and voice clarity.
voice commands. This approach eliminates the • Conversion to Mel-Frequency Cepstral
need for traditional input devices like keyboards Coefficients (MFCCs), which represent the
and mice, making computing more intuitive, unique features of human speech.
accessible, and efficient. It is particularly useful • Segmentation of speech into phonemes
for people with physical disabilities, for better accuracy in speech-to-text
professionals who need hands-free operations, conversion.
and users seeking a more immersive computing
experience. Feature Extraction and Classification
To develop such a system, a well-defined Once the input data is pre-processed, features
methodology and structured workflow are are extracted to classify gestures and recognize
required. This includes data acquisition, speech.
preprocessing, feature extraction, classification,
command mapping, and execution. By -Gesture Feature Extraction and Recognition:
integrating computer vision for gesture
• Key points of the hand (e.g., fingertips,
recognition and speech recognition technologies,
palm position) are extracted using Media
this system can interpret user inputs in real-time
Pipe, Open Pose, or Tensor Flow.
and translate them into actionable commands.
• Feature vectors representing hand
Methodology
positions, movement direction, and shape
are generated.
1. Data Acquisition
• Classification is performed using Machine
The first step in developing a gesture and voice-
Learning (ML) models like CNN
based control system is acquiring real-time input
(Convolutional Neural Networks), Support
data. This data comes from:
Vector Machines (SVM), or Decision Trees.
Gesture Input:
• Captured through a webcam, depth sensor
3. Voice Feature Extraction and Recognition:
(like Kinect), or infrared cameras (like Leap
• Spectrograms of audio signals are
Motion Controller).
generated and analysed.
• Frames are extracted in real-time for
• Speech-to-text conversion is performed
further processing.
using deep learning models like RNN
(Recurrent Neural Networks) or
Transformer-based models like Whisper.

192
Gesture and voice-based personal computer control system

• The recognized text is compared with • The speech input is converted into text
predefined voice commands. and matched against predefined
commands using NLP.
4. Command Mapping and Action Execution
After recognizing the gesture or speech Step 4: Mapping to System Commands
command, it must be mapped to a specific • Recognized gestures and voice
computer action. commands are compared to a command
database.
• A predefined command dictionary stores • If a match is found, the system executes
all possible gesture-action and voice-action the corresponding action.
mappings.
• When a command is recognized, the system Step 5: Executing the Command
translates it into corresponding system • The system uses automation libraries to
actions such as: trigger the appropriate command.
• File Operations: Opening, closing, deleting • The user receives visual feedback (GUI
files. changes) or audio confirmation.
• Media Control: Play, pause, adjust volume,
switch tracks. ADVANTAGES OF GESTURE AND VOICE-
• System Navigation: Switching applications, BASED PC CONTROL
minimizing/maximizing windows, 1. Hands-Free Operation: Allows users to
controlling the cursor. interact with their computer without touching a
• Custom User Commands: Executing user- keyboard or mouse.
defined automation tasks. 2. Improved Accessibility: Beneficial for
people with physical disabilities or those who
5. System Integration and Execution cannot use traditional input devices.
The recognized commands are executed using 3. Enhanced Productivity: Professionals in
system automation tools, including: healthcare, engineering, and industrial settings
• PyAutoGUI: Automates mouse movements can operate computers more efficiently.
and keyboard inputs. 4. More Natural Interaction: Mimics real-
• Windows Speech Recognition API / Linux world interactions, making computing more
Voice Commands: For OS-level control. intuitive and engaging.
• OpenCV + Tensor Flow Integration: For 5. Multi-Modal Input: Combines gesture and
real-time hand gesture execution. voice commands for better flexibility.

Workflow Of The System CHALLENGES AND FUTURE SCOPE

The system follows a structured workflow from Challenges
data collection to command execution. - Gesture Recognition Accuracy: Variations in
Step 1: Capturing User Input lighting, hand positions, and backgrounds can
• The system starts by continuously reduce detection accuracy.
capturing video frames for hand - Voice Recognition in Noisy Environments:
gestures and audio input for voice Background noise may interfere with speech
commands. recognition.
• Inputs are processed in real-time, - Hardware Limitations: High-performance
ensuring low latency. cameras and microphones are required for
accurate detection.
Step 2: Pre-processing the Data
• The captured gesture images are filtered, Future Improvements
and the background is removed to focus - AI-Powered Enhancements: Use deep learning
only on the hand. models like Vision Transformers (ViTs) for more
precise gesture recognition.
• Voice signals are denoised and
converted into meaningful - Better Noise Reduction: Advanced speech
enhancement techniques can improve voice
representations for analysis.
recognition.
Step 3: Recognizing Gestures and Speech - Integration with Wearable Devices: Smart
gloves or AR/VR devices can improve user
• The system classifies the gesture using a
experience.
deep learning model trained on various
hand positions and movements.

193
 International Journal on Advanced Electrical and Computer Engineering

TECHNOLOGIES AND TOOLS USED

Component Technology/Library Used
Gesture Detection OpenCV, Media Pipe, Tensor Flow
Hand Tracking Media Pipe Hand Tracking, Open Pose
Gesture Recognition CNN, SVM, Decision Trees
Voice Processing Google Speech API, CMU Sphinx, Deep Speech

Speech Recognition RNN, Transformer-based models

Automation Tools PyAutoGUI, Windows Speech API, Linux Commands

CONCLUSION
The Gesture and Voice-Based Personal Computer RESULT
Control System represents a significant leap in Dash Board of Project
human-computer interaction (HCI), making
computing more natural, intuitive, and
accessible. By leveraging computer vision for
gesture recognition and speech processing
technologies, this system enables users to
interact with their computers hands-free,
eliminating the need for traditional input devices
like keyboards and mice. This system is
particularly beneficial for individuals with
physical disabilities, professionals in hands-free
work environments, and users who seek a more
immersive computing experience. The References
integration of gesture recognition using OpenCV Mitra, S., & Acharya, T. (2007). "Gesture
and Media Pipe, along with voice recognition Recognition: A Survey." IEEE Transactions on
powered by deep learning models, allows for Systems, Man, and Cybernetics, Part C:
seamless, real-time execution of system Applications and Reviews, 37(3), 311-324.
commands. doi:10.1109/TSMCC.2007.893280
Through a well-defined system architecture, the
solution ensures efficient data acquisition, pre- Wu, Y., & Huang, T. S. (1999). "Vision-Based
processing, feature extraction, classification, and Gesture Recognition: A Review." International
execution of user commands. The use of machine Gesture Workshop. Springer, Berlin, Heidelberg.
learning models, speech recognition frameworks,
and automation tools like PyAutoGUI further Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T.,
enhances the system’s efficiency and usability. Finocchio, M., Moore, R., ... & Blake, A. (2011).
Despite its many advantages, the system faces "Real-time human pose recognition in parts from
certain challenges, such as gesture recognition single depth images." Proceedings of the IEEE
accuracy in varying lighting conditions, voice Conference on Computer Vision and Pattern
recognition in noisy environments, and Recognition (CVPR).
hardware limitations. Future improvements
could include AI-powered enhancements, better Rabiner, L. R. (1989). "A Tutorial on Hidden
noise reduction techniques, integration with Markov Models and Selected Applications in
wearable devices (such as smart gloves or AR Speech Recognition." Proceedings of the IEEE,
headsets), and support for additional languages 77(2), 257-286. doi:10.1109/5.18626
and dialects in speech recognition. In conclusion,
the Gesture and Voice-Based Personal Computer Koller, O., Zargaran, S., Ney, H., & Bowden, R.
Control System has the potential to revolutionize (2016). "Deep Sign: Hybrid CNN-HMM for
the way users interact with their computers. As Continuous Sign Language Recognition." British
advancements in artificial intelligence, deep Machine Vision Conference (BMVC).
learning, and sensor technologies continue, the
system will evolve to become even more efficient, Google Speech-to-Text API. (n.d.). Retrieved from
accurate, and widely adopted, paving the way for [[Link]
a touchless, intelligent, and futuristic computing text]([Link]
experience.
Media Pipe Hand Tracking. (n.d.). Google
Research. Retrieved from

194
Gesture and voice-based personal computer control system

[[Link]
ps://[Link]/mediapipe)

OpenCV Library. (n.d.). Retrieved from

[[Link]

Zhang, Z. (2012). "Microsoft Kinect Sensor and

Its Effect." IEEE Multimedia, 19(2), 4-10.
doi:10.1109/MMUL.2012.24

PyAutoGUI: Automating the GUI. (n.d.). Retrieved

from
[[Link]
[Link])

195