HOVER: Virtual Mouse

Harsha R1, Manu P2, Mohan R3, Nishith P Shetty4, Asst. Prof. Sanjay Kumar5

1,2,3,4 BE CSE (Artificial Intelligence & Machine Learning) Student, Don Bosco Institute of Technology, Bengaluru,
Karnataka, India
5Assistant Professor, Dept. of CSE (Artificial Intelligence & Machine Learning), Don Bosco Institute of Technology,

Bengaluru, Karnataka, India

------------------------------------------------------------------------------------------------------------------------------------------
Abstract - "HOVER: Virtual Mouse" demonstrates the
potential of gesture-based interfaces in creating more 1.1 Objective
natural and immersive interactions with technology, paving
The main objectives include:
the way for future developments in touchless control systems.
 Develop a real-time gesture-based control system that
The system captures video input through a standard webcam,
utilizes hand recognition to perform essential computer
processes the frames to detect hand landmarks, and
functions such as cursor movement, volume adjustment,
interprets specific gestures to perform corresponding
and media navigation.
actions. Key gestures include moving the cursor, clicking,
 This project aims to provide an intuitive, accessible, and
scrolling, and executing keyboard shortcuts such as pressing
hands-free alternative to traditional input devices like
space, taking screenshots, and controlling media playback.
keyboards and mouse, enhancing user interaction.
Key Words: Gesture Recognition, Hand Tracking,  It addresses the limitations of physical devices,
particularly for users with disabilities or in situations
Touchless Control, Media-pipe, OpenCV, Real-Time
requiring contactless operation, while showcasing the
Interaction, Accessibility, AI-Powered Interface, potential of computer vision for seamless human-
Computer Vision, Cross-Platform Compatibility. computer interaction.

1. INTRODUCTION 1.2 Limitations Of Existing System

The "HOVER: Virtual Mouse" project is a pioneering  Gesture Recognition in Low-Light Conditions: One of
endeavor in the realm of touchless human computer the main challenges faced during testing was the
interaction. This innovative system utilizes advanced system’s ability to recognize hand gestures accurately
computer vision and machine learning techniques to in low-light environments
translate hand gestures into mouse and keyboard actions,  Accuracy of Gesture Recognition for Fast Movements:
providing an intuitive and efficient interface for users. The The system sometimes struggled with recognizing
primary components of this system include OpenCV for fast or rapid hand movements, resulting in missed
real-time video capture and image processing, Mediapipe gestures or delayed actions.
for robust hand landmark detection, and a Kalman Filter for  The Dependency on Web Camera Quality: The system
smoothing cursor movements. These technologies work in relies heavily on the quality of the webcam used.
concert to create a seamless user experience, enabling
 The System Performance on Lower-End Hardware:
precise control and a variety of functionalities through
While the system performed well on higher-end
simple hand gestures. hardware.
The HOVER system is designed to recognize a variety of
1.3 Scope
hand gestures and translate them into corresponding
mouse and keyboard actions. For instance, moving the
While the "HOVER: Virtual Mouse" project has achieved
index finger controls the cursor, while specific
significant milestones in creating a gesture based control
configurations of finger positions trigger actions like
system, there are several areas where further
clicking, scrolling, and executing keyboard shortcuts.
enhancements could improve its performance, usability,
Volume control is implemented by measuring the distance
and versatility. These improvements would not only
between specific landmarks on both hands, allowing for
address the limitations observed during testing but also
dynamic adjustment of audio levels. These functionalities
expand the system's capabilities to support a wider range
are achieved through a series of if-else conditions that
of expected usage and other major applications and user
interpret the landmark data and execute the appropriate
needs.
commands.
 2. PROPOSED SYSTEM
2.1 Design Details
hand landmarks. The module computes the relative
The system design for the "HOVER: Virtual Mouse" project is
positions of the hand's key landmarks and processes
built upon a combination of computer vision, machine
them to recognize different gestures, such as moving the
learning, and real-time input processing. The design
cursor or clicking. These steps ensure high-quality input
emphasizes intuitive, touchless interaction by translating
for machine learning models, improving performance.
hand gestures into meaningful computer commands such as
mouse movement, volume control, and media navigation. The
iii. Model Training: Cursor Control Module: This module is
system is broken down into several components: hand
responsible for translating the hand gesture movements
gesture detection, gesture interpretation, cursor control,
into mouse cursor movements on the screen. It maps the
action execution, and feedback provision. The system is
movement of the user's hand to the cursor's position
broken down into several components: hand gesture.
and uses smoothing algorithms to ensure the cursor
moves smoothly. The system uses PyAutoGUI to control
the mouse position on the screen. The hand movement
data is transformed into screen coordinates, allowing
the system to move the cursor. The Kalman filter is
applied to smooth the cursor's motion and minimize
jitter caused by inaccurate hand movement.

iv. Model Development: Action Execution Module Role:

This module is responsible for executing the commands
mapped from hand gestures, such as performing mouse
clicks, scrolling, and adjusting the volume. The system
uses PyAutoGUI for mouse clicks and scrolling. For
volume control, the PyCaw library is used to adjust the
Fig -1: Working of System system's audio settings based on the distance between
the thumb and index finger. The module interacts with
2.2 Methodology the operating system to perform these actions based on
The development of Medi-Genius: AI-Powered Health gesture [Link] the actions based on gesture
Insights follows a structured methodology involving recognition, including mouse clicks, scrolls, and volume
data collection, pre-processing, model training, testing, control, ensuring real-time interaction with the system.
and deployment. The platform integrates machine
learning techniques with a web-based interface to v. Model Evaluation: The HOVER undergoes rigorous
provide efficient and accessible healthcare insights. model evaluation to ensure accuracy, reliability, and
effectiveness Functionality: The module overlays the
i. Webcam Input module: This module handles the detected hand landmarks on the webcam feed, provides
capture of real-time video frames from the webcam, visual indicators for recognized gestures, and updates
which is the primary source of input for detecting hand the cursor position as the user moves their hand. It also
gestures. The webcam feed is continuously processed handles error detection, notifying the user if a gesture
to detect hand movements, finger positions, and is not recognize. Ensures that the system gives feedback
gestures. Functionality: The system utilizes OpenCV to to the user, confirming the recognition of thumb and
access the webcam and capture video frames. The index finger. The module interacts with the operating
frames are passed to the hand tracking system system to perform these actions based on gesture input.
(Mediapipe) for further processing. Responsibility: captures the video stream in real-time, and maintains
Ensures the webcam is initialized properly, captures the frame rate suitable gestures actions.
the video stream in real-time, and maintains the frame
rate suitable for gesture recognition.

ii. Hand Gesture Recognition Module: Role: This

module is responsible for detecting hand landmarks
and interpreting the hand's position to identify specific
gestures. Using the Mediapipe framework, it extracts
key points from the user's hand, such as the positions
of the fingers and joints. Mediapipe’s hand tracking
model is applied to the input frames to detect and track
vi. Deployment: Then there's geometry, which comes
into play when detecting gestures. The program
calculates distances between key hand landmarks
using the Euclidean distance formula, helping
determine whether fingers are spread apart or close
together. This is what allows it to recognize gestures
like controlling volume—when fingers move closer or
farther apart, the program maps that change to a
volume adjustment. Speaking of mapping,
interpolation is another key element. Since the
webcam and the screen have completely different
coordinate systems, the program needs to translate
hand positions into something that makes sense for
screen movement. By using [Link](), it effectively
scales hand positions from the camera feed to match
the screen dimensions, ensuring accurate cursor
control..

vii. Implementation: The implementation of the

"HOVER: Virtual Mouse" project involved combining
several technologies and frameworks to create a fully
functional gesture-based control system. The
following steps describe how the system was
implemented: Setting Up the Environment: The
development environment was set up using Python as
the programming language due to its powerful
libraries for computer vision, machine learning, and
GUI automation. The Visual Studio Code IDE was
chosen for coding due to its flexibility and extensive
support for Python. Required libraries were installed,
including: o OpenCV for handling webcam input and
image processing. o Mediapipe for hand tracking and
gesture recognition. o PyAutoGUI for controlling the
mouse cursor and simulating clicks. o PyCaw for
controlling system volume. o NumPy for numerical
operations and data manipulation. o FilterPy for
Kalman filtering to smooth the cursor’s movement.

viii. Result: The "HOVER: Virtual Mouse" project

successfully demonstrates the potential of gesture-
based control systems, providing a hands-free,
touchless alternative to traditional input devices such
as keyboards and mice. Through the integration of
advanced computer vision and machine learning
technologies, the system enables intuitive control of
computer functions, including cursor movement,
volume adjustment, and media navigation. By using
hand gestures, users can interact with their computers
in a more natural and accessible way, enhancing
usability, particularly for individuals with disabilities
in this within itself also or in environments
3) CONCLUSION:
REFERENCES:
The "HOVER: Virtual Mouse" project successfully
demonstrates the potential of gesture-based control
[1] Ö. Aydın, E. Karasarlan, “Touchless Control Systems: A
systems, providing a hands-free, touchless alternative
Review of Hover-Based Virtual Mouse Technology,”
to traditional input devices such as keyboards and
Journal of Emerging Technologies and Innovation
mice. Through the integration of advanced computer
Research, December 2022..
vision and machine learning technologies, the system
[2] S. Raj, K. M. Sharma, “Contactless Human-Computer
enables intuitive control of computer functions,
Interaction: A Hover-Based Virtual Mouse System,”
including cursor movement, volume adjustment, and
International Conference on Emerging Trends in
media navigation. By using hand gestures, users can
Computing and Communication, August 2023
interact with their computers in a more natural and
[3] M. Deshmukh, A. Patel, P. Roy, “Vision-Based Virtual Mouse
accessible way, enhancing usability, particularly for
Using Hand Gesture Recognition,” International Journal of
individuals with disabilities or in environments where
Computer Applications, March 2022. Kamakhya Narain
physical contact with devices is not ideal. The project
Singh,
successfully achieved its primary objective of creating
[4] NJibendu Kumar Mantri., “An intelligent recommender
an intuitive and responsive touchless control interface,
system using machine learning association rules and rough
highlighting the power of machine learning algorithms
set for hand gesture prediction from incomplete symptom
such as Mediapipe and the Kalman Filter in real-time
set”., Decision Analytics Journal., February 2024.
applications. The gesture recognition system proved to
[5] V. Reddy, S. N. Gupta, “Enhancing User Interaction with a
be accurate and effective under normal lighting
Hover-Controlled Virtual Mouse,” Journal of Interactive
conditions, and the system showed promise as a tool
Computing and Applications, July 2023.
for hands-free interaction. The integration of libraries
[6] Ömer Aydin, Enis Karaarslan., “OpenAI ChatGPT Generated
like PyAutoGUI for action execution and PyCaw for
Literature Review: Digital Twin in Gesture Prediction”.,
volume control further streamlined the overall
Journal of Emerging Technologies and Innovation
experience, making it possible for users to perform
Research., December 2022.
essential tasks without the need for traditional input
methods.

4) FUTURE SCOPE:

The "HOVER: Virtual Mouse" project has achieved

significant milestones in creating a gesture based
control system, there are several areas where further
enhancements could improve its performance,
usability, and versatility. These improvements would
not only address the limitations observed during
testing but also expand the system's capabilities to
support a wider range of applications and user needs.
Below are potential directions for future
enhancements: Improved Gesture Recognition
Accuracy: One of the key limitations identified during
testing was the system’s ability to accurately recognize
gestures in low-light conditions. Future versions of the
system could integrate better lighting perform a
broader range of actions, such as navigating web pages,
controlling media players, or interacting with specific
applications. demands of real-time hand tracking and
gesture recognition. To address this, future
enhancements could involve optimizing the code for
better performance on lower-end systems. This could
include reducing the resolution of video frames,
optimizing the machine learning model for faster
processing, or utilizing hardware acceleration through
GPUs or specialized processors