Gesture Controlled Mouse Using Python

Revanth Reddy Nidhi Patel Dheeraj Kumar

Department of C.S.E Department of C.S.E Department of AI and DS
Parul University Parul University Parul University
Vadodara, INDIA Vadodara, India Vadodara, INDIA
210303105395@[Link] nidhi.patel270183@[Link] 210303124676@[Link]

Manikanta Vinayak
Department of AI and DS
Parul University
Vadodara, INDIA
210303124569@[Link]

Abstract— A. Working Mechanism

In the age of contactless technology, gesture-controlled mouse
are innovative approaches to human-computer interactions by
1) Hand Detection & Tracking:
eliminating the need for physical peripherals such as traditional • The system captures live video from a webcam and
mice and touch pads. Provides. This paper uses computer vision processes each frame.
and machine learning to track hand movements in real time • Media Pipe Hand Tracking is used to detect and track
and control cursors on the screen with natural hand gestures.
Through a webcam, you use Media Pipe’s hand tracking API
key points on the user’s hand.
to process data using maps, from hand gestures to movements, 2) Gesture Recognition:
clicks, cards, and more. The index finger controls the movement • Specific hand gestures are identified using landmark po-
of the cursor and the thumb and index finger click on the
mouse. Additionally, the position of the middle finger compared
sitions (e.g., index finger extended for movement, thumb
to the index finger allows for smooth scrolling, providing a and index finger touching for clicking).
seamless, intuitive navigation experience. Fields such as gaming, 3) Mouse Control:
automation, and augmented reality. By combining real-time
• The system maps the detected hand movements onto the
image processing with smooth control mechanisms, the mouse
with a gesture-controlled mouse represents one step toward screen and simulates mouse operations like:
creating a futuristic, touch-free interface that improves user – Moving the cursor.
experience and convenience. – Left- and right-click actions.
Index Terms—Computer Vision, Gesture Recognition, Hand – Scroll up and down.
Tracking, Media Pipe, OpenCV, Real-time Processing, Virtual
Mouse, Touch-less Interface, Python Automation, PyAutoGUI – Drag-and-drop functionality.
II. K EY T ECHNOLOGIES U SED
I. I NTRODUCTION • OpenCV: Handles video capture and image processing.
• Media Pipe: A machine learning framework for efficient
In recent years, advances in computer vision and human- hand tracking.
computer interaction (HCI) have paved the way for innovative • PyAutoGUI / Autopy: Controls mouse functions pro-
input mechanisms that go beyond traditional devices such grammatically.
as keyboards and mice. One such innovation is the gesture-
controlled mouse, which allows users to control a computer’s III. LITERATURE REVIEW
cursor and perform various mouse operations using hand ges- Several approaches have been developed in the past to
tures. [2]This eliminates the need for physical touch, providing implement gesture-based virtual mouse control. Early systems
a more intuitive and contactless way to interact with digital relied on hardware-based methods such as glove-based gesture
systems. recognition, where users had to wear gloves embedded with
This project aims to develop a gesture-based virtual mouse sensors. These gloves would collect data and recognize hand
using Python and computer vision techniques, making use of movements. However, such methods had several drawbacks,
Open CV, Media Pipe, and PyAutoGUI (or Autopy) to track including inconvenience, limited mobility of the hand, and
hand movements and translate them into mouse actions. Using potential skin allergies due to prolonged use. Furthermore,
real-time image processing and machine learning-based hand the requirement for additional hardware made the system less
tracking, this system enables users to perform actions such as accessible and expensive.
cursor movement, clicking, scrolling, and dragging simply by Another approach involved color-based tracking, where
moving their hands in front of a webcam. users attached colored markers to their fingertips for gesture
recognition. [1]This method faced challenges in real-world • [8]Media Pipe Hands is a deep learning-based framework
applications due to lighting variations, inconsistent marker developed by Google that efficiently detects and tracks 21
detection, and difficulties in distinguishing background noise. hand landmarks in real time.
As a result, these approaches were not suitable for real-time • It uses a pre-trained [9]neural network that processes each
applications and lacked accuracy in performing precise mouse frame from the webcam, detecting the position of fingers,
operations. palms, and hand orientation.
Recent advances in computer vision (CV) and artificial • The output of this model includes x, y, and z coordinates
intelligence (AI) have led to software-based solutions that do of each landmark, which helps in identifying different
not require additional hardware. One of these contributions gestures.
came from the [3] Google MediaPipe framework, which in- 2) Convex Hull Algorithm (for Fingertip Detection):
troduced real-time hand tracking and gesture recognition using • The Convex Hull technique is applied to identify the
deep learning. This framework significantly improved the effi- outermost points of the hand contour.
ciency of gesture-based interfaces by detecting hand landmarks • It helps in detecting fingertips by forming a boundary
with high accuracy and [Link] current system builds on around the hand and identifying points where fingers
these advancements and leverages OpenCV, MediaPipe, and extend outward.
PyAutoGUI to create a gesture-controlled virtual mouse that • This algorithm is useful in recognizing open/closed hand
operates using a standard webcam. The system detects and gestures, which are essential for controlling mouse func-
tracks hand movements to perform essential mouse operations tions like clicks and scrolling.
such as cursor control, left-click, right-click, double-click, drag
and drop, scrolling, and system volume/brightness control. 3) Euclidean Distance Calculation (Cursor Movement Con-
Unlike traditional methods, this approach eliminates the need trol):
for gloves or external markers, [4]making it cost-effective, • The cursor movement is controlled by calculating the
user-friendly, and efficient. Euclidean distance between the detected fingertip and the
A key challenge in gesture recognition systems is ensuring previous cursor position.
real-time responsiveness. Some previous work required frames • This ensures smooth and precise movement, preventing

to be stored before processing, which increased latency and erratic behavior.

reduced usability for dynamic applications. The proposed sys- • The movement is mapped to the screen resolution to

tem processes frames in real-time, ensuring smooth and precise ensure accurate positioning.
control. It uses contour detection, [5] convex hull techniques, 4) Hand Gesture Recognition for Mouse Operations: The
and fingertip tracking to accurately recognize gestures. system assigns specific functions to different gestures based
Recent studies have also explored integrating deep learn- on finger positions and movements:
ing models such as YOLOv5 for gesture-based interfaces. • Index finger extended → Moves the cursor
While these methods enhance accuracy, they require high • Index and middle fingers extended (V-shape) → Performs
computational power and extensive dataset training. The cur- a left-click
rent approach balances accuracy and efficiency by utilizing • Thumb and index finger touching (Pinch gesture) → Drag
lightweight yet effective algorithms, making it accessible for and drop function
[6]real-time Human-Computer Interaction (HCI) applications. • All fingers open → Right-click
In summary, this research enhances the usability and ac- • Two fingers moved up/down → Scroll function
cessibility of gesture-controlled interfaces by implementing a • Fist closed → Stop cursor movement
camera-based, real-time, hardware-free virtual mouse. [7]The
system aims to provide seamless and intuitive control, improv- B. Tools and Libraries Used
ing human-computer interaction without additional peripher- 1) OpenCV (Open Source Computer Vision Library):
als.
• OpenCV is used for image processing and computer
IV. ALGORITHMS AND TOOLS USED vision tasks.
• It captures real-time video from the webcam and prepro-
To develop the Gesture Controlled Mouse Using Python,
cessed frames for gesture recognition.
various computer vision techniques, algorithms, and tools were
• Functions such as Gaussian blur, thresholding, and con-
utilized to ensure accurate real-time hand tracking and gesture
tour detection are applied to extract hand features.
recognition. [8]The system leverages OpenCV, MediaPipe,
and PyAutoGUI to process hand gestures and perform mouse 2) Media Pipe:
operations seamlessly. Below is a detailed explanation of the • Media Pipe provides a lightweight and efficient frame-
algorithms and tools used in the project. work for hand tracking.
• It runs on CPU and GPU, making it suitable for real-time
A. Algorithms Used applications.
1) Hand Tracking and Gesture Recognition - Media Pipe • The 21-point hand landmark model helps in detecting
Hands: hand position accurately.
 3) PyAutoGUI: 2) Hand Detection and Landmark Extraction:
• PyAutoGUI is a Python library used to simulate mouse • The MediaPipe Hands model is employed to detect hand

and keyboard interactions. landmarks in each frame.

• It allows the system to control the cursor, perform clicks, • It provides 21 key points corresponding to different parts

drag operations, and scroll actions based on detected of the hand (fingertips, joints, palm, etc.).
gestures. • The landmark coordinates are normalized and mapped to
screen resolution for accurate tracking.
4) NumPy:
3) Gesture Recognition and Classification:
• NumPy is used for mathematical operations and array
• Specific hand gestures are assigned to different mouse
manipulations in the gesture recognition process.
functions:
• It helps in processing coordinate data efficiently.
– Index finger extended → Cursor movement
5) Autopy (Alternative for Cursor Control): – Index and middle fingers extended (V-shape) → Left-
• Autopy is an alternative library that allows programmatic click
control of the mouse pointer. – Thumb and index finger touching (Pinch gesture) →
• It is used to implement cursor movement mapping across Drag and drop
different screen resolutions. – All fingers open → Right-click
6) Tkinter (GUI for Settings and Calibration – Optional): – Two fingers moved up/down → Scrolling
– Closed fist → Stop cursor movement
• Tkinter is used for creating a simple user interface for ad- • Gesture classification is performed using distance calcula-
justing sensitivity, tracking speed, and other parameters. tions between landmarks and the Convex Hull algorithm
• Users can modify settings to optimize gesture recognition for fingertip detection.
performance.
4) Mapping Hand Movements to Screen Coordinates:
• The detected hand position (from the webcam) is mapped
V. P ROPOSED M ETHODOLOGY
to the screen resolution for accurate cursor control.
The Gesture Controlled Mouse Using Python aims to pro- • The Euclidean distance formula is used to ensure smooth

vide a real-time, [10]hardware-free virtual mouse system by movement by reducing sudden jumps.
utilizing computer vision and AI-based hand tracking tech- • Scaling factors are applied to match the webcam frame

niques. The proposed methodology focuses on hand gesture size with the screen size.
recognition using a webcam and translating these gestures 5) Executing Mouse Functions:
into corresponding mouse actions, such as movement, clicks, • Once a gesture is recognized, corresponding mouse ac-
scrolling, and drag operations. [11]The system eliminates tions are performed using PyAutoGUI or Autopy.
the need for physical peripherals, making human-computer • The library simulates real mouse operations:
interaction more efficient and intuitive. – [Link](x, y) for cursor move-
ment
A. System Architecture – [Link]() for left-click
– [Link]() for right-click
The system consists of the following key components: – [Link](x, y) for drag-and-drop
• Real-time video capture using a webcam functionality
• Hand detection and tracking using the MediaPipe Hands • A smooth transition effect is applied to avoid abrupt
framework cursor jumps.
• Gesture recognition for identifying specific mouse actions 6) System Optimization and Calibration:
• Mapping hand movements to screen coordinates
• A settings interface (using Tkinter) can be provided
• Executing mouse functions using PyAutoGUI or Autopy
for adjusting tracking sensitivity and gesture recognition
The methodology follows a structured pipeline to ensure accuracy.
smooth and accurate gesture-based control of the mouse. • The system filters noise from background movements
using thresholding and frame averaging.
B. Step-by-Step Implementation • The frame rate (FPS) is optimized for real-time perfor-
mance.
1) Real-time Video Capture:
• The system uses a webcam to capture live video frames. C. Flowchart of the System
• OpenCV ([Link]) is used to continuously re- The system follows a structured workflow, as shown below:
trieve frames from the camera feed.
• Each frame is processed to detect hand landmarks and 1) Start
extract gesture features. 2) Capture video frame from webcam
 Fig. 1. Flow chart.

Fig. 3. Flow chart.

Fig. 2. Flow chart.

3) Detect hand and extract landmarks using MediaPipe

4) Identify gestures based on landmark positions
5) Map gestures to corresponding mouse actions
6) Perform the mouse operation using PyAutoGUI
7) Continue processing frames in real-time
8) End (when user exits the program)

D. Advantages of the Proposed System

• No external hardware required (only a webcam)
• Real-time, high-speed performance
• Highly intuitive and user-friendly
• Eliminates the need for physical touch (useful in hygiene-
sensitive environments)
• Adaptable for accessibility applications
VI. S OME OF THE F IGURES OF THE PROJECT
VII. R ESULTS AND I NFERENCES
The proposed Gesture Controlled Mouse system was suc-
cessfully implemented and tested under various conditions.
The system utilized hand gestures to perform essential mouse
functions such as cursor movement, left-click, right-click,
double-click, and drag-and-drop. The key observations and
results are summarized below:
A. Experimental Setup Fig. 4. Algorithm
The system was tested using a webcam with a resolution
of 720p under different lighting conditions. The processing
was performed using Python with OpenCV and MediaPipe, VIII. C ONCLUSION
ensuring real-time response. The proposed Gesture Controlled Mouse system suc-
B. Performance Analysis cessfully demonstrates an innovative approach to Human-
Computer Interaction (HCI) [13]using computer vision and
The system was evaluated based on accuracy, response time,
hand gesture recognition. By leveraging the capabilities of
and usability. The results are as follows:
OpenCV, MediaPipe, and PyAutoGUI, the system enables
users to perform essential mouse functions such as cursor
TABLE I
P ERFORMANCE M ETRICS OF THE G ESTURE C ONTROLLED M OUSE movement, clicking, dragging, and scrolling without the need
for physical peripherals.
Parameter Value
Cursor Movement Accuracy 95.3%
The implementation eliminates the requirement for external
Left Click Accuracy 93.8% hardware like gloves or colored markers, [14] making it a
Right Click Accuracy 92.1% cost-effective and user-friendly solution. Unlike traditional
Drag and Drop Success Rate 90.4%
Response Time 0.15 seconds
hardware-based approaches, this vision-based system ensures
a seamless and real-time interaction experience with high
accuracy.
C. Observations Despite its promising results, [15]the system has some
• The system performed well in good lighting conditions limitations, such as sensitivity to lighting conditions, back-
but had slight detection issues in low-light environments. ground noise, and camera quality. Future improvements could
• Real-time processing was achieved with minimal latency. focus on enhancing [16]gesture recognition accuracy, reducing
• The accuracy of gesture recognition varied slightly based processing latency, and integrating machine learning models
on hand positioning and distance from the camera. for adaptive gesture control.
• Background noise (other objects in the frame) sometimes In conclusion, this research highlights the potential of
affected the detection accuracy. gesture-based interfaces in reshaping the way [17]humans
interact with computers. As technology advances, such sys-
D. Inferences tems could be further optimized for applications in gaming,
From the above results, we infer that: accessibility tools, virtual reality (VR), and smart home con-
• The gesture-controlled mouse system is [18]highly accu- trol, paving the way for a more intuitive and immersive user
rate and provides a viable alternative to traditional input experience.
devices.
• Further improvements in environmental adaptability and
ACKNOWLEDGMENT
robustness can enhance the system’s usability. We sincerely thank PARULUNIVERSITY & C.S.E Dept
• With additional training and fine-tuning, the model can be for their support and resources in conducting this research. We
integrated into various [19]Human-Computer Interaction also extend our gratitude to our mentors Nidhi Patel and col-
(HCI) applications. leagues for their valuable guidance and feedback throughout
the project. Additionally, we acknowledge the contributions of
open-source communities and frameworks such as OpenCV,
MediaPipe, and PyAutoGUI, which played a crucial role in
the development of this system.

R EFERENCES
[1] V. Thakkar, R. Dangar, M. Singh, and N. Patel, ”DAPSys Soft-
ware,” in 2024 Parul International Conference on Engineering
and Technology (PICET), Vadodara, India, 2024, pp. 1-6. doi:
10.1109/PICET60765.2024.10716093.
[2] H. Bari and N. Patel, ”Generalized Immutable Ledger (GILED)
using Blockchain Technology,” in 2023 IEEE International
Students’ Conference on Electrical, Electronics and Computer
Sciences (SCEECS), Bhopal, India, 2023, pp. 1-7. doi:
10.1109/SCEECS57921.2023.10062983.
[3] D. L. Quam et al., “Gesture Recognition with a Dataglove,” in IEEE
Conference on Aerospace and Electronics, 1990, pp. 755-760.
[4] G. Wang et al., “Optical Mouse Sensor-Based Laser Spot Tracking
for HCI Input,” in Proceedings of the Chinese Intelligent Systems
Fig. 5. Comparison of accuracy across different gestures Conference, 2015, pp. 329-340.
[5] M. Baldauf and P. Frohlich, “Supporting Hand Gesture Manipulation
The implementation demonstrates that AI-driven gesture of Projected Content with Mobile Phones,” in European Conference on
recognition can efficiently replace conventional hardware- Computer Vision, 2013, pp. 381-390.
[6] R. Matlani, R. Dadlani, S. Dumbre, S. Mishra, and A. Tewari, “Virtual
based input devices, opening doors for further research in Mouse Hand Gestures,” in International Conference on Technology
contactless computing and assistive technology. Advancements and Innovations, 2021, pp. 340-345.
 [7] M. Yeshi, P. Kale, B. Yeshi, and V. Sonawane, “Hand Gesture Recog-
nition for Human-Computer Interaction,” in International Journal of
Scientific Development and Research, 2016, pp. 9-13.
[8] S. Shriram, B. Nagaraj, J. Jaya, S. Sankar, and A. P. Ajay, “Deep
Learning-Based Real-Time AI Virtual Mouse System Using Computer
Vision to Avoid COVID-19 Spread,” in Journal of Healthcare Engineer-
ing, 2021, pp. 3076-3083.
[9] N. S. Raj, V. Gobbur, P. Praveen, R. Patil, and V. Naik, “Implementing
Hand Gesture Mouse Using OpenCV,” in International Research Journal
of Engineering and Technology, 2020, pp. 4257-4261.
[10] U. Sneha, B. Monika, and M. Ashwini, “Cursor Control System Using
Hand Gesture Recognition,” in International Journal of Advanced Re-
search in Computer and Communication Engineering, 2013, pp. 2278-
1021.
[11] M. Krishnamoorthi, S. Gowtham, K. Sanjeevi, and R. V. Revanth, “Vir-
tual Mouse Using YOLO,” in International Conference on Innovative
Computing, Intelligent Communication and Smart Electrical Systems,
2022, pp. 1-7.
[12] K. S. Varun, I. Puneeth, and T. P. Jacob, “Virtual Mouse Implementation
Using OpenCV,” in International Conference on Trends in Electronics
and Informatics, 2019, pp. 435-438.
[13] F. Quek et al., “Towards a Vision-Based Hand Gesture Interface,” in
Proceedings of Virtual Reality Software and Technology, 1994, pp. 17-
31.
[14] R. M. Tharsanee, R. S. Soundariya, A. S. Kumar, M. Karthiga, and
S. Sountharrajan, “Deep Convolutional Neural Network-Based Image
Classification for COVID-19 Diagnosis,” in Data Science for COVID-
19, Academic Press, 2021, pp. 117-145.
[15] A. Newell, K. Yang, and J. Deng, “Stacked Hourglass Networks for
Human Pose Estimation,” in European Conference on Computer Vision,
Springer, Cham, 2016, pp. 483-499.
[16] V. Ramakrishna, D. Munoz, M. Hebert, J. A. Bagnell, and Y. Sheikh,
“Pose Machines: Articulated Pose Estimation via Inference Machines,”
in European Conference on Computer Vision, Springer, Cham, 2014,
pp. 33-47.
[17] G. Tharani, R. Gopikasri, R. Hemapriya, and M. Karthiga, “Gym Posture
Recognition and Feedback Generation Using Mediapipe and OpenCV,”
in International Journal of Advance Research and Innovative Ideas in
Education, 2022, pp. 2053-2057.
[18] K. H. Shibly, S. Kumar, M. A. Islam, and S. I. Showrav, “Design and
Development of Hand Gesture-Based Virtual Mouse,” in International
Conference on Advances in Science, Engineering and Robotics Tech-
nology, 2019, pp. 1-5.
[19] V. K. Sharma et al., “Virtual Mouse Control Using Hand Class Gesture,”
GIS Science Journal, 2020.
[20] E. Sankar et al., “Virtual Mouse Using Hand Gesture,” IJSREM, vol.
07, no. 05, May 2023.
[21] J. Prithvi et al., “Gesture Controlled Virtual Mouse with Voice Automa-
tion,” IJERT, vol. 12, no. 04, April 2023.
[22] B. K. R. Sandra et al., “Gesture Control-Virtual-Mouse,” IRJMETS, vol.
4, no. 4, April 2022.
[23] D. Rathod et al., “Mouse Control Using Hand Gesture Recognition,”
IJRPR, vol. 4, no. 3, March 2023.
[24] A. V. Karthik et al., “Virtual Mouse,” IJRPR, vol. 4, no. 10, October
2023.
[25] S. Shaikh et al., “Gesture Recognition Based Virtual Mouse and Key-
board,” IJPREMS, vol. 3, no. 5, May 2023.
[26] T. Geetha et al., “Human Computer Interaction Using Hand Gesture,”
IJAEM, vol. 3, no. 5, May 2021.
[27] I. Grishchenko and V. Bazarevsky, “MediaPipe Holistic: Simultaneous
Face, Hand and Pose Prediction on Device,” Google Research, USA,
2020.
[28] D. Gupta et al., “An Interactive Computer System with Gesture-Based
Mouse and Keyboard,” 2020.
[29] S. Nasr-Esfahani et al., “Hand Gesture Recognition for Contactless
Device Control in Operating Rooms,” arXiv preprint arXiv:1611.04138,
2016.