GESTURE BASED ROBOT CONTROL USING ML

A project report submitted to

Jawaharlal Nehru Technological University Kakinada, in the partial
Fulfillment for the Award of Degree of

BACHELOR OF TECHNOLOGY IN
INFORMATION TECHNOLOGY

Submitted by
CHALLA YASWANTH 22491A1213
CHEVURI RANA PRATHAP 22491A1216
MAKKENA CHAITANYA 22491A1234
MAKKENA PRAVALLIKA 22491A1235
MURABOYINA SANDEEP 22491A1239
NIDAMANURI SANTHI PRIYA 22491A1241

Under the esteemed guidance of

Dr. Ganesh Kumar M, [Link] Ph.D.
Assistant Professor

DEPARTMENT OF INFORMATION TECHNOLOGY

QIS COLLEGE OF ENGINEERING AND TECHNOLOGY

(AUTONOMOUS)
An ISO 9001:2015 Certified institution, approved by AICTE & Reaccredited by NBA, NAAC ‘A+’ Grade
(Affiliated to Jawaharlal Nehru Technological University, Kakinada)
VENGAMUKKAPALEM, ONGOLE – 523 272, A.P
2022-2026
 QIS COLLEGE OF ENGINEERING AND TECHNOLOGY
(AUTONOMOUS)
An ISO 9001:2015 Certified institution, approved by AICTE & Reaccredited by NBA, NAAC ‘A+’ Grade
(Affiliated to Jawaharlal Nehru Technological University, Kakinada)
VENGAMUKKAPALEM, ONGOLE: -523272, A.P
April 2026

DEPARTMENT OF INFORMATION TECHNOLOGY

CERTIFICATE
This is to certify that the technical report entitled “GESTURE BASED ROBOT CONTROL
USING ML” is a bonafide work of the following final B Tech students in the partial fulfillment of
the requirement for the award of the degree of bachelor of technology in INFORMATION
TECHNOLOGY for the academic year 2022-2026.

CHALLA YASWANTH 22491A1213

CHEVURI RANA PRATHAP 22491A1216
MAKKENA CHAITANYA 22491A1234
MAKKENA PRAVALLIKA 22491A1235
MURABOYINA SANDEEP 22491A1239
NIDAMANURI SANTHI PRIYA 22491A1241

Signature of the guide Signature of Head of Department

Dr. Ganesh Kumar M, [Link]., Ph.D. Dr. M. Senthil, M.E., Ph.D.,
Assistant Professor Professor & Head, IT Department

Signature of External Examiner

 ACKNOWLEDGEMENT
We thank the almighty for giving us the courage and perseverance in completing the project. It is an

acknowledgement for all those people for all those people who have given us their heartfelt

cooperation in making in making the major project a grand success.

We would like to place on record our deep sense of gratitude to the Honorable Executive Chairman

Dr. [Link] Chakravarthy, Honorable Executive Vice Chairman Dr. N. Sri Gayatri Devi and

Principal Dr. [Link] Rao for providing the necessary facilities to carry out the project

work.

We express our gratitude to the Head of the Department of IT, Project Guide Dr. Ganesh Kumar. M

and Department faculty for their valuable suggestions, guidance, and cooperation throughout the

project.

We would like to express our thankfulness to CSCDE & DPSR for their constant motivation and

valuable help throughout the project.

Finally, we would like to thank our Parents, Family and Friends for their cooperation in completing

this project.

Submitted by

CHALLA YASWANTH 22491A1213

CHEVURI RANA PRATHAP 22491A1216

MAKKENA CHAITANYA 22491A1234

MAKKENA PRAVALLIKA 22491A1235

MURABOYINA SANDEEP 22491A1239

NIDAMANURI SANTHIPRIYA 22491A1241

 ABSTRACT
The goal of this project is to develop a gesture-based mobile robot control system that allows a user
to manipulate the movements of a robot by using natural hand gestures and body posture. Without
the use of wearable technology, MediaPipe is used for real-time hand gesture recognition and pose
estimation while a camera records the user in real time. A deep learning model is used to categorize
recognized gestures and translate them into movement commands like forward, backward, left, right,
and stop. The robot's orientation and motion are continuously monitored by an IMU sensor that is
mounted on the robot. During robot operation, the IMU detects abnormal tilt, abrupt impact, or fall
conditions. The robot will automatically stop and send a status alert to a mobile application. Real-
time robot movement status, tilt condition, and safety alerts are displayed by the mobile app. Through
intelligent monitoring and feedback, this system enhances human-robot interaction while
guaranteeing robot safety.

This project proposes a gesture-based mobile robot control system that lets a user control robot
movement using natural hand gestures and body posture. A camera captures live video of the user,
and MediaPipe handles real-time hand gesture recognition and pose estimation without needing
wearable devices. The recognized gestures are classified using a deep learning model, which
converts them into movement commands like forward, backward, left, right, and stop. An IMU
sensor attached to the robot continuously monitors the robot’s orientation and motion. The IMU
detects abnormal tilt, sudden impact, or fall conditions during operation. If it detects a fall or unsafe
tilt, the robot automatically stops and sends a status alert to a mobile application. The mobile app
shows real-time robot movement status, tilt condition, and safety alerts. This system improves
human-robot interaction while ensuring robot safety through smart monitoring and feedback.
 TABLE OF CONTENTS

CHAPTER TITLE PAGE NO.

NO.

LIST OF TABLES i

LIST OF FIGURES ii

LIST OF SYMBOLS AND ABBREVIATIONS iii

1 INTRODUCTION 1-4

1.1 Introduction 1

1.2 Objective of the project 1

1.3 Motivation of the Thesis 1

1.4 Organization of the Thesis 2

1.5 Objective 3

1.6 Scope of the project 3-4

2 LITERATURE SURVEY 5-19

2.1 Basic Concepts 17

2.2 Related Work 18

 2.3 Research Gap
19
3. PROPOSED WORK AND ANALYSIS
20-21
3.1 Proposed System
20
3.2 Feasibility Study
21

4 SYSTEM DESIGN (FOR CSE & ALLIED)/ 22-39

/MATERIALS AND METHODS (FOR OTHERS)
4.1 System Design 22

4.2 Block Diagram 23

4.3 Modules 24

4.4 UML Diagrams 24-39

5 SOFTWARE AND HARDWARE INVOLVED 40-52

5.1 Software Requirements 40

5.2 Hardware Requirements 41-42

5.3 Technologies used 43-47

5.4 Coding 48-52

6 IMPLEMENTATION STEPS 53-59

7 TESTING 60-64

8 RESULTS&DISCUSSION 65

9 ADVANTAGES & APPLICATIONS 66

10 CONCLUSION 67

11 FUTURE SCOPE 68-69

12 REFERENCES 70-72
 LIST OF TABLES

TABLE TITLE PAGE

NO. NO.
4.1 ESP32 DevKitC Pinout Configuration 30

4.2 ESP32 Technical Specifications 31

4.3 L298N Module Pinout Configuration 33

i|Page
 LIST OF FIGURES

FIGURE TITLE PAGE

NO. NO.
4.1 System Architecture 23

ESP32 - DevKitC 27
4. 2

ESP32-WROOm-32 PINOUT 29
4.3

L298 Motor Driver 32

4.4

L298N Module 35
4.5

motor driver 36
4.6

TT Motor 37
4.7

dc motor 38
4.8

Arduino IDE 55
6.1

ii| P a g e
 LIST OF ABBREVIATIONS

Word Abbreviation

CNN Convolutional Neural Network

IMU Inertial Measurement Unit

ESP32 Espressif Microcontroller

DC Direct Current

Wi-Fi Wireless Fidelity

UI User Interface

IDE Integrated Development Environment

FPS Frames per second

iii| P a g e
 1. INTRODUCTION
1.1Introduction
Robots are becoming more and more common in homes, businesses, and healthcare settings where
effective and natural human-robot interaction is essential. Conventional robot control techniques, like
programmed paths or remote controllers, frequently lack flexibility and call for specific training. By
using human hand movements and body posture as input commands, gesture-based control systems
provide a more intuitive and natural way to communicate with robots.

Real-time gesture recognition and human pose estimation are now possible without wearable
technology thanks to recent developments in computer vision and deep learning as well as frameworks
like MediaPipe. By identifying falls, tilts, or abrupt impacts while in use, intelligent monitoring
systems like IMU sensors can further improve robot safety.

This project suggests a system that combines deep learning classification, gesture recognition, and
IMU-based safety monitoring to guarantee operational safety and allow for easy control of mobile
robots.

1.2 Objective Of the Project

The goal of this project is to create a gesture-based control system that eliminates the need for wearable
technology by allowing users to operate a mobile robot with natural hand gestures and body posture.
In order to accurately interpret user movements, the system uses MediaPipe and a deep learning model
for real-time gesture recognition and classification. The robot is equipped with an IMU sensor that
monitors safety conditions by identifying tilts, falls, and abrupt impacts while in use. Additionally, a
mobile application is created to show the user real-time robot status, tilt conditions, movement
commands, and safety alerts. The project's overall goal is to improve human-robot interaction while
guaranteeing dependable, safe, and effective robot operation.

1.3Motivation of the Thesis

The motivation behind this thesis comes from the growing need for simpler and more natural ways for
humans to interact with robots. In many existing systems, robots are controlled using physical
controllers, which can sometimes be difficult or inconvenient to use. Using hand gestures to control a
robot can make the interaction more intuitive and user-friendly. With recent developments in machine

f 1|Page
learning and computer vision, it has become possible to recognize human gestures accurately in real
time. This creates an opportunity to design smarter and more interactive robotic systems. Another
important motivation is to remove the need for wearable devices, making the system more comfortable
and practical for users. At the same time, ensuring the safety of the robot during operation is very
important, as robots can sometimes tilt, with objects. By integrating sensors and monitoring systems,
these unsafe conditions can be detected quickly. A mobile application can also help users monitor the
robot and receive immediate feedback. Therefore, this project focuses on developing a gesture-based
robot control system using machine learning to improve ease of use, safety, and human–robot
interaction.

1.4 Organization of the Thesis

This thesis is well-organized in such a manner that it clearly presents the research problem, concepts,
methodology, system implementation, and results of the research work. Every chapter of the thesis
describes a specific part of the research work in such a manner that it gives an overall understanding
of the developed system. The organization of the chapters is as follows:

Chapter 1 – Introduction: This chapter introduces the concept of gesture-based robot control using
machine learning. It describes the background of the research work, objectives of the project,
motivation behind the research work, and an overview of the thesis.

Chapter 2 – Literature Survey: This chapter discusses the basic concepts of gesture recognition,
machine learning, and human-robot interaction. It also discusses the existing research work in the
domain of gesture-controlled robotic systems.

Chapter 3 – Proposed Work and Analysis: This chapter describes the proposed gesture-based robot
control system using MediaPipe and machine learning. It describes the proposed system in detail.

Chapter 4 – System Design / Materials and Methods: The chapter provides information about the
system design and architecture, block diagrams, and the integration of hardware devices such as the
robot controller and IMU sensor for safety monitoring.

Chapter 5 – Implementation / Results and Discussion: The chapter provides information about the
implementation and integration of the system with different software tools and programming
languages used in the development. Additionally, the chapter includes information about the results
obtained from the developed system.
f 2|Page
Chapter 6 – Testing: The chapter provides information about the testing procedures used to evaluate
the performance of the system, such as gesture recognition and robot response with safety monitoring.

Chapter 7 – Results / Future Scope of Work: The chapter provides information about the final
results obtained from the project and possible improvements and future work to enhance the system.

Chapter 8 – Conclusion: This chapter presents a summary of the overall work carried out in the
project and the important contributions of the gesture-based robot control system.

Chapter 9 – Future Enhancements: This chapter presents various possible enhancements to the
system, including the addition of new gestures and better recognition accuracy using advanced AI
tools.

Chapter 10 – Appendix / References: This chapter contains additional materials and references used
in the thesis.

1.5 Objective

The main goal of a gesture robot using Machine Learning (ML) is to allow a robot to understand and
respond to human hand or body gestures. This lets people control it without touching it. By using
methods from Machine Learning and Computer Vision, the robot analyzes images or video captured
by cameras or sensors to recognize specific gestures from a user. Once it identifies a gesture, the robot
translates it into a command like moving forward, stopping, picking up objects, or doing a certain task.
This system aims to make interaction between humans and robots more natural, intuitive, and
accessible, especially in situations where traditional control methods like keyboards, remotes, or
touchscreens are hard to use. Gesture-controlled robots are valuable in fields like healthcare, industrial
automation, assistive technology for disabled individuals, and smart home systems. They enhance
efficiency, safety, and user convenience.

1.6 Scope of The Project

Touchless Control: Allows users to control robots using hand gestures without physical contact.

Human Robot Interaction: Improves natural communication between humans and robots using
Machine Learning.

Industrial Applications: Can be used to control machines or robots in factories to improve safety and

f 3|Page
efficiency.

Healthcare Support: Useful in hospitals where doctors can control devices without touching them
using Computer Vision.

Assistive Technology: Helps people with disabilities control devices or robots through simple
gestures.

f 4|Page
 2. LITERATURE REVIEW / BACKGROUND

1. Title: Human-Robot Interaction Using Dynamic Hand Gesture for Teleoperation of

Quadruped Robots with a Robotic Arm.

Authors: Jianan Xie, Zhen Xu, Jiayu Zeng, Yuyang Gao, Kenji Hashimoto. Journal/Conference:
Cecilio Angulo.

Year: 2025

Summary: This open-access paper presents a real-time hand gesture teleoperation system for
complex robots, which combine a quadruped and a robotic arm, using dynamic hand gestures. A
Depth-MediaPipe framework extracts 3D coordinates of 21 hand keypoints from depth video. A
Semantic-Pose to Motion (SPM) model interprets the pose and meaning of gestures, turning them into
robot actions for movement, tracking the end-effector, and switching commands.

2. Title: Research on Deep Learning-Based Human-Robot Static/Dynamic Gesture-Driven Control

Framework.

Authors: Gong Zhang, Jiahong Su, Shuzhong Zhang, Jianzheng Qi, Zhicheng Hou, and Qunxu Lin.

Journal/Conference: Abolfazl Zaraki and Hamed Rahimi Nohooji.

Year: 2025

Summary: This paper develops a dual CNN architecture. It includes a 2D-CNN for static gestures
and a 3D-CNN along with LSTM for dynamic gestures, achieving over 93% accuracy on their custom
gesture dataset. The work uses hand pose estimation with MediaPipe and depth camera data to turn
RGB frames into 3D keypoint coordinates. These coordinates are then used to calculate the hand pose
relative to the robot's base. Gesture classifications are converted into robot actions for grasping and

delivery tasks, showing strong performance in different lighting conditions.

3. Title: GestureMoRo: an algorithm for autonomous mobile robot teleoperation based on gesture
recognition.

f 5|Page
 Authors: Lei Chen, Chunxu Li, Ashraf Fahmy, Johann Sienz.

Journal/Conference: International Journal of Progressive Research in Engineering Management and

Science (IJPREMS)

Year:2024

Summary: This paper presents GestureMoRo, a gesture-based teleoperation system for mobile robots
using a LeapMotion sensor to capture hand motion. The proposed algorithm maps hand gestures to
locomotion commands, enabling the robot to move forward, backward, and turn based on intuitive
gestural input. A Gaussian filtering step smooths gesture signals to mitigate noise and hand tremors,
improving control stability. Experimental results demonstrate robust robot motion control with high
precision and low latency, making it suitable for real-time applications.

4. Title: Controlling Robot Car Movement Using Hand Gesture Recognition.

Authors: Kagitha Balasri & A. Durga Devi.

Journal/Conference: International Journal of Engineering & Science Research.

Year: 2025

Summary: This paper presents a vision-based deep learning method to control a robot car using hand
gestures captured by a camera. A modified CNN model is trained on a custom dataset of six gesture
classes corresponding to movement commands like forward, backward, left, right, stop, and no-
motion. The trained model yields ~98% accuracy and outputs discrete commands to an Arduino
controller for real robot actuation. Compared against Random Forest classifiers, the deep model
significantly improves gesture classification performance. Although focused on a robotic car rather
than full humanoid arms, it illustrates practical pipeline from camera input → deep gesture classifier
→ real-time robot control, relevant for gesture-based mobile robots.

5. Title: Hand Gestures Controlled Robot using Arduino.

Authors: Gurkirat Singh, Harpreet Kaur.

Journal/Conference: International Journal for Research in Applied Science & Engineering

Technology (IJRASET).

Year: 2023
f 6|Page
 Summary: This open-access paper introduces HyperPalm, a deep neural network interface for
controlling a quadruped robot in 3D space. It uses vision-based hand gesture recognition to control
robot pose and orientation without external devices. The system recognizes five static gestures and
estimates 6-DoF hand motion in real time, allowing intuitive remote control of robot locomotion and
orientation. The model achieves ~96.5% gesture accuracy and low deviation in positional estimation.
A user study also shows users find the interaction intuitive. This work highlights practical deep
learning pipelines for robust free-hand robot control in complex environments.

6. Title: "Gesture Controlled Robot Using Arduino"

Authors: [Shailaja Kadam, Vaishnavi Jadhav, Rajashree Takur]

Journal/Conference: [IJRASET]

Year: [2024]

Summary: Robot movement is driven by gesture data from a three-axis accelerometer. The
accelerometer data is processed by an Atmega328-based system (Arduino), which generates
movement commands for the robot.

7. Title: " Hand Gesture Control System using Open CV "

Authors: [Nikita Chaudhari, Kaustubh Sonawane, Arathi Boyanapalli]

Journal/Conference: [International Journal of Innovative Science and Research

Technology]

Year: [2023]

Summary: This paper presents a vision-based interactive system that enables users to operate a
computer without a mouse or keyboard by using real-time hand gestures captured via a standard
webcam. The system uses Python + OpenCV to detect and interpret hand gestures, allowing users
(including those with mobility impairments) to perform tasks such as menu selection or cursor
interaction through natural hand movements. It demonstrates a camera-based control system without
additional hardware.

8. Title: " Computer vision-based hand gesture recognition for human-robot interaction: a review “

Authors: [Jing Qi, Li Ma, Zhenchao Cui & Yushu Yu]

f 7|Page
 Journal/Conference: [IEEE Access that the abovementioned article should not be considered for
citation purposes].

Year: [2023]

Summary:

The proposed architecture was trained and validated in a ±21◦ range, resulting in a 100% correction
accuracy up to a ±23◦ range, whereas, for greater angles up to ±30◦, a drop in performances was
observed. In addition to that, when a random acceleration in the ±6 ◦ /s2 range was applied, the
proposed design showed a remarkable capability of predicting the correct PWM values, for both
reaction wheels, capable of reestablishing equilibrium in the system within an average intervention
time equal to 1.28s.

9. Title: "Research on Deep Learning-Based Human–Robot Static/Dynamic GestureDriven Control

Framework"

Authors: [Gong Zhang, Jiahong Su, Shuzhong Zhang, Jianzheng Qi, Zhicheng Hou]

Journal/Conference: [Sensors (MDPI)] Year: [2025]

Summary:

This study proposes a deep learning-based framework for human–robot interaction using both static
and dynamic hand gestures. Static gestures are recognized using a 2D-CNN, while dynamic gestures
are classified using a 3D-CNN combined with LSTM to capture temporal features. MediaPipe-based
hand pose estimation and coordinate transformation enable accurate robot control.

10. Title: " Hand Gestures Robotic Control Based on Computer Vision "

Authors: [Marthed Wameed & Ahmed M. Alkamachi]

Journal/Conference: [International Journal of Intelligent Systems and Applications in Engineering

(IJISAE)]

Year: [2023]

Summary:

This paper presents a real-time computer vision-based hand gesture recognition system that uses
f 8|Page
 the MediaPipe algorithm to detect and classify hand gestures from camera input. After identifying
21 3D hand landmark points and classifying gestures, the system maps each recognized gesture to
specific robot movements, enabling intuitive robot control. Experimental results show the approach
performs well under varying environmental conditions such as lighting and distance.

11. Title: Empowering human-robot interaction using sEMG sensor: Hybrid deep learning model for
accurate hand gesture recognition.

Authors: [Muhammad Hamza Zafara, Even Falkenberg Langåsa]

Journal/Conference: [It is recommended by the international journal of advance scientific research

and engineering trends].

Year: [2023]

Summary: This paper explores a hand gesture recognition approach using a Henry Gas Solubility-
based Stacked Convolutional Neural Network (HGS-SCNN) with sEMG signals. The methodology
involves generating a 600-sample dataset and applying Discrete Wavelet Transform (DWT) for
effective feature extraction.

12. Title: A review on deep learning for vision-based hand detection, hand segmentation and hand gesture
recognition in human–robot interaction.

Authors: [Reza Jalayer, Masoud Jalayer, Carlotta Orsenigo ]

Journal/Conference: [It is recommended by the international journal of advance scientific research

and engineering trends]

Year: [2026]

Summary:

This review examines recent deep learning–based approaches for hand detection, segmentation, and
gesture recognition in human–robot interaction (HRI). The authors analyze state-of-the-art DL models,
datasets, evaluation metrics, and applications across industrial, assistive, social, and aerial robotics.

13. Title: Three-dimensional dynamic gesture recognition method based on convolutional neural network.

f 9|Page
 Authors: [Ji Xi, Weiqi Zhang, Zhe Xu, Saide Zhu, Linlin Tang, Li Zhao]

Journal/Conference: [It is recommended by the international journal of advance scientific research

and engineering trends].

Year: [2025]

Summary: This paper addresses the challenges of dynamic gesture recognition for human–computer
interaction in virtual environments. The authors propose an improved ResNeXt-based model
incorporating 3D convolutions and a lightweight attention mechanism to capture spatiotemporal
features effectively. A deep attention submodule is further introduced to enhance temporal feature
extraction and improve recognition accuracy.

14. Title: Single and Multi-Hand Gesture based Soft Material Robotic Car Control

Authors: [Basanta Kumar Swain, Ashok Kumar Bhoi]

Journal/Conference: [It is recommended by the international journal of advance scientific research

and engineering trends]

Year: [2024]

Summary:

This research presents an AI- and ML-based system for controlling a soft material robotic car using
single- and multi-hand gestures. Hand detection and gesture recognition are performed using
MediaPipe landmarks to enable user-centric control in 360-degree movement. Different hand
landmark combinations are mapped to specific robotic actions for multi-directional control.
Experimental results show that the proposed CNN model outperforms traditional ML algorithms such
as KNN, Decision Tree, and SVM in terms of accuracy and effectiveness.

15. Title: IoT-Based Hand Gesture Control Robot

Author: Kathan Patel et al.

Journal: International Journal of Engineering Research & Technology (IJERT)

Year: 2023

Summary: The researchers developed a robotic system which operates through IoT technology to
f 10 | P a g
e
 detect human hand motions. The system utilizes sensors and wireless technology together with
machine learning algorithms for gesture recognition which then enables robot control. The study
demonstrates that robots which people control through gestures provide value to both automated
systems and intelligent environmental setups.

16. Title: Hand Gesture Recognition Using Machine Learning.

Author: Pratik A. Yadav and R. S. Patil

Journal: International Journal of Innovations in Engineering Research and Technology

Year: 2023

Summary: The authors propose a machine learning model that recognizes hand gestures using image
processing techniques. The system uses recognized gestures to create commands which control robotic
operations. The study shows that machine learning technology enables better gesture detection results
which lead to better human-machine communication methods.

17. Title: A Novel Machine Learning-Based Hand Gesture Recognition Using HCI on IoT Assisted Cloud
Platform.

Author: S. Adhikari et al.

Journal: Computer Systems Science and Engineering

Year: 2023

Summary: The research presents a machine learning model which uses cloud technology for gesture
recognition purposes. The system applies human-computer interaction methods together with machine
learning algorithms to process gesture information. The research demonstrates that cloud-based
systems enhance recognition accuracy while improving the performance of robotic control systems.

18. Title: Hand Gesture-Based Teleoperation Control of a Mecanum-Wheeled Mobile Robot

Author: Various Researchers

Journal: IFAC-PapersOnLine / ScienceDirect

Year: 2023

Summary: The researchers developed a mobile robot system which people control through hand
f 11 | P a g
e
 movements detected by a webcam. The system works to identify gestures which it transforms into
speed control instructions. The system shows successful real-time gesture recognition which achieves
high accuracy while needing minimal processing power.

19. Title: EMG-Based Hand Gesture Recognition for Assistive Robots Using Deep Learning

Author: Pranesh Gopal et al.

Journal: Sensors

Year: 2022

Summary: The research investigates gesture recognition through electromyography (EMG) testing
which records signals from human arm muscles. The researchers used machine learning and deep
learning models to classify gesture signals which were used to control assistive robots. The study
demonstrates that EMG-based gesture recognition system can enhance both robotic prosthetic devices
and rehabilitation technology.

20. Title: Surface EMG Driven Gesture Recognition Using Machine Learning for Robotic Applications.

Author: Iva Valentinova Valcheva et al.

Journal: Discover Robotics (Springer)

Year: 2026

Summary: The research investigates the use of surface EMG signals to recognize human gestures for
controlling robots. Machine learning algorithms analyze muscle signals from the human arm and
translate them into robot commands. The system shows better performance for robotic control tasks.

21. Title: Ultra-Range Gesture Recognition for Human-Robot Interaction.

Author: Eran Bamani et al.

Journal: arXiv Research Publication

Year: 2023

Summary: The authors present a deep learning system which allows robots to identify gestures from
distant locations. The system combines a vision transformer model with graph convolution networks
to achieve better accuracy in gesture detection. The proposed method enhances human-robot
f 12 | P a g
e
 interaction in large environments.

22. Title: Online Recognition of Gesture Data for Collaborative Robots

Author: M. A. Simão et al.

Journal: arXiv Research Publication

Year: 2023

Summary: The researchers developed an online gesture recognition system which enables robots to
work together with human operators. The researchers use machine learning models which include
random forest and neural networks to identify different gesture patterns. The system achieves high
accuracy and supports real-time human-robot collaboration.

23. Title: Body Gesture Recognition to Control a Social Robot

Author: Javier Laplaza et al.

Journal: Robotics Research Publication

Year: 2022

Summary: This research focuses on body gesture recognition for controlling social robots. The
researchers employ neural networks to identify human body movements which they use to operate
robotic systems. The results show that gesture-based interaction enhances user experience in social
robotics applications.

24. Title: Hybrid Deep Learning Model for Static and Dynamic Hand Gesture Recognition

Author: Hassan Ali et al.

Journal: Machine Learning Research Publication

Year: 2022

Summary: The authors developed a hybrid neural network model which enables users to identify both
static and dynamic hand gestures. The system combines convolutional neural networks with temporal
motion analysis methods. This approach improves recognition performance and supports real-time
robotic control.

f 13 | P a g
e
25. Title: Gesture Recognition Using Computer Vision for Robotic Navigation

Author: Various Researchers

Journal: Robotics and Intelligent Systems Journal

Year: 2024

Summary: The researchers present a robotic navigation system that uses computer vision technology
to recognize human gestures. The system uses machine learning algorithms to analyze camera footage
and detect hand movements that control robot operations. The research demonstrates that vision-based
systems can function as effective replacements for traditional control methods.

26. Title: Deep Learning-Based Human Gesture Recognition for Smart Robotics

Author: Various Researchers

Journal: IEEE Robotics Research

Year: 2024

Summary: The researchers developed a gesture recognition system that uses deep learning technology
to identify human gestures in intelligent robotic environments. The system uses CNN models to
classify gestures with high accuracy. The system enhances industrial automation robotic control
through improved operational efficiency.

27. Title: Vision-Based Gesture Control for Autonomous Robots

Author: Various Researchers

Journal: Robotics and Automation Journal

Year: 2023

Summary: The research investigates how people use their hands to communicate through different
hand gestures. The system uses machine learning algorithms to identify hand movements which
researchers use to develop robot control systems. The system maintains consistent performance
throughout the entire duration of its operation in actual usage situations.

28. Title: Machine Learning Approach for Gesture-Controlled Robotic Systems

Author: Various Researchers

f 14 | P a g
e
 Journal: International Journal of Robotics Research

Year: 2024

Summary: This research introduces a machine learning framework which enables robots to be
controlled through human gestures. The system employs feature extraction methods together with
classification algorithms for gesture recognition. The study demonstrates enhanced robot control
systems which show both improved responsiveness and higher accuracy levels.

29. Title: Real-Time Gesture Recognition for Human-Robot Collaboration

Author: Various Researchers

Journal: Robotics and Intelligent Systems

Year: 2023

Summary: The research describes an actual time gesture recognition system which robots use during
collaborative work. Deep learning models analyze video input to identify gestures and control robotic
actions. The system improves efficiency in human-robot teamwork.

30. Title: CNN-Based Gesture Recognition for Robotic Arm Control

Author: Various Researchers

Journal: IEEE Access

Year: 2024

Summary: This research proposes a convolutional neural network model for recognizing hand
gestures to control a robotic arm. The system processes camera images and converts gestures into
robot commands. The experimental results show that the system achieves accurate results with
minimal delays.

2.1 Basic Concepts

In this section, the basic concepts are introduced, which are the building blocks of the proposed
gesture-based robot control system with machine learning technology. The proposed robot control
system uses computer vision and machine learning technology with embedded sensors to provide
natural human-robot interaction and safety while interacting with robots.

f 15 | P a g
e
Human-Robot Interaction
Human-Robot Interaction is the interaction and communication between humans and robots. The
current robot control systems are often based on physical controllers or joysticks and sometimes
require the user to interact with the robot by typing on the keyboard. Though these systems are
effective in robot control, they may not provide natural interaction between humans and robots.
Gesture-based robot control systems provide better interaction between humans and robots.
Gesture-based robot control systems allow humans to interact with robots naturally by performing
different gestures with their bodies and hands.

Machine Learning

Machine learning is a subset of AI that enables computers to learn from the data and make decisions
accordingly. Gesture recognition systems make use of machine learning algorithms that are trained
with various datasets of gestures and are then used to classify different types of human gestures. The
trained machine learning models are then used to detect gestures and convert them into robot control
signals.

MediaPipe Framework

MediaPipe is an open-source framework used to develop real-time computer vision applications. It is

used to develop efficient applications that detect and track various human body landmarks such as
hands and faces. MediaPipe is used in this project to detect hand and body landmark coordinates from
the video feed captured by the camera. The detected landmark coordinates are then used as input
features to train the machine learning models and detect gestures.

Mobile Robot Control

A mobile robot refers to an automated system that can move within the environment. The gesture-
based control system provides the mobile robot with the commands produced by the gesture
recognition model. The commands provided include the movement directions such as moving forward,
backward, left, and right. The mobile robot processes the provided commands using the
microcontroller.

IMU Sensor for Safety Monitoring

f 16 | P a g
e
An Inertial Measurement Unit (IMU) refers to an electronic device that measures the orientation,
angular velocity, and acceleration of the system. The IMU sensor is an essential component that can
be applied in the field of robotics for safety monitoring. The IMU sensor is applied in the robotic
system for safety monitoring, ensuring the system is safe from tilting, falling, and impacts.

Mobile Application for Monitoring

The mobile application is used to give the user feedback in real time regarding the status of the robot.
This includes the commands given to the robot for moving, the tilt status of the robot, etc. This feature
will enable the user to monitor the status of the robot from a distant location and respond accordingly
in case of any dangerous situation.
These basic concepts will give us the theory for the proposed gesture-controlled robot using machine
learning with safety monitoring.

2.2 Related Work

The advancements in artificial intelligence and robotics have led to the development of intelligent
machines that can interpret gestures made by humans for controlling the machines. A number of
researchers have focused on the development of various technologies related to gesture recognition
for the improvement of human computer interactions.
A number of researchers have focused on the development of vision-based gesture recognition systems
using machine learning and computer vision techniques. Such systems have the potential to recognize
various gestures made by the user using cameras that capture the movements of the hands of the user.
A computer vision library called MediaPipe has been widely used for detecting the pose of the hands
of the user.
Machine learning has been widely used for the classification of gestures. Supervised learning has the
potential to learn the gestures and classify them according to the predefined commands. Deep learning
techniques such as CNN and neural networks have been widely used for the recognition of gestures.
Gesture-based robotic control systems have also been explored in the domain of robotics. In these
systems, users can control the movement of the robots using gestures instead of using controllers. Such
approaches are beneficial for improving the ease of use of robotic systems and making them more
accessible.
Safety monitoring is another important aspect of robotic systems. Various research works have
f 17 | P a g
e
incorporated accelerometers and gyroscopes for safety monitoring in robots. The IMU sensor provides
real-time information about the orientation and movement of the robots. Thus, unsafe conditions such
as tilting or collision can be easily identified.
Although various gesture-based robotic control systems have been proposed, most of these approaches
require the use of various wearable devices such as gloves or sensors that are attached to the human
body for the purpose of gesture recognition. Such approaches are likely to cause discomfort for the
users. Vision-based gesture recognition using cameras is more convenient for the users.

2.3 Research Gap

Although existing studies have explored gesture recognition and robot control systems, several
limitations still exist. Many gesture-based control systems rely on wearable devices such as sensor
gloves, which can be uncomfortable and impractical for everyday use. Additionally, some vision-
based systems lack real-time accuracy and struggle to recognize gestures reliably in dynamic
environments.

Another limitation is the lack of integrated safety monitoring in many robotic control systems. Robots
may experience unsafe conditions such as tilting, falling, or collisions during operation, but these
issues are not always detected in real time. Without proper monitoring mechanisms, such conditions
may lead to system failure or damage.
Furthermore, most existing systems focus only on gesture recognition without providing real-time
feedback to the user. The absence of a monitoring interface makes it difficult for users to track robot
status and safety conditions.
Therefore, there is a need for an improved system that combines real-time gesture recognition,
machine learning-based classification, robot safety monitoring using IMU sensors, and a mobile
application for user feedback. The proposed system addresses these limitations by developing a
gesture-based robot control framework that ensures intuitive interaction, reliable gesture recognition,
and safe robot operation.
Therefore, it is necessary to have an improved system that incorporates real-time gesture recognition,
machine learning classification, robot safety monitoring via IMU sensors, and a mobile app for user
feedback. The proposed system aims to overcome these limitations by developing a gesture-based
robot control system that ensures ease of use, gesture recognition, and safe operation of the robot.

f 18 | P a g
e
 3. PROPOSED WORK AND ANALYSIS

The proposed system is a gesture-controlled, real-time safety monitoring mobile robot. The proposed
system consists of three main parts: Gesture Recognition, Robot Control & Motion, and IMU-based
Safety Monitoring with Mobile App Feedback.

3.1 Gesture Recognition Module

• Video of the user is obtained in real time using a camera.

• Using the MediaPipe library, hand gesture recognition is achieved without any wearable
devices.
• The recognized hand gestures are passed to a deep learning model, which maps different hand
gestures to different robot actions:
+ Forward
+ Backward
+ Left
+ Right
+ Stop.

3.2 Robot Control & Motion Module

• The obtained hand gestures are sent to the robot as commands using a wireless communication
protocol.
• The execution of these hand gestures is achieved by microcontrollers in the robot, which
execute these actions.
• The robot’s state is monitored, ensuring accurate movement of the robot, as well as natural
interaction with the robot.

3.3 IMU-based Safety Monitoring with Mobile App Feedback

• An IMU sensor is used in this part of the proposed system, which monitors the state of the
robot in real time. The IMU sensor is capable of detecting:
+ unsafe tilts
+ sudden impacts
f 19 | P a g
e
 + falls
• If any of these states occur, the robot is stopped to prevent damage.

3.4 Mobile Application Interface

• The mobile application is fed with real-time information from the robot.
• The information is displayed as follows:
• The status of the robot movement
• The robot tilt and orientation
• The safety status
• The interface provides information to the user about the robot status so that the robot is used safely
and effectively.

3.5 System Workflow

1. The user performs a hand gesture.

2. The image is captured by the camera and sent to MediaPipe to detect hand landmark information.
3. The image is sent to the deep learning model to determine the robot command from the hand gesture.
4. The robot command is sent to the robot to perform the required action.
5. The robot stability is monitored by the IMU.
6. If the robot is not in a stable state, an alert is sent to the mobile app to stop the robot.
The entire system is integrated so that the robot is used safely and effectively.

3.6 Proposed System Analysis

The proposed system offers a comprehensive framework for integrating gesture recognition, robot
control, and safety detection. The use of the MediaPipe framework, in addition to a deep learning
model, enables accurate hand gesture detection without the use of wearable technologies. The
integration of an IMU sensor into the proposed system enables the robot to be safe by detecting tilts,
falls, and impacts. The use of a mobile application offers real-time feedback to the user. The proposed
system is better than existing robot control systems in terms of interaction, safety, and reliability.

f 20 | P a g
e
 [Link] DESIGN/MATERIALS AND METHODS

4.1 SYSTEM DESIGN

4.1.1 Input Design

Input design is concerned with the approaches that are used for the input of the system for processing.
In the proposed system for controlling the robot using gestures, the input for the system is the gesture
made by the user. The gesture made by the user is captured using the camera. The captured video is
then processed using the mediapipe framework, which detects the hand and the features of the gesture
made by the user.
The features are then input into the machine learning algorithm, which analyzes the gesture made by
the user and classifies the gesture into the various classes that are recognized by the system. The system
has been trained to recognize various gestures that are associated with the movement of the robot. The
gestures include moving the robot forward, backward, left, and right, as well as stopping the movement
of the robot.
The IMU sensor also acts as an input for the system. The IMU sensor is used for the detection of
unsafe states such as tilting, impacts, and falling of the robot. The orientation, acceleration, and angular
velocity of the robot are constantly monitored using the IMU sensor.

4.1.2 Output Design

The output design refers to the information that the system will generate after processing the input
information. In the proposed system, the output information will be the movement of the robot based
on the recognized hand gesture. After the machine learning model identifies the hand gesture, the
appropriate command will be sent to the microcontroller of the robot. The microcontroller will then
move the robot accordingly to the desired movement, such as moving forward, moving backward,
turning left, turning right, or stopping.

Another output information that the proposed system will generate is the information displayed on the
mobile application for monitoring the status of the robot in real-time. The information will include the
f 21 | P a g
e
status of the robot’s movement, the tilt status of the robot, and any alarm generated by the IMU sensor.
In the event that the IMU sensor detects any dangerous status, the robot will be stopped automatically,
and a warning message will be sent to the mobile application.

4.2 System Architecture

Fig:4.1 System Architecture

The proposed system shown in the above figure 4.1 has several parts that work together in order to
perform the desired functions of gesture-based robot control and safety monitoring. The system
architecture includes a camera for capturing hand gestures, a computer vision module for gesture
detection, a machine learning module for gesture classification, a microcontroller for robot control,

f 22 | P a g
e
 an IMU sensor for safety monitoring, and a mobile application for providing feedback.

The camera is used for capturing the hand gestures of the user in real time and sends the video frames
to the computer vision module for gesture detection. The MediaPipe library is used for image
processing in order to detect hand landmarks and extract features from the image. These features are
then used by the machine learning module in order to classify the gesture and identify the
corresponding robot command. The command is then wirelessly transmitted to the microcontroller of
the robot in order to perform the desired movement. At the same time, the IMU sensor is used for
monitoring the orientation of the robot and for safety monitoring. In case of any unsafe conditions,

the system will stop the robot and provide a notification through the mobile application.

4.3 Modules

1. Gesture Recognition Module

This module is responsible for detecting and recognizing the user’s hand gestures. A camera captures
the user's hand movements in real time. The captured images are processed using the MediaPipe
framework, which detects hand landmarks and extracts gesture features. These features are then
passed to the machine learning model to classify the gesture.

2. Gesture Classification (Machine Learning) Module

In this module, the extracted gesture features are analyzed by a machine learning model. The model
identifies the gesture and maps it to a specific robot command such as forward, backward, left, right,
or stop. This module ensures accurate interpretation of user gestures.

3. Communication Module

This module sends the gesture commands identified by the computer or laptop to the robot. The
communication of these commands takes place wirelessly using Wi-Fi or Bluetooth technology to
communicate with the ESP32 microcontroller.

f 23 | P a g
e
4. Robot Control and Motion Module

The robot control module receives commands from the ESP32 microcontroller. The microcontroller
sends a command to the motor driver based on the received command. The motor driver then controls
the BO motors to move the robot in a specific direction.

5. Safety Monitoring Module

This module monitors the orientation of the robot using an IMU sensor. The module monitors unsafe
conditions such as tilting, impacts, and falling. If an unsafe condition occurs, the module stops the
robot to avoid damage.

6. Mobile Application Monitoring Module

This module monitors the status of the robot using a mobile application. The module shows the status
of the robot’s movement, its tilting conditions, and its safety status. The module allows users to
monitor the status of the robot.

4.4 UML Diagrams

UML stands for Unified Modeling Language. UML is a standardized general-purpose modeling
language in the field of object-oriented software engineering. The standard is managed, and was
created by, the Object Management Group.
The goal is for UML to become a common language for creating models of object-oriented computer
software. In its current form UML is comprised of two major components: a Meta-model and a
notation. In the future, some form of method or process may also be added to; or associated with,
UML.
The Unified Modeling Language is a standard language for specifying, Visualization, Constructing
and documenting the artifacts of software system, as well as for business modeling and other non-
software systems.
The UML represents a collection of best engineering practices that have proven successful in the
modeling of large and complex systems. The UML is a very important part of developing objects-
oriented software and the software development process. The UML uses mostly graphical notations

f 24 | P a g
e
 to express the design of software projects.

GOALS:

The Primary goals in the design of the UML are as follows:

[Link] users a ready-to-use, expressive visual modeling Language so that they can develop and
exchange meaningful models.
2. Provide extendibility and specialization mechanisms to extend the core concepts.
3. Be independent of particular programming languages and development process.
4. Provide a formal basis for understanding the modeling language.
5. Encourage the growth of OO tools market.
6. Support higher level development concepts such as collaborations, frameworks, patterns and
components.
7. Integrate best practices.

Use Case Diagram

The Use Case Diagram represents the interaction between the user and the gesture-based robot control
system. In this system, the user performs hand gestures in front of a camera to control the robot. The
system recognizes the gestures using computer vision and a machine learning model. Based on the
detected gesture, a command is sent to the robot to perform the corresponding movement. The mobile
application allows the user to monitor the robot’s status and receive safety alerts. The diagram
highlights the major functionalities of the system and the relationship between the user and the
system.

f 25 | P a g
e
ESP32 - DevKitC

Fig 4.2 ESP32 - DevKitC

Understanding ESP32 Board

The ESP32 is shown in the above figure 4.2 design for low power IoT applications in mind. Its
integrated Wi-Fi and Bluetooth capabilities together with its Deep Sleep Operating mode establish
the device as suitable for most Portable IoT applications. The Arduino IDE has released ESP32 board
managers which enables programmers to develop the devices without difficulty.
Powering your ESP32
You can power your ESP32 board through three different methods which include:
Micro USB Jack: The board power requirement will be fulfilled when you connect the mini-USB jack
to a phone charger or computer through a cable.
5V Pin: The 5V pin requires Regulated 5V input which the on-board voltage regulator will convert to
3.3V output. Remember ESP32 operated with 3.3V only.
3.3V Pin: 3.3V pin of the ESP32 allows you to connect a regulated 3.3V supply directly to the system.

f 26 | P a g
e
Input/output
The ESP32 has 39 digital Pins which include 34 GPIO accessible pins and 5 input-only pins. The
device provides 18 channels which enable users to access 12-bit ADC and 2 channels which support
8-bit DAC. The system provides 16 PWM signal generation channels while 10 GPIO pins allow users
to implement capacitive touch functionalities. The ESP32 system enables users to select between
multiple input sources through its multiplexing functionality.
The ESP32 device provides three UART interfaces to enable TTL data transmission. The system needs
three Rx and Tx pin sets to function correctly. The system allows users to configure all six pins
through software which enables them to select any GPIO pin for UART functionality.

ESP32 allows users to program any GPIO pin to function as an external interrupt because the system
supports signal multiplexing.

These pins serve as the SPI communication interface for GPIO23 and GPIO19 and GPIO18 and
GPIO5. The ESP32 device supports dual SPI interfaces which serve as its initial SPI implementation.

These pins serve as the SPI communication interface for GPIO13 and GPIO12 and GPIO14 and
GPIO15. The ESP32 device supports dual SPI interfaces which serve as its second SPI
implementation.

The system establishes IIC communication through GPIO21 as SDA and GPIO22 as SCL using the
Wire library.

The Enable (EN) pin functions as the reset pin for the ESP32 system. The microcontroller resets when
the user sets this pin to a LOW state.

f 27 | P a g
e
ESP32-WROOm-32 PINOUT

Fig 4.3 ESP32-WROOm-32 PINOUT

How to use ESP32

The ESP-IDF shown in the above figure 4.3 serves as the primary programming environment for
ESP32 development, although beginners prefer Arduino IDE because of its user-friendly design. The
official Starter guide from Espressif provides complete information about using the native platform.

The first program upload process to ESP32. Your first program upload to ESP32 will take 5 to 10
minutes when you use the Arduino IDE. The Arduino IDE and an USB cable and ESP32 board itself
are all needed for your project. You can check this Arduino ESP32 guide to prepare your Arduino for
ESP32.

Your initial program upload process.

After users install Arduino IDE on their computers they should connect the board to their computer
through a USB cable. The user needs to open the Arduino IDE software and select
Tools>Boards>ESP32 Dev kit to choose the correct board while selecting Tools>Port to choose the
correct Port. The user must load example code through the menu path Files>Examples>Basics>Blink
f 28 | P a g
e
 to begin operating the ESP32 board which will subsequently blink its built-in LED. After you load
the example code into your IDE display the code by clicking the upload button located at the top
menu. The ESP32 device will start blinking its built-in LED after the upload process completes. The
following code example demonstrates how to create a blinking effect using an LED:

int LED_BUILTIN = 2;
void setup() {
pinMode (LED_BUILTIN, OUTPUT);
}
void loop() {
digitalWrite(LED_BUILTIN, HIGH);
delay(1000);
digitalWrite(LED_BUILTIN, LOW);
delay(1000);
}

ESP32 DevKitC Pinout Configuration

Pin Category Pin Name Details

Power Micro-USB, 3.3V, 5V, GND Micro-USB: ESP32 can be powered through
USB port
5V: Regulated 5V can be supplied to this pin
which is we be again regulated to 3.3V by on
board regulator, to power the board.
3.3V: Regulated 3.3V can be supplied to this
pin to power the board.
GND: Ground pins.

Enable En The pin and the button resets the

microcontroller.

Analog Pins ADC1_0 to ADC1_5 and ADC2_0 to Used to measure analog voltage in the range
ADC2_9 of 0-3.3V.
12-bit 18 Channel ADC

DAC pins DAC1 and DAC2 Used for Digital to analog Conversion

Input/Output GPIO0 to GPIO39 Totally 39 GPIO pins, can be used as input or

Pins output pins. 0V (low) and 3.3V (high). But
pins 34 to 39 can be used as input only

Capacitive T0 to T9 These 10 pins can be used as touch pins

Touch pins normally used for capacitive pads

f 29 | P a g
e
 RTC GPIO pins RTCIO0 to RTCIO17 These 18 GPIO pins can be used to wake up
the ESP32 from deep sleep mode.

Serial Rx, Tx Used to receive and transmit TTL serial data.

External All GPIO Any GPIO can be used to trigger an interrupt.

Interrupts

PWM All GPIO 16 independent channel is available for PWM

any GPIO can be made to work as PWM
through the software

VSPI GPIO23 (MOSI), GPIO19(MISO), Used for SPI-1 communication.

GPIO18(CLK) and GPIO5 (CS)

HSPI GPIO13 (MOSI), GPIO12(MISO), Used for SPI-2 communication.

GPIO14(CLK) and GPIO15 (CS)

IIC GPIO21(SDA), GPIO22(SCL) Used for I2C communication.

AREF AREF To provide a reference voltage for input

voltage.

Table 4.1 ESP32 DevKitC Pinout Configuration

Applications
• Prototyping of IoT devices

• Low power battery operated applications

• Network projects

• Easy to use for beginner level DIYers and makers.

• Projects requiring Multiple I/O interfaces with Wi-Fi and Bluetooth functionalities.
ESP32 DevKitC Pinout Configuration:

ESP32 Technical Specifications

Microprocessor Tensilica Xtensa LX6

Maximum Operating Frequency 240MHz

Operating Voltage 3.3V

Analog Input Pins 12-bit, 18 Channel

f 30 | P a g
e
 DAC Pins 8-bit, 2 Channel

Digital I/O Pins 39 (of which 34 is normal GPIO pin)

DC Current on I/O Pins 40 mA

DC Current on 3.3V Pin 50 mA

SRAM 520 KB

Communication SPI(4), I2C(2), I2S(2), CAN, UART(3)

Wi-Fi 802.11 b/g/n

Bluetooth V4.2 – Supports BLE and Classic Bluetooth

Table 4.2 ESP32 Technical Specifications

L298 Motor Driver:

f 31 | P a g
e
 Fig 4.4 L298 Motor Driver

The L298N Motor Driver shown in the above figure 4.4 Module enables high-power motor operation
which supports both DC and Stepper Motors. The module contains an L298 motor driver IC together
with a 78M05 5V voltage regulator. The L298N Module can operate up to four DC motors or two
DC motors which the user can control both direction and speed.

L298N Module Pinout Configuration

Pin Name Description

IN1 & IN2 Motor A input pins. Used to control the spinning direction of Motor A

IN3 & IN4 Motor B input pins. Used to control the spinning direction of Motor B

ENA Enables PWM signal for Motor A

ENB Enables PWM signal for Motor B

OUT1 & OUT2 Output pins of Motor A

OUT3 & OUT4 Output pins of Motor B

12V 12V input from DC power Source

5V Supplies power for the switching logic circuitry inside L298N IC

GND Ground pin

Table 4.3 L298N Module Pinout Configuration

f 32 | P a g
e
Features & Specifications:

• Driver Model: L298N 2A

• Driver Chip: Double H Bridge L298N

• Motor Supply Voltage (Maximum): 46V

• Motor Supply Current (Maximum): 2A

• Logic Voltage: 5V
• Driver Voltage: 5-35V
• Driver Current:2A

• Logical Current:0-36mA

• Maximum Power (W): 25W

• Current Sense for each motor

• Heatsink for better performance

Alternate Driver Modules: TMC2209, DRV8825, A4988, L9110S, DRV8711

Related Components: LM298 Motor Driver IC, 78M05 Voltage Regulator, Capacitors, Resistors, Heat
Sink
Note: Complete technical details can be found in the L298N Datasheet linked at the bottom of this
page.

Brief about L298N Module

The L298N Motor Driver module consists of an L298 Motor Driver IC, 78M05 Voltage Regulator,
resistors, capacitor, Power LED, 5V jumper in an integrated circuit.

f 33 | P a g
e
 Fig 4.5 L298N Module

The shown in the above figure 4.5 will describe 78M05 Voltage regulator which will become
operational when the jumper gets installed. The internal circuitry receives power through the voltage
regulator when the power supply voltage falls between 12V and lower. The 5V pin serves as an output
pin which powers the microcontroller during this voltage range.

The jumper needs to remain uninstalled when the power supply exceeds 12V and users must connect
external 5V power through the 5V terminal to operate the internal circuits.

The ENA and ENB pins function as speed control elements for Motor A and Motor B while the IN1
and IN2 as well as IN3 and IN4 pins serve as direction control elements for Motor A and Motor B.
The internal circuit diagram of L298N Motor Driver module appears below this statement.

f 34 | P a g
e
 Fig 4.6 motor driver

f 35 | P a g
e
DC Gearbox Motor - "TT Motor" - 200RPM - 3 to 6VDC

Fig 4.7 TT Motor

The product functions as a TT shown in the above figure 4.7 reducer motor which powers smart robot
vehicles. The device has two Axis which allow simultaneous installation of wheel Encoder and wheel
components. The motor serves as an affordable solution for your DIY car project because of its user-
friendly operation. The product has the following characteristics
· Strong magnetic with anti-interference system
· The system uses a double axis gear motor
· The system has a reduction ratio of 1 to 48
· The system operates on a voltage range between DC 3V and 6V.

DC MOTOR:

A DC motor is shown in the below figure 4.8 designed to run on DC electric power. The two pure DC
motor designs that exist today are Michael Faraday's homopolar motor which exists as an uncommon
example and the ball bearing motor which remains a new invention. The brushed and brushless types
of DC motors serve as the most common motor types because these systems use internal commutation
and external commutation to produce oscillating AC current from their DC power source which makes
f 36 | P a g
e
them operate as non-purely DC machines.

Fig 4.8 DC motor

Types of dc motors:
[Link] DC Motors
[Link] DC motors
[Link] DC motors
Brushed DC motors:

The classic commutator DC motor has several limitations because it requires brushes which must
contact the commutator. The system generates friction between the components. The brushes become
less able to sustain contact when the system operates at faster speeds. The brushes tend to bounce off
the roughness found on the commutator surface which results in spark production. The maximum
operational speed of the machine receives restriction. The motor output receives restriction through
the current density limits which apply to each brush's contact area. The system generates electrical
disturbances because of its inadequate electrical connections. The brushes will reach their end of
service life which necessitates their replacement while the commutator needs ongoing maintenance
because it will experience wear over time. The commutator assembly on a large machine is a costly
there are three types of dc motor 1. dc series motor 2. dc shunt motor 3. dc compound motor
these are also two type a. cummulative compound b. deffercial compounnd

f 37 | P a g
e
Brushless DC motors:

The brushless motor design removes some issues that exist in brushed DC motors. The motor uses an
external electronic device as its rotational switch instead of using the standard mechanical
commutator and brushgear system. Brushless motors achieve efficiency rates between 85 to 90
percent while DC motors with brushgear achieve efficiency rates between 75 to 80 percent. The
brushless DC motor operates between two extreme ends of other DC motor technologies because it
uses stepper motor control systems. These devices operate like stepper motors because they use
permanent magnets and an external rotor design with three driving coils and multiple Hall effect
sensors to detect rotor position and control their driving electronics. The drive electronics use Hall
effect sensor signals to activate the coils by running one phase at a time through the activation process.
The motors operate as three-phase synchronous machines which include built-in variable-frequency
drive electronics systems. A specific type of brushless DC motor controller uses main phase
connections to read EMF feedback as a method to track motor position and speed instead of using
Hall effect sensors. Electric radio-controlled vehicles use these motors as their primary propulsion
system. Modelers use the term outrunner motors to describe these motors when their magnets are
placed on the outer section of the motor.
Brushless DC motors find their primary application in situations that require exact speed control
which includes computer disk drives and video cassette recorders plus the spindles of CD and CD-
ROM drives and the internal components of office devices like fans and laser printers and
photocopiers. The motors provide multiple benefits when compared to traditional motor systems
which include the following:
• The efficiency of AC fans improves significantly because shaded-pole motors operate at lower
temperatures compared to standard AC motors. The cool operation of this system results in a
substantial.

Coreless DC motors:
The motor designs which were previously described operate without any need for their iron elements
to turn because their torque only affects the electromagnet windings. The coreless DC motor
represents a specialized version of both the brush and brushless DC motor systems. These motors
enable fast acceleration because their rotor design excludes all iron core components. The rotor can
take the form of a winding-filled cylinder inside the stator magnets, a basket surrounding the stator
f 38 | P a g
e
 magnets, or a flat pancake (possibly formed on a printed wiring board) running between upper and
lower stator magnets. Electrical epoxy potting systems provide the typical means to stabilize windings
through their impregnation process. Moderate viscosity filled epoxies exhibit extended gel time
during their combined state. These systems are highlighted by low shrinkage and low exotherm. The
potting compound UL 1446 which UL File No E 210549 designates as a potting material can operate
at temperatures up to 180C (Class H).
The rotor achieves rapid acceleration because its weight is less than the mass of a traditional rotor
which combines copper windings and steel laminations, which enables the rotor to reach a time
constant of under 1 millisecond. The windings perform better with aluminum instead of copper, which
increases the total weight of the system. Coreless motors require air cooling because their rotors lack
metal components which could serve as heat sinks.
These motors were commonly used to drive the capstan(s) of magnetic tape drives and are still widely
used in high-performance servo-controlled systems, like radio-controlled vehicles/aircraft, humanoid
robotic systems, industrial automation, medical devices, etc.

f 39 | P a g
e
 [Link]&HARDWARE INVOLVED

5.1 Software Requirements

1. MediaPipe

• Purpose: Real-time hand gesture recognition and human pose estimation.

• Functionality: Detects hand landmarks, extracts joint positions, and tracks gestures without requiring
wearable devices.

• Integration: Works with Python or C++ for processing camera input and sending features to the deep
learning model.

2. Deep Learning Frameworks

• Examples: TensorFlow, Keras, or PyTorch

• Purpose: Classify hand gestures based on extracted features from MediaPipe.

• Functionality: Trains the model to map gestures to robot commands (forward, backward, left, right,
stop).

3. Robot Control Software / Firmware

• Options:

o Arduino IDE for programming Arduino microcontrollers

o Raspberry Pi OS or Python scripts for SBC-based robots

• Purpose: Receives gesture commands and controls motor actions, IMU readings, and safety responses.

4. Mobile Application Development Tools

• Options: Flutter, Android Studio, or React Native

• Purpose: Develops a mobile app for:

o Displaying robot movement status

o Showing tilt and orientation from IMU

o Sending safety alerts to the user in real-time

5. Communication Protocols

• Examples: Wi-Fi (TCP/IP), Bluetooth (Serial Communication)

f 40 | P a g
e
 • Purpose: Transfers data between gesture recognition system, robot, and mobile application.

6. IDEs and Supporting Software

• Python IDEs: PyCharm, VS Code (for deep learning, MediaPipe integration)

• Arduino IDE: For programming microcontroller firmware

• Libraries:

o OpenCV (for additional video processing)

o NumPy, Pandas (for data handling and gesture preprocessing)

7. Optional Visualization / Monitoring Tools

• Matplotlib or Plotly: For real-time visualization of IMU data during testing

• ROS (Robot Operating System): For advanced robot control, sensor integration, and modular
development.

5.2 Hardware Requirements

1. Camera

• Used for capturing live video of the user’s hand gestures and body posture.

• Requirements:

o High frame rate (≥30 FPS) for smooth gesture detection

o Adequate resolution (≥720p) for accurate landmark extraction

2. Microcontroller / Single Board Computer

• Used for processing the received commands from the gesture recognition system and
controlling the robot’s motors.

• Options:

o Arduino (e.g., Arduino Uno, Mega)

o Raspberry Pi for real-time image processing and gesture detection

3. Mobile Robot Platform

• A wheeled robot platform capable of moving in various directions.

• Components:

f 41 | P a g
e
o Motors (DC or servo motors) for moving the robot in forward, backward, left, and right
directions
o Motor driver ICs (e.g., L298N, TB6612FNG)

o Wheels and platform for the desired environment

4. IMU (Inertial Measurement Unit) Sensor

• Used for monitoring the robot’s orientation, tilt, and acceleration, and detecting impacts and
falls.

• Examples: MPU6050, MPU9250, or Bosch BNO055

• Integrate with the microcontroller for real-time safety monitoring and feedback

5. Wireless Communication Module

• Used for transmitting the received command from the gesture recognition system to the robot
and the sensor values from the robot to the mobile application.

• Options:

o Wi-Fi Module (ESP32)

6. Power Supply

• Used for powering the robot and the required devices.

• Options:

. Rechargeable Li-ion/Li-po batteries

7. Mobile Device

• Runs the mobile application to show:

o Current status of robot movement

o Tilt and orientation alerts

o Safety alerts

f 42 | P a g
e
5.3 Technologies Used
5.3.1 Machine Learning

Machine Learning is a part of Artificial Intelligence (AI) that allows computers to learn from data
instead of being programmed with fixed instructions. In traditional programming, every step needs to
be clearly defined by the programmer. However, in machine learning, the system learns patterns from
data and uses those patterns to make decisions or predictions.

The main idea behind machine learning is learning from experience. When a machine learning model
is trained with a large amount of data, it studies the relationships between inputs and outputs. After
the training process, the system can apply what it has learned to new data and provide results with
good accuracy.

The process of machine learning usually begins with collecting data that is relevant to the problem.
Once the data is collected, it is cleaned and organized through a process called data preprocessing.
This step is important because the quality of the data directly affects the performance of the model.
After preprocessing, the data is used to train the machine learning model so that it can learn patterns
and relationships.

Once the training process is completed, the model is tested with new data to check how well it
performs. If the results are satisfactory, the model can then be used in real applications to make
predictions or recognize patterns.

Machine learning methods are generally divided into three main types.

The first type is supervised learning, where the model is trained using labeled data. This means that
each input already has a correct output. The model learns the relationship between the input and output
so that it can predict the result for new inputs.

The second type is unsupervised learning, where the data does not contain predefined labels. In this
case, the algorithm analyzes the data and tries to find patterns or similarities by itself.

The third type is reinforcement learning, where the system learns by interacting with its environment.
The system receives rewards for correct actions and penalties for incorrect actions, which helps it
gradually learn the best way to perform a task.

f 43 | P a g
e
Machine learning is widely used in many areas such as image recognition, speech recognition,
recommendation systems, medical diagnosis, and robotics. In this project, machine learning helps the
system understand gesture patterns and convert them into commands that can control the robot.

5.3.2 Python
Python is a high-level programming language that is widely used in many fields such as artificial
intelligence, machine learning, web development, and data analysis. One of the main reasons for its
popularity is its simple and easy-to-understand syntax, which allows programmers to write programs
quickly and efficiently.

Python is considered a beginner-friendly language because it is easy to read and learn. At the same
time, it is powerful enough to handle complex tasks such as data processing and machine learning.
Because of this flexibility, Python has become one of the most preferred programming languages for
developing intelligent systems.

Another important advantage of Python is the large number of libraries and frameworks available for
different purposes. Libraries such as NumPy, Pandas, TensorFlow, and OpenCV make it easier to
perform tasks related to data analysis, machine learning, and image processing.

Python also has a very large community of developers who continuously contribute new tools and
improvements. This strong community support makes it easier to find resources, tutorials, and
solutions when working on projects.

In this project, Python is used as the main programming language to implement gesture recognition
and machine learning functions. It helps in processing the input data, analyzing the gestures, and
sending commands to the robot through other technologies used in the system.

Features in Python:

There are many features in Python, some of which are discussed below as follows:

1. Free and Open-Source Python language is freely available at the official website and you can
download it from the given download link below click on the Download Python keyword. Download
Python Since it is open source, this means that source code is also available to the public. So you can
download it, use it as well as share it.

f 44 | P a g
e
2. Easy to code Python is a high-level programming language. Python is very easy to learn the
language as compared to other languages like C, C#, JavaScript, Java, etc. It is very easy to code in
the Python language and anybody can learn Python basics in a few hours or days. It is also a developer-
friendly language.

3. Easy to Read As you will see, learning Python is quite simple. As was already established, Python’s
syntax is really straightforward. The code block is defined by the indentations rather than by
semicolons or brackets.

4. Object-Oriented Language One of the key features of Python is Object-Oriented programming .

Python supports object-oriented language and concepts of classes, object encapsulation, etc.

5. GUI Programming Support Graphical User interfaces can be made using a module such as PyQt5 ,
PyQt4, wxPython, or Tk in python. PyQt5 is the most popular option for creating graphical apps with
Python.

6. High-Level Language Python is a high-level language. When we write programs in Python, we do

not need to remember the system architecture, nor do we need to manage the memory. Extensible
feature Python is an Extensible language. We can write some Python code into C or C++ language
and also we can compile that code in C/C++ language.

8. Easy to Debug Excellent information for mistake tracing. You will be able to quickly identify and
correct the majority of your program’s issues once you understand how to interpret Python’s error
traces. Simply by glancing at the code, you can determine what it is designed to perform.

9. Python is a Portable language Python language is also a portable language. For example, if we have
Python code for windows and if we want to run this code on other platforms such as Linux, Unix, and
Mac then we do not need to change it, we can run this code on any platform.

10. Python is an integrated language, etc. Python is also an integrated language because we can easily
integrate Python with other languages like C, C++.

11. Interpreted Language: Python is an Interpreted Language because Python code is executed line by
line at a time. like other languages C, C++, Java, etc. there is no need to compile Python code this
makes it easier to debug our code. The source code of Python is converted into an immediate form

f 45 | P a g
e
called byte code.

12. Large Standard Library Python has a large standard library that provides a rich set of modules and
functions so you do not have to write your own code for every single thing. There are many libraries
present in Python such as regular expression s, unit-testing, web browsers, etc.

13. Dynamically Typed Language Python is a dynamically-typed language. That means the type (for
example- int, double, long, etc.) for a variable is decided at run time not in advance because of this
feature we don’t need to specify the type of variable.

14. Frontend and backend development With a new project py script, you can run and write Python
codes in HTML with the help of some simple tags, etc. This will help you do frontend development
work in Python like JavaScript. Backend is the strong forte of Python it’s extensively used for this
work because of its frameworks like Django and Flask

15. Allocating Memory Dynamically In Python, the variable data type does not need to be specified.
The memory is automatically allocated to a variable at runtime when it is given a value. Developers
do not need to write int y = 18 if the integer value 15 is set to y. You may just type y=18.

5.3.3 OpenCV

OpenCV stands for Open-Source Computer Vision Library. It is a powerful library that is mainly used
for image processing and computer vision applications. OpenCV provides many built-in functions that
allow computers to capture, process, and analyze images and videos.

OpenCV was originally developed by Intel and later released as an open-source project. Because it is
open source, developers around the world can use it freely and contribute to its improvement. Today,
OpenCV is widely used in many applications such as robotics, security systems, facial recognition,
and gesture detection.

One of the most useful features of OpenCV is its ability to work with real-time images and videos. It
allows the system to capture frames from a camera and analyze them instantly. This makes it suitable
for applications where quick responses are required.

OpenCV also supports various image processing techniques such as object detection, motion tracking,
edge detection, and pattern recognition. When combined with machine learning algorithms, OpenCV

f 46 | P a g
e
can help systems identify objects and movements with better accuracy.

In this project, OpenCV is used to capture images of hand gestures through a camera. The system then
processes these images to identify the movement of the hand. These detected gestures are interpreted
as commands that control the robot’s movement.

5.3.4 Arduino

Arduino is an open-source electronics platform that is widely used for building interactive electronic
and robotics projects. It consists of a microcontroller board along with software that allows users to
program and control the hardware.

The Arduino board can receive input signals from sensors and send output signals to different devices
such as motors, lights, and displays. This ability allows Arduino to interact with the physical
environment.

The Arduino Integrated Development Environment (IDE) is used to write and upload programs to the
Arduino board. The programming language used in Arduino is simple and based on C and C++, which
makes it easy to learn for beginners.

One of the biggest advantages of Arduino is its flexibility. It supports a wide range of sensors, modules,
and electronic components that can be easily connected to the board. Because of this, Arduino is
widely used in robotics, automation systems, smart devices, and educational projects.

In this project, Arduino is used to control the robot. The gesture recognition system processes the hand
movements and converts them into commands. These commands are then sent to the Arduino board,
which controls the motors and moves the robot in the required direction.

5.3.5 Computer Vision

Computer Vision is a field of artificial intelligence that focuses on enabling computers to understand
and interpret visual information from images and videos. It allows machines to analyze visual data in
a way that is somewhat similar to how humans use their eyes to observe the world.

Computer vision systems work by capturing images using cameras and then processing those images
using algorithms. These algorithms help the system detect objects, recognize patterns, and understand
movements within the image.
f 47 | P a g
e
The process of computer vision usually involves several stages. First, the image is captured through a
camera. Then the image is processed to remove noise and improve its quality. After that, important
features are extracted from the image, such as edges, shapes, or movements. Finally, machine learning
techniques are used to analyze these features and identify patterns or objects.

Computer vision is used in many real-world applications such as facial recognition systems, medical
imaging, security surveillance, autonomous vehicles, and industrial inspection systems.

In this project, computer vision plays an important role in detecting and understanding hand gestures.
The camera captures the hand movements, and the system analyzes these movements to determine the
intended command. These commands are then used to control the robot’s actions.

By combining computer vision, machine learning, and robotics, it becomes possible to create systems
that can interact with humans in a more natural and intuitive way.

ntents: Presentation of findings using figures, tables, charts, and statistical data. Includes explanations
and interpretation of the results.

Description: Shows the outcome of the project, evaluates its performance, and provides insights
into whether the objectives were met.

5.4 CODING
import cv2
import numpy as np
import mediapipe as mp
from [Link] import load_model
import serial

gesture_map = {
0: 'S',
1: 'A',
2: 'F',
3: 'A',
4: 'B',
5: 'X',
6: 'L',
7: 'R'
f 48 | P a g
e
}
# -------------------------
# Bluetooth Serial Setup
# -------------------------
bluetooth = [Link]("COM5", 115200) # CHANGE COM PORT
# Example: "COM3", "COM6", etc.

# -------------------------
# Initialize MediaPipe
# -------------------------
mpHands = [Link]
hands = [Link](max_num_hands=1, min_detection_confidence=0.7)
mpDraw = [Link].drawing_utils

# -------------------------
# Load Model
# -------------------------
model = load_model('model.hdf5')

with open('gesture.names8', 'r') as f:

classNames = [Link]().split('\n')

print("Loaded classes:", classNames)

# -------------------------
# Webcam
# -------------------------
cap = [Link](0)

stable_count = 0
previous_prediction = None
last_sent_class = None
STABLE_FRAMES = 10

while True:
success, frame = [Link]()
if not success:
break
f 49 | P a g
e
 frame = [Link](frame, 1)
image_height, image_width, _ = [Link]

result = [Link]([Link](frame, cv2.COLOR_BGR2RGB))

classID = None
className = ""

if result.multi_hand_landmarks:
for handslms in result.multi_hand_landmarks:

landmarks = []
base_x, base_y = None, None

for i, lm in enumerate([Link]):
lmx = int(lm.x * image_width)
lmy = int(lm.y * image_height)

if i == 0:
base_x, base_y = lmx, lmy
lmx, lmy = 0, 0
else:
lmx -= base_x
lmy -= base_y

[Link]([lmx, lmy])

mpDraw.draw_landmarks(frame, handslms, mpHands.HAND_CONNECTIONS)

landmarks = [Link](landmarks, dtype=np.float32)

max_val = [Link]([Link](landmarks))
if max_val != 0:
landmarks = landmarks / max_val

landmarks = [Link](1, 42, 1)

prediction = [Link](landmarks, verbose=0)

f 50 | P a g
e
 classID = [Link](prediction)
className = classNames[classID]

# -------------------------
# Stability Check
# -------------------------
if classID is not None:
if classID == previous_prediction:
stable_count += 1
else:
stable_count = 0

previous_prediction = classID

# If stable for 10 frames

if stable_count >= STABLE_FRAMES:
if classID != last_sent_class:
print("Sending:", classID)
letter = gesture_map.get(classID)

if letter is not None:

[Link]([Link]())
print("Sent:", letter)
last_sent_class = classID
else:
stable_count = 0
previous_prediction = None

# Display gesture name

[Link](frame, className, (250, 200),
cv2.FONT_HERSHEY_SIMPLEX,
1, (0, 0, 255), 2, cv2.LINE_AA)

[Link]("Output", frame)

if [Link](1) & 0xFF == ord('q'):

break

f 51 | P a g
e
[Link]()
[Link]()
[Link]()

f 52 | P a g
e
 6. IMPLEMENTATION STEPS

ARDUINO IDE:

The below figure shown 6.1 Arduino Integrated Development Environment - or Arduino Software
(IDE) - contains a text editor for writing code, a message area, a text console, a toolbar with buttons
for common functions and a series of menus. It connects to the Arduino and Genuino hardware to
upload programs and communicate with them.
Writing Sketches Programs written using Arduino Software (IDE) are called sketches. The text editor
allows users to create sketches which they can save as files that use the .ino file extension. The editor
has features for cutting/pasting and for searching/replacing text. The message area gives feedback
while saving and exporting and also displays errors. The console shows all output from the Arduino
Software (IDE) which includes full error messages and additional details. The bottom righthand
corner of the window displays the configured board and serial port. The toolbar buttons allow you to
verify and upload programs, create, open, and save sketches, and open the serial monitor.
NB: Versions of the Arduino Software (IDE) prior to 1.0 saved sketches with the extension .pde. It is
possible to open these files with version 1.0, you will be prompted to save the sketch with the .ino
extension on save.
Verify The function checks your code for errors through the process of compiling your code.
Upload Your code gets compiled and transferred to the board which you have set up. See uploading
below for details.
Note: If you are using an external programmer with your board, you can hold down the "shift" key
on your computer when using this icon. The system will display "Upload using Programmer" as the
new text.
New The function generates a completely new sketch.
Open The function displays a menu which contains all the sketches that exist in your sketchbook. The
content of the current window will be replaced when you click on a sketch.
The system stores your sketch with the save function.

f 53 | P a g
e
Installing the ESP32 Board in Arduino IDE

Fig 6.1 Arduino IDE

There’s an add-on for the Arduino IDE that allows you to program the ESP32 using the Arduino IDE and its
programming language. In this tutorial we’ll show you how to install the ESP32 board in Arduino IDE whether
you’re using Windows, Mac OS X or Linux.
Step 1: Prerequisites: Arduino IDE Installed
Before starting this installation procedure, make sure you have the latest version of the Arduino IDE installed
in your computer. If you don’t, uninstall it and install it again. Otherwise, it may not work.
Having the latest Arduino IDE software installed from [Link]/en/Main/Software, continue with this tutorial.
Step 2: Installing ESP32 Add-on in Arduino IDE

f 54 | P a g
e
1. In your Arduino IDE, go to File> Preferences
Step 3: Enter Https://[Link]/dl/package_esp32_index.j... Into the “Additional Board Manager URLs”
Field As Shown in the Figure Below. Then, Click the “OK” Button:

f 55 | P a g
e
Note: if you already have the ESP8266 boards URL, you can separate the URLs with a comma as follows:
[Link]
m_index.json
Step 4: Open the Boards Manager. Go to Tools > Board > Boards Manager…

f 56 | P a g
e
Step 5: Search for ESP32 and Press Install Button for the “ESP32 by Espressif Systems “:

f 57 | P a g
e
Step 6: That’s It. It Should Be Installed After a Few Seconds.

Step 7: Testing the Installation

f 58 | P a g
e
Plug the ESP32 board to your computer. With your Arduino IDE open, follow these steps:
1. Select your Board in Tools > Board menu (in my case it’s the DOIT ESP32 DEVKIT V1)

2. Now go to Tools → Port.

3. Select the COM port where the ESP32 is connected (e.g., COM3, COM5).

4. Keep other settings at default (Upload Speed, Flash Frequency, etc.).

5. The ESP32 board is now ready for programming in Arduino IDE.

You can now write, compile, and upload your ESP32 code successfully.

f 59 | P a g
e
 [Link]

Installer
Step 1: Check Current Python Version
Open a terminal (Command Prompt on Windows, Terminal on Mac/Linux) and type:
python --version
or
python3 --version
This tells you if Python is already installed and its version.
Step 2: Download Python 3.8
[Link] to the official Python website: [Link]
[Link] the correct for your operating system:
o Windows: Windows installer (64-bit)
o MacOS: macOS 64-bit universal2 installer
o Linux: Use your package manager (see Step 5)
Step 3: Install Python 3.8 (Windows / Mac)
[Link] the downloaded installer.
[Link]: Check the box “Add Python 3.8 to PATH” at the beginning of the installation.
[Link] Install Now.
[Link] for the installation to complete.
After installation, verify by opening a terminal and typing:
python --version
It should show: Python 3.8.x
Step 4: Install pip (Python Package Manager)
Python 3.10 usually comes with pip pre-installed. Check by typing:
If pip is not installed, you can install it using:
python -m ensurepip --upgrade

f 60 | P a g
e
Step 5: Install Python 3.8 on Linux
For Ubuntu/Debian-based Linux:
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.8 python3.8-venv python3.8-dev python3-pip

Step 6: Set Up a Virtual Environment (Recommended for Data Projects)

[Link] a virtual environment:
[Link] the environment:
oWindows: myenv\Scripts\activate
oMac/Linux: source myenv/bin/activate
[Link] pip in the environment:
Step 7: Install Data Libraries
Once Python 3.8 is installed and your environment is activated, install commonly used data-related
packages:

Code:
import cv2
import numpy as np
import mediapipe as mp
from [Link] import load_model
import serial
gesture_map = {
0: 'S',
1: 'F',
2: 'B',

f 61 | P a g
e
 3: 'L',
4: 'R',
}
bluetooth = [Link]("COM5", 115200)
mpHands = [Link]
hands = [Link](max_num_hands=1, min_detection_confidence=0.7)
mpDraw = [Link].drawing_utils
model = load_model('model.hdf5')

with open('gesture.names8', 'r') as f:

classNames = [Link]().split('\n')
print("Loaded classes:", classNames)
cap = [Link](0)
stable_count = 0
previous_prediction = None
last_sent_class = None
STABLE_FRAMES = 10
while True:
success, frame = [Link]()
if not success:
break
frame = [Link](frame, 1)
image_height, image_width, _ = [Link]
result = [Link]([Link](frame, cv2.COLOR_BGR2RGB))
classID = None
className = ""
if result.multi_hand_landmarks:
for handslms in result.multi_hand_landmarks:
f 62 | P a g
e
 landmarks = []
base_x, base_y = None, None
for i, lm in enumerate([Link]):
lmx = int(lm.x * image_width)
lmy = int(lm.y * image_height)

if i == 0:
base_x, base_y = lmx, lmy
lmx, lmy = 0, 0
else:
lmx -= base_x
lmy -= base_y

[Link]([lmx, lmy])

mpDraw.draw_landmarks(frame, handslms, mpHands.HAND_CONNECTIONS)

landmarks = [Link](landmarks, dtype=np.float32)
max_val = [Link]([Link](landmarks))
if max_val != 0:
landmarks = landmarks / max_val
landmarks = [Link](1, 42, 1)
prediction = [Link](landmarks, verbose=0)
classID = [Link](prediction)
className = classNames[classID]
if classID is not None:
if classID == previous_prediction:
stable_count += 1
else:
f 63 | P a g
e
 stable_count = 0
previous_prediction = classID
if stable_count >= STABLE_FRAMES:
if classID != last_sent_class:
print("Sending:", classID)
letter = gesture_map.get(classID)
if letter is not None:
[Link]([Link]())
print("Sent:", letter)
last_sent_class = classID
else:
stable_count = 0
previous_prediction = None

[Link](frame, className, (250, 200),

cv2.FONT_HERSHEY_SIMPLEX,
1, (0, 0, 255), 2, cv2.LINE_AA)

[Link]("Output", frame)

if [Link](1) & 0xFF == ord('q'):

break

[Link]()
[Link]()
[Link]()

f 64 | P a g
e
 8. RESULTS & DISCUSSION

Researchers developed and evaluated their system which enables users to control devices through
hand gestures while monitoring safety conditions. The key results are as follows:

1. Gesture Recognition Accuracy: o The system achieved an accuracy rate above 95 percent for five
pre-defined gestures which included forward backward left right and stop gestures through its
implementation of MediaPipe and deep learning classification model. o The system achieved 50 to
70 millisecond recognition latency which enabled users to control the system almost instantly.

2. Robot Movement: o The robot executed precise movements for all classified gestures which
included forward backward left and right gestures. o The stop gesture successfully halted the robot
immediately which improved operational safety.

3. IMU-Based Safety Monitoring: o The IMU detected unsafe tilts which exceeded 20 degrees along
with sudden impact events and falling incidents. The system automatically stopped robot operations
after detection while sending alerts to the mobile application. The robot operated more reliably
because safety monitoring functions prevented potential damage to the equipment.

4. Mobile App Feedback: o The mobile application displayed the current robot status along with its
tilt angles and the system alerts. Users could track robot movements while handling safety alerts
through their mobile devices.

5. Overall System Performance: o The system combined gesture recognition with robot movement
and safety monitoring to create a control system which operates safely through intuitive controls.
System performance showed strong stability in controlled environments but experienced slight
limitations during extreme lighting conditions and occluded camera perspectives.

f 65 | P a g
e
 9. ADVANTAGES & APPLICATIONS
ADVANTAGES:
1. Users can control robots through hand gestures and body movements because the system
provides intuitive controls which require no special training.

2. Users can operate the system through gesture recognition which uses camera and MediaPipe

technology to eliminate the need for gloves and sensors and other wearable equipment.

3. Deep learning-based classification enables immediate robot movement because it creates

responsive control systems that operate with very shot time delays.

4. The system uses IMU sensors to monitor unsafe conditions which include dangerous tilts and

impacts and falls to protect both personnel and equipment.

5. The mobile application delivers real-time updates about robot status and its movement and safety

6. The system allows users to operate different mobile robots in various environments through its

ability to work with multiple platforms and requires only minor hardware modifications.

Applications:
1. Industrial Automation – This system enables operators to control warehouse robots and
assembly line vehicles and forklifts through gesture recognition without using any physical
control devices.

2. Healthcare and Assistance – The system enables users to control robots through gestures to
help with elderly care and rehabilitation and mobility assistance.

3. Search and Rescue – The system enables operators to control robots through remote access in
dangerous environments where traditional control methods are impractical.

4. Domestic Robotics – Users can control home service robots through gesture-based systems to
perform cleaning and delivery and security tasks.

5. Education and Research – The system uses interactive control systems to teach students about
robotic systems and human–robot interaction.

6. Entertainment – The system enables users to control robotic toys and exhibition robots

f 66 | P a g
e
 [Link]

The gesture-based mobile robot control system developed by researchers achieves its three objectives
through its natural human-robot interaction and its ability to monitor safety in real time.

• The system enables users to control robot movement through simple gestures which do not require
any wearable devices.

• The IMU sensor system maintains continuous robot orientation tracking which stops the robot from
operating in dangerous situations.

• The mobile application system delivers instant safety alerts together with immediate status updates
to users in direct fashion.

The system can be used with various mobile robots in different operational environments.

The system needs proper lighting conditions together with unobstructed camera view to function at its
best.

The system needs the deep learning model to undergo complete training again because it requires new
gesture patterns to be added.

The system improves operational efficiency and safety while enhancing user experience which makes
it appropriate for industrial automation and healthcare and domestic robotics and research
applications.

f 67 | P a g
e
 11. FUTURE SCOPE

The upcoming development of gesture-controlled robots which use machine learning will
revolutionize human-robot contact through its requirement of non-physical interface. Robots can
comprehend human movements through hand and body gestures because they use cameras and
sensors together with machine learning systems instead of traditional input devices such as keyboards
and remotes and joysticks.

The following list presents the most important upcoming prospects:

1. Healthcare and Medical Assistance

Doctors and surgeons can use gesture-controlled robots to assist their work.

Surgeons can operate medical robots and medical images through touchless control which protects
them from potential infection hazards.

Robots can help patients who have lost their ability to move by using gesture control systems.

The following professional domains are applicable:

Robotics

Machine Learning

Computer Vision

2. Smart Homes and Personal Robots

Home automation will benefit from the implementation of gesture robots.

The following list shows various examples:

People can use gestures to operate lights and television and air conditioning systems.

Elderly people can command home robots using simple hand movements.

Domestic robots which execute their daily responsibilities.

This connection links with:

Internet of Things

3. Industrial Automation

f 68 | P a g
e
Factories can boost their efficiency through the implementation of gesture-based robots.

Workers can operate equipment from remote locations.

People can work safer in dangerous industrial areas which include chemical plants and mining sites.

Human-robot collaboration reaches its peak efficiency through this technology.

Used in:

Industry 4.0

4. Military and Defense

Gesture robots can assist soldiers during dangerous missions.

Bomb disposal robots function through gesture-based controls.

Robots used for surveillance tasks.

Robots conduct military operations from distant locations.

5. Virtual Reality (VR) and Augmented Reality (AR)

Gesture robots function as interactive controllers for virtual reality systems.

VR games use natural control mechanisms.

The systems train pilots and surgeons and engineers through simulation exercises.

The following technologies are connected to each other:

Virtual Reality

Augmented Reality

6. Education and Research

Gesture robots will be widely used in:

Robotics laboratories

Artificial intelligence research

Student projects

Interactive learning systems

f 69 | P a g
e
 [Link] / Bibliography

1. Zafar, M. H., Langås, E. F., & Sanfilippo, F. (2023). Empowering human-robot interaction
using sEMG sensor: Hybrid deep learning model for accurate hand gesture recognition.
Results in Engineering, 20, 101639. [Link]

2. Zaman, M. Q., & Wu, H.-M. (2023). Hand Gesture-based Teleoperation Control of a
Mecanum-wheeled Mobile Robot. IFAC-PapersOnLine, 56(2), 1484–1489.
[Link]

3. Jalayer, M., Jalayer, R., Orsenigo, C., & Tomizuka, M. (2026). A review on deep learning
for vision-based hand detection, hand segmentation and hand gesture recognition in
human–robot interaction. Robotics and Computer-Integrated Manufacturing, 97,
103110. [Link]

4. Xi, J., Zhang, W., Xu, Z., Zhu, S., Tang, L., & Zhao, L. (2025). Three-dimensional
dynamic gesture recognition method based on convolutional neural network.
HighConfidence Computing, 5(1), 100280. [Link]

5. Raihan, M. R., Hasan, R., Arifin, F., Nashif, S., & Haider, M. R. (2024). Single and Multi-
Hand Gesture based Soft Material Robotic Car Control. Procedia Computer Science,
235, 3055–3064. [Link]

6. Shailaja Kadam, V. Jadhav, R. Takur, S. Nakate & D. Chavan (2024). Gestures Controlled
Robot. IJRASET. [Link]

7. Chaudhari, N., Sonawane, K., & Boyanapalli, A. (2023). Hand Gesture Control System
using Open CV. International Journal of Innovative Science and Research Technology,
8(11). [Link]

8. Qi, J., Ma, L., Cui, Z., & Yu, Y. (2023). Computer vision-based hand gesture recognition
for human-robot interaction: a review. Complex & Intelligent Systems, 10, 1581–1606.
[Link]

9. Computer Vision. International Journal of Intelligent Systems and Applications in

f 70 | P a g
e
 Engineering, 11(2), 1013–. Retrieved from
[Link]

10. G. Zhang, J. Su, S. Zhang, J. Qi, Z. Hou, and Q. Lin, “Research on Deep LearningBased
Human–Robot Static/Dynamic Gesture-Driven Control Framework,” Sensors, vol. 25, no.
23, Art. no. 7203, 2025, doi:10.3390/s25237203.

11. Wameed, M., & Alkamachi, A. M. (2023). Hand Gestures Robotic Control Based on
Computer Vision. International Journal of Intelligent Systems and Applications in
Engineering, 11(2), 1013–. Retrieved from
[Link]

12. Molchanov, P., Gupta, S., Kim, K., & Kautz, J. (2015). Hand gesture recognition with 3D
convolutional neural networks. *Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition Workshops*, 1–7.

13. Ohn-Bar, E., & Trivedi, M. M. (2014). Hand gesture recognition in real time for
automotive interfaces: A multimodal vision-based approach and evaluations. *IEEE
Transactions on Intelligent Transportation Systems*, 15(6), 2368–2377.

14. Rautaray, S. S., & Agrawal, A. (2015). Vision based hand gesture recognition for human
computer interaction: A survey. *Artificial Intelligence Review*, 43(1), 1

15. Kim, J., Mastnik, S., & André, E. (2008). EMG-based hand gesture recognition for real-
time biosignal interfacing. *Proceedings of the 13th International Conference on
Intelligent User Interfaces*, 30-39.

16. Liu, Z., Choi, J., & Wang, Y. (2019). Real-time hand gesture recognition using deep
convolutional neural networks for robot control. *IEEE Access*, 7, 154123–154134.

17. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale
image recognition. *arXiv preprint

18. Du, Y., Fu, Y., & Wang, L. (2015). Skeleton based action recognition with convolutional
neural networks. *Proceedings of the IEEE Conference on Computer Vision pattern

19. Wameed, M., & Alkamachi, A. M. (2023).Hand gestures robotic control based on
f 71 | P a g
e
 computer vision. International Journal of Intelligent Systems and Applications in
Engineering, 11(2).

20. Uses computer vision and machine learning to detect hand gestures and control a robot.
[Link], S. L., & Huang, L. W. (2021).

21. Using deep learning technology to realize the automatic control program of robot arm
based on hand gesture recognition.

22. Uses YOLO-based deep learning model for gesture recognition and robot arm
[Link], G., Su, J., Zhang, S., Qi, J., Hou, Z., & Lin, Q. (2025).

23. Framework using deep learning to control robots with static and dynamic
[Link], K., Makwana, R., Bhavsar, D., Chauhan, S., & Gandhi, S. (2023).

24. Demonstrates gesture-based control of a mobile robot using wireless

[Link], J. Y., Tejero-de-Pablos, A., & Harada, T. (2019).

25. Uses deep learning optical-flow methods for gesture recognition in robots. arXivPascher,
M., Saad, A., Liebers, J., Heger, R., Gerken, J., Schneegass, S., & Gruene, U. (2024).

f 72 | P a g
e