“Real-time Hand Pose Estimation and Simulation for
Immersive AR/VR Interraction”
Bachelor of Technology
in
Electrical and Electronics Engineering
UE21CS421AA1 – Research Writing and Ethics
Submitted by:
RAJ SHEKHAR SINGH PES1UG21EE070
June - Nov 2024
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
FACULTY OF ENGINEERING
PES UNIVERSITY
(Established under Karnataka Act No. 16 of 2013)
Electronic City, Hosur Road, Bengaluru – 560 100, Karnataka, India
1
ABSTRACT
In traditional VR games, players often rely on clunky button and joystick controls,
which can break immersion. To solve this, we're developing an AR-VR glove that
uses AI and machine learning. This glove recognises gestures and adapts to player
behaviour in real time, making gaming more natural and engaging.
Our project builds on recent advancements in VR technology, aiming to enhance
player experiences by introducing a more intuitive control system. By integrating AI
and machine learning into an AR-VR glove interface, we're creating a seamless way
for players to interact with virtual environments.
Existing VR control methods can feel disconnected and limit immersion. With our
AR-VR glove, players can enjoy a more immersive gaming experience that fosters
deeper emotional connections with the virtual world. By recognising gestures and
adapting to player behaviour, our glove offers a natural and intuitive way to navigate
VR environments.
The gaming industry is seeing a surge in demand for immersive experiences, and
traditional controls need to catch up. By leveraging AI and machine learning in our
AR-VR glove, we're addressing this problem head-on, providing players with a more
intuitive and engaging way to interact with virtual worlds.
We're crafting a shooting range game for gamers and military training, enhancing
eye-hand coordination. With immersive gameplay and eSports potential, our game
offers skill development and competitive gaming in an accessible format.
2
CONTENTS
ABSTRACT 5
1 INTRODUCTION 7
1.1 BACKGROUND 7
1.2 PROBLEM STATEMENT 7
1.3 OBJECTIVES/DELIVERABLES
1.4 CHALLENGES 9
2 PROPOSED PROJECT 10
2.1 FUNCTIONALITY OF THE PROPOSED PROJECT 10
2.2 WORKING OF THE PROJECT 11
2.3 CONSTRAINTS OF WORKING ON PROJECT 12
3 LITERATURE SURVEY & EVALUATION OF VARIOUS METHODS
3.1 LITERATURE REVIEW 13
4 HARDWARE & SOFTWARE ARCHITECTURE
4.1 BLOCK DIAGRAM 22
4.2 PICTURES 23
4.3 COMPARISON AND ESTIMATION OF OVERALL EXPENSES OF THE
PROJECT 24
4.4 SOFTWARE ARCHITECTURE 33
4.4.1 SOFTWARE TOOLS USED 35
5 CONCLUSION 37
6 FUTURE SCOPE 38
7 REFERENCES 39
3
TABLE OF FIGURES
Figure 1 (a) under-side of the glove with 6 vibrothermal units
(b) top-down view of the glove showing component layout
(c) side view of glove illustrating haptic force delivery system 14
Figure 2 Framework with haptic sensation 15
Figure 3 Schematics of multi-functional sensing glove with triboelectric finger
and palm sensors, and piezoelectric mechanical stimulator. 17
Figure 4 Prototype of H glove 19
Figure 5 Hardware block diagram 22
Figure 6 Glove hardware model 22
Figure 7 Circuit Diagram 23
Table 1 Cost comparison of 24
Figure 8 Arduino uno 26
Figure 9 Arduino Nano 26
Figure 10 MPU 6050 27
Figure 11 NRF 24L01 27
Figure 12 flex sensor 28
Figure 13 ERM Coin vibration motor 28
Figure 14 Rechargeable battery 29
Figure 15 10 microfarad capacitor 29
Table 2 Timeline of each phase of the project 36
4
1. INTRODUCTION
1.1 Background
Our project aims to advance the capabilities of human-computer interaction through
the development of a two-hand AR/VR haptic glove system. This immersive
technology enhances the user's ability to experience realistic touch sensations, such as
force, texture, and spatial awareness, within a virtual environment. Earlier haptic
systems mainly offered basic vibrotactile feedback, like that found in mobile devices.
However, recent innovations in sensor technology and hardware integration have
made it possible to deliver more sophisticated, lifelike interactions. By designing this
haptic glove system specifically for an interactive VR shooting range game, we aim to
create a compelling and accessible VR experience that provides both high engagement
and intuitive control.
This system uniquely integrates advanced sensors, real-time data processing, and
interactive algorithms to create an unparalleled VR experience. The dual-hand
configuration ensures users can navigate, interact, and control their virtual
surroundings naturally and seamlessly. The AR/VR glove's versatility extends its
utility beyond gaming, offering applications in training simulations, medical
procedures, and virtual prototyping.
1.2 Problem Statement
Traditional VR control methods, which primarily use buttons and joystick controllers,
can limit immersion and feel unnatural. These tools often break the continuity of the
virtual experience, making interactions feel mechanical rather than fluid. Our
approach addresses these limitations by developing a haptic glove system that
provides a natural and intuitive control mechanism.
By combining AI, machine learning, and advanced sensor integration, our project
enables real-time gesture recognition and haptic feedback, allowing users to interact
more realistically with virtual environments. This system focuses on achieving
5
seamless synchronization between physical and virtual interactions, leveraging
dual-hand functionality where each glove is assigned specific tasks to maximize
immersion and usability. Moreover, the potential to integrate this technology with
emerging AR applications broadens the scope of its impact.
1.3 Objectives/Deliverables
● Orientation Accuracy: Achieve <5% orientation tracking error through sensor
fusion and calibration algorithms.
● Latency Optimization: Implement real-time multiplayer networking with
<100ms latency to ensure fluid interactions.
● Portability: Design hardware components weighing <2kg for enhanced user
comfort and mobility.
● Game Development: Develop a VR shooting range game with <500MB
installation size to ensure widespread accessibility.
● System Stability: Ensure software stability with <1% crash rate during
rigorous testing scenarios.
● Comprehensive Testing: Conduct thorough testing covering >95% of potential
scenarios to ensure reliability.
● User-Centric Design: Incorporate user feedback to refine ergonomics and
functionality, emphasizing intuitive and immersive interactions.
1.4 Challenges
General Challenges
● Portability: Reducing the size and weight of the device to enhance user comfort
and usability.
● Scalability: Designing the system to accommodate future upgrades and
additional features without requiring significant rework.
● Circuit Design: Crafting an efficient and durable circuit layout that minimizes
power consumption and heat generation.
Tracking Hand Movement and Flexion
6
● Accuracy: Developing algorithms capable of capturing precise hand and finger
movements in real-time.
● Sensor Drift: Implementing techniques to mitigate drift in IMU sensors for
sustained accuracy.
● Flex Sensor Limitations: Addressing inconsistencies in flex sensor readings to
ensure reliable gesture recognition.
Additional Sensor Integration Challenges
● Compatibility: Ensuring seamless integration of multiple sensors without signal
interference.
● Latency Issues: Reducing latency in data transmission for real-time
responsiveness.
● Structural Integrity: Designing a robust glove structure that withstands
prolonged usage while maintaining flexibility.
● Circuit Complexity: Simplifying the design to enhance repairability and
reduce manufacturing costs.
Power Management
● Efficiency: Ensuring prolonged battery life through optimized power usage
without compromising performance.
● Stability: Incorporating safeguards to handle voltage fluctuations and prevent
disruptions during intensive operations.
Data Processing and Algorithm Optimization
● Real-Time Algorithms: Developing efficient algorithms for gesture recognition
and haptic feedback to ensure smooth and natural interactions.
● Resource Optimization: Minimizing computational overhead to ensure
consistent performance even on low-power hardware.
User Interface Design
● Intuitiveness: Creating an interface that is easy to learn and use while offering
advanced functionalities for experienced users.
7
● Adaptability: Designing the interface to accommodate different hand sizes and
usage scenarios for a personalized experience.
Environmental Adaptability
● Versatility: Ensuring reliable operation across various environmental conditions,
including extreme temperatures and humidity.
● Durability: Protecting hardware components from physical damage and wear
over extended periods.
●
Ergonomics and Comfort
● Designing for user comfort during prolonged use is essential for
satisfaction.
PROPOSED PROJECT
2.1. FUNCTIONALITY OF THE PROPOSED PROJECT
Our proposed two-hand AR/VR haptic glove system is a low-cost, scalable solution
designed to enhance user interaction within virtual environments. The gloves integrate
a variety of sensors and actuators, including flex sensors for finger tracking,
MPU6050 IMUs for spatial orientation, and vibration actuators for realistic touch
feedback. By dividing functionalities between the left and right gloves, we achieve a
more intuitive control experience.
Components:
● Microcontrollers: Arduino Uno for the left hand and Arduino Nano for the
right hand.
● Sensors: MPU6050 IMUs for spatial tracking and orientation, and flex sensors
for finger movement detection.
8
● Wireless Communication: NRF24L01 modules are used to transmit data
between gloves and the VR system.
● Power Supply: AAA battery setup with a 10-microfarad capacitor for stable
power delivery.
● Actuators: Vibrothermal feedback for enhancing touch sensations within the
VR environment.
DUAL-HAND FUNCTIONALITY
To maximize the immersive experience, we utilize both the left and right hands, each
assigned specific functionalities:
● Left Hand Functionality: The left glove provides real-time haptic feedback
and controls in-game actions, such as aiming, shooting, and reloading. This
glove allows users to perform interactive tasks with realistic touch feedback,
utilizing GY-521 and flex sensors connected to an Arduino Uno.
● Right Hand Functionality: The right glove enables look-around functionality
and controls spatial navigation. This glove uses GY-521 connected to an
Arduino Nano, allowing users to naturally explore the virtual environment by
orienting the camera and controlling character movement through gestures.
Need for Dual-Hand Functionality: Using both hands allows for a wider range of
control and interaction options within the VR environment. Assigning different
functions to each hand enables a more natural and immersive experience, as users can
perform simultaneous actions—such as aiming with the left hand while navigating
with the right. This setup mimics real-life interactions, making the VR experience
more intuitive and engaging.
9
2.2. WORKING OF THE PROJECT
The project uses a sensor-equipped glove system to capture real-time hand and finger
movements, allowing users to interact naturally in a virtual environment. Key stages
in the process are as follows:
● Stage 1: Hand Orientation and Position Capture
○ The MPU6050 sensor in each glove captures hand orientation and
spatial movement, providing data on rotation (pitch, roll, yaw). This data
is converted into quaternion representations for accurate and smooth
hand tracking.
● Stage 2: Finger Flexion Detection
○ Flex sensors detect the degree of finger bending, producing analog
signals corresponding to each joint’s movement. These signals are
processed into digital values, enabling accurate hand gestures like
grasping and pointing.
● Stage 3: Data Processing and Transmission
○ Data from the sensors is processed by Arduino microcontrollers and
combined using a multiplexer. The processed data is transmitted
wirelessly through NRF24L01 modules to the main VR system.
● Stage 4: Data Interpretation in Unreal Engine
○ The VR system receives and interprets the data using Unreal Engine,
translating hand movements and gestures into virtual actions.
Quaternion data from the MPU6050 sensor is used to display hand
orientation, while flex sensor data animates finger movements.
● Stage 5: Real-Time Interaction and Feedback
○ The left glove, responsible for actions like shooting and reloading,
provides haptic feedback through vibration actuators. The right glove
enables look-around functionality, allowing users to control their
perspective and navigate the virtual environment intuitively.
By combining real-time movement capture with VR integration in Unreal Engine, the
glove system offers an immersive, natural interaction experience within virtual
environments.
10
2.3. CONSTRAINTS OF WORKING ON PROJECT
Sensor Integration for Precise Orientation Tracking
The integration of sensors, such as IMUs and flex sensors, is critical for accurately tracking hand
orientation and movements. The challenge lies in ensuring seamless communication between these
components to minimize errors and latency, which are crucial for a smooth and immersive VR
experience.
Calibration Algorithms to Minimize Tilt and Yaw Drift Errors
Drift errors in tilt and yaw measurements are common challenges with IMUs. Developing robust
calibration algorithms is essential to correct these errors over time and maintain precise tracking. This
involves periodic recalibration and leveraging advanced software techniques, such as sensor fusion, to
combine data from multiple sensors.
Sensor Fusion Techniques for Accurate Data Amalgamation
To provide a cohesive representation of hand movements, data from different sensors must be
combined effectively. Sensor fusion techniques, such as Kalman filtering or complementary filtering,
are necessary to ensure accurate and reliable data integration. These techniques require computational
efficiency to process data in real time.
Multiplayer Networking for Collaborative VR Gameplay
Implementing a multiplayer VR experience introduces networking challenges, such as synchronizing
hand movements and interactions across multiple users. Ensuring low-latency data transmission and
avoiding packet loss are crucial for a seamless and collaborative experience. Additionally, robust
server-client architecture must handle simultaneous data streams without degrading performance.
Compact and Lightweight Hardware Design
The hardware components of the system, including microcontrollers, sensors, and actuators, must be
compact and lightweight to ensure user comfort. Achieving this without compromising functionality
requires careful component selection and optimized circuit design. Heat dissipation and power
consumption are additional factors to consider.
Integration of Wireless Connectivity for Mobility
Wireless connectivity, such as NRF24L01 or Bluetooth modules, is essential for enabling free
movement within the VR environment. Challenges include maintaining a stable connection,
minimizing power usage, and ensuring low latency. Future advancements may explore more advanced
wireless technologies for enhanced range and reliability.
11
3. LITERATURE SURVEY & EVALUATION OF VARIOUS
WAYS
3.1 LITERATURE REVIEW
Paper [1]: “Low Cost and Scalable Haptic VR Glove”Lara Alotaibi, Maria
Alabdulrahman, Ahmed Abul Hasanaath, Salahudean B. Tohmeh, Nazeeruddin
Mohammad Department of Computer Engineering College of Computer Engineering
and Science, Prince Mohammad Bin Fahd University, Khobar, KSA
14th IEEE International Conference on Computational Intelligence and
Communication Networks
DOI: 10.1109/cicn.2022.59 2022 14th International Conference on Computational
Intelligence and Communication Networks (CICN) | 978-1-6654-8771-9/22/$31.00
©2022 IEEE | DOI: 10.1109/CICN56167.2022.10008349
Virtual Reality (VR) is a subset of computer graphics where computers generate
simulated environments. VR allows users to experience a virtual three-dimensional
world where scenes and objects seem real. Wearable human-machine interfaces (HMI)
like helmets, gloves, etc are essential in making the experience feel realistic
and interactive. Gloves in particular stimulate the sense of touch, which in some ways,
is equally as important as visual stimulation. Developing a glove is a multi-faceted
problem. Accurately tracking hand movement and delivering contextually appropriate
feedback to the user in response to the events in the VR environment are the
two main challenges. This paper proposes a scalable and low-cost haptic VR glove
design that can accurately track and convey hand movement and finger flexion data.
The design also incorporates haptic primary colours (HPC), i.e. The design can
stimulate the full gamut of touch senses, namely force, vibration, and temperature.
The design makes use of flex sensors to track finger flexion. Inertial measurement
units (IMUs) were used to track hand movement and rotation. A Bluetooth module
communicates with a microcontroller connected to the VR environment. A
vibro-thermal unit delivers a sense of vibration as well as temperature change. Servo
motors were used to restrict finger movement in order to simulate force feedback. A
prototype was built which was based on the proposed design. The prototype was
12
tested in an integrated game development environment. It accurately emulated hand
movement and reliably delivered a sense of touch by means of vibration and force
feedback from the dedicated motors.
Figure 1 (a) under-side of the glove with 6 vibration units (b) top-down view of the glove
showing component layout (c) side view of glove illustrating haptic force delivery system
13
Paper [2]: “A Comprehensive Review of Haptic Gloves: Advances, Challenges,
and Future Directions” Published in: 2023 Second International Conference on
Electronics and Renewable Systems (ICEARS)
Date of Conference: 02-04 March 2023
Date Added to IEEE Xplore: 05 April 2023
ISBN Information:DOI: 10.1109/ICEARS56392. 2023.10085607
Publisher: IEEE Conference Location: Tuticorin, In
This work reviews haptic gloves, a technology revolutionizing robotic teleoperation
by enabling realistic touch feedback. We explore the evolution of haptics from early
vibrotactile responses to current capabilities of mimicking shape, texture, and force.
By incorporating haptic gloves, robotic tasks in fields like surgery, space exploration,
and manufacturing benefit from improved precision, accuracy, and reduced operator
cognitive load. Future advancements in haptics promise even more realistic feedback,
further solidifying haptic gloves as a cornerstone of dexterous robotic control.
This version incorporates the following improvements:
Figure 2 Data Flow
Figure 2 Framework with haptic sensation
14
Key points/Takeaways:
Stronger opening: Emphasises the transformative nature of haptic gloves.
Conciseness: Combines sentences while maintaining clarity.
Highlights: Emphasises key benefits (precision, accuracy, reduced workload) for
different fields.
Conclusion: Positions haptic gloves as a foundation for future advancements
Paper[3]: “Sensory-Glove-Based Human Machine Interface for Augmented
Reality (AR) Applications” Published in: 2020 IEEE 33rdInternational Conference
on Micro Electro Mechanical Systems (MEMS)
Date of Conference: 18-22 January 2020
Date Added to IEEE Xplore: 06 April 2020
ISBN Information: ISSN Information:
DOI: 10.1109/MEMS46641 .2020.9056153
Publisher: IEEE Conference Location: Vancouver, BC, Canada
This paper presents a novel glove-based Human-Machine Interface (HMI) for virtual
reality (VR) and augmented reality (AR) applications. The key innovation lies in
using easy-to-implement triboelectric sensors for multi-dimensional gesture
recognition and piezoelectric stimulators to deliver haptic feedback for user
interaction in the virtual world. Additionally, the HMI incorporates machine learning
for real-time object recognition, allowing users to manipulate virtual objects in
VR/AR space through sensory information captured by the glove. This integration of
gesture recognition, haptic feedback, and object recognition paves the way for a more
immersive and interactive VR/AR experience.
Improvements made:
Stronger opening: Clearly states the purpose and innovation of the proposed HMI.
Action verbs: Emphasise functionalities like recognition, manipulation, and feedback.
Focus on benefits: Highlights the contribution to a more immersive VR/AR
experience.
Conciseness: Streamlined the information while retaining key points.
15
Figure 3 Schematics of multi-functional sensing glove with triboelectric finger and palm sensors,
and piezoelectric mechanical stimulator.
Paper[4]: “Virtual reality simulation modelling for a haptic glove”
Published in: Proceedings Computer Animation 1999
Date ofConference:26-29 May 1999
Date Added to IEEE Xplore: 06 August 2002
Print ISBN:0-7695-016 7-2 Print ISSN: 1087-4844
DOI:
10.1109/CA.1999.
781212
Publisher: IEEE
Building on the recent advancements in haptic feedback for VR, this paper explores
the importance of physical modelling for realistic manipulation of virtual objects. We
present a VR system utilising a haptic glove (Rutgers Master II) and a novel method
for modelling virtual hand interactions, including contact surfaces, object hardness,
and surface deformation. This enhanced fidelity is particularly crucial for dexterous
tasks. As an example, we showcase an orthopaedic rehabilitation library where users
interact with dynamic objects with simulated plasticity. This work paves the way for
more immersive and interactive VR applications, particularly those requiring precise
manipulation. This improved abstract incorporates the following enhancements:
Stronger opening: Emphasises the focus on physical modelling for realistic
manipulation.
Clarity: Clearly states the VR system components (haptic glove, modelling method).
Application: Highlights the practical use case in an orthopaedic rehabilitation library.
Benefits: Emphasises the contribution to more immersive and interactive VR,
specifically
for dexterous tasks.
Conciseness: Streamlined the information while retaining key points.
16
The development of virtual reality (VR) simulations for rehabilitation exercises,
focusing on interactions with virtual objects and haptic feedback. These simulations
involve tasks like squeezing a rubber ball, playing a ball game, and moulding virtual
"power putty." The system allows users to select parameters such as exercise difficulty
and time limits. The simulations feature high-resolution virtual hands and deformable
objects, with hand gestures triggering interactions. A novel haptic interaction model
using mesh-based calculations is proposed, facilitating realistic object deformations
and haptic feedback. Future research directions include refining mesh parameters and
applying the method to new haptic interfaces like elbow and knee devices. The goal is
to enhance rehabilitation simulations with more realistic haptic interactions and
physical modelling.
Paper[5]: “Touching Virtual Reality: A Review of Haptic Gloves”
Published in:ACTUATOR 2018; 16th International Conference on NewActuators
Date of Conference:25-27 June 2018
Date Added to IEEE Xplore: 23 September 2018
Print ISBN:978-3-8007-4675-0
Publisher: VDE
Conference Location: Bremen,German
The allure of physically interacting with virtual objects in VR environments drives the
development of haptic gloves. This surge in VR market demand has fueled innovation
in this technology, with numerous commercial glove announcements in recent years.
However, not all solutions address the same needs, and technical hurdles remain,
particularly concerning actuation technology. This paper tackles these challenges. We
present a critical review of existing and past haptic glove projects, comparing their
features and performance. Finally, we offer insights into
the future directions of this rapidly evolving field.
Improvements made:
Stronger opening: Emphasises the user desire for touch in VR and the market
response.
Conciseness: Streamlined information while maintaining key points.
Action verbs: Emphasise functionalities like reviewing and comparing.
Focus: Maintains focus on technical aspects and future outlook.
17
Figure 4 prototype of the H glove
Paper [6]: "Haptic Glove Systems in Combination with Semi-Immersive Virtual
Reality for Upper Extremity Motor Rehabilitation after Stroke: A Systematic
Review and Meta-Analysis"
Authors: Diego Fernández-Vázquez, Roberto Cano-de-la-Cuerda, Víctor
Navarro-López
Publication: PMC9408073, 2022
This systematic review investigates the impact of combining haptic gloves with
semi-immersive VR for motor rehabilitation in stroke patients. It highlights how VR
combined with haptic feedback enhances neuroplasticity and motor recovery, offering
improvements over conventional rehabilitation techniques. The review concludes that
the haptic glove's feedback enhances patient interaction and motor function recovery
in post-stroke rehabilitation.
Improvements Made:Emphasis on VR and haptic glove synergy for medical
rehabilitation.
Conciseness: Directly addresses the benefits in neuroplasticity and motor recovery.
Action Verbs: Uses terms like "enhances," "improves," and "facilitates."
Focus: Concentrates on clinical impact and patient rehabilitation outcomes.
Paper [7]: Smart Haptic Gloves for Virtual Reality Surgery Simulation: A Pilot
Study on External Ventricular Drain Training.
18
Authors: Jonah Boutin, Jafer Kamoonpuri, Reza Faieghi, Joon Chung, Sandrine de
Ribaupierre, Roy Eagleson
Conference: Canadian Neurosurgery Rookie Bootcamp, 2022
DOI: PMC10806798
Summary: This study evaluates the use of smart haptic gloves in a VR surgery
simulator for training neurosurgery residents. The simulator allows users to experience
realistic tactile feedback during the External Ventricular Drain Placement (EVD)
procedure. The study measured performance metrics and user experience, showing
that haptic gloves improved realism and training effectiveness, demonstrating the
gloves’ value in enhancing surgical simulation for educational purposes.
Improvements Made:
Emphasis on haptic feedback's contribution to realism in surgical training.
Conciseness: Highlights EVD procedure and key metrics for training effectiveness.
Action Verbs: Focuses on "improving," "enhancing," and "demonstrating."
Focus: Centers on practical application in surgical training and measurable outcomes.
19
4. HARDWARE & SOFTWARE ARCHITECTURE
4.1 BLOCK DIAGRAM
Figure-5 Hardware block diagram
4.2 PICTURES
● 3D Model of the hardware
Figure 6 - Glove Hardware Model.
20
● Circuit
Left Hand Circuitry Right Hand Circuitry
Figure 7 - Circuit Diagram.
Hardware Used:
1. Synthetic Glove: A standard glove material used as the base for sensor integration.
2. Microcontrollers:
○ Arduino UNO: Primary microcontroller for managing inputs and data flow.
○ Arduino NANO: Secondary microcontroller used to handle additional sensor
data.
3. MPU6050 (6-Axis Accelerometer and Gyroscope): Used for tracking hand orientation
and movement, providing real-time spatial data.
4. NRF24L01 (Receiver and Transmitter): Wireless communication modules used for
transmitting sensor data between the glove and connected devices.
5. Flex Sensors (9 Total): Integrated on fingers for detecting bending and rotation.
6. AAA Battery: Power supply for microcontroller and sensors.
7. 10µF Capacitor: Used for voltage stabilization in the circuit.
21
Software Used:
1. Arduino IDE: Programming environment for writing, compiling, and uploading code
to the Arduino microcontrollers.
2. Unity Editor: Used to create and manage the virtual environment where haptic
interactions are visualized and simulated.
3. Blender: 3D modeling software used to design virtual objects and environments for
interaction within Unity.
4. [Link] (Tinkercad Circuits): Virtual platform for designing and simulating the
electronic circuit layout before physical implementation.
4.3 COMPARISON AND ESTIMATION OF OVERALL
EXPENSES OF THE PROJECT
SI No. Functionalities Referred Projects Spent Model Components Estimated
(Quantity) Cost (Quantity) Cost
1 Microcontroller Arduino Mega 3200 Arduino Nano - (1) 1900+550
2560-(1) Arduino Uno - (1)
2 Rotation(Fingers) Flex sensor(9) 11880 Flex Sensors(9) 2475
3 Communication NA NA NRF(3) 600
4 Hand Tracking MPU9250(1) 600 MPU6050(2) 360
5 Multiple I/Ps NA NA Multiplexer 74150N IC 150
6 Glove Synthetic Material 600 Synthetic Glove 475
7 Rotation(Joints) Servo motors(4) 4480 NA NA
8 Battery Source NA NA Rechargeable Battery(2) 300
9 Wiring Jumper cables 600 Jumper cables 150
10 Housing unit PLA Material 1200 3D Printed 950
TOTAL COST- Referred Project- 22560 Project Model -7910
Table 1: Cost comparison of Low Cost and Scalable Haptic VR Glove and the
proposed project
22
REQUIREMENTS OF PROJECT AND USE CASES
The software required for this project are:-
● Arduino IDE - Arduino Integrated Development Environment (IDE) is a
software platform used for programming Arduino microcontroller boards. It
provides an easy-to-use interface for writing, compiling, and uploading code to
Arduino boards. Arduino IDE supports the Arduino programming language,
which is based on C/C++, and offers a range of libraries and tools for
developing projects involving sensors, actuators, and other electronic
components.
● Unreal Engine - Unreal Engine is a powerful and widely-used game engine
developed by Epic Games, popular for creating high-quality 3D applications,
including virtual reality (VR), augmented reality (AR), and immersive
simulations. The Unreal Editor is the main interface for developers, providing a
robust set of tools to design and build interactive and visually stunning
environments. It enables the development of realistic game environments,
complex gameplay mechanics, asset integration, and performance
[Link] Engine supports a wide range of platforms, including PC,
mobile devices, gaming consoles, and VR/AR headsets, making it versatile for
cross-platform development. Its advanced rendering capabilities and support
for real-time physics and animations make it ideal for projects requiring high
visual fidelity and interactive experiences, such as VR simulations and training
applications.
● Blender - Blender is a free and open-source 3D computer graphics software
used for creating animated films, visual effects, video games, and 3D printed
models. It offers a wide range of features for modeling, sculpting, rigging,
animation, rendering, and compositing. Blender's flexible and powerful toolset
makes it suitable for a variety of tasks, from creating character models and
environments to producing complex visual effects sequences. It is widely used
by artists, designers, and animators for both personal and professional projects.
● [Link] - also known as Tinkercad Circuits, is an online platform developed
by Autodesk that allows users to design, simulate, and prototype electronic
circuits in a virtual environment. It offers a user-friendly interface with a
drag-and-drop system for adding components such as resistors, capacitors,
LEDs, microcontrollers, and sensors to a virtual breadboard.
23
Hardware required for this project are:
● Synthetic Glove:
A synthetic glove is a wearable device typically made from synthetic
materials such as nylon or polyester. It is used to house sensors and electronics
for capturing hand movements and gestures, often used in applications like
motion capture, virtual reality, or human-computer interaction.
● Microcontrollers:
a) Arduino UNO: Arduino UNO is a popular microcontroller board
based on the ATmega328P microcontroller. It features digital and analog
input/output pins, onboard power regulation, and a USB interface for
programming and communication with a computer. It is widely used for
prototyping and DIY electronics projects.
Figure 8 - Arduino Uno.
b) Arduino NANO: Arduino NANO is a compact and cost-effective
microcontroller board based on the ATmega328P. It offers similar features to
Arduino UNO but in a smaller form factor, making it suitable for projects with
space constraints or embedded applications.
24
Figure 9 - Arduino Nano.
● MPU6050:
MPU6050 is a 6-axis accelerometer and gyroscope module. It combines
a 3-axis accelerometer and a 3-axis gyroscope into a single chip, allowing for
accurate measurement of motion, orientation, and tilt. It is commonly used in
applications such as motion tracking, robotics, and wearable devices.
Figure 10 - MPU6050.
● NRF24L01:
NRF24L01 is a low-cost wireless transceiver module operating in the
2.4GHz ISM band. It provides a simple and efficient way to establish
communication between microcontrollers or other electronic devices wirelessly
over short distances. It is commonly used in projects requiring wireless data
transmission, such as remote control systems and sensor networks.
Figure 11 - NRF24L01.
25
● Flex Sensors:
Flex sensors are resistive devices that change resistance when bent or
flexed. They are typically made from a flexible substrate coated with
conductive material. Flex sensors are commonly used for measuring the degree
of bending or flexing in applications such as robotics, motion capture, and
wearable devices.
Figure 12 - Flex Sensor.
● ERM Coin Vibration Motor::
The ERM (Eccentric Rotating Mass) Coin Vibration Motor is a
compact and efficient device commonly used to provide haptic feedback in
portable electronics, such as mobile phones, wearables, and handheld devices.
ERM motors generate vibrations by rotating an unbalanced mass, which creates
an oscillating force. As the mass rotates, it causes the entire motor to shake,
producing a tactile sensation that can be used as feedback in various
applications.
26
Figure 13 - ERM Coin Vibrator
● Rechargeable Battery:
A rechargeable battery is a compact and portable power source
commonly used in electronic projects and devices requiring low to moderate
power consumption. It provides a convenient power supply for
microcontrollers, sensors, and other electronic components.
Figure 14 - Rechargeable Battery.
● 10 Microfarad Capacitor:
A 10 microfarad capacitor is an electronic component used to store and
release electrical energy. Capacitors are commonly used in electronic circuits
for filtering, smoothing, and decoupling purposes. A 10 microfarad capacitor
specifically would be used for tasks such as voltage regulation or noise
reduction.
27
Figure 15 - 10 Microfarad Capacitor.
HARDWARE ARCHITECTURE
The haptic glove system is designed with components for sensing, processing,
communication, and feedback delivery. Key components are:
1. Microcontrollers:
○ Arduino Uno: Used in the left glove to process sensor data and control
ERM vibrational motor feedback.
○ Arduino Nano: Used in the right glove to handle spatial orientation and
look-around functionalities for improved navigation.
2. Sensors:
○ MPU6050 IMU Sensors: Track spatial movements and hand orientation
in real time, used in both gloves for accurate movement tracking.
○ Flex Sensors: Positioned on each finger to detect finger bending,
allowing the system to capture hand gestures like grasping, pointing, and
more.
3. Wireless Communication:
○ NRF24L01 Modules: Enable wirelessly transmitted data between the
gloves and the VR system, ensuring low-latency communication for
real-time interaction.
4. Power Supply:
○ Powered by AAA batteries with a 10-microfarad capacitor to ensure
stable power delivery and prevent fluctuations during high power
demand.
5. Actuators:
○ ERM Vibrational Motors: Installed in the left glove to provide haptic
feedback through vibrations, enhancing tactile interaction in the VR
environment.
28
Dual-Hand Functionality
The glove system uses both hands with specific functionalities for an enhanced VR
experience:
1. Left Hand Functionality:
○ Equipped with flex sensors, an MPU6050 IMU, and ERM vibrational
motors. It handles actions like aiming, shooting, and reloading,
providing tactile feedback and immersion during VR gaming.
2. Right Hand Functionality:
○ Uses flex sensors and an MPU6050 IMU. It primarily controls spatial
navigation (look-around functionality), allowing users to explore the
virtual environment and interact with objects through natural hand
gestures.
3. Dual-Hand Interaction:
○ By distributing functions across both gloves, users can perform
simultaneous actions, such as aiming with the left hand while exploring
with the right, replicating real-world interactions in VR.
Working of the Project
The system works in five key stages:
1. Stage 1: Hand Orientation and Position Capture:
○ Each glove’s MPU6050 tracks hand orientation, converting rotational
data into quaternion representations for accurate and continuous
movement tracking in VR.
2. Stage 2: Finger Flexion Detection:
29
○ Flex sensors detect finger bending, converting analog signals into
digital data for accurate gesture recognition (gripping, pointing, etc.).
3. Stage 3: Data Processing and Transmission:
○ Data from the sensors is processed by the Arduino microcontrollers. It is
then transmitted wirelessly to the VR system using NRF24L01
modules.
4. Stage 4: Data Interpretation in Unreal Engine:
○ Unreal Engine interprets the transmitted data, mapping hand orientation
from the MPU6050 sensors and finger movements from the flex sensors
to animate gestures in VR.
5. Stage 5: Real-Time Interaction and Feedback:
○ The left glove provides haptic feedback using ERM vibrational
motors during actions like shooting, while the right glove allows
dynamic perspective control and navigation in the virtual world.
Enhancing Realism Through Integrated Feedback
The dual-hand system, combining hand movement tracking and haptic feedback,
improves user interaction within VR environments, making it more intuitive and
lifelike. By mimicking real-life interactions, the system reduces the learning curve and
enhances immersion.
30
4.4 SOFTWARE ARCHITECTURE
1. Animation Class (Control Rig)
○ Purpose: The Control Rig in the Animation Class is responsible for
managing and animating the skeletal structure of the VR character’s
hands and fingers. It uses Inverse Kinematics (IK) to synchronize VR
hand and finger movements with real-world actions captured by the
sensors.
○ Functionality:
■ Fine-Tuned Control: Provides precise control over hand and
finger animations, allowing realistic gestures, grips, and finger
flexion in the virtual environment.
■ Local and Global Transformations: Adjusts both local
(finger-specific) and global (whole-hand) bone transformations,
including rotation and positioning.
■ Inverse Kinematics (IK): Calculates joint rotations to achieve
target positions, ensuring that VR hand movements align
accurately with user input.
■ Dependent vs. Independent Movements:
■ IK (Dependent): Controls whole-hand movements, such
as reaching a target.
■ Forward Kinematics (FK) (Independent): Manages
individual finger movements, allowing each finger to
move independently.
○ Outcome: This class translates sensor data into smooth, realistic hand
and finger animations in VR, closely reflecting the user’s real-world
gestures and improving immersion.
2. Player Class
○ Purpose: The Player Class manages the character’s overall movement
and interaction within the VR environment based on glove input data,
facilitating body motion and positioning for interactive hand
movements.
31
○ Functionality:
■ Movement Logic: Maps glove coordinate data (X, Y, Z) to VR
hand movements, ensuring spatial alignment between the user's
physical movements and the virtual representation.
■ Integration with Unreal Engine’s Character Movement
Component: Utilizes Unreal Engine’s movement component to
apply velocity, direction, and rotation derived from sensor data.
■ Dynamic Positioning: Continuously updates the VR hand
location to enable seamless and natural interaction with virtual
objects.
○ Outcome: This class converts glove spatial data into VR character
actions, allowing immersive and interactive control of hand movements
in the VR environment.
3. Arduino Class
○ Purpose: The Arduino Class acts as a data bridge between the glove
sensors (e.g., IMUs, flex sensors) and the VR software. It processes
sensor data and transmits it to the VR system for real-time integration.
○ Functionality:
■ Sensor Data Acquisition:
■ IMUs: Capture hand orientation and rotation, enabling
accurate tracking of spatial movements.
■ Flex Sensors: Detect finger bending and joint angles to
facilitate realistic gesture recognition.
■ Serial Communication Interface: Establishes a real-time serial
connection to transmit sensor data directly to the VR software.
■ Data Transmission Speed: Operates at a high frame rate to
ensure virtual hand movements are synchronized with the user’s
physical actions, creating a fluid VR experience.
○ Outcome: This class enables accurate and continuous data flow from
glove sensors to VR, ensuring real-time responsiveness and an
immersive experience through precise hand movement tracking.
32
4.4.1 SOFTWARE TOOLS USED
● Arduino IDE - Arduino Integrated Development Environment (IDE) is a
software platform used for programming Arduino microcontroller
boards. It provides an easy-to-use interface for writing, compiling, and
uploading code to Arduino boards. Arduino IDE supports the Arduino
programming language, which is based on C/C++, and offers a range of
libraries and tools for developing projects involving sensors, actuators,
and other electronic components.
● Unreal Engine - Unreal Engine is a powerful and widely-used game
engine developed by Epic Games, popular for creating high-quality 3D
applications, including virtual reality (VR), augmented reality (AR), and
immersive simulations. The Unreal Editor is the main interface for
developers, providing a robust set of tools to design and build interactive
and visually stunning environments. It enables the development of
realistic game environments, complex gameplay mechanics, asset
integration, and performance [Link] Engine supports a
wide range of platforms, including PC, mobile devices, gaming
consoles, and VR/AR headsets, making it versatile for cross-platform
development. Its advanced rendering capabilities and support for
real-time physics and animations make it ideal for projects requiring
high visual fidelity and interactive experiences, such as VR simulations
and training applications.
● Blender - Blender is a free and open-source 3D computer graphics
software used for creating animated films, visual effects, video games,
and 3D printed models. It offers a wide range of features for modeling,
sculpting, rigging, animation, rendering, and compositing. Blender's
flexible and powerful toolset makes it suitable for a variety of tasks,
from creating character models and environments to producing complex
33
visual effects sequences. It is widely used by artists, designers, and
animators for both personal and professional projects.
● [Link] - also known as Tinkercad Circuits, is an online platform
developed by Autodesk that allows users to design, simulate, and
prototype electronic circuits in a virtual environment. It offers a
user-friendly interface with a drag-and-drop system for adding
components such as resistors, capacitors, LEDs, microcontrollers, and
sensors to a virtual breadboard.
1. Initial Prototype: Basic Sensor Integration
The first prototype focused on the integration of fundamental sensor functionality.
This stage emphasized establishing the foundational tracking needed for hand
movements within Unity. While the initial design was limited by wired connections, it
successfully captured and transmitted essential sensor data, demonstrating the
feasibility of real-time hand tracking. This prototype laid the groundwork for
subsequent improvements in both hardware and software design.
2. Prototype 2: Transition to Wireless Functionality
To enhance user comfort and mobility, the second prototype introduced wireless
communication capabilities. By incorporating NRF24L01 modules, this version
eliminated the constraints of wired connections, enabling smoother, more natural hand
movements within VR. The transition to wireless functionality significantly improved
the user experience, providing a glimpse of the system's potential for immersive VR
interactions. Enhanced firmware ensured stable and reliable data transmission during
prolonged use.
3. Enhanced Simulation with Unreal Engine
The third prototype leveraged the advanced graphical and simulation capabilities of
Unreal Engine. This version introduced Inverse Kinematics (IK) for precise hand
simulation, enabling realistic skeletal animations and accurate representation of finger
and hand movements in the virtual environment. The separation of functionalities
between the left and right hands facilitated intuitive navigation and interaction in VR.
This iteration also saw the addition of haptic feedback elements, further increasing
immersion and realism.
34
4. Final Prototype: Integrated Hardware and Immersive VR Environment
The final prototype combined all previous advancements into a cohesive and robust
system. Featuring custom-designed housing for hardware protection and optimized
level design within Unreal Engine, this version offered a seamless blend of
functionality, comfort, and immersion. The dual-hand system allowed users to
perform complex interactions, such as aiming and shooting with tactile feedback,
while exploring virtual environments. With enhanced durability, portability, and
ergonomic design, the final prototype achieved a balanced and polished VR
experience. This iteration demonstrated the system's readiness for real-world
applications and potential commercialization.
PROJECT TIMELINE
Table 2: Timeline of each phase of the project
5. CONCLUSION
● This research paper presents the development and evaluation of a low-cost, scalable
haptic VR glove system, designed to enhance immersion and interactivity in virtual
environments. By integrating flex sensors, IMUs, and vibro-thermal units, this system
35
successfully translates natural hand movements and gestures into VR with high
fidelity. Through iterative prototype development, we addressed key challenges in
sensor integration, wireless functionality, and skeletal animation control, ultimately
achieving a responsive and immersive VR experience. The combination of hardware
components and software classes—Arduino Class for data transmission, Animation
Class for skeletal control, and Player Class for VR interaction—demonstrates the
effectiveness of our approach in capturing real-world hand gestures and rendering
them in VR. The final prototype, utilizing Unreal Engine, delivers accurate hand
simulation with intuitive interaction and high visual realism, providing a significant
advancement in VR haptic technology. This haptic glove system shows promising
applications across multiple fields, including VR gaming, medical training,
manufacturing, and robotic teleoperation. Future improvements could involve refining
haptic feedback quality, enhancing wireless performance, and further reducing
hardware weight and size for extended use. Our findings lay the groundwork for
further exploration in human-machine interaction, advancing VR technology’s ability
to replicate complex tactile experiences and enhance training, simulation, and
interactive applications in virtual environments.
6. FUTURE SCOPE
● Enhanced Immersive Experience: Integrating haptic feedback for vibrations and gun
recoil into the shooting range game will significantly enhance the immersive
experience. Users will feel tactile sensations in their hands, making the gameplay
more realistic and engaging.
● Wireless Integration: Making the glove wireless using NRF24L01 sensor and flex
sensors for finger rotation computation allows for greater freedom of movement and
eliminates the need for cumbersome wires. This enhances user comfort and flexibility
during gameplay.
● Precise Hand Pose Estimation: Utilizing the MPU6050 sensor for hand pose
estimation ensures accurate tracking of hand movements and gestures. This enables
precise control over virtual objects and interactions within the game environment,
enhancing user immersion and gameplay precision.
● Expandability and Customization: The modular nature of the project allows for future
expansion and customization. Additional sensors or haptic feedback mechanisms
can be integrated to further enhance the gaming experience. Moreover, the system
36
can be adapted for different types of games or applications beyond just the shooting
range genre.
● Potential for Commercialization: With the rising popularity of AR/VR gaming, a fully
developed and polished product incorporating these features could have significant
commercial potential. It could be marketed as a high-end gaming accessory for
enthusiasts or incorporated into VR arcade experiences.
● Research and Development Opportunities: The project opens avenues for further
research and development in the field of immersive technologies. It could lead to
advancements in haptic feedback systems, hand pose estimation algorithms, and
wireless communication protocols for VR applications.
Applications Beyond Gaming
The glove system has potential applications in:
● Medical Training: Simulating surgeries or medical procedures for practice without
risk.
● Rehabilitation: Assisting patients with motor skill recovery by simulating exercises in
a virtual environment.
● Industrial Training: Providing workers with virtual hands-on training for complex
machinery or hazardous environments.
● Robotics: Controlling robotic arms with precision using real-time hand movement
tracking.
● Education: Creating interactive educational simulations for learning complex
concepts in physics, chemistry, or biology.
Environmental Adaptability
Future designs can incorporate adaptability to various environmental conditions, such as
waterproofing for outdoor use or temperature resistance for extreme conditions. This would
make the system viable for use in diverse fields, including space exploration and underwater
research.
37
User Feedback Integration
Incorporating user feedback mechanisms to dynamically adjust haptic feedback intensity and
gesture sensitivity based on user preferences can enhance the usability and personalization
of the system. This feedback loop could use AI to make real-time adjustments during use.
Multi-User VR Interaction
Future iterations could enable multi-user interaction within the same virtual environment.
This would allow for collaborative VR applications, such as team-based gaming, virtual
meetings, or joint training exercises, where multiple users interact with shared virtual objects.
7. REFERENCES
1. Low Cost and Scalable Haptic VR Glove, Published in: 2022 14th International
Conference on Computational Intelligence and Communication Networks (CICN),
Date of Conference: 04-06 December 2022, Date Added to IEEE Xplore: 13 January
2023, DOI: 10.1109/CICN56167.2022.10008349, Publisher: IEEE, Conference
Location: Al-Khobar, Saudi Arabia
[1] Alexander C. Abad, Manex Ormazabal, David Reid, and Anuradha Ranasinghe.
An untethered multimodal haptic hand wearable. In 2021 IEEE Sensors, pages 1–4,
2021.
[2] Thomas Alexander, Martin Westhoven, and Jessica Conradi. Virtual environments
for competency-oriented education and training. In Advances in Human Factors,
Business Management, Training and Education, pages 23–29. Springer, 2017.
[3] Valve Corporation. Head mounted display, 2015.
[4] Wolfgang Daum, Thomas Gunther, Jorn Husert, and Patrick Scherr. Motion
sensor which produces an asymmetrical signal in response to symmetrical movement,
1987.
[5] Michael Efraimidis and Katerina Mania. Wireless embedded system on a glove
for hand motion capture and tactile feedback in 3d environments. In SIGGRAPH Asia
2020 Posters, pages 1–2. 2020.
38
2. A Comprehensive Review of Haptic Gloves: Advances, Challenges, and Future
Directions (Published in: 2023 Second International Conference on Electronics and
Renewable Systems (ICEARS),Date of Conference: 02-04 March 2023,Date Added
to IEEE Xplore: 05 April 2023,DOI:
10.1109/ICEARS56392.2023.10085607,Publisher: IEEE, Conference Location:
Tuticorin, In. Yuan Guo, Dangxiao Wang, Senior Member, IEEE,
3. Sensory-Glove-Based Human Machine Interface for Augmented Reality (AR)
Applications, Published in: 2020 IEEE 33rd International Conference on Micro
Electro Mechanical Systems (MEMS),Date of Conference: 18-22 January 2020,Date
Added to IEEE Xplore: 06 April 2020,DOI:
10.1109/MEMS46641.2020.9056153,Publisher: IEEE, Conference Location:
Vancouver, BC, Canada. B. O’Flynn et al., “Novel smart sensor glove for arthritis
rehabiliation,” 2013 IEEE Int. Conf. Body Sens. Networks, BSN 2013, pp. 1–6, 2013.
4. Virtual reality simulation modeling for a haptic glove, Published in: Proceedings
Computer Animation 1999,Date of Conference: 26-29 May 1999,Date Added to IEEE
Xplore: 06 August 2002,Print ISBN:0-7695-0167-2,Print ISSN: 1087-4844,DOI:
10.1109/CA.1999.781212,Publisher: IEEE.
5. Touching Virtual Reality: A Review of Haptic Gloves, Published in: ACTUATOR
2018; 16th International Conference on New Actuators, Date of Conference: 25-27
June 2018,Date Added to IEEE Xplore: 23 September 2018,Print
[1] S. Lederman and R. Rowse. “The physiology and psychophysics of touch. In Proceedings
of the NATO Advanced Research Workshop on Sensors and Sensory Systems for Advanced
Robots, volume F43, pages 71-91, 1988.
[2] [Link] cybergrasp, March 2018
[3] M. Turner, D. Gomez, M. Tremblay, and M. Cutkosky, “Preliminary tests of an
armgrounded haptic feedback device in telemanipulation,” in Proc. ASME Dyn. Syst.
Control Div., 1998, vol. DSC-64, pp. 145– 149
[4] M. Bouzit, G. Popescu, G. Burdea and R. Boian, “The Rutgers Master II-ND
Force Feedback Glove”, Proceedings of IEEE VR 2002 Haptics Symposium, Orlando
FL, March 2002
[5] C. Pacchierotti, S. Sinclair, M. Solazzi, A. Frisoli, V. Hayward, and D.
Prattichizzo, “Wearable Haptic Systems for the Fingertip and the Hand: Taxonomy,
Review, and Perspectives”, IEEE Transactions on Haptics (ToH), IEEE, 2017, 10 (4),
pp.580 - 600ISBN:978-3-8007-4675-0,Publisher: VDE, Conference Location:
Bremen, German.
6. S. Lederman and R. Rowse. “The physiology and psychophysics of touch. In
Proceedings of the NATO Advanced Research Workshop on Sensors and Sensory
39
Systems for Advanced Robots, volume F43, pages 71-91, 1988.
7. Alexander C. Abad, Manex Ormazabal, David Reid, and Anuradha Ranasinghe. An
untethered multimodal haptic hand wearable. In 2021 IEEE Sensors, pages 1–4, 2021.
[2] Thomas Alexander, Martin Westhoven, and Jessica Conradi. Virtual environments
for competency-oriented education and training. In Advances in Human Factors,
Business Management, Training and Education, pages 23–29. Springer, 2017
8. Ziqi Wang, Xiuping Yang, Haitong Wang, Yuru Zhang, Senior Member, IEEE,
Weiliang Xu, Senior Member, IEEE “Achieving High Stiffness Range of Force
Feedback Gloves using Variable Stiffness Mechanism.” 2019 IEEE World Haptics
Conference (WHC) Tokyo, Japan.
40