Indian Institute of Technology, Guwahati
781039 Guwahati,Assam,India
AI-Based Gesture Recognition for Assistive Communication Using Eye Movements
Pawar Maheshwari - 220107061
Komala Madhavi - 220107048
Goddati Sai Likhitha -220107041
Motivation
• Individuals suffering from quadriplegia or severe motor disabilities often lose the ability to use their hands for communication and
computer interaction.
• Traditional input devices such as keyboards and mice require physical movement, making them inaccessible for such users.
• Eye tracking provides an alternative interface that allows users to interact with computers using only eye movements.
• Advances in computer vision and machine learning enable accurate detection of facial landmarks and eye motion using a standard
webcam.
• This project aims to develop an AI-based system that allows users to type text using a gaze-controlled virtual keyboard.
State of the art Core technological idea
• The system enables text input using eye gestures
• Early eye tracking systems used specialized infrared detected through computer vision techniques.
cameras and expensive hardware. • Main components of the system:
• Modern approaches use computer vision algorithms to • Webcam captures real-time frames of the user’s face.
detect eye movements using standard cameras. • Face detection identifies the face region in each frame.
• Facial landmark detection algorithms identify key points • Facial landmark detection locates eye positions.
on the face to locate the eyes. • Eye blinking detection identifies when the user selects
• By analyzing eye movement and blinking patterns, it is a letter.
possible to estimate gaze direction. • Gaze detection determines whether the user is looking
• Such systems enable touchless human–computer left or right.
interaction and assistive communication technologies. • A virtual keyboard interface allows users to type
characters using eye gestures.
Implementation & Achievements
• The system is implemented1 mm using Python with OpenCV
and Dlib libraries.
• A webcam captures real-time video frames.
• Facial landmark detection identifies 68 key points on the
face.
• Eye landmarks are used to detect blinking and gaze
direction.
• The keyboard interface highlights letters sequentially on
the screen.
• Users can select a letter by blinking their eyes when the
desired letter is highlighted.
• The system successfully enables typing of words using
eye movements.
Reflection&Conclusion Outlook
Future improvements may include:
• The project demonstrates that eye-tracking
• Integrating predictive text to improve typing speed
interfaces can be implemented using low-cost
• Using deep learning models for more accurate gaze
computer vision techniques. detection
• The system provides an assistive communication • Developing mobile or web-based versions of the system
tool for individuals with severe physical disabilities. • Integrating speech output for assistive communication
• Lighting conditions and camera resolution can affect
detection accuracy.
• Future improvements could enhance robustness and Reference
[1] Sergio Canu, “Gaze Controlled Keyboard using OpenCV,” Pysource Blog.
usability. s ,
[2] V. Kazemi and J. Sullivan, “One Millisecond Face Alignment with an Ensemble of
Regression Trees,” IEEE CVPR 2014.
[3] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools.
[Link]