Report On

Motion controlled game system using computer vision

Submitted in partial fulfillment of the requirements of the Mini project in
Semester III of Second Year Artificial Intelligence and Data Science (AI&DS)

By

Parth Puri (Roll No. 24)

Divyah Mandavia (Roll No. 11)
Devashree Pawar (Roll No. 22)

Mentor
Prof. Priyanka Bhoir

University of Mumbai

Vidyavardhini's College of Engineering & Technology

Department of Artificial Intelligence and Data Science (AI&DS)

(A.Y. 2021-22)
 Vidyavardhini's College of Engineering & Technology

Department of Artificial Intelligence and Data Science (AI&DS)

CERTIFICATE

This is to certify that the Mini Project entitled “Motion controlled game system using
computer vision” is a Bonafede work, Parth Puri (Roll No. 24), Devashree Pawar (Roll
No. 22) and Divyah Mandavia (Roll No. 11) submitted to the University of Mumbai in
partial fulfillment of the requirement for the award of the degree of “Bachelor of
Engineering” in Semester III of Second Year “Artificial Intelligence and Data Science
(AI&DS)”.

______________________
Prof. Name Surname
Mentor

________________ _______________
Dr Thaksen Parvat Dr. H.V. Vankudre
Co-ordinator of Department Principal
 Vidyavardhini's College of Engineering & Technology

Department of Artificial Intelligence and Data Science (AI&DS)

Mini Project Approval

This Mini Project entitled “Motion controlled game system using computer
vision” by Parth Puri (Roll No. 24), Devashree Pawar (Roll No. 22) and Divyah
Mandavia (Roll No. 11) is approved for the degree of Bachelor of Engineering in
in Semester III of Second Year Artificial Intelligence and Data Science (AI&DS)

Examiners

1………………………………………
(Internal Examiner Name & Sign)

2…………………………………………
(External Examiner name & Sign)

Date:

Place:
 Contents

Abstract ii

Acknowledgments iii

List of Abbreviations iv

List of Figures v

List of Tables vi

List of Symbols vii

1 Introduction 1
1.1 Introduction
1.2 Problem Statement & Objectives
1.3 Scope

2 Literature Survey 8

2.1 Survey of Existing System/SRS

2.2 Limitation Existing system or Research gap
2.3 Mini Project Contribution

3 Proposed System (e.g., New Approach of Data Summarization) 10

3.1 Introduction
3.2 Architecture/ Framework/Block diagram
3.3 Algorithm and Process Design
3.4 Details of Hardware & Software
3.5 Experiment and Results for Validation and Verification
3.6 Analysis
3.7 Conclusion and Future work.

References 16
4 Annexure

4.1 Published Paper /Camera Ready Paper/ Business pitch/proof of concept

Abstract:

This research deals with presenting a design of a game controller interface based on visual processing. Visual
define the world, each visual has its own story, it contains a lot of crucial information that can be useful in many
ways. This information can be obtained with the help of the technique known as Visual Processing. It is the core
part of computer vision which plays a crucial role in many real-world examples like robotics, self-driving cars,
and object detection. The mechanical and electric approach of playing a video game has been changed, i.e., a
joystick, with camera inputs. This research helps in exploring a field of artificial intelligence, i.e., Computer
Vision. Computer vision is a field of artificial intelligence (AI) that enables computers and systems to derive
meaningful information from digital images, videos and other visual inputs — and take actions or make
recommendations based on that information Computer vision works much the same as human vision, except
humans have a head start. Human sight has the advantage of lifetimes of context to train how to tell objects
apart, how far away they are, whether they are moving and whether there is something wrong in an image. Our
system is based entirely on Computer Vision and does not use any marker system such as any signs or symbols.
Motivation is one of the key factors when having to do exercises for hand mobility, because they have to be
done consistently. The application is visually attractive and simple to use. we propose a human-computer
interaction (HCI) system for gaming purpose via a camera. Human–computer interaction (HCI) is research in
the design and the use of computer technology, which focuses on the interfaces between people (users)
and computers. HCI researchers observe the ways humans interact with computers and design technologies that
allow humans to interact with computers in novel ways. The whole system is comprised of three modules: hand
tracking, gesture recognition and integration with the controls in our system, specifically, hand detection is
based entirely on computer vision.
Acknowledgement:
List of Figures:

Sr no. Figure Name Page no.

1 Closed hand within the up triangle 13

2 Closed hand within the down triangle 13

3 Closed hand within the left triangle 14

4 Closed hand within the right triangle 14

1.) Introduction:

1.1) Introduction

This project presents a design of a game controller interface based on visual processing. Visual processing
allows us to transform and manipulate thousands of Visual at a time and extract useful insights from them. It
has a wide range of applications in almost every field. Python is one of the widely used programming
languages for this purpose. Its amazing libraries and tools help in achieving the task of visual processing
very efficiently. Through this article, you will learn about classical algorithms, techniques, and tools to
process the image and get the desired output. In vision-based interfaces for video games, gestures are used
as commands for the games instead of pressing buttons on a keyboard or moving a mouse.
Computer vision-based interface to a game holds a promise of rich natural interaction and thus provides a
realistic gaming experience. In this interface, the unintentional movement and continuous gestures must be
supported to provide the user with more natural interface. To explore the vision-based systems we have
worked on the game of Battle City. We have coded this project using Python. We have included NumPy and
OpenCV libraries.
This project is considered to be under the field of gesture recognition which is a field which is gaining
widespread popularity as it makes it very easy for the user get some particular tasks done. NumPy is a
Python library used for working with arrays. It also has functions for working in domain of linear algebra,
Fourier transform, and matrices. NumPy was created in 2005 by Travis Oliphant. It is an open-source
project and you can use it freely. NumPy stands for Numerical Python.
OpenCV is the huge open-source library for the computer vision, machine learning, and image processing
and now it plays a major role in real-time operation which is very important in today’s systems. By using it,
one can process images and videos to identify objects, faces, or even handwriting of a human. When it
integrated with various libraries, such as NumPy, python is capable of processing the OpenCV array
structure for analysis. To Identify image pattern and its various features we use vector space and perform
mathematical operations on these features. Gesture recognition has indeed created very widespread use in
the gaming industry and is surely moving ahead.

1.2) Problem Statements and Objectives:

The current game system consists of manual input using optical devices such as keyboard and mouse, the
implementation of a system that recognizes the motion gestures to perform various operations that we will
make the current system effortless.

1.3) Scope:
This project has wide ranging applications over the gaming performance of a player. It directly establishes
Human Computer Interaction (HCI). It takes the gaming experience of a player on a next level. The
background of the implementation can be also used to develop an interesting gaming application for the
children in which they can get immersed and have a fabulous experience. It can be also implemented in
various games according to the controls that is one of the best things about this project. As well as gaming
can be more smooth, entertaining, realistic and etc.
2.) Literature Survey:

2.1) Survey of existing system/SRS:

[1] In this paper, we propose a human-computer interaction (HCI) system for entertainment or education via
a depth-sensing camera. The whole system is comprised of three modules: hand detection, hand tracking,
and gesture recognition. In our system, specifically, hand detection is based entirely on computer vision and
do not use any markers. We utilize the Kalman filter and depth data from the Kinect to predict the hand
position making the tracking smooth and robust. And we extract the apparent gestural features to recognize
gestures, which are adaptable and quick.
Advantage: The most significant advantage of this paper is that its gesture-based HCI system is its natural
interaction.
Disadvantage: This system works a little bit slow.

[2] This paper proposes a motivational application for people who have suffered hand trauma and aims to
help them to do exercises for regaining hand mobility.
Advantages: The primary purpose of the application is to incite the user to perform as many hand exercises
as possible.
Disadvantages: The main drawback consists in the fact that the system needed a 6-camera module and hand-
held sensors to detect movement.

[3] This paper has been taken up to demonstrate an emerging field of Human Computer Interaction i.e.,
Gesture Recognition. Gesture recognition has found its way into many applications ranging all across from
basic home automation to navigation and gaming. In this paper, I have explored the capabilities of gesture
recognition by demonstrating the classic Snake game as a gesture-controlled snake.
Advantages: This paper gives light on conceivable outcomes of utilizing computer vision to take care of
genuine issues are many.
Disadvantages: The disadvantage of this paper system requirement.

[4] This research presents a design of a game controller interface based on visual processing. Mechanical
and electrical interfaces of a video game can be replaced with a camera input. Tests were performed for a
car simulation game. The game player operates a mock wheel with colored markings on it to the
camera.
Advantages: The main advantage of this paper is that it helps fast pace gamers to gain more control over the
game as compared to traditional method.
Disadvantages: This system can get more complex by adding additional gestures.
2.2) Limitation Existing system:
The system has a limitation on hand gesture. Only static gestures are supported. In addition, there are
restrictions on the multi-hand tracking and recognition. And multi-user cooperation or multi-hand
competition is becoming more and more of a trend. The system gets slow which causes FPS fluctuations and
it happens a frequent time.

2.3) Mini Project Contribution:

A quick, lightweight method in terms of computation resources for detecting and recognizing a set of
gestures, under different light conditions. A game controlled by the gestures mentioned above, which aims
to motivate the user into doing hand exercises for hand mobility.
This will take the gaming world to the next level. It will enhance the control over the game compared to the
traditional method controller.
3.) Proposed System:

3.1) Introduction:
The project is implemented completely using Python. Libraries like python 3.x, cv2, TensorFlow, NumPy,
gym, gym_super_mario_bros, gym_chrome_dino and etc. have contributed much towards this project.
Setting up the key binding using the python libraries and then integrating the key bindings with the game.
the controller gets ready. The main and the most important thing to complete the project a camera for
capturing and tracking the movements of the user is also a need of the project. Implementing the
landmarks on the hand is the most important thing in this project. The camera works like an interaction
bridge between the system and the user. Eventually that makes the whole system work perfectly.

3.2) Process Design:

Live video capture using camera is the first step. Implementing landmark points on hand is the most
important thing for tracking the movements of the user. The next thing setting up key binding using python
library. Importing libraries like OpenCV, NumPy, TensorFlow are the most important libraries in the
project. Integrating the key bindings with the code. Integrating the code with OpenCV helps to recognize
the movements of the user. Setting up data set for the game. Setting up the code as an application. And one
of the most important step Testing of the game makes the controller the best of the best.

3.3) Details of Hardware and Software:

Recommended System Requirements
Software Requirements – Python 3.7 or above is required. The most important Libraries like OpenCV,
NumPy, TensorFlow and etc. are required.
Hardware Requirements – A personal computer with a camera is must, having processor intel core i5 5th
gen and above. And graphics card is required.
3.4) Experiment and Results for Validation and Verification:
Experiment: Code-
Result:
1. Closed hand within the up triangle -> Go up

FIG 3.4.1
2. Closed hand within the left triangle -> Go down

FIG 3.4.2
3. Closed hand within the left triangle -> Go left

FIG 3.4.3
4. Closed hand within the right triangle -> Go right

FIG 3.4.4
3.6) Analysis:

Gesture recognition has found its way into many applications ranging all across from basic home automation to
navigation and gaming. the field of gesture recognition which is a field which is gaining widespread popularity
as it makes it very easy for the user get some particular tasks done. Gesture recognition has slowly but surely
created a stronghold in the gaming industry and now moving beyond it in every aspect. Together with computer
vision it is enabling for a much better user experience and dynamic and user-friendly interfaces.

3.7) Conclusion and Future work:

1. Conclusion:
With the help of OpenCV it has become very easy to implement hand gestures which has made this task
faster and simpler. This project helps us to build and learn various aspects of programming. This also makes
gaming as a life changing experience.

2. Future Work:
The first part that is creating a controller system for a game is done and testing it on a simple game has
worked perfectly. Second and the third part in this project would be firstly optimizing the code for achieving
better results than the earlier testing. And finally creating software out of that.
3.8) References:
REFERENCES
[1] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A Discriminative Feature Learning Approach for Deep Face
Recognition,” Proc. of European Conference on Computer Vision (ECCV), pp. 499-515, 2016.

[2] Y. Li, “Hand gesture recognition using Kinect,” Proc. of IEEE Control and Decision Conference (ICDC),
pp. 196-199, 2017.

[3] J. Wu and J. Cheng, “Bayesian Co-Boosting for Multi-modal Gesture Recognition,” Journal of Machine
Learning Research, vol. 15, no. 1, pp. 3013-3036, 2017.

[4] Q. Pu, S. Gupta, S. Gollakota, and S. Patel, “Whole-home gesture recognition using wireless signals,” Proc.
of International Conference on Mobile Computing & NETWORKING ACM, pp.27-38, 2013.

[5] C. H. Morimoto and M. R. M. Mimica, “Eye gaze tracking techniques for interactive applications ☆,”
Computer Vision & Image Understanding, vol. 98, no. 1, pp. 4-24, 2005.