MULTITOUCHLESS INTERFACE
SEMINAR REPORT
Submitted by
VIDHYA VIJAYAN
11413076
To
The University of Kerala
in partial fulfillment of requirements for the award of the Degree
Of
Bachelor of Technology in Computer Science & Engineering
Department of Computer Science & Engineering
Lourdes Matha College of Science & Technology
Thiruvananthapuram - 695 574
November 2014
LOURDES MATHA COLLEGE OF SCIENCE & TECHNOLOGY
KUTTICHAL, THIRUVANANTHAPURAM
CERTIFICATE
This is to certify that the report entitled Multitouchless Interface submitted by
Vidhya Vijayan, S7 CS (11413076) to the University of Kerala in partial fulfillment of the
requirements for the award of the Degree of Bachelor of Technology in Computer Science &
Engineering is a bonafide record of the seminar presented by Vidhya Vijayan.
Seminar Co-ordinator Head of the Department
Mrs. Priya Sekhar S Dr. C.G. Sukumaran Nair
Associate Professor Prof &HOD
Dept of Information Technology ept of CSE
LMCST, Trivandrum LMCST, Trivandrum
ACKNOWLEDGEMENT
I had the opportunity to present a Seminar Paper titled Seminar Title as a part of our
[Link] degree curriculum for the seventh semester Information Technology students of
Lourdes Matha College of Science & Technology, Thiruvananthapuram.
I would like to take this opportunity to express my deep, sincere and profound
gratitude to our respected Principal, [Link] Kattady for providing all the facilities and
moral support to complete my seminar.
I would like to express my sincere thanks to Dr. C.G. Sukumaran Nair, Prof. & Head
of Department of Computer Science & Engineering who made it possible for me to utilize the
facilities offered by the Department of Computer Science & Engineering to the fullest.
Finally yet importantly, I would like to express my gratitude to my seminar co-
ordinator [Link] Sekhar S, Associate Professor, Dept. of Information Technology for her
valuable assistance.
I am highly indebted to my guide Name, Assistant Professor, Dept. of Information
Technology for her excellent guidance and cooperation.
I am also obliged to entire faculty and friends, who paved the ways for the successful
presentation of the seminar.
Above all, I thank Almighty for showering His blessings upon me.
Vidhya Vijayan
ABSTRACT
A proposed multitouchless interface accurately tracks users'
hands and fingers in 3d, using a Microsoft Kinect sensor. The system tracks up to four hands
simultaneously on a standard desktop computer at 30 fps, with approximately 5-mm accuracy. From this
basic tracking, the system identifies gestures and uses them for interaction. The making sense
demonstrator applied this technology to let users quickly summarize and explore complex multimedia
data taken from hard drives. Touchless user interface commanding the computer via body motion and
gestures without physically touching a keyboard, mouse or screen. For example, Microsoft's Kinect is a
touchless game interface. Although not tethered to the computer by a cord, products such as the Wii are
not touchless. They use a wireless controller that is handheld.
INTRODUCTION
Multitouch technology is now commonplace across modern
devices and provides direct interaction with an applications GUI. However, interaction is limited to the
2d plane of the display. The detection and tracking of fingertips in 3d, independent of the display, opens
up multitouch to new application areas, ranging from medical analysis in sterile environments to home
entertainment and gaming. Furthermore, the additional dimension of interaction provides new avenues
for UI design.
The Leap Motion, an infrared stereo sensor tailored for hand
interaction, accurately tracks fingertips, enabling interaction with existing multitouch applications.
However, it has a limited workspace allowing only the hands to be tracked. Alternatively, using a generic
depth sensor allows both the body and hands to be tracked for interaction. Cem Keskin and his colleagues
created a means of full hand pose estimation that can also classify hand shapes. This approach, however,
requires significant quantities of synthetic data. (For more on previous research on hand pose estimation
and interactive displays,
In contrast, this approach detects only fingertips but does so
without machine learning and a large amount of training data. Its real-time methodology uses geodesic
maxima on the hands surface instead of its visual appearance, which is efficient to compute and robust
to both the pose and environment. With this methodology, we created a Multitouchless interface that
allows direct interaction with data through finger tracking and gesture recognition.
2. TRACKING FINGERTIPS
A Microsoft Kinect serves as the sensor. First, we capture the depth
image and calibrate the point cloud. We locate and separate hand blobs with respect to the image domain.
Processing each hand in parallel, we build a weighted graph from the real-world point information for
each hands surface. An efficient shortest-path algorithm traverses the graph to find candidate fingertips.
We then filter these candidates on the basis of their location relative to the center of the hand and the
wrist, and use them in a temporally smoothed model of the fingertip locations.
2.1 HAND AND FOREARM SEGMENTATION
For each color frame from the Kinect, we locate the users face using a standard
face detector. This provides the subjects location for each frame. Using the point cloud derived from
the calibrated depth assume is the user. We smooth the depth over a 20ms window to define the back
plane for segmentation, which lets us remove the body and background from the depth image. The
remaining depth space is the workspace for gesture-based interaction.
We cluster any points in this space into connected blobs. We ignore blobs smaller
than potential hand shapes; the remaining blobs are candidates for the users arms. We then classify the
points in each blob as part of a hand subset or wrist subset. This classification uses a depth threshold of
one-quarter of the arms total depth.
2.2 HAND CENTER LOCALIZATION
The hands center serves as the seed point for distance computation.
Simply using the centroid of points would result in the seed point shifting when the hand is
opened and closed. We use the hands chamfer distance, measured from the external boundary
to find a stable center. (the chamfer distance is a transform that computes the distance between
sets of points. In this context, it computes each points distance from the closest boundary
point.) A hands center is the point with the greatest distance in the chamfer image.
2.3 CANDIDATE FINGERTIP DETECTION
By mapping the hands surface and searching for extremities, we can find
the fingertips, excluding the closed fingers. We conduct this search by mapping the distance
from the hands center to all other points on the hands surface. The geodesic distances with the
greatest local value correspond to geodesic maxima. We search for up to five extrema to account
for all open fingertips.
Practice, however, the wrist forms additional extremities with a similar
geodesic distance. So, we greedily compute the first seven extremities; this accounts for each
fingertip, including two false positives. When the fingers are closed, the tips arent of interest
because the fingers contribute to the fists formation. The extremity normally associated with a
folded finger forms an additional false positive that we filter out at a later stage.
To find the candidate fingertips, we first build a weighted undirected graph
of the hand. Each point in the hand represents a vertex in the graph. We connect these vertices
with neighbouring vertices in an 8-neighborhood fashion, deriving the edge cost from the
euclidean distance between their world coordinates.
(
a
)
(
b
)
(
c
)
(
d
)
(
e
)
(
e 1
Figure 2. Multiple hand shapes and their first seven extremities. (a) An open palm with two extrema belonging to the wrist.
(
b) Pinching. (c) Open pinching. (d) A pointing finger, with one other extremum approaching
the tip. (e) Self - occlusion. ( e 1 )
The extrema
path
2.4 NONFINGERTIP REJECTION
Next, we filter the candidate fingertips points to a subset of valid
fingertips. We combine a penalty metric derived from the path taken during dijkstras
algorithm with a candidates position relative to the hands center.
Finding the penalty for each candidate requires the covariance of
the hand and arm points, translated to the wrist location. (the covariance models the variation
in pixels for the chosen body part.) Using the covariance to map the wrist in this manner takes
into consideration the variability of hand shapes. We translate the covariance to form an
elliptical mask centered around the wrist. If a pixel has a Mahalanobis distance within three
standard deviations of the wrist, we mark it as 1; otherwise, we mark it as 0
Figure 3. The covariance of several hand shapes translated to the wrist.
(a) An image of an open palm, showing a long principle axis for a full
hand and arm. (b) An image of a fist, revealing a circular boundary.
(c) An image of two extended fingers, demonstrating wrist rotation.
2.5 FINGER ASSIGNMENT AND TRACKING
To track and assign the fingertips between frames, we use a Kalman filter.
(a kalman filter takes a series of noisy measurements observed over time and combines these
estimates to give a more accurate prediction. It often includes assumptions about the type of
motion and noise that might be observed, which it uses to smooth the predictions.) We update
this model using the point correspondence that minimizes the change between consecutive
frames. As tracking is in 3d, we must check all possible permutations when matching points.
When assigning five or fewer points, searching all permutations requires
fewer operations than using the hungarian algorithm (an optimization algorithm that efficiently
solves the assignment problem). The hand has five fingers, resulting in only 120 permutations
in the worst case.
We use each detected fingertips world position to update a bank of 3d
kalman filters that persist from one frame to the next. To update this model, we pair detected
extrema with the kalman filter predictions by selecting the lowest-cost assignment between
pairs.
We use the smoothed model to output each fingertip as a 3D coordinate in
millimeters. For visualization, we can project these coordinates back to the image domain (see
Figure 5).
Its more accurate to track the fingertips in 3D than in the image domain because the additional
depth information and filter estimation provide subpixel accuracy. If we tracked the points in
the image plane, the resulting fingertips would be prone to jitter, owing to the quantization of
the image.
2.6 TRACKING VALIDATION
Figure 5. Multiple fingertips and the hands center tracked over 10 frames. Each lines length
demonstrates the distance covered In the previous 10 frames.
Our tests, written in C++, were performed on a standard desktop
pc with a 3.4-ghz Intel core i7 CPU. To validate tracking performance, we used sequences
captured from a range of users. We labelled these videos with approximately 15,000 ground
truth markers, against which we measured the tracked fingers.
Over all the sequences, the average error was 2.48 cm. However,
figure 6 shows that most fingertips were accurate to within 5 mm. Our approach operated at
30 fps for each hand, with up to four hands processing in parallel.
3. RECOGNIZING GESTURES
Gestures for interaction typically are common across multitouch
devices and provide familiarity that lets users easily perform complex tasks. However, 3d
interaction differs fundamentally, and 2 d gestures dont necessarily translate well to 3d.
The development of gestures requires special consideration
because theyre integral to the user experience. They should be simple and semantically similar
to the task they represent; a good example would be using a pinching gesture to zoom.
However, for large-scale displays, a single-handed, two-finger pinch zoom does not scale well
to the visualizations size; a two-handed, single-finger zoom is more appropriate.
To provide flexible multitouchless gesture recognition, we place
middleware between the finger-tracking and application levels. We chose the gesture
vocabulary to generalize across applications. To minimize the complexity new users
experience when the interaction modality changes, we adapted the most common 2d gestures.
We can extend the system to recognize additional types and
combinations of gestures. However, more complex gestures involving transitionsfor
example, a horizontal swipe followed by a downward swipecould be better represented
using a probabilistic approach such as hidden Markov models.
3.1 BASIC GESTURES
From hand and finger tracking, we know the presence and
position of the users head, hand, and fingers (see figure 7a); additional skeletal information is
available from the kinect. We use a representation of a hands velocity to detect a zoom gesture
(see figure 7b) and horizontal and vertical swipes (see figure 7c), single- or double-handed. As
we know the number of fingers visible on each hand, our system can recognize these gestures
on the basis of specific finger configurations. For example, at the application level, users
employ one finger per hand for the zoom gesture, but the system detects swipes regardless of
the number of fingers.
Users can also employ an extended thumb and index finger
to control a cursor. Then, to select an object placed under the index finger, the user retracts the
thumb to lie alongside the palm. The system also easily detects a grasping gesture as the rapid
transition from five to zero fingers. Users can employ this gesture with an open hand to open
a menu or with the index finger to select an item. They can rotate a hand to control a circular
context menu.
3.2 LIMITATIONS
Because the Kinect sensor is designed for full body pose
estimation, it has inherent drawbacks for our approach. For example, it has difficulty mapping the
hands surface when fingers are directed toward it. This forms a hole in the depth information that
our approach does not account for. So, we avoid gestures in which users point at an onscreen object.
In addition, we parametrically tuned the distance threshold
described in the section Nonfingertip rejection to be invariant to adult users. In the event of a child
using the system, this might lead to failure owing to the smaller hand.
It uses limited number of gestures. Need to improve the
system in several ways. It is better if we use the Kinects skeletal tracker to establish more accurate
hand and forearm segmentation instead of relying on face detection.
More complex hand shapes, moving toward full hand pose
estimation are not possible in this interface. Investigation on this problem is going on now. Wider
range of detected hand shapes would also allow recognition of more gestures, extending the systems
capabilities.
4. APPLICATIONS
Primary use of multitouchless interaction has been in the
Making Sense project (www. [Link]), which employs data-mining and machine
learning tools to allow the discovery of patterns that relate images, video, and text. It
combines multitouchless interaction and visualization to let analysts quickly
visualize data,
summarize the data or subsets of the data,
discriminatively mine rules that separate one dataset from another,
find commonality between data,
project data into a visualization space that shows the semantic similarity of the different items,
and
group and categorize disparate data types.
CONCLUSION
Multitouchless interface that allows direct interaction
with data through finger tracking and gesture recognition. Extending interaction beyond
surfaces within the operators physical reach provides greater scope for interaction. We can
easily extend the users working space to walls or the entire environment. Our approach also
achieves real-time tracking of up to four hands on a standard desktop computer. This opens up
possibilities for collaboration, especially because the workspace isnt limited by the displays
size. Primary use of multitouchless interaction has been in the Making Sense project
They plan to improve the system in several ways. Instead
of relying on face detection, well use the Kinects skeletal tracker to establish more accurate
hand and forearm segmentation. Well also incorporate a probabilistic framework in which we
can combine several hypothesis of fingertips, improving detection and accuracy.
Another area of investigation is the classification of
more complex hand shapes, moving toward full hand pose estimation. A wider range of
detected hand shapes would also allow recognition of more gestures, extending the systems
capabilities.
REFERENCES
1 F. Weichert et al., Analysis of the Accuracy and Robustness of the Leap Motion Controller, Sensors,
vol. 13, no. 5, 2013, pp. 63806393.
2 C. Keskin et al., Real Time Hand Pose Estimation Using Depth Sensors, Proc. 2011 IEEE Computer
Vision Workshops, 2011, pp. 12281234.
3 P. Krejov and R. Bowden, Multi-touchless: RealTime Fingertip Detection and Tracking
Using Geodesic Maxima, Proc. IEEE 10th Intl Conf. and Workshops Automatic Face and
Gesture Recognition, 2013, pp.