Human Action Recognition in Surveillance
Human Action Recognition in Surveillance
Abstract: Abstract: Human action recognition is an important area of research in the field of computer vision due to its extensive
applications like security surveillance; content based video retrieval and annotation, human computer interaction, human fall detection,
video summarization, robotics, etc. The surveillance system deals with the monitoring and analysing the human behaviour and activities.
The main aim of the smart surveillance system is to recognize anomalous behaviour in given scene and provide real time intimation to
relevant person. We have designed and tested Smart Surveillance System for College Corridor Scene (3S2CS). The system recognises the
anomalous behaviour and an intimation is provided in the form of Firebase Cloud Messaging (FCM) alert on the android mobile phone to
the authorised user. This paper mainly discuss the methodologies used for the human action recognition. The basic step is to provide video
as an input. These videos are further divided into number of frames. The videos are used for training and for each video, Scale Invariant
Feature Transform (SIFT) is applied for extracting features and developing feature vectors. The actions are classified using K Nearest
Neighbour (KNN) and Support Vector Machine (SVM) classifier. Two standard offline datasets considered for testing are Weizmann and
UTD-MHAD. For real time scenario we have created dataset in college campus called as College Corridor dataset. It contains student
activities like falling, fighting, walking, running, sitting and other general actions. If falling or fighting action is detected, the notification
is sent to the authorized user who has installed ActionDetector android application and registered a device for the same. Action recognition
accuracy is 92.91% using SVM and 90.83% using KNN.
Keywords: Computer Vision, SIFT, SVM, KNN, Human Action Recognition, FCM Alert
1
human activity are extracted and the clusters are obtained using
Assistant Professor at PICT, Pune, India
2 density based clustering method [5] [6]. Many times feature
Student of PICT, Pune, India
3
Student of PICT, Pune, India extraction methods are used to describe the human actions. The
4
Student of PICT, Pune, India features are usually synthesized into descriptors. Shape contexts
5
Student of PICT, Pune, India and SIFT descriptors are most popular descriptors in this area [7].
* Corresponding Author Email: gvkale@[Link]
International Journal of Intelligent Systems and Applications in Engineering IJISAE, 2019, 7(1), 60–65 | 7
Generally, these descriptors encode the body looks and some notification and registration token which is required by FCM.
context of motion. Lowe and David G categorizes human motion Service. Apache tomcat is an open source web server that
using hybrid of spatial-temporal and static features [8]. The system implements Java Server Pages (JSP) specifications. We are using
designed by Niebles [Link] classifies the activities and detects FCM for sending notification to the registered device. FCM is a
abnormality using approach that is introduced for multi-class SVM cross-platform solution provided by Google for messages and
classification, which checks for the states that are unreachable notifications for ios, web applications and android. A message can
from the current state and avoids them [10]. The system is designed transfer a payload of up to 4KB to a client application using FCM
by Iqbal et. al to detect old people fall detection by extracting for an instant messaging. This service is used to inform the client
features using SIFT and then making comparisons using Hidden about suspicious action is detected we are using with the
Morkov Model (HMM) [11]. The nearest neighbour (NN) authentication of the user. Currently surveillance system and the
classifiers, is among the simplest and most efficient classification authorised user are assumed in same wireless network at college
rules and are widely used in practice KNN mainly used for campus. However, android application can be extended to use from
statistical pattern recognition and the supervised neural network anywhere using internet.
learning points of view [12]. The KNN classification algorithm is
3.1. Scale Invariant Feature Transform (SIFT):
applied for handwritten digit recognition efficiently [13]. The
different complexity of action. Hierarchical approaches have SIFT is used for feature extraction which extracts key points and
shown great success for the recognition of complicated actions and compute its descriptors. SHIT features are invariant to direction,
interactions. The techniques like bag-of-words, and HMM that scaling, and also partly invariant to lighting variations and affine
have shown success in speech and text recognition are successfully distortion. This helps to accurately recognize objects with noise
applied for action recognition [14]. Three main technologies used and fractional occlusion. It has good recall rates, included in
for capturing videos are RGB camera, depth sensor and wearable OpenCV library. It is -relatively efficient as compared to older
device. It is found that the popularity of RGB camera in HAR algorithms. The features are extracted for the trained videos and
research has dropped while both depth and wearable sensors are the feature vectors are developed. Test videos are given as input to
the substitutes. On the other hand, the use of Kinect sensor (depth system and SIFT features are extracted. These features further
sensor) into HAR system is promising and somewhat expensive. given as input for action recognition system. SIFT finds out key
Some challenges involved in human activity recognition are points from images and extract the descriptor for each keypoint.
application domain, occlusion, background and recording settings, 128-bit descriptor is stored for each key point.
learning paradigm usage, variations in Inter and Intra class, etc. There are mainly four steps involved in SIFT algorithm [17]
[15]. Chaquet et. al have given comparison between the different
datasets and for the sake of clarity, the main characteristics of the I. Find Scale Space Extrema – An internal representation of
28 public video datasets for human action and activity recognition original image.
described [16]. We have considered the Weizmann and UTD-
II. Keypoint Localization and Filtering – It selects significant
MHAD standard datasets before going for the real time
key points removes least significant points.
applications, and created College Corridor dataset in college
III. Orientation Assignment – It removes effects of rotation and
campus. The suspicious activities are notified to the end user on
scaling.
smart phone application the system with real time application of
IV. Descriptor Creation – It is done using histograms of
College Corridor scenario.
orientations.
3. Smart Surveillance System for College Corridor
Scene (3S2CS) 3.2. Action Classification Method
S3C2S system mainly consists of three different layers that are Action classification and prediction is important steps in human
client layer, application layer and database server layer. The client motion data analysis. Researcher for human motion classification
layer includes the user android application to receive notifications. successfully applies the different classifiers. This work uses KNN
Only authorised users can receive alert about anomalous action and SVM classifier applied for classification activity video in
detection. different action classes. The classifier is built from the training set
Application layer consists of main human action Recognition associated with class labels. The training dataset is vectors in a
system. multidimensional feature space, each with a class label.
At database server layer, there is Google server and apache tomcat 3.2.1. K-Nearest Neighbor Classifier
local server to handle the services of human action recognition As its name suggest the KNN algorithm finds the nearest matching
system. For College Corridor scenario, dataset is created at PICT
from training dataset. The distance is calculated between the new
corridor with six action classes fighting, falling, walking, running,
feature vector and every vector of the training set. Euclidian
sitting and general action class. If actions like fighting or falling
distance measure is used for this purpose. Simple majority vote of
are detected by the system then, the notification is sent to the user the nearest neighbors of every point is computed using KNN
who has installed the android application and registered for the classifier. Data class label of the most representative class is
same. assigned to the key points extracted from query (test) video. It
The system takes video as an input, performing grayscale
implements learning based on the k nearest neighbours of each
conversion and pre-processing steps, feature extraction is done
query point where k is integer value given by user. The basic step
using SIFT. These extracted features are used for action
used for nearest neighbour’s classification is uniform weights, that
classification purpose. SVM and KNN classifiers are used for is, the value assigned to a query point is computed from a simple
activity classification. The system results show the comparison majority vote of the nearest neighbours. Sometimes, it is better to
between SVM and KNN algorithms used for dataset classification. weight the neighbours such as nearer neighbours contribute more
Apache tomcat server is installed for maintaining the database of
to the fit. This can be accomplished by specifying weight values.
International Journal of Intelligent Systems and Applications in Engineering IJISAE, 2019, 7(1), 60–65 | 61
The ‘uniform’ value of weight assigns uniform weights to each
neighbour.
Figure 1. Architecture diagram of smart surveillance system for College Corridor Scenario (S3C2S)
International Journal of Intelligent Systems and Applications in Engineering IJISAE, 2019, 7(1), 60–65 | 62
Table 2. Results of 3S2CS system on Weizmann, UTD-MHAD and
College Corridor Dataset
(a): Confusion Matrix for results using SVM on Weizmann Dataset
Predicted Classes
Skip
Bend
Run
Wave2
Jack
Side
Jump
Wave1
Walk
Jump
Skip 8 0 0 0 0 0 0 0 0 0
Bend 0 9 0 0 0 0 0 0 0 0
Run 0 0 9 0 0 0 0 0 0 0
Actual Classes
(c) College Corridor Scenario Wave2 0 0 0 9 0 0 0 0 0 0
Figure 2. Sample frames from (a) (b, (c) dataset Jack 0 0 0 0 9 0 0 0 0 0
Side 0 0 0 0 0 9 0 0 0 0
Institute of Science (Faculty of Mathematics and Computer
Jump 0 0 0 0 0 0 9 0 0 0
Science, Israel) records it in 2005. Database of 90 low-resolution
Wave1 0 0 0 0 0 0 0 9 0 0
(180 x 144, deinterlaced 50 fps) video sequences collected on a
Walk 0 0 0 0 0 0 0 0 9 0
nine different persons. Ten different actions are walking, running,
jumping, siding, skipping, bending, one hand waving, two hand Jump 0 0 0 0 0 0 0 0 0 9
waving, jumping in place and jumping jack. It contains both (b). Confusion Matrix for results using KNN on Weizmann Dataset
periodic and non-periodic actions.
Predicted Classes
Table 1. Summarization of Activity Datasets
Skip
Bend
Run
Wave2
Jack
Side
Jump
Wave1
Walk
Jump
Dataset Actors Repetition Actions Sequences
Weizmann 9 1 10 90 Skip 7 0 0 0 0 0 0 0 0 0
UTD-MHAD 8 4 27 192 Bend 0 9 0 0 0 0 0 0 0 0
College Corridor Dataset 4 10 6 240 Run 0 0 9 0 0 0 0 0 0 0
Actual Classes
Wave2 0 0 0 9 0 0 0 0 0 0
3.3.2. UTD-MHAD Dataset Jack 0 0 0 0 9 0 0 0 0 0
This dataset was collected using a Microsoft Kinect sensor and a Side 0 0 0 0 0 9 0 0 0 0
wearable inertial sensor in an indoor environment. The dataset Jump 0 0 0 0 0 0 9 0 0 0
contains 27 actions as jogging, squat, boxing, clapping, walking, Wave1 0 0 0 0 0 0 0 9 0 0
pushing, swipe left, swipe right, sit to stand, jogging in place, right Walk 0 0 0 0 0 0 0 0 9 0
hand waving, etc. performed by 8 subjects. Each subject has Jump 0 0 0 0 0 0 0 0 0 9
repeated every action 4 times. The dataset is published by
department of electrical engineering, university of Texas at Dallas (c). Confusion Matrix for results using SVM on UTD-MHAD Dataset
for research purpose in 2015. We have considered 6 different Predicted Classes
actions jogging, boxing, clapping, squat, walking, pushing by 4
Boxing
Clapping
Jogging
Pushing
Squat
Walking
Clapping
Jogging
Pushing
Squat
Walking
4. Results
Weizmann, UTD-MHAD and College Corridor datasets are Boxing 29 2 0 0 0 0
Actual Classes
tested using KNN and SVM classifiers and their results are
Clapping 0 30 0 0 0 0
analyzed and discussed in this section.
Jogging 0 0 31 0 1 1
Pushing 2 0 0 32 0 0
Squat 0 0 0 0 31 0
Walking 0 0 1 0 0 29
International Journal of Intelligent Systems and Applications in Engineering IJISAE, 2019, 7(1), 60–65 | 63
Table 2 (e ) Confusion Matrix for results using SVM on College Corridor using SIFT as it is scale invariant algorithm and it can accurately
Dataset represents object even along with noise and fractional occlusion.
Predicted Classes Then the extracted features are given to the SVM and KNN
algorithms for action classification. Training and testing is
Running
Walking
Fighting
General
Falling performed on three different datasets. The results are collected for
Sitting
these two classifiers on three datasets. The SVM with RBF (radial
basis function) kernel gives better results as compared to the KNN
Falling 36 2 0 2 0 1
classification algorithm. We have obtained good prediction
Actual Classes
General
Falling
Acknowledgment
Sitting
Table 2 shows the performance of both classifiers on three [3] Tran, D., & Sorokin, A. (2008, October). Human activity recognition
mentioned datasets. All 90 sequences given in Weizmann dataset with metric learning. In European conference on computer
vision (pp. 548-561). Springer, Berlin, Heidelberg.
are considered to test the algorithms. 10 actions X 9 persons = 90
sequences. Recognition results by SVM and KNN classifiers are [4] Arikan, O., Forsyth, D. A., & O'Brien, J. F. (2003, July). Motion
synthesis from annotations. In ACM Transactions on Graphics
shown in table 2(a) and table 2(b). Action recognition accuracy is
(TOG) (Vol. 22, No. 3, pp. 402-408). ACM.
98.88% using SVM and 97.78% using KNN. Even though UTD-
MHAD has 27 actions, we have considered selected six actions to [5] Zhang, H., & Yoshie, O. (2012, July). Improving human activity
recognition using subspace clustering. In 2012 International
test 3S2CS system. Considered six actions are – boxing, clapping, Conference on Machine Learning and Cybernetics (Vol. 3, pp.
jogging, pushing, squat, walking. 6 actions X (4 male + 4 female) 1058-1063). IEEE.
X 4 repetitions = 192 action sequences are tested. Recognition
[6] Kailing, K., Kriegel, H. P., & Kröger, P. (2004, April). Density-
results by SVM and KNN classifiers are shown in table 2(c) and connected subspace clustering for high-dimensional data.
table 2(d). Action recognition accuracy is 96.35% using SVM and In Proceedings of the 2004 SIAM international conference on data
93.23% using KNN. College Corridor Scenario dataset has 6 mining (pp. 246-256). Society for Industrial and Applied
classes X 10 actors X 4 repetitions = 240 sequences and data is Mathematics.
tested real time. Recognition results by SVM and KNN classifiers [7] Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and
are shown in table 2(e) and table 2(f). Action recognition accuracy object recognition using shape contexts. IEEE Transactions on
Pattern Analysis & Machine Intelligence, (4), 509-522.
is 92.91% using SVM and 90.83% using KNN. From recognition
results, it is observed that SVM outperform over the KNN for all [8] Lowe, D. G. (2004). Distinctive image features from scale-invariant
three datasets. keypoints. International journal of computer vision, 60(2), 91-110.
It has also been observed that overall recognition rate for our [9] Niebles, J. C. (2007, June). A hierarchical model of shape and
dataset is less as compared to other two standard datasets. Main appearance for human action classification. In 2007 IEEE
Conference on Computer Vision and Pattern Recognition (pp. 1-8).
reason behind this is involvement of more than one actor in dataset.
IEEE.
Scenario like fighting always involve more than one person and
dataset involving multiple persons is more realistic. Also, slow [10] Palaniappan, A., Bhargavi, R., & Vaidehi, V. (2012, April).
Abnormal human activity recognition using SVM based approach.
running is miss-classified as walking and fast walking is miss- In 2012 International Conference on Recent Trends in Information
classified as running. Classifier is confused between falling down Technology (pp. 97-102). IEEE.
and sitting down for few cases.
[11] Iqbal, J. M., Lavanya, J., & Arun, S. (2015). Abnormal Human
Activity Recognition using Scale Invariant Feature
5. Conclusion Transform. International Journal of Current Engineering and
Technology, 5(6), 3748-3751.
The system gives overview of human action classification using
[12] Laaksonen, J., & Oja, E. (1996, June). Classification with learning
supervised learning algorithm. The paper mainly discuss computer k-nearest neighbors. In Proceedings of International Conference on
vision algorithms SIFT, KNN and SVM. Features are extracted Neural Networks (ICNN'96) (Vol. 3, pp. 1480-1483). IEEE.
International Journal of Intelligent Systems and Applications in Engineering IJISAE, 2019, 7(1), 60–65 | 64
[13] Ravi Babu, U., & Venkateswarlu, Y. (2014). Aneel Kumar
Chintha,“Handwritten Digit Recgnition using K-Nearest Neighbout
Classifier”, IEEE. In World Congress on Computing and
Communication Technologies (pp. 60-65).
[14] Ann, O. C., & Theng, L. B. (2014, November). Human activity
recognition: a review. In 2014 IEEE International Conference on
Control System, Computing and Engineering (ICCSCE 2014) (pp.
389-393). IEEE.
[15] Subetha, T., & Chitrakala, S. (2016, February). A survey on human
activity recognition from videos. In 2016 International Conference
on Information Communication and Embedded Systems
(ICICES) (pp. 1-7). IEEE.
[16] Chaquet, J. M., Carmona, E. J., & Fernández-Caballero, A. (2013).
A survey of video datasets for human action and activity
recognition. Computer Vision and Image Understanding, 117(6),
633-659.
[17] Hsu, C. W., & Lin, C. J. (2002). A comparison of methods for
multiclass support vector machines. IEEE transactions on Neural
Networks, 13(2), 415-425.
[18] Chang, C. C., & Lin, C. J. (2011). LIBSVM: a library for support
vector machines. ACM transactions on intelligent systems and
technology (TIST), 2(3), 27.
[19] Chen, C., Jafari, R., & Kehtarnavaz, N. (2015, September). UTD-
MHAD: A multimodal dataset for human action recognition
utilizing a depth camera and a wearable inertial sensor. In 2015
IEEE International conference on image processing (ICIP) (pp.
168-172). IEEE.
[20] S. Tasdemir, Artificial Neural Network Model For Prediction Of
Tool Tip Temperature And Analysis, International Journal of
Intelligent Systems and Applications in Engineering, 2018, 6(1): 92-
96.
[21] C. Aci, C. Ozden, Prediction The Severity Of Motor Vehicle
Accident Injuries In Adana-Turkey Using Machine Learning
Methods And Detailed Meteorological Data, International Journal of
Intelligent Systems and Applications in Engineering, 2018, 6(1): 72-
79.
[22] F. Khademi, Predicting strength of recycled aggregate concrete
using artificial neural network, adaptive neuro-fuzzy inference
system and multiple linear regression, Int. J. Sustain. Built Environ.
5 (2), (2016) 355–369, doi: 10.1016/[Link].2016.09.003.
[23] F. Khademi, M. Akbari, S.M. Jamal, Prediction of compressive
strength of concrete by data-driven models, i-manager’s J. Civ. Eng.
5 (2), (2015), 16.
[24] M. Nehdi, M. Bassuoni, Fuzzy logic approach for estimating
durability of concrete, Proc. Ins. Civ. Eng.-Constr. Mater. 162 (2)
(2009) 81–92, doi: 10.1680/coma.2009.162.2.81.
[25] B. Vakhshouri, S. Nejadi, Predicition of compressive strength in
light-weight self-compacting concrete by ANFIS analytical model,
Arch. Civ. Eng. 61 (2), (2015) 53–72, doi:10.1515/ace-2015-0014.
[26] I. Mansouri, Predicting behavior of FRP-confined concrete using
neuro fuzzy, neural network, multivariate adaptive regression
splines and M5 model tree techniques, Mater. Struct. 49 (10), (2016)
4319–4334, doi: 10.1617/s11527-015-0790-4.
[27] A. Sadrmomtazi, J. Sobhani, M. Mirgozar, Modeling compressive
strength of EPS lightweight concrete using regression, neural
network and ANFIS, Constr. Build. Mater. 42, (2013), 205–216, doi:
10.1016/[Link].2013.01.016.
[28] EN 197-1. Cement - Part 1: Composition, specifications and
conformity criteria for common cements, 2011.
International Journal of Intelligent Systems and Applications in Engineering IJISAE, 2019, 7(1), 60–65 | 65