Multimedia Systems (2022) 28:2319–2334

[Link]

REGULAR PAPER

Map modeling for full body gesture using flex sensor and machine
learning algorithms
Arash Hekmat1 · Zhang Zuping1 · Hashim Saleh Sharaf Al‑deen1

Received: 2 December 2021 / Accepted: 18 April 2022 / Published online: 13 June 2022
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2022

Abstract
This study investigates how we can provide a database map model for full-body gestures by utilizing the hierarchical nested
set model and its features to cover all aspects of existing gestures and motions. A mapping model allows us to execute any
appropriate gesture pattern for each motion state via a hierarchically nested set tracking feature that executes at a varied
speed or time. The nested set model allows us to distinguish each node location with its right and left values, which aids
us in controlling the power to motors at a certain time and speed while taking data quantity into account by eliminating
the time and interaction data for speed and motor management. The main issue in this study is that instead of reprocessing
data for each movement change, machine learning methods are used to create a map model from classified data. This paper
discusses the connection between sensors and databases for exchanging data models in the form of a map that may be used
to interact between different positions of the robot's parts based on sensor data. For example, suppose a robot falls down
and the sensors such as the gyroscope, accelerometers, touch sensor, camera (image processing), or voice recognition are
set as an input command to understand the current position. Then, using artificial modeling in our database, we can control
the robot to return to the standard position, such as standing up on its legs. We used motion flex sensor gloves to record all
gestures at varied motion speeds and execution durations, and then we ran three different classification algorithms on the
recorded data to achieve the best data categorization. Finally, based on a nested set model for the whole body, we provided a
database map with those classified data gathered from the sensor, and as a consequence, we made a comparison with parent
and child categorization to highlight the complexity and data collection differences between these two techniques.

Keywords Body gesture recognition · Gesture pattern · Map model · Hierarchical nested set

1 Introduction satisfaction. These issues can be overcome by the utiliza-

tion of a set of gestures that are used to grasp as natural,
Human–Computer interaction (HCI) mainly illustrates that is immediate association of shape of the body parts
the interaction between humans and computers in novel configuration or movement [2]. However, the cost of artifi-
ways with less complexity. Improvised interaction between cially extracting large-scale gesture samples is prohibitive,
humans and machines is unavoidable these days. As user limiting the use of gesture recognition [7, 8]. Furthermore,
interfaces are managed, one can see how electronic devices improvements in the user interface (UI), such as regulating
are becoming more intelligent. Any gesture-based HCI [1] the volume of a car audio system or playing games with
can be successful and efficient if human users are used. Kinect, or simulating animated characters with motion cap-
Complex or unnatural movements to use the interface can ture cameras, all rely on body motions and movements being
be learned using conventional input, ensuring accuracy and captured by various sensors. It wasn't just at our fingertips
anymore, and it wasn't just communicating in a limited way.
It was all over us, with several small nano radar chips that
Communicated by Y. Kong.
could detect micro motions on our fingers.
* Zhang Zuping Body gesture recognition, specifically hand gesture rec-
zpzhang@[Link] ognition, is a method for describing these movements as a
mathematical dataset that can be accessed by computers.
1
School of Computer Science and Engineering, Central South One purpose of this research is to make human–computer
University, Changsha, Hunan, China

13
Vol.:(0123456789)

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2320 A. Hekmat et al.

interaction easier, while another is to gather and use data [22, 30, 43–48]. Then the result would be declared on that
for controlling humanoid robots such as Boston Dynam- selection of thresholds. Literature [24] relies on a kind
ics, as well as graphical animations and virtual reality (VR) of a recurrent neural network (RNN) [23] known as long
objects. Sensor-Based Recognition (SBR) algorithms and short-term memory (LSTM) architecture which is specifi-
vision-based recognition (VBR) algorithms are the two types cally designed to address the vanishing gradient problem
of gesture recognition algorithms [6, 10, 12]. and also uses this network with one input layer, one hidden
The VBR algorithms can be used to capture gestures from layer, and one output layer [49–54]. The interconnected
images to perform the recognition process. However, the hidden layer consists of memory cells and respective gate
extraction of information and data classification is effec- units. More than the LSTM, the flex sensor spots can be
tive, but it undergoes high computation complexity while effectively adapted by the PairNet algorithm [26]. This is
performing both training and inference operations. The SBR a type of unique 1D convolution neural network [23], in
approach depends on sensing cameras and commonly used which the convolution layers with stride size 2 and kernel
sensors for this approach. Accelerometers[21], gyroscopes size 1 × 2 were adopted. Further, the PairNet algorithm
[16], flex sensors or electromyography (EMG) [18], or ultra- can be used to enhance the field acceptance with lower
sonic distance sensors [13, 14] are some examples of the computational complexities while performing the gesture
sensors for this field of study. High dataset classification spotting than the traditional one.
filters have been established in several of these types of sen- The state-of-art gesture techniques are mainly based
sors. However, many of these approaches do not recognize on raw data retrieval from sensors and classifying them
the priority of each sequence read from the sensor. Only data using machine learning algorithms. The main principle of
accepted are passed from filtering algorithms. these techniques is to operate the sensors using a machine
Robotics and AI now focus on the modeling of the per- learning algorithm. To overcome this drawback, the raw
sonality of humans. computer vision with AI provides sim- data from the sensor is saved to the database, and based on
ple means to capture the data of a person. Dynamics and these motions using the artificial neural network (ANN),
kinematics of robotics offer in-depth analysis of human the gesture of the robot is identified. Further, the contribu-
movements including biomechanical information. Normally tions of this paper are described below:
a Skeleton-based dynamic hand gesture recognition tech-
nique recognizes the gestures by the data gathered from the • The state-of-art systems mainly detected the objects
flex sensor [19]. The details are then saved in the database via region-based classification, localization, and slid-
and used when needed. Every movement along with the ing windows which assigns the region of interest with
speed that needs to be applied to the motors can be com- a high probability value and this process is also very
puted. Figure 1 shows the hand as a sample of body move- time consuming and computationally expensive. To
ment joints data. overcome this drawback, a map model is created using
One of the methods that are used for gesture recog- a neural network algorithm to identify the normal and
nition relies on sensory data variances [3, 4]. Variances abnormal behavior of the user with high accuracy.
with values below the maximum are considered raw data • The movements of the body parts are recorded and
converted into a machine-readable format. This step is
crucial in identifying the actor's movements.
• A nested set model is formed which is a graph-based
pattern to mimic the right and left sides of the actor's
body movement. The nested set model tracks the
object's current position and can also control the mod-
el's gesture. For example, if a robot falls to the ground,
voice commands can be used to return it to its former
location. In this way, the interaction between the data-
base and motors is reduced thus reducing the compu-
tational complexity.
• A real-time robotic hand model is deployed in this work
which activates as per the voice command provided to it.

The rest of this paper is arranged accordingly. Section 2

Fig. 1  Expansion of the whole body as a single node for each joint
presents the related work regarding the works conducted
or muscles state and movement with different values and modes for in a similar domain. Section 3 provides details about the
describing the data mapping model working mechanism of the proposed methodology in-depth

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Map modeling for full body gesture using flex sensor and machine learning algorithms 2321

and Sect. 4 provides the different experiments conducted. correction output code support vector machines (ECOC-
Section 5 concludes the paper. SVM) and K-nearest neighbour (KNN) classifiers. Ehrnsper-
ger et al. [42] utilized the naïve gesture recognition technol-
ogy to identify different gestures such as foot kick, hand
2 Review of related works swiping, etc. To execute the gesture recognition algorithms
in a low-cost microcontroller unit, they are providing a new
This research area is divided into two categories: data col- design approach for scalable neural networks.
lection from motion capture using specific cameras [6] based
on graphical images and video recording, and data collection 2.2 Vision‑based adaptive hand gesture recognition
from Skeleton-Based activity from special sensors such as system
flex sensors or other sensors [5].
Damdoo et al. [29] proposed a real-time vision-based
2.1 Recognition of hand gesture using machine approach for recognizing hand gestures based on machine
learning algorithms and flex sensor learning using MATLAB. They proposed an image capture
model with eight steps: color conversion process, back-
Syed et al. [5] presented a control-centric technique for ground separation processing, noise reduction, applying
improved dynamic goniometry and limited control of a bit/binary threshold value, frame accumulation and process-
bionic hand using flex sensor based on Hammerstein–Wie- ing, frame processing, and classification process. Here in
ner model on their structure of hand's gesture, which consists this step frames are continuously fed to a Recursive Neural
of a linear dynamic system and two static nonlinearities. Network (RNN) for the classification process and the ges-
The imposed method is compared with the raw data that is ture is analyzed. Here RNN is used which consists of basic
calibrated from the finger, as well as the associated temporal three tiers. Each tier in this network consists of 200 neurons
and spatial gesture data from both the bionic hand and the node. Reale et al.[36] developed a vision-based HCI model
glove, in terms of the integral of exact errors and misunder- to monitor the eye gaze, mouth motion, and hand pointing
standings between the bionic hand and flex sensor. For this, via different control components. A two-camera system is
they utilized an initial inverse dynamic design strategy via implemented which detects the face and eye region. The
the Wiener model. gaze pointing and the hand finger-pointing are the main con-
Wei-Chieh et al. [20] provided a method for implement- cepts formulated in this work.
ing a continuous finger motion detection system. In this sys- Zhu et al. [37] analyzed the similarity between the hand
tem, flex sensors are utilized to precisely recognize a series gestures via the Kinect sensors and they segmented the 3D
of activities as well as to develop a wireless smart glove hand shapes from the occupied backgrounds. They obtain the
using flex sensors. The experiments were carried out for 3D point information from multiple scales since the global
both training and testing sets. The authors described that and local shape distribution is necessary. A dynamic time
the method effectively recognizes the sequence of finger warping algorithm is used for hand gesture recognition and
gestures even for more complicated transitions. Yen-Cheng the experiments are conducted on different benchmark data-
et al. [25] stated a novel neural network (NN) algorithms sets. Chen et al. [38] developed a hand gesture recognition
based systems for hand gesture recognition. The data from system that is insensitive to varied backgrounds and environ-
the sensory accelerometers and gyroscopes were classified mental illumination. This system is mainly developed using
by using the adopted algorithm. This also utilizes the Pair the biologically inspired neuromorphic vision sensor which
Net-based CNN for handling pairing operations. The authors outputs a sequence of asynchronous events. The visual data
described that the work along with Pair Net achieved better is analyzed via the wearable glove. The active LED makers
classification results than the traditional CNN and Recurrent are used to visualize the palm and fingers. The restricted spa-
NN (RNN) methods. tiotemporal particle filter algorithm is used to track the hand
Jia et al. [40] integrated the supervised and unsupervised movements precisely in the temporal domain. Zhang et al.
techniques for classifying 10 different electromyograms [39] used short-term sampling neural networks for hand ges-
(EMG) signals based on hand gesture classes. For the clas- ture recognition. Every video input is segmented into frame
sification process, a modified fuzzy c-means (FCM) clus- groups based on a predetermined value to learn the short-
tering algorithm is utilized. Wong et al. [41] introduced a term features. Both the short-term and long-term features
low-cost capacitive sensor device to identify hand gestures. are fed as an input to the Convolutional Neural Network
The wearable capacitive sensor prototype designed helps to (ConvNet). However, these techniques mainly rely on large
retrieve the capacitance values from the electrodes placed computational resources and a large volume of data to train
on the fingers. To gain more insights into the sensing data, a deep learning model. These constraints make the models
they applied two machine learning algorithms namely error hard to be deployed in real-time. The long-term features are

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2322 A. Hekmat et al.

then learned using the LSTM network from which the final layers. Finally, the skeleton sequences are merged into a
results are predicted. single-layer perception and finalized the decision based on it.
Most of the recent researchers are interested in deep
learning methods for skeleton-based dynamic hand ges-
ture recognition. Núñez et al. [11] used a combination of
2.3 Hand pose estimation with ConvNets
CNN and LTSM for dynamic hand gesture recognition
and action recognition from skeletal data. The authors
The basic hand gesture recognition approach utilizes vari-
described that the model training is performed by using
ous extract the hand pose features along with the hidden
CNN. Ma et al. [9] concentrated on addressing noisy skel-
Markov model (HMM). Meanwhile, due to the develop-
eton data and using an LSTM network together with a
ment of a deep learning-based approach, the research-
nested interval unscented Kalman filter (UKF) to improve
ers use the method to perform the recognition operations
performance in order of removing the noise from datasets.
[31, 33]. Besides, visual features from the computerized
Thus, their method attains the advantage of both hand-
approaches were performed by the convolutional neural
crafted features and deep learning methods to obtain opti-
networks (CNN). For video modeling, a 3D convolutional
mal features for hand gesture recognition.
neural network (3DCNN) has been utilized, which utilizes
The major challenge associated with the existing tech-
spatiotemporal filters along with the conventional CNNs.
niques is the need for temporal and hand-crafted spatial
The literature [32, 34] utilizes this type of approach. The
features. Nowadays, various sensor designs have been pre-
main feature of 3DCNN is to directly generate hierarchi-
sented with different sensor technology and a high amount
cal determinations of spatiotemporal data. however, the
of information has been generated. The main challenge is
demerits of using this type of approach were the need for
to identify the appropriate sensor to implement the design
more parameters. The training portion is also arduous due
with minimal time to offer fast processing. The redundant
to the inclusion of the additional kernel. Hence, it has been
feature elimination is also an important challenge faced
replaced with domain adaption on pre-trained instances.
by the existing techniques. The resampling and interpola-
Recent research about hand pose data extraction is based on
tion techniques are also used which is not possible in a
calculating the machine learning algorithms [2, 15] based
live recognition system. To overcome these complexities
on cameras or sensors.
the proposed methodology integrated ANN with different
To estimate 3D hand posture, Guijin et al. [7] presented
technologies and ANN is mainly selected due to its capa-
a region ensemble network (REN) for direct 3D coordinate
bility to identify the interrelationship that exists between
regression. Because convolutional networks (ConvNet) with
the input without any prior knowledge. The ANN weights
complicated architecture are not appropriate for 3D coordi-
play a prominent role in feature selection which improves
nate regression, they presented a Region Ensemble Network
the classifier performance.
(REN). In the first step their algorithm partitions, the last
convolutional outputs of ConvNet into several grid regions,
and then results from separate fully-connected (FC) regres-
sors on each region are integrated by another FC layer to
3 Methods and experimental setup
perform the estimation. Similarly, Zhang et al. [8] stated
skeletal manifold into ConvNets and trained end-to-end to
Our technique section is split into two parts. The initial
render a sequential prediction model.
step of our study focuses on installing the flex sensor, col-
lecting data from it, and then categorizing the data using
three algorithms on various users to arrive at the best
2.4 Activity recognition with neural networks result for our objectives. The second section focuses on
how to push these distinct values into our database, which
Yong et al. [17] describe how to incorporate human activities reflect the value of the last layer in our nested set model
as a trajectory of skeletal joints in their hierarchical recurrent and then employ them as a map to reach all of the body
neural network (RNN) architecture. The authors stated that gesture positions. The software used for our research is
the RNN is modeled along with the long-term contextual the Nodejs framework based on Javascript for gathering
information of temporal sequences and also delineated an the data from flex-sensors and Mysql as a database and the
end-to-end hierarchical RNN for skeleton-based action rec- r-studio is used for extracting the results and also charts.
ognition. In addition, the human skeleton was separated into A Raspberry Pi model 4 is used as hardware because of its
five pieces and fed with five equal subnets. While the num- GPIO pins, which are beneficial for attaching sensors to
ber of layers has been increased, the extracted layer by the ADC integrated circuits (IC MCP3008) in order to capture
subnets is hierarchically merged to be the inputs of higher raw data, which is covered in detail in Sect. 3.2.

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Map modeling for full body gesture using flex sensor and machine learning algorithms 2323

3.1 Artificial neural network for gesture recognition gyroscope inputs [22]. An artificial neural network (ANN)
[11, 23] technique was used in this study to extract all of the
So the primary concept behind their work is to save the raw needed acceptable values from raw data, and then save the
data collected by the sensor in a database, and then apply processed and mature data in the database as a hierarchi-
an artificial neural network (ANN) to the data to detect the cal nested set model, as shown in Fig. 3. Figure 3 depicts
intended gesture, as illustrated in Fig. 2. how the process of reading data and writing to the database
So, as the result of this observation, we proposed a changes depending on our model.
method that creates a map model based on the hierarchical
nested set approach to first avoid ambiguity in the database. 3.2 Flex sensor installation to raspberry pi
The second reason is to structure the base model in such and glove and classifying the data
a way that it can track all of the skeleton joints as a single
node state, and the third reason is to reduce the amount of 3.2.1 Installation the flex sensors on the glove and its
unusable raw data and extra data content stored in a data- requirements and wiring part
base to reduce data manipulation and increase data reading
speed. The last challenge with this method is developing a The major component utilized in this study is a 4.5-in.
model that is based on the data responder and responsive to flex sensor (SEN—08,606) for three fingers (fore, mid-
external system demands like speech recognition commands, dle, and ring) and a 2.2-in. flex sensor (SEN—10,264) for
image processing input commands, or accelerometer and two fingers (thumb and little). The hardware prototype
with flex sensors mounted on fingers is represented in
the Fig. 4. The terminal resistance adapted to the bending
of Flex's sensors, assisting in the recognition of move-
ment of a specific region of the human body. Since the
flex sensor does not possess polarized terminals, there
will be no positive and negative terminals. From the fig-
ure, it is encountered that pin number P1 is connected
to the positive power source and P2 is connected to the
ground. Moreover, the resistances of the sensor increase
with the increasing bent/Flex in the Flex sensor. The con-
nection between the IC mcpp3008 and analog to digital
converter IC is made by using the interface. Hence after
the proper installment of the sensor in the IC, the con-
nection of Raspberry Pi 4 (minicomputer) has been made.
With the help of Raspberry Piremote control via Wi-Fi, a
Fig. 2  Demonstrating the relationship between data storage and using
an artificial neural network (ANN) algorithm on the stored data for specifically designed code is exported into the Raspberry
reaching the intended gesture Pi based on the [27] Nodejs code which when there is a

Fig. 3  Changing the priority

of two states of artificial neural
network (ANN) with database
storage and providing the map
model

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2324 A. Hekmat et al.

Fig. 4  Installation of the flex sensor to MCP3008 analog to digital converter IC (ADC IC) and wiring it to the Raspberry Pi 4

slight change in the bend of the Flex sensor, there is a 3.2.2 Data extraction from sensor and analysis
change observed in the readings obtained as output from
the Raspberry Pi. Figure 5 depicts the features and char- The speed-Hz configuration option is used to specify the
acteristics of the flex sensors used. SPI clock frequency in Hertz when connecting to the rasp-
berry pi master input and slave output (MISO/MOSI) for
reading the value from the flex (SEN—08,606) sensor[27].
The default SPI clock frequency for the MCP3008 is

Fig. 5  Glove design and fea-

tures. a The glove made and the
two different sizes of the flex
sensor (2.2 in. and 4.5 in.). b
Features of the sensor

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Map modeling for full body gesture using flex sensor and machine learning algorithms 2325

Fig. 6  Different hand gestures.

a to b from normal finger pose
to bend each finger step by step
till reaching the gesture d all
fingers and flex sensor com-
pletely bend

Table 1  Gathering values from all finger sensors at gesture (a) sizes 4.5 in. and 2.2-in. a smaller size needs a higher resistor
Thumb Fore Middle Ring Little
around 40 K to normalize the data gathering from 4.5 with
20 K resistor. The values are attained by the flex sensor after
95.8064 59.0322 55.8064 56.7741 78.7096 the installation section, while it is in the typical position
96.4516 58.7096 54.8387 56.7741 80.6451 gesture, as shown in Fig. 6 and Table 1.
95.4838 59.0322 55.1612 57.0967 79.9999 According to Fig. 6 by bending the fingers the data
94.8387 57.4193 56.4516 58.1645 81.6129 slightly starts to change while keeping the same tolerance.
96.7741 59.3548 55.4838 56.7741 82.5032 So the values for all 5 fingers at the stable position (no move-
95.8064 58.7096 55.8064 57.0967 79.9999 ment or no finger bending) is selected to draw (a) as shown
96.1290 57.1967 55.1612 57.4193 80.6451 in Fig. 6 is the intended position. Periodical values vary in
95.4838 58.7096 55.1612 58.1645 81.4521 the sequence of 1-to-4-number at the integer part and 8 dif-
94.8387 57.1967 54.8387 57.0967 82.9032 ferences at the decimal part over a period time of observa-
94.8387 58.7096 55.4838 57.0967 81.4521 tion at each point. In the charts below in Fig. 7 the line chart
96.1290 59.3548 56.4516 58.1645 78.7096 is demonstrated according to the Table 1 data at gesture (a)
95.8064 57.1967 55.4838 56.7741 79.9999 for all fingers.

3.2.3 Data classification using three distance metrics

1,350,000 Hz but lowering it to 20,000 Hz gives a more
accurate and precise bending reading. Normally, it's not After all of these observations, the data are classed and nor-
important to lower the clock speed to read a value. The malized using Mahalanobis Distance, Euclidean Distance,
default clock rate speed of 1350000 Hz for the MCP3008 and multilayer neural network to get the best classification
has been taken from the MCP3008 datasheet. The maximum results and compare each user's result. The setup includes
rate at voltage = 2.7 V is 75 ksps and each sample needs an multiple estimations from the sensors. However, these meas-
18-bit transfer. 75,000 × 18 = 1,350,000. 1,350,000 Hz is an urements include eight features for each instance, and thus
old usage of frequency in the above circuit pattern as VDD attains five-dimension feature vectors. Bayesian Classifier
is 3.3 V. Also, to make the all data gathering from both is used for the classification of the sensor measurement and

Fig. 7  Line-chart for Table 1.

All fingers are at the position of
figure gesture (a)

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2326 A. Hekmat et al.

is categorized into 27 classes. The processing is conducted layer has 8 nodes, the first hidden layer contains 50 nodes,
on a resource constraints microcontroller and hence it is the second hidden layer contains 25 nodes, and the output
ineluctable to make some assumptions for the class distri- layer contains 25 nodes. Like the Mahalanobis classifier,
butions. If each class is considered to be Gaussian same the neural network also works on large datasets. It gives
covariance matrix, then Mahalanobis Distance Classifier is more precision with hidden layers added at the trade of high
a better option due to its computational efficiency as given in computational power.
(1), The Mahalanobis distance between two d-dimensional Moreover, neural networks are also utilized for the clas-
numerical vectors h and h′ can be determined as, sification of finger gestures. The simplest learning approach
is supervised learning, which uses labeled inputs to train
d2 (h, h� ) = (h − h� )T C−1 (h − h� ) (1) the neural network. The labeled samples are then utilized
C is a d × d dimension matrix in the range 2 which performs to develop a data model or generalizable rules that may be
better than the identity matrix. The covariance matrix applied to unlabeled data later on. The convolutional neural
acquired for the data for analysis is designated as ∑, and network (ConvNets) is a kind of neural network [7] which
the euclidean distance may be calculated using the formula offers several significant benefits over traditional neural net-
C = ∑-1. The C value obtained using this method mainly works, particularly when it comes to data classification. The
contributes to the actual definition of Mahalanobis distance. key data inputs, for example, would be images, which are
Furthermore, if the constraints are distributed and each fea- made up of big data.
ture has the same variance, the Euclidean Distance Classifier The problem with image classification is that it soon
is the best alternative and is more computationally effective becomes computationally prohibitive to train with some
than the Mahalanobis Distance Classifier. Thus Euclidean models when there are large images with multiple 3-channel
distance is determined as, RGB hues [2, 15]. CNN totally bypasses a filter over the first
image that imposes matrix multiplication on a subset of the
pixels in the first image, and it continues the process until all
√
√N
subsets are reunited. The filter focuses on acquiring the most
√∑
Euclidean distance = √ (h − h� ) (2)
i=1 significant features while leaving out the less important ones.
The convolution layer is the process of applying a filter to
Means and variances for each class are obtained using the initial input pixels data. After the convolution layer has
(3), and the covariance matrix is obtained using (4), of each been applied, the pooling layer will be trained to minimize
attribute for respective classes. the spatial size of the convoluted features. The reduction in
∑N � � complexity has also resulted in a reduction in dimensional-
hi
Mean� ∶ h� = i=1 , (3) ity, which reduces the computational cost of analyzing the
N data set and makes the approach more efficient.
∑N � As a result, a kernel runs through all subsets of serial
input data obtained from the sensor once again at this layer.
�� �
h − h� g − g�
Covariance matrix ∶ 𝜎xy = a = i=1
, (4)
N There are two types of pooling kernels available, one of
which we are employing here. The first is Max Pooling,
hi and gi is the ith sample data of two classes (h, g), n is the which retains the subset's maximum value. The average
total number of samples for each class, h′ is the mean value pooling kernel is the second kernel, and it accomplishes
of the respective class and 𝜎ij is the standard deviation of ‘h’ exactly what we anticipate. To lower the difference value
row and ‘g’ column of the covariance matrix. The distance for each user, it preserves the average value of all the data in
between the test instance and the estimated mean values of the subset. The conceptual processes of the pooling phase
all attributes in each class is measured by these classifiers. are depicted in Fig. 8b. After all of the classified data had
The resulting classified class is the one that is the minimum appeared, all five fingers chart locations were categorized
distance from the test sample. Mahalanobis distance classi- according to Table 4.
fier takes into account the covariance of the classes in addi-
tion to the mean, and is, set, therefore, more powerful given 3.3 Database collection map model
there is enough data to estimate covariance. The neural net-
work is one of the high computational classifiers been used Preparing and recognizing data from a group of persons
in the area of recognition. It has the capability of storing needs some primary considerations and also depends on
knowledge in the training phase using synaptic weights. The the two main features: first identify the actors whose dif-
number of neurons in the input layer is equal to the number ferent gestures are going to be captured and the second one
of features in each sample vector. The network processes the is the software and the way of recording the positions. [35]
test data with stored data and computes the output. The input This interaction is analyzed in such a way that each actor

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Map modeling for full body gesture using flex sensor and machine learning algorithms 2327

Fig. 8  The pooling phase of

convolution neural networks

performs in a particular location for every scene. The geo- structure is useful [17, 19]. In this section, we divide the
metrical figures can be estimated with the machine learning- body into different sub-categories [28] as we can see in
based computational approach which can be used to rec- Fig. 9. A graph-based pattern is based on a nested set
ognize the recorded body parts. It also converts the data model that assigns a right and left value to each node as
into a machine-processable form. The steps explained below a result of acknowledging the current position, tracking
show the formal description of body gestures to recognize each position change movement from one gesture to the
the actors’ movements. next, and controlling and reducing computational actions
for interacting with databases and motors. In brief, the
1. Using the triangle Tk the spatiotemporal description of main goal for this creation of the map model mostly is to
an actor’s movement as a sequence of body gestures are interact as an artificial resource that responds to some out-
generated. Moreover, based on the individual’s positions coming command, like image processing command from a
(p) and directions the route can be formed at each time camera resource or voice recognition commands from the
sequence. Each position is described in a coordinate sys- microphone, or even some sensors like a gyroscope for
tem, and the previous location of the actors is checked stabilizing the balance.
to ensure the accuracy of the location of each scene. The nested model data classification scheme with the
2. Directional, whole-system, proximal, or distance fea- left and right values shown in Fig. 9 is quite important
tures and connectivity containment employing proxim- for accessing the state of each node prior to the intended
ity locations are used to provide spatial connections and state, as well as calculating the position and performing
restrictions. data analysis for movement control. The right side, which
3. The recorded data is utilized to check the positions of is not indicated in this model, has the same structure as the
the actor’s knees, shoulders, head, torso, neck, elbows, left side. The body category table is mainly covered in this
hands, femur balls, and feet. The normal vector pointing work which consists of the primary table, and the fingers
away from the body parts is estimated using the triangles table, which has the applicable normalized values for each
designed with vertices on the head, arms and legs, upper finger position, because creating a model for the entire
body, lower body, whole body, and three edges. Besides, body is too large at this time. The body_category table
the body gestures are constructed by moving the limbs (Table 2) categorizes all body gestures into the nested set
and recording trajectories. Meanwhile, the gesture vec- model.
tors f can be evaluated by using the triangles and their The finger table shown in Table 3 is an aggregate table
normal vectors, and also the trajectories tk(va), tk(vb), that contains the last executable values depending on its cat-
tk(vc), and the distances d(vk1, vk2). egory_id or its parent id in the 4th column which is a foreign
key to the body_category in the category_id column. Table 3
3.3.1 Nested set model description shows that after classifying data for each finger, 155 rows
of data were imported into the table. So, the first and the
The neural classifier outperformed all other users in gen- last value for each finger is the only values that are needed.
erating the most correct value for each finger position, Here, the letter (T) stands for the thumb, (F) stands for the
according to the data. So, for this purpose first, we are fore-finger, (M) stands for the middle finger, (R) stands for
going to describe the structure for this issue and why this the ring finger, and (L) stands for the little finger.

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2328 A. Hekmat et al.

Fig. 9  Data classification using

the nested set model

Table 2  Nested set categorization table for human body model Table 3  The aggregate table with the category_id as a foreign key is
related to Table 2
Class-id Part name Left Right
Finger_id Finger Value Category_id
1 Body 1 30
2 Left_side 2 27 1 T1 95.25 9
3 Body_gesture 3 24 … … … 9
4 Shoulder 4 5 27 T27 192.74 9
5 Elbow 6 7 28 F1 58.45 10
6 Wrist 8 9 … … … 10
7 Hnd 10 23 59 F31 155.35 10
8 Fingers 11 22 60 M1 55.23 11
9 Thumb 12 13 … … … 11
10 Fore 14 15 93 M33 158.59 11
11 Middle 16 17 94 R1 57.58 12
12 Ring 18 19 … …. … 12
13 Little 20 21 125 R31 154.48 12
14 Facial_gesture 25 26 126 L1 80.53 13
15 Right_side 28 29 … … … 13
155 L29 171.47 13
This table might be expanded to the other form depending on catego-
rizing the intended index

collect all of the positions of each finger with aggregate

3.3.2 Nested set aggregated tables and function its category from the body category table, as seen below:
SELECT [Link],COUNT([Link])
The relevant table can be filled out using the data gathered FROM body_category AS node,
in Sect. 3.2.3 of the classifying section, to its aggregate body _category AS parent,
table, at each layer, based on this sub-category model [28]. fingers
Let us assume a character for each finger, as described WHERE [Link] BETWEEN [Link] and [Link]
in the previous section (TFMRL-five finger). To put it AND [Link] = [Link]
another way, one should first write all of the appropriate GROUP BY [Link]
locations categorized values in our finger stable for that ORDER BY [Link];
action. After that, using a very easy query, one can just The number of each category node that has a direct rela-
tionship to its parent is determined by running the above

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Map modeling for full body gesture using flex sensor and machine learning algorithms 2329

Fig. 10  Some samples for hand

gesture body language like
Thumb’s Up and OK or grab
a pen

query. For example, for the first root and parent of the tree, Table 4  Different gestures model table
which is 155, all rows from the finger table are fetched, and Gesture_id Value Keywords
the number for the category thumb is 27. This strategy is
also applicable to the wrist table and elbow table. 1 T25F29M2R1L1 OK—fine
2 T1F31M33R31L29 Thumbs up—good
3.3.3 Selection of gesture set or activity table 3 T27F31M33R31L29 Punch

In this section, we will use the freely accessible gesture

reference database to train benchmark classifiers, param- Table 5  Body class-table based on the adjacency list categorization
eters, and models to gain insight into the nature of human model
movement [1]. Furthermore, a user database may be built Class-id Part name Parent
by making the following assumptions to test HCI-related
algorithms centered on gestures, such as the Naturalness of 1 Body NULL
a Gesture Set. We chose naturally prevalent gestures that are 2 Left_side 1
commonly generated throughout childhood, such as thumbs 3 Body_gesture 2
up or money, as shown in Fig. 10, and not obtained by using 4 Shoulder 3
appropriate programs such as ASL, divers' hand signals, or 5 Elbow 3
soldiers' hand signals. It can be also extended to some activi- 6 Wrist 3
ties like grabbing something or holding an object in a unique 7 Hand 3
way (for example grabbing a pen in our hand). 8 Fingers 7
By mapping the data, one can use a single query request 9 Thumb 8
to get a certain hand pattern, such as punch mode, holding 10 Fore 8
something, [1], and so on. The information will be as fol- 11 Middle 8
12 Ring 8
lows: each node's state is represented by a character (T, F,
13 Little 8
M, R, L). As a result, Si will be a set of states represented by
14 Facial_gesture 2
each sensor's id; for example, consider the Table 5 structure
15 Right_side 1
for the gesture sets used.
According to Table 4, the string code of the gesture (C) in
Fig. 10, is this value below which has the direct:
fourth scenarios is for supporting multiple users without
T25F29M2R1L1
and with speed constraints, respectively. For each scenario,
(T25) means state pose for the Thumb finger with the the subsets of the set can be established by providing the
node id of 25 and (R1) means Ring finger with the node id indexes of people and speeds (G). For each subset, the
of 1. Also, several motions may be saved for a period of separation measure JAB and the confusion ratio for each
time before being deployed using the mechanical hand pro- pair of gestures were calculated. This method allowed us
totype. To analyze the differences between the participants to reduce the amount of computation time and respond
four scenarios are listed: 1. One person with unlimited quickly to input commands. Nevertheless, the keyword
speed, 2. One person with normal speed, 3. More people column is used which is quite important for this problem
with unlimited speed, 4. More people with normal speed and represents each gesture as a value to impose on the
4. The speed limit is eliminated for a single individual in motors for movement control.
the first scenario, and the stability of the gesture template
is measured. Second, the speed limit is set and measured
in the same manner. The interface used in the third and

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2330 A. Hekmat et al.

3.3.4 Comparison between hierarchical nested set vs Table 6  Comparative study of the performances of all the classifiers
adjacency list model based on the number of training samples
Dataset FCM (%) REN (%) REN (%) Proposed (%)
In this section, the hierarchical nested set model is compared
User-1 77.57 62.16 89.95 94.3
with the recursive functionality of the adjacency list. The
User-2 75.35 63.78 96.53 99.5
authors of the literature [28] proposed a recursive function
User-3 37.63 58.36 85.47 95.8
of the adjacency list model as an indicator for each item
User-4 61.25 68.88 89.78 98.9
in the table, allowing us all to recognize the relationship
User-5 65.29 64.36 93.15 95.72
to its parent. According to Table 5, the first-row item, the
Average 63.41 63.5 91.57 96.84
class ‘body’, provides NULL value for its parent. The adja-
cency list model can be quite simple to understand, however,
it is hard to implement based on the database structure. The
relationship between the depth and the number of increasing
parent nodes is the most detrimental element of this type of
clustering. The complexity of running the query increases
for each node in each layer, and for each additional depth,
one must use yet another join query on its table to reach the
children of that depth.
Furthermore, the complete tree's hierarchical data can be
used with some type of indentation. The SQL form can be
used to acquire this information as shown below.
SELECT [Link] AS lev1, [Link] AS lev2, t3. name AS
lev3, t4. name AS lev4
FROM body_category AS t1
LEFT JOIN body_category AS t2 ON [Link] = t1.
category_id
LEFT JOIN body_category AS t3 ON [Link] = t2.
category_id Fig. 11  Comparing the accuracy of algorithms on different training
LEFT JOIN body_category AS t4 ON [Link] = t3. samples at specific thresholding
category_id
WHERE [Link] = ‘body’;
In comparison to the query provided in Sect. 3.2.3, the user-specific data (testing of user 1 only takes data from user
query's complexity is very high, and it is not reasonable for 1). The Neural Network obtains a superior result, followed
the machine to conduct this type of query for each request by the REN classifier, which achieves higher accuracy, as
or command that arrives at the system. seen in the table. While the FCM classifier obtains superior
results for small data partitions, the FCM has not acquired
the needed quantity of data and hence does not function
4 Evaluate and compare the results accurately for all of the parameters taken. Similarly, as the
number of training samples grows, the performance of the
In this section, the results are evaluated for different classi- Neural Networks outperforms the other three classifiers. Fig-
fier algorithms and the dataset model separately and com- ure 11 illustrates the graphical representation of the overall
pare each of the results with similar methods. performances of all the taken classifiers.

4.1 Classifying results
4.2 The nested set map model evaluation
Following all of these studies, our goal is to use the nodejs
framework to apply the three previously stated classifying In this section according to Fig. 12, the quality of the tree is
algorithms for five different users, resulting in the following evaluated with these two metrics:
values for each user for each method as shown in Table 6.
To compare the results and determine which algorithm has 1. The coherence
the most coordination point and accuracy, as well as all of 2. The coverage of the tree
the user's hand data captured through [Link] 6 dis-
plays the outputs of the accuracy comparison employing For both approaches with this below sample tree:

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Map modeling for full body gesture using flex sensor and machine learning algorithms 2331

Table 8  Runtime for identifying different gestures

Category_id Name Runtime
(seconds)

1 body 0.025
2 left_side 0.025
3 body_gesture 0.023
4 shoulder 0.023
5 elbow 0.026
6 wrist 0.021
7 hand 0.021
8 fingers 0.023
Fig. 12  The structure of discrete data sample for demonstrating the 9 thumb 0.021
evaluation between two different models deployed on the data 10 fore 0.021
11 middle 0.024
12 ring 0.025
Table 7  Data retrieval examples in different levels
13 little 0.026
Dataset Level 1 Level 2 Level 3 Level4 Total nodes 14 facial_gesture 0.024
Adjacency list 6 6 6 1 19 15 right_side 0.023
Nested set 1 2 6 1 10

The coherence: in our experiments, the number of nodes

is set as ten for both the nested set and adjacency list models
at four levels of child-parental data. Therefore, 10 nodes
are available in 4 levels. In this step, simply requesting a
complete tree in both ways produces the results presented
in Table 7.
The coverage of tree: to investigate the coverage of all
nodes wherever in the tree (for example the child node posi-
tion at each level) one have to find out that to reach each
node at any level what is the cost of data redundancy and
what is the rate of query complexity and execution time
to determine each node position. At this point, one must
acknowledge the problem in the context of the current exam-
ple (Fig. 12). A total of 10 nodes is only employed in the
four levels, which will become more consistent in the real
Fig. 13  Comparative analysis using ERR
full-body data map model. In the nested set model, however,
we have the left and right values, which are the best factors
for quickly reaching any appropriate node at the lowest cost. classifiers such as FCM [40], KNN [41], REN [7], and
Runtime analysis: the runtime is computed for the dif- ECOC-SVM [42]. The intentional error made by the robot is
ferent body gestures shown in Table 3 and the results are evaluated using the ERR and the results obtained are shown
presented in Table 8. The runtime value is the average value in Fig. 13. The average EER and its standard deviation (SD)
taken to analyze a total of 100 images for a specific ges- value is also analyzed for both random and planned error and
ture. Based on the runtime computed one can see that our the results obtained are depicted in Table 9. The proposed
proposed framework can be efficient when implemented in methodology shows minimal values in terms of both average
real-time. BRR and SD. The techniques such as FCM and KN show
The Equal error rate (ERR) is mainly used to compute the the lowest performance when compared to other techniques.
robustness of the robot designed where both the False Rejec- The proposed methodology is evaluated with the exist-
tion Rate (FRR) and False Acceptance Rate (FAR) match. ing techniques in terms of recognition rate and the results
A lower ERR value specifies that the gestures of the robot obtained are shown in Fig. 14. The recognition rate is
are highly robust. The efficiency of the proposed system mainly derived based on the ability of the system to ana-
is analyzed by comparing it with the different state-of-art lyze the full body gesture. The utilization of the nested set

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2332 A. Hekmat et al.

Table 9  Performance evaluation of the robot in terms of random and Table 10  Ablation study for different set of features and settings
intentional error using different techniques
Combination Accuracy (%) Recall (%) Precision (%)
Techniques Robustness evaluation (average EER
value(%) ± SD) C1 85 84 83
C2 84 83 82
Random error Intentional error
C3 84 83 82
FCM [40] 3.56 ± 0.02 5.65 ± 0.35 C4 85 83.5 84
KNN [41] 4.21 ± 0.05 6.33 ± 0.78 C5 87 85 86
REN [7] 5.63 ± 0.09 7.54 ± 0.25 C6 86 84 85
ECOC-SVM [42] 4.21 ± 0.05 5.45 ± 0.15 C7 85 83 84
Proposed ANN integrated 0.89 ± 0.02 3.25 ± 0.12 C8 86 84 85
with C9 87 86 84
C10 90 89 89
C11 89 87 87
C12 88 86 86
C13 89 88 88
C14 90 89 90
C15 90.85 90 89.5

C6: Classification via three distance metrics

C7: Features (Hand, Fingers, Thumb, forefinger, middle
finger, and ring finger)
C8: Features (facial_gesture and right_side)
C9: Features (body_gesture, body, and left_side)
C10: Features (shoulder, elbow, wrist, and hand)
C11: Entire features (C6-C11)
The results were obtained for different performance
metrics such as accuracy, recall, and precision as shown
in Table 10. The proposed model offers improved perfor-
mance of different human part gesture recognition in terms
Fig. 14  Comparison analysis using recognition rate of accuracy, recall, and precision. An increased number of
features give improved results for the proposed methodol-
ogy as shown for C15. The proposed models mainly offer
model for gesture set classification increases the recogni- improved results due to the presence of the equal sample
tion rate of the proposed methodology a little bit further distribution and the ability of the classifier to grasp the dif-
by clearly deriving the edge details. The KNN and FCM ferent body gestures.
methods recognition rate decreased mainly due to the illu-
mination of hand gestures. The performance of the REN
and ECOC-SVM techniques decreases when subjected to 5 Conclusion
identifying the facial and body gestures. The best perfor-
mance of the proposed ANN model implemented in the In this paper, a low-cost technique is proposed that uses a
flex sensor with map modeling is also evaluated using the neural network algorithm to extract user data from a flex
ablation study for different feature sets and diverse set- glove (sensor) and then fill the database with normalized
tings. The parameters of the ANN were trained for a total data using a nested set model. The nested set model divides
of 500 epochs with a learning rate of 0.01 and a batch size the gestures of the body to left and right. On the left side,
of 64. The combinations for which the ablation study is there are two types of gestures regarding the body and face.
conducted is presented below: Because developing a model for the full body is too large at
C1: Single person with unlimited speed this time, we will just cover the body category table, which
C2: Single person with normal speed is the primary data, and the fingers table, which contains the
C3: Multiple people with unlimited speed appropriate normalized values for each finger position. The
C4: Single person with normal speed hierarchical nested set model is compared with the recursive
C5: Gesture ID (1,2,3). function of the adjacent list model. The nested set map is

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Map modeling for full body gesture using flex sensor and machine learning algorithms 2333

evaluated using the coherence and the coverage of the tree 2. Supancic, J.S., Rogez, G., Yang, Y., Shotton J., Ramanan, D.:
to evaluate the performance of both the nested set and adja- Depth-based hand pose estimation: data, methods, and challenges.
In: 2015 IEEE International Conference on Computer Vision
cency list models using 10 nodes at four different levels. The (ICCV), 2015, pp. 1868–1876. [Link]
results show that the implementation of the adjacency matrix 2015.​217
is intricate due to its complicated database structure. The 3. Tan, D.J., et al.: Fits like a glove: rapid and reliable hand shape
nested model helps to reach the appropriate node with the personalization. In: 2016 IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2016, pp. 5610-5619, [Link]
lowest cost due to the presence of its left and right values. doi.​org/​10.​1109/​CVPR.​2016.​605
The runtime efficiency achieved for the 15 different classes 4. Chen, X., Wang, G., Guo, H., Zhang, C., Wang, H., Zhang, L.:
for the proposed model is more than satisfactory for the Mfa-net: Motion feature augmented network for dynamic hand
real-time implementation. The efficiency of the proposed gesture recognition from skeletal data. Sensors 19(2), 239 (2019)
5. Syed Mubarak Ali, S.A.A., Ahmad, N.S., Goh, P.: Flex sensor
methodology is compared with different existing techniques compensator via Hammerstein–Wiener modeling approach for
such as FCM, KNN, REN, and ECOC-SVM in terms of improved dynamic goniometry and constrained control of a bionic
accuracy, recognition rate, and ERR. The proposed model hand. Sensors 19(18), 3896 (2019)
offers a lower ERR value when compared to other tech- 6. Müller, M., Röder, T., Clausen, M., Eberhardt, B., Krüger, B.,
Weber, A. Mocap database HDM05. Institut für Informatik II,
niques in terms of random (0.89% ± 0.02) and intentional Universität Bonn 2(7) (2007)
error (3.25% ± 0.12). The high recognition rate and low ERR 7. Guo, H., Wang, G., Chen, X., Zhang, C.: Towards good practices
value show the robustness of the proposed technique. For for deep 3D hand pose estimation. ArXiv:abs/1707.07248 (2018)
a combination of the Hand, Fingers, body_gesture, body, 8. Zhang, Y., Xu, C., Cheng, L.: Learning to search on manifolds for
3D pose estimation of articulated objects. ArXiv:abs/1612.00596
facial_gesture, right_side left_side, Thumb, forefinger, mid- (2016)
dle finger, ring finger, shoulder, elbow, and wrist gesture 9. Ma, C., Wang, A., Chen, G., Xu, C.: Hand joints-based gesture
features, the proposed model achieves accuracy, recall, and recognition for noisy dataset using nested interval unscented
precision values of 90.85%, 90%, and 89.5% respectively. In Kalman filter with LSTM network. Vis. Comput. 34, 1053–1063
(2018)
the future, we plan to conduct these experiments in thermal 10. Wang, C., Liu, Z., Chan, S.: Superpixel-based hand gesture recog-
images by altering their complex visual effects with different nition with Kinect depth camera. IEEE Trans. Multimedia 17(1),
pre-processing and feature extraction techniques. 29–39 (2015). [Link]
11. Núñez, J.C., Cabido, R., Pantrigo, J.J., Montemayor, A.S., Vélez,
J.F.: Convolutional neural networks and long short-term memory
for skeleton-based human activity and hand gesture recognition.
Funding Not applicable. Pattern Recognit. 76, 80–94 (2018)
12. Zhu, G., Zhang, L., Shen, P., Song, J.: Multimodal gesture rec-
Declarations ognition using 3-D convolution and convolutional LSTM. IEEE
Access 5, 4517–4524 (2017). [Link]
Conflict of interest The authors declare that they have no conflict of 2017.​26841​86
interest. 13. Ghosh et al.: On automatizing recognition of multiple human
activities using ultrasonic sensor grid. In: 2017 9th International
Human and animal rights This article does not contain any studies with Conference on Communication Systems and Networks (COM-
human or animal subjects performed by any of the authors. SNETS), 2017, pp. 488–491. [Link]
ETS.​2017.​79454​40
Informed consent Informed consent was obtained from all individual 14. Ghosh, A., Chakraborty, A., Chakraborty, D., et al.: UltraSense: A
participants included in the study. non-intrusive approach for human activity identification using het-
erogeneous ultrasonic sensor grid for smart home environment. J.
Consent to participate Not applicable. Ambient Intell. Human. Comput. (2019). [Link] 007/​
s12652-​019-​01260-y
Consent for publication Not applicable. 15. Lefebvre, G., Berlemont, S., Mamalet, F., Garcia, C.: Blstm-rnn
based 3D gesture classification. In: Artificial Neural Networks and
Availability of data and materials Data sharing is not applicable to this Machine Learning, pp. 381–388. Springer, New York, 2013.
article as no new data were created or analyzed in this study. 16. Gupta, H.P., Chudgar, H.S., Mukherjee, S., Dutta, T., Sharma,
K.: A continuous hand gestures recognition technique for human–
Code availability Not applicable. machine interaction using accelerometer and gyroscope sensors.
IEEE Sens. J. 16(16), 6425–6432 (2016). [Link]
JSEN.​2016.​25810​23
17. Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural net-
work for skeleton based action recognition. In: 2015 IEEE Confer-
References ence on Computer Vision and Pattern Recognition (CVPR), 2015,
pp. 1110–1118. [Link]
1. Głomb, P., Romaszewski, M., Opozda, S., Sochan, A.: Choosing 18. Zhang, X., Chen, X., Li, Y., Lantz, V., Wang, K., Yang, J.: A
and modeling the Hand Gesture Database for a natural user inter- framework for hand gesture recognition based on accelerometer
face. In: Gesture Workshop (2011) and EMG sensors. IEEE Trans. Syst. Man Cybern. Part A Syst.
Hum. 41(6), 1064–1076 (2011). [Link] oi.o​ rg/1​ 0.1​ 109/T
​ SMCA.​
2011.​21160​04

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

2334 A. Hekmat et al.

19. Ma, C., Zhang, S., Wang, A., Qi, Y., Chen, G.: Skeleton-based 39. Zhang, W., Wang, J., Lan, F.: Dynamic hand gesture recogni-
dynamic hand gesture recognition using an enhanced network tion based on short-term sampling neural networks. IEEE/CAA
with one-shot learning. Appl. Sci. 10(11), 3680 (2020) J. Autom. Sin. 8(1), 110–120 (2020)
20. Chuang, W.C., Hwang, W.J., Tai, T.M., Huang, D.R., Jhang, Y.J.: 40. Jia, G., Lam, H.K., Ma, S., Yang, Z., Xu, Y., Xiao, B.: Classi-
Continuous finger gesture recognition based on flex sensors. Sen- fication of electromyographic hand gesture signals using modi-
sors 19(18), 3986 (2019) fied fuzzy C-means clustering and two-step machine learning
21. Xu, R., Zhou, S., Li, W.J.: MEMS accelerometer based nonspe- approach. IEEE Trans. Neural Syst. Rehabil. Eng. 28(6), 1428–
cific-user hand gesture recognition. IEEE Sens. J. 12(5), 1166– 1435 (2020)
1173 (2012). [Link] 41. Wong, W.K., Juwono, F.H., Khoo, B.T.T.: Multi-features capaci-
22. Gupta, H.P., Chudgar, H.S., Mukherjee, S., Dutta, T., Sharma, tive hand gesture recognition sensor: a machine learning approach.
K.: A continuous hand gestures recognition technique for human- IEEE Sens. J. 21(6), 8441–8450 (2021)
machine interaction using accelerometer and gyroscope sensors. 42. Ehrnsperger, M.G., Brenner, T., Hoese, H.L., Siart, U., Eibert,
IEEE Sens. J. 16(16), 6425–6432 (2016) T.F.: Real-time gesture detection based on machine learning clas-
23. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT sification of continuous wave radar signals. IEEE Sens. J. 21(6),
Press, Cambridge (2016) 8310–8322 (2020)
24. Tai, T.M., Jhang, Y.J., Liao, Z.W., Teng, K.C., Hwang, W.J.: Sen- 43. Gowthul Alam, M.M., Baulkani, S.: Reformulated query-based
sor-based continuous hand gesture recognition by long short-term document retrieval using optimised kernel fuzzy clustering algo-
memory. IEEE Sens. Lett. 2(3), 1–4 (2018) rithm. Int. J. Bus. Intell. Data Min. 12(3), 299 (2017)
25. Chu, Y.C., Jhang, Y.J., Tai, T.M., Hwang, W.J.: Recognition of 44. Sundararaj, V.: An efficient threshold prediction scheme for wave-
hand gesture sequences by accelerometers and gyroscopes. Appl. let based ECG signal noise reduction using variable step size fire-
Sci. 10(18), 6507 (2020) fly algorithm. Int. J. Intell. Eng. Syst. 9(3), 117–126 (2016)
26. Jhang, Y.J., Chu, Y.C., Tai, T.M., Hwang, W.J., Cheng, P.W. and 45. Gowthul Alam, M.M., Baulkani, S.: Geometric structure informa-
Lee, C.K.: Sensor based dynamic hand gesture recognition by tion based multi-objective function to increase fuzzy clustering
PairNet. In: 2019 International Conference on Internet of Things performance with artificial and real-life data. Soft Comput. 23(4),
(iThings) and IEEE Green Computing and Communications 1079–1098 (2019)
(GreenCom) and IEEE Cyber, Physical and Social Computing 46. Sundararaj, V.: Optimised denoising scheme via opposition-based
(CPSCom) and IEEE Smart Data (SmartData), IEEE. p. 994– self-adaptive learning PSO algorithm for wavelet-based ECG
1001. (2019) signal noise reduction. Int. J. Biomed. Eng. Technol. 31(4), 325
27. [Link] (2019)
28. http:// ​ m ikeh ​ i llyer. ​ c om/ ​ a r tic ​ l es/ ​ m anag ​ i ng- ​ h iera ​ r chic​ 47. Aswini, J., Yamini, B., Jatothu, R., Nayaki, K.S. and Nalini, M.,
al-​data-​in-​mysql/ 2021. An efficient cloud‐based healthcare services paradigm for
29. Damdoo, R., Kalyani, K., Sanghavi, J.: Adaptive hand gesture chronic kidney disease prediction application using boosted sup-
recognition system using machine learning approach. Biosci. Bio- port vector machine. Concurrency and Computation: Practice and
technol. Res. Commun. 13, 106–110 (2020) Experience, p.e6722.
30. Al-Hammadi, M., et al.: Deep learning-based approach for sign 48. Hassan, B.A., Rashid, T.A.: Datasets on statistical analysis and
language gesture recognition with efficient hand gesture represen- performance evaluation of backtracking search optimisation algo-
tation. IEEE Access 8, 192527–192542 (2020). [Link] oi.o​ rg/1​ 0.​ rithm compared with its counterpart algorithms. Data Brief 28,
1109/​ACCESS.​2020.​30321​40 105046 (2020)
31. Alhussein, M., Muhammad, G.: ‘Voice pathology detection using 49. Hassan BA (2020) CSCF: a chaotic sine cosine firefly algorithm
deep learning on mobile healthcare framework.’ IEEE Access 6, for practical application problems. Neural Comput Appl 1–20
41034–41041 (2018) 50. Rejeesh, M.R.: Interest point based face recognition using adap-
32. Muhammad, G., Alhamid, M.F., Long, X.: Computing and pro- tive neuro fuzzy inference system. Multimed. Tools Appl. 78(16),
cessing on the edge: smart pathology detection for connected 22691–22710 (2019)
healthcare. IEEE Network 33(6), 44–49 (2019) 51. Sundararaj, V., Muthukumar, S., Kumar, R.S.: An optimal clus-
33. Hou, R., Chen, C., Shah, M.: An end-to-end 3D convolutional ter formation based energy efficient dynamic scheduling hybrid
neural network for action detection and segmentation in videos. MAC protocol for heavy traffic load in wireless sensor networks.
2017. arXiv:1​ 712.0​ 1111 [Online]. [Link] rxiv.o​ rg/a​ bs/1​ 712.0​ 1111 Comput. Secur. 77, 277–288 (2018)
34. Varol, G., Laptev, I., Schmid, C.: Long-term temporal convolu- 52. Sundararaj, V., Anoop, V., Dixit, P., Arjaria, A., Chourasia, U.,
tions for action recognition. IEEE Trans. Pattern Anal. Mach. Bhambri, P., Rejeesh, M.R., Sundararaj, R.: CCGPA-MPPT:
Intell. 40(6), 1510–1517 (2017) Cauchy preferential crossover-based global pollination algorithm
35. Sasaki, K. et al.: An automated structural approach to support for MPPT in photovoltaic system. Prog. Photovolt. Res. Appl.
theatrical performances by introducing gesture recognition to a 28(11), 1128–1145 (2020)
cuing system. CRIWG/CollabTech (2019) 53. Vinu, S.: Optimal task assignment in mobile cloud computing by
36. Reale, M.J., Canavan, S., Yin, L., Hu, K., Hung, T.: A multi- queue based ant-bee algorithm. Wirel. Pers. Commun. 104(1),
gesture interaction system using a 3-D iris disk model for gaze 173–197 (2019)
estimation and an active appearance model for 3-D hand pointing. 54. Manikandan, N., Gobalakrishnan, N., Pradeep, K.: Bee optimi-
IEEE Trans. Multimedia 13(3), 474–486 (2011) zation based random double adaptive whale optimization model
37. Zhu, C., Yang, J., Shao, Z., Liu, C.: Vision based hand gesture for task scheduling in cloud computing environment. Comput.
recognition using 3D shape context. IEEE/CAA J Autom. Sinica. Commun. 187, 35–44 (2022)
8, 1600–1613 (2019)
38. Chen, G., Xu, Z., Li, Z., Tang, H., Qu, S., Ren, K., Knoll, A.: A Publisher's Note Springer Nature remains neutral with regard to
novel illumination-robust hand gesture recognition system with jurisdictional claims in published maps and institutional affiliations.
event-based neuromorphic vision sensor. IEEE Trans. Autom. Sci.
Eng. 18(2), 508–520 (2021)

13

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:

1. use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
2. use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
3. falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
4. use bots or other automated methods to access the content or redirect messages
5. override any security feature or exclusionary protocol; or
6. share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at

onlineservice@[Link]