512 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO.

3, MARCH 2021

Deep Learning for EMG-based Human-Machine

Interaction: A Review
Dezhen Xiong, Daohui Zhang, Member, IEEE, Xingang Zhao, Member, IEEE, and Yiwen Zhao

Abstract—Electromyography (EMG) has already been broadly intramuscular EMG (iEMG), according to the electrodes’
used in human-machine interaction (HMI) applications. location. The former is collected from the surface of human
Determining how to decode the information inside EMG signals
robustly and accurately is a key problem for which we urgently skin, while the latter is collected from needle electrodes
need a solution. Recently, many EMG pattern recognition tasks planted inside the human muscle. sEMG has been widely used
have been addressed using deep learning methods. In this paper, for hand gesture classification [2], [3], silent speech recogni-
we analyze recent papers and present a literature review tion [4], [5], stroke rehabilitation [6], [7], robot control [8],
describing the role that deep learning plays in EMG-based HMI.
[9], and other applications, mainly because it is cheap and
An overview of typical network structures and processing
schemes will be provided. Recent progress in typical tasks such as easy to collect and it provides a method for more natural
movement classification, joint angle prediction, and force/torque human-machine collaboration.
estimation will be introduced. New issues, including multimodal Many approaches, such as video, inertial measurement units
sensing, inter-subject/inter-session, and robustness toward (IMU), and EMG, can be used to decode the movement
disturbances will be discussed. We attempt to provide a
comprehensive analysis of current research by discussing the intention of humans. The video-based method requires
advantages, challenges, and opportunities brought by deep relatively higher computational resources, and it can be easily
learning. We hope that deep learning can aid in eliminating affected by environmental factors such as light change,
factors that hinder the development of EMG-based HMI systems. background noise, and camera position. The IMU-based
Furthermore, possible future directions will be presented to pave method can estimate joint angles while moving with high
the way for future research.
precision. For example, the Noraxon motion capture system1
Index Terms—Accuracy, deep learning, electromyography (EMG), estimates the human joint angle using an IMU attached to the
human-machine interaction (HMI), robustness.
body. However, it has a larger time delay compared with
I. Introduction EMG signals, which occur approximately 50–100 ms earlier
[10], before the action happens. Moreover, it is invalid under
LECTROMYOGRAPHY (EMG) is the recording of
E electric signals generated during muscle contraction.
EMG contains a large amount of information and reflects the
some conditions, such with rehabilitation training of patients
after stroke or prosthetic hand control of amputees, because it
cannot predict actions when the limbs do not move. In
movement intentions of a subject. EMG can be viewed as the
contrast, an EMG provides a method for obtaining a more
summation of the motor unit action potential (MUAP) with
natural and fluent human-machine interaction (HMI) that
noise, and can be decomposed into motor unit (MU), which
reflects human intent physiologically.
are the minimum entity of the human muscle [1]. It can be
In [11], a review of EMG pattern recognition algorithms
classified into two classes, i.e., surface EMG (sEMG) and
was presented. According to this paper, the typical EMG
Manuscript received August 4, 2020; revised October 29, 2020; accepted pattern recognition pipeline can be divided into three
November 19, 2020. This work was supported in part by the National Natural substages: 1) Preprocessing. The EMG data will be filtered to
Science Foundation of China (U1813214, 61773369, 61903360), the Self-
planned Project of the State Key Laboratory of Robotics (2020-Z12), and remove noise and keep the useful information unchanged. 2)
China Postdoctoral Science Foundation funded project (2019M661155). Feature extraction. Time, frequency, or time-frequency
Recommended by Associate Editor Hui Yu. (Corresponding author: Daohui domain features will be extracted for intention recognition. 3)
Zhang and Xingang Zhao.)
Citation: D. Z. Xiong, D. H. Zhang, X. G. Zhao, and Y. W. Zhao, “Deep
Classification or regression. Feature extraction is of vital
learning for EMG-based human-machine interaction: a review,” IEEE/CAA J. importance because it determines the ceiling of the
Autom. Sinica, vol. 8, no. 3, pp. 512–533, Mar. 2021. recognition performance, which leads to a rise in feature
D. Z. Xiong is with the State Key Laboratory of Robotics, Shenyang engineering, which aims to provide a feature set that is
Institute of Automation, Institutes for Robotics and Intelligent Manufacturing,
Chinese Academy of Sciences, Shenyang 110016, and also with the optimal for representing the information from EMG to achieve
University of Chinese Academy of Sciences, Beijing 100049, China (e-mail: better performance. Nevertheless, it is a very time-consuming
xiongdezhen@[Link]). task that requires professional knowledge to find the optimal
D. H. Zhang, X. G. Zhao, and Y. W. Zhao are with the State Key
Laboratory of Robotics, Shenyang Institute of Automation, Institutes for
feature set, which thus promotes great interest in deep
Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, learning.
Shenyang 110016, China (e-mail: zhangdaohui@[Link]; zhaoxingang@[Link]; Deep learning belongs to representation learning, which
zhaoyw@[Link]).
aims to create a better representation from input data using
Color versions of one or more of the figures in this paper are available
online at [Link]
1 [Link]
Digital Object Identifier 10.1109/JAS.2021.1003865

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 513

multiple layers of processing blocks such as neural networks overview of general processing procedures will be provided
[12]. It has achieved many benchmark achievements in and compared with traditional methods.
computer vision [13], speech recognition [14], machine 2) A thorough review of EMG-based HMI tasks will be
translation [15], and so on. Unlike machine learning-based introduced. We will talk about three tasks, namely, discrete
algorithms, which need to extract features from input data for movement classification, joint angle estimation, and
classification or regression tasks, deep learning can extract force/torque estimation.
high-level abstract features automatically from input data 3) New topics, such as inter-session/subject, electrode shift,
while using multiple hidden layers. The whole process is multimodal sensors fusion, will be discussed to convey the
usually end-to-end, which is quite convenient in multifarious latest progress. Applications in physical systems will also be
applications. considered.
In recent years, the deep learning-based scheme has been 4) The advantages, challenges, and opportunities to solve
widely used in EMG recognition. Many pieces of research questions in EMG recognition through deep learning will be
have applied deep neural networks for EMG processing. summarized in Section VI. Moreover, four future directions
There are some survey papers related to deep learning-based that we believe are important for future development will be
EMG pattern recognition tasks. Buongiorno et al. [16] wrote a covered.
brief survey about deep learning in EMG processing, which The remainder of this article is organized as follows:
included tasks such as hand gesture recognition, sleep stage Section II covers the basic knowledge of deep learning-based
identification, speech, and emotion classification. This paper decoding approaches. Section III introduces EMG-based tasks
mainly focuses on EMG-based classification applications. that can be addressed with deep learning methods. Section IV
Phinyomark et al. [17 ] discuss the problem of EMG presents new hot-spot topics in this field. Section V covers
processing under the rapid development of big data and deep applications in real systems. Section VI presents a discussion
learning. Faust et al. [18] review deep learning in health-care that is relevant to the main issues in this paper. Section VII
applications with biomedical signals, including EMG, EEG gives the conclusions of this article and the prospects for
(Electroencephalogram), ECG (electrocardiogram), and EOG future work.
(electrooculogram). Mahmud et al. [19 ] summarize the
application of deep learning methods, reinforcement learning II. Basic Knowledge and Scheme
methods, and deep reinforcement learning in the biological This section discusses deep learning-based EMG
field with biomedical signals including EEG, ECG, and EMG. recognition procedure, which mainly includes three parts:
Deep learning has been broadly used in biomedical signal deep neural networks widely used in EMG decoding, normal
pattern recognition in fields including EEG, ECG, EMG, etc. processes of EMG preprocessing, and the whole scheme.
For the issue of deep learning-based EMG pattern recognition,
previous reviews, including [16]–[19], mainly concern A. Basics of Deep Learning
movement classification tasks. Other sub-areas including Neural networks have a long history that can even be traced
continuous angle estimation, force/torque estimation, back to the 1940s. Since then, many new network structures,
multimodal sensing, inter-session/subject, robustness, and such as multiple layer perceptron (MLP), recurrent neural
applications are not concerned. One of the motivations of this networks (RNN), and convolutional neural networks (CNN),
paper is to provide a comprehensive map of current research have been proposed for fitting the input data with the
involving deep learning-based EMG recognition for HMI corresponding labels. Recently, deep neural networks have
tasks. Another motivation is to distinguish the role of deep shown outstanding performance in many research areas, as
learning in EMG-based HMI tasks from other biomedical described in [12]. They can be used to classify objects into
signals like EEG or ECG to analyze the benefits deep learning corresponding types or regress data into continuous sequences
brings to us in EMG-based HMI and how it can help us in the through an end-to-end method without feature extraction and
future. selection. This section will introduce deep neural networks
This paper attempts to provide a comprehensive review of that are usually used in EMG processing. A few types of
deep learning in EMG pattern recognition for human-machine neural networks, including CNN, RNN, autoencoder (AE),
interfaces. By illustrating typical applications, such as deep belief network (DBN), and mixed structures, will be
movement classification, joint angle prediction, and so on, this introduced in brief. Furthermore, deep transfer learning, which
study attempts to present penetrating analyses of the functions shows great potential in EMG decoding, will be presented.
of deep learning in EMG-based HMI. It also analyzes the As is known, the networks mentioned above have a long
challenges and the corresponding solutions to make up for history since they were first proposed. They are basic
disadvantages, and discusses the chance that it will provide us components of deep learning, but they are not equal to deep
more stable and accurate HMI systems. The primary learning. The notion of “ deep learning” originates in 2006
contributions of our article can be summarized in the when Hinton et al. [20] proposed a novel and fast method for
following parts: training deep belief networks using unsupervised greedy
1) The general scheme and basic knowledge needed for training methods. Deep learning involves the learning of high-
deep learning in EMG-based HMI will be introduced. Typical level characterizations of input data using multiple hidden
processing schemes, frequently used network structures, and layers. The deeper a network is, the larger the number of
preprocessing methods will be introduced in general. An hidden layers it includes.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
514 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

1) CNN: LeCun et al. [21 ] proposed the CNN for the first Ot Ot − 1 Ot Ot + 1
time in the 1980s, which was used to classify handwritten
digits. Many deep learning models, such as VGGNet, LeNet, ht ht − 1 ht ht + 1
AlexNet, and the Google Inception series, have been designed Unfold
based on the CNN. It usually consists of two operations: Xt Xt − 1 Xt XSt + 1
convolution and pooling. Multiple filters are used for the
convolutions to extract edges, corners, or other high-level
Fig. 2. The structure of the RNN.
features from an image automatically. Then, a pooling layer
follows, such as max-pooling, which selects the largest this question, the long-short term memory (LSTM) [23]
number in a box, which aims at keeping the most significant network, which contains the forget gate, was introduced. The
features of the original picture while decreasing the number of forget gate can determine the proportion of preceding
input dimensions. After undergoing a few layers of information that should remain or be thrown away. In
convolution and pooling, the abstract features extracted by the addition, gated recurrent units (GRU) [24] have a similar
CNN will be used for tasks such as classification through effect as the LSTM, but they require less computing cost. A
several fully connected layers and an output layer. CNN is not normal RNN predicts only the output of a specific moment
the same as MLP, which is composed by fully connected based on the information from the past, but the current output
layers, where every neuron connects with all of the neurons in can also be associated with the input in a future moment. The
the next layer. CNN has only local connections among the bidirectional RNN (Bi-RNN) [25] can resolve this question by
neurons and adjacent layers, instead. It also shares the same stacking two RNNs in the forward and backward directions
parameters for different parts of the image. These together, to make decisions that are based on not only
characteristics save on the number of parameters and thus previous states but also future states.
promote more efficient training. An example of 2D-CNN Unlike the CNN, which normally views EMG as an
based image classification is shown in Fig. 1 . In addition to “image”, RNN takes the EMG data as a sequence. It can
the 2D CNN, there is 3D or 1D CNN, which can be used for obtain information among the adjacent inputs. The EMG is
handling a 3D spatial array or a 1D sequence, respectively. biologically time-dependent, which implies that the temporal
information extracted by the RNN can also be used for
Cup
Pen
intentional recognition.
Fu

3) AE: The auto-encoder (AE), which was initially proposed

l

Tea
ly lay
co er

Dog in 1987, is the first type of neural network that benefits from
nn
ec

unsupervised pre-training [26]. AE has been used in fault

ted

Pooling detection [27], medical image processing [28], and other

Convolution Pooling Convolution
applications. It contains two parts: an encoder and a decoder.
Fig. 1. The structure of a CNN. The input data of the encoder is usually the same as the output
label of the decoder. This type of network attempts to ensure
CNN is of vital importance for EMG decoding using deep that the differences between the input data and output labels
learning methods. Most research decodes human intention are minimized with loss functions, such as mean square error
with an “EMG image” using a CNN. The features learned by (MSE). The procedure is unsupervised, and can learn the
the CNN have resulted in state-of-the-art performance for structure of the input data without the corresponding labels.
EMG recognition. By adjusting the network structures, a After the pre-training process, the encoder will remain for a
better result can be accomplished. further operation, such as being stacked with an output layer
2) RNN: The original RNN is called the Elman network, and fine-tuned by the back-propagation algorithm or as a
which was presented by Elman et al. [22 ] in 1990. Ordinary feature extractor combined with machine learning algorithms
neural networks fit input data to their labels individually, and for data pattern recognition. Fig. 3 illustrates the simple
they are not concerned with the relationships between the structure of the AE. The number of neurons at the top of the
different individual input data instances. The RNN was encoder is usually less than the input, which will lead to a
proposed to model the temporal information inside a decline in the data dimension. Sometimes, the AE is used for
sequence, especially the relationship of the current input and data dimension reduction, similar to principal component
former input. It is composed of an input layer, an output layer, analysis (PCA). There are many variants of AE, such as sparse
and hidden layers, similar to the MLP. The unusual aspect is auto-encoder [29] and denoising auto-encoder [30].
that the current nodes of the hidden layers are connected with The AE can be used to extract hidden information inside an
the former nodes. The structure of the RNN is illustrated in EMG to obtain better performance. It can be used for feature
Fig. 2, in which the current input Xt together with the state of extraction from raw data or for feature mining from hand-
the previous hidden layer St–1 will be sent into the current crafted features. The features stand for the inherent
hidden layer. Thus, the information between the input information of the EMG data, which has nothing to do with
sequences can be learned by the network. the target labels.
One drawback is that the RNN cannot remember content 4) DBN: Before the introduction of the DBN, we will talk
very long due to gradients disappearing or exploding. To solve about the restricted Boltzmann machine (RBM), which is the

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 515

Encoder network will be fine-tuned by the back-propagation method to

Input Decoder Output
fit the corresponding labels. The DBN has many applications,
such as health diagnosis [31], speech recognition [32], and
natural language processing [33].
Similar to the AE, the DBN attempts to learn a better
distribution from the EMG data without prior knowledge of
the corresponding tasks. Both raw data and EMG features
have been used for EMG analysis.
5) Mixed Network: There are various deep neural networks
that are geared to mixed networks that are composed of the
networks enumerated above that have displayed good
performance in EMG pattern recognition. The outcome of
mixed networks is ordinarily better than for one structure
Hidden layers alone because higher dimensional EMG features can be
extracted.
Fig. 3. Structure of the auto-encoder network with three hidden layers. Combining two types of networks, such as CNN stacked
with RNN, is the most popular mixed structure used in EMG-
basic component of the DBN. The RBM belongs to a special based applications [34]–[36]; they can extract both spatial and
type of Markov Random Field, which is comprised of two temporal information of EMG data at the same time. Network
layers: the visible layer and the hidden layer, as depicted in 1 is connected to network 2, where the output of network 2
Fig. 4 (a). The connections among the two layers of the RBM will be placed into several fully connected layers, which can
are bidirectional, while no neuron connections exist inside the create a fusion of extracted features for classification or
visible layer or hidden layer. The weights and bias of the regression, as described in Fig. 5 (a). Analogous research that
RBM are usually trained by a contrastive divergence learning combines deep networks with machine learning methods [37],
method iteratively. The RBM is an unsupervised machine [38] can also be illustrated similarly. Other network structures,
learning method that can reconstruct data without a predefined such as a dual-stream network [39], [40] or multi-stream
label, which is similar to AE. It can extract a better network [41], [42], assemble several blocks for feature
distribution of the input data, and thus, it can be adopted to extraction and combine features to make a final decision. The
pre-train deep neural networks. input data will then be placed into all of the sub-networks, and
The DBN is cascaded by multiple RBMs, as shown in the results are fused by fully connected layers. A
Fig. 4 (b). The Network will be trained layer-to-layer using a diagrammatic sketch of this structure is shown in Fig. 5 (b).
greedy algorithm. For each RBM, the hidden layer will turn
into the visible layer for the next RBM, which means that the

Fully connected
former output layer will be the next input layer. After several

Output
layers of stacking, an output layer such as softmax will be EMG Network 1 Network 2
added onto the top of the stacked RBMs. In the end, the entire Input

Hidden (a)
layer

Visible Network 1
layer
(a)
Network 2
Output
Fusion

Output
EMG
…

Input
Hidden 3
RBM 3
Network n
Hidden 2
(b)
RBM 2
Fig. 5. Mixed structures of deep neural networks. (a) is the structure in
Hidden 1 series; (b) is the multi-stream network if n > 2, and dual-stream network if
RBM 1 n = 2.

Input These mixed networks usually show better performance

(b) than single type. More hidden information in the EMG data
can be separated from these complex structures. However, the
Fig. 4. Structure of the DBN. (a) is the structure of the RBM; (b) is the main shortcoming of this approach is that it contains too many
structure of the DBN stacked by three RBMs and an output layer. parameters, which results in a very high computational cost.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
516 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

6) Deep Transfer Learning: The applications of machine movements under the guidance of prompt information. For
learning algorithms are usually under the assumption that the joint angle estimation tasks, spatial movement capture
training dataset and the testing dataset are analyzed within the systems, such as the Vicon motion capture system5 or IMU
equivalent feature space, and their feature sets obey together with EMG electrodes, will be adopted to obtain the
indistinguishable probability distributions. Nonetheless, this angles of human joints and EMG signals at the same time. For
situation is almost impossible for EMG recognition because of force/torque estimation tasks, a system that can record the
the many interchangeable factors under various scenarios. forces implemented by human muscles and EMG signals will
Transfer learning, which attempts to solve new questions with be designed for simultaneous EMG-force data acquisition.
the help of the knowledge learned before, permits different These systems capture EMG signals with their predefined
distributions of the source dataset and target dataset, even for labels, such as joint angle or force, synchronously for human
different tasks [43]. It is usually used under the assumption intention prediction.
that the source problem is the same as the target problem to 2) Filtering and Segmenting: The raw data contains a large
some extent. For example, the knowledge that helps to amount of noise, which makes filtering necessary. The
classify a dog from a cat can also help to distinguish an bandpass Butterworth filter with different bounds, such as
airplane from a car. 5–500 Hz [51], [52], 10–450 Hz [53], or 10–500 Hz [54], are
Deep transfer learning attempts to improve transfer learning often used for rejecting environmental noise. Other studies
using deep neural networks. This learning method can be select high-pass filters together with a low pass filter for noise
classified into four categories, namely, instance-based transfer removal [55], [56]. Then, the data will be rectified, and a 50
learning, mapping-based transfer learning, network-based Hz/60 Hz notch filter will be used to dislodge the disturbance
transfer learning, and adversarial transfer learning, according of the power line. Nevertheless, any type of processing could
to [44]. It improves the applicability of deep learning, which is lose valid information from EMG data, which leads to that
exceedingly data-dependent, while retaining the ability to some studies feed raw signals into deep neural networks [34],
learn features using deep neural networks. Deep transfer [40], [57].
learning has been applied to object detection [45], [46], image After filtering, the sliding window method is usually
classification [47], [48], and so on. It also shows great selected to segment the EMG signals into a series of
potential for EMG pattern recognition under data-shift envelopes. A window with length W slides across the EMG
conditions caused by cross-subject, electrode shift, and so on, signals with a step length of T, which is depicted in Fig. 6 (a).
which will be further discussed in Sections V-B and V-C. To guarantee efficient real-time performance, the length of the
sliding window is usually within 300 ms [58]. Windowing is
B. EMG Signal Processing Approach the approach that is most often used in machine learning or
1) EMG Data Acquisition: One source of an EMG dataset is deep learning-based EMG pattern recognition schemes.
the publicly open datasets. There are several benchmark
datasets, such as Ninapro2, CapMyo [49], cls-hdemg (CSL) W t1 t2 t3 t4 t5
WW
[50], which have been extensively applied for assessing the
performance of the proposed algorithm by many researchers.
Ninapro is most likely the largest dataset for EMG-based hand
gesture recognition, which includes ten sub-datasets for now.
DB1 to DB7 are for hand gesture classification. DB8 is for
finger angle regression. DB9 is the kinematic data captured
with Cyberglove-II. The last one, named MeganePro, is a TT
multimodal dataset for prosthetics control. Amputees have (a) (b)

access to this project thus those datasets are of helpful to Fig. 6. Illustration of the segmentation method. (a) is the sliding window
improve the quality of their life. CapMyo and CSL are EMG method with window length W and sliding step T; (b) is the instantaneous
datasets that were captured by electrode arrays, and the signal EMG method with ti as the sampling time.
is in high density, called HD-EMG. These datasets are easy to
access and make it easy to evaluate the algorithms’ The instantaneous value of EMG signals, which can be
performance, thus contributing substantially to deep learning- interpreted by Fig. 6 (b), has also been proven to be effective
based EMG recognition schemes. for gesture classification [49], [59]. Geng et al. [59 ] showed
In addition to the publicly open datasets, self-made datasets that the instant EMG value can be used for hand gesture
are also widely used for performance evaluation. There are classification using a CNN for the first time. Their scheme
sensors that are normally used for EMG capture, such as the shows good performance in sparse channel EMG, such as
Myo armband3, Delsys Trigno4, and so on. For gesture Ninapro DB1 and DB2, or HD-EMG, such as CapMyo.
classification tasks, sensors will be attached to the skin to Compared with the sliding window-based scheme, it can be
acquire EMG signals while the subject performs various more natural and fluid for HMI due to having less time-delay.
Other studies [60], [61] attempt to estimate the limb angle
2 [Link] during movement with instant EMG data, which is down-
3 [Link]
4 [Link] 5 [Link]

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 517

sampled to 100 Hz. In general, it has been verified that instant preprocessing, feature extraction and selection, in addition to
EMG data contains effective information that can be used for pattern recognition. Bi et al. [77 ] classify EMG-based
EMG recognition directly. However, the performance of continuous motion prediction methods into model-based
single-frame data is relatively poor, and therefore, it usually techniques and model-free techniques. Farina et al. [78]
chooses the majority voting method as an assistant strategy summarize the ways to control prostheses with EMG signals
[59]. into the following classes: proportional control, pattern
The numerical value of EMG data is small, which is often recognition, direct neural control by EMG decomposition, and
regularized using normalization algorithms such as min-max multimodal sensor fusion.
normalization, z -score normalization, or conversion into a For the methods besides deep learning, there are two main
fixed range. In addition to these, the following methods are strategies that are usually used by researchers, including the
often used to construct a better format of the input data. machine learning-based method, which is often accompanied
3) Data Reconstruction Methods: The raw EMG data is by various pattern recognition algorithms, and the model-
relatively noisy, which makes it difficult for pattern based method, which chooses kinematics, dynamics, or
recognition. Unlike applications in computer vision, where musculoskeletal models for intention identification. The
raw data is put into the networks and decoded end-to-end, machine learning-based method has been widely adopted to
EMG signals are regularly reconstructed into new formats. recognize the patterns inside EMG signals. This approach is
The distinction between data reconstruction and feature depicted in Fig. 7 (a). The model-based approach is popular in
engineering is that the former is normally in a 2D format, continuous movement estimation of the joint angle, force,
while the latter is typically in a 1D format. The structure of torque, and so on. The whole process of the model-based
the reconstructed EMG is similar to raw EMG data, while the strategy is described in Fig. 7 (b). Compared with the machine
feature extraction method reduces the dimension of input data. learning-based method and the deep learning-based method, it
The following formats are often chosen for EMG conversion: models the relationship between EMG and human motion
a) Time Domain: The original data after processing contains using musculoskeletal, kinematics, or dynamics model [77],
information and can be used for pattern recognition directly. which needs accurate representation of human limbs. The
Some papers [59], [62], [63] convert it into gray-scale images processing step contains preprocessing, and feature extraction
with the value range of [0, 255]. [77], which is similar to the machine learning method, and
b) Frequency Domain: The Fourier transform (FT), fast thus they share the same drawbacks of feature engineering.
Fourier transform (FFT), and discrete Fourier transform The model-based method requires more prior knowledge
(DFT) are often used to obtain the spectrum of the about the human limb and involves more complicated
preprocessed EMG signals, which could reflect the amplitude parameter identification than the machine/deep learning
at different frequency levels [64]–[66]. method, which limits its application.
c) Time-Frequency Domain: Approaches such as the
wavelet transform (WT), continuous wavelet transform EMG data EMG data EMG data
(CWT), wavelet packet transform (WPT), discrete wavelet
transform (DWT), and short-time Fourier transform (STFT) Pre-processing
Processing Processing
could abstract time-frequency domain features of EMG
signals for further operation [67]–[70]. This approach is more Feature extraction
informative than using time-domain features or frequency Model Deep neural networks
domain features, although it is more time-consuming. Machine learning
d) Others: Some research [41] constructs the EMG image
Result Result Result
using human-designed features. A few researchers choose
(a) (b) (c)
classical features directly to feed into deep neural networks
[71]–[73]. Moreover, there are new formats, such as fused Fig. 7. Process of EMG-based human intention recognition. (a) is the
time-domain descriptors (fTDD) [74], [75] and Hilbert space- machine learning-based method; (b) is the model-based method; (c) is the
filling curves [76], which are used for next step processing. deep learning-based method.
These reconstruction methods could project EMG data into
a more discriminant space, where different movements have a The deep learning-based method contains no complex
larger gap with one another. Although raw data can be used feature extraction or feature selection procedures, as shown in
for recognition with deep neural networks, a new format for Fig. 7 (c). Deep neural networks will be used for movement
input achieves better performance. The new format for input prediction after three steps of processing, as illustrated in part
data is worthwhile to explore because it can boost B of this section. This approach relaxes the demand for
discrimination performance. feature engineering and kinematic modeling, which brings
about new options in eliminating the original faults with
C. Algorithm Schemes Comparison traditional approaches.
For the EMG pattern recognition strategy, there are several
partition criteria according to different standards and III. Human-Machine Interface With Deep
application standards. Simão et al. [11] partition the machine Learning and EMG
learning-based approach with three main procedures: In this section, we will discuss EMG-based HMI tasks with

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
518 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

deep learning as a recognition technique. First, we discuss An issue is that deep CNN models have too many
EMG-based discrete movement classification questions. The parameters, which could cause problems for real-time
movements are predefined into several classes of postures, applications. Chen et al. [67] propose a compact CNN model
including being open-handed, having a fist, or relaxed. named EMGNet that has fewer parameters but better
However, this is not convenient for tasks such as upper limb classification accuracy, which is a benefit for online working.
motion prediction, which requires us to have knowledge of an Similar research is addressed [53], which proposes an
accurate position of the arm in real time. A more natural and embedded CNN to decode the HD-EMG signals on a single
smooth interaction approach that estimates the joint angle embedded system. A filter kernel with size 3 × 3 or 1 × 1 is
continuously through EMG will be presented in the next selected in most CNN-based networks. However, a filter with
section. Next will be the estimation of force/torque during size m × n, where m is not equal to n, exists in some studies
body movement, which is similar to joint angle prediction. [40], [85]. There is a narrow gap between the length and width
Other tasks will be discussed briefly at the end. of the EMG image, which leads to the idea of larger and
Generally speaking, deep learning-based EMG recognition “thinner” filters.
assignments can be split into two types: classification tasks The RNN, which is usually selected to process temporal
and regression tasks. The typical examples of classification information for tasks such as natural language processing has
questions are hand gesture classification, body movement also been applied to this question [57], [72], [86]–[88]. Nasri
discrimination, sign language processing, and so on, which et al. [57 ] propose a GRU-based scheme to process EMG
predefine the label into several classes. Regression questions segments by the sliding window method, and the accuracy
such as real-time joint angle, force, and torque estimation, reaches 77.85% for 6 gestures performed by 35 subjects. Koch
where the label varies in a fixed range in real-time. et al. [72 ] present a scheme using an RNN with novel
weighting loss. The features of the EMG are extracted, and the
A. Movement Classification performance outperforms the most up-to-date approaches
The human hand is of great significance in our daily life. using 3 types of datasets. Simão et al. [86] process the single
Hand gesture recognition is the most common task of EMG- frame EMG with several types of deep learning methods, such
based HMI, and hence, various datasets for hand gesture as RNN, LSTM, and GRU, to extract temporal information
classification, such as Ninapro and CSL, are presented for inside the EMG. Samadani et al. [87 ] compare several
performance evaluation. Multifarious deep learning methods optimization methods, including the bidirectional recurrent
have been widely employed for this task. layer and attention mechanism, together with a step-wise
CNN is often selected for gesture classification, and it learning rate. The highest accurary achieve is 86.7% for 18
normally views EMG signals as an image. Park et al. [79] gestures with the Ninapro DB2 using bidirectional LSTM (Bi-
chose CNN to classify hand gestures of different users, and LSTM). Alfaro-Ponce et al. [88] compare the performance of
the results show better performance compared with support the time-delay neural network (TDNN), differential neural
vector machine (SVM) for both adaption and non-adaption network (DifNN), and complex-valued neural network
conditions. This study is the first time that deep learning was (CVNN) for two different physiological signals, including
used in EMG-based HMI tasks [80], [81]. Afterward, more EMG and foot pressure of the gait for Parkinson disease (PD)
researchers paid attention to this field. Atzori et al. [82] patients, and the accuracy of all three networks was greater
evaluate the performance of a simple CNN on EMG data of than 95%.
Ninapro DB1, DB2, and DB3, which contains approximately In addition to the CNN and RNN mentioned earlier, which
50 hand gestures collected from 67 healthy subjects and 11 belong to the supervised learning approach, unsupervised deep
amputees in total. The performance of CNN is superior to the neural networks, such as auto-encoder and deep belief
average accuracy of the traditional methods but worse networks, have also been used for movement classification.
compared with the result achieved by SVM. This finding The AE-based schemes can be classified into two types
implies it is possible that better results can be achieved with a according to the input format: hand-crafted feature-based
larger network for computer vision and object recognition methods [89] and raw data-based methods [90]. Rehman et al.
tasks. [89] apply stacked sparse auto-encoders (SSAE) to multiday
Olsson et al. [62 ] propose a CNN-based multi-labeled EMG recordings to improve the performance. The results of
classification scheme with HD-EMG as the input. The multi- the SSAE is better than linear discriminant analysis (LDA) for
label methods express complex movements as a summation of both intact and disabled subjects with four time-domain
multiple simpler movements. In this paper, 16 independent features. Rehman et al. [90 ] compare the hand gesture
movements are used to model the state of the hand. The classification performance of CNN, SSAE, and LDA. They
accuracy reaches 78.7% in 14 healthy subjects. Zhai et al. [83] also evaluated the SSAE with raw EMG signals and time-
propose a CNN-based strategy with self-recalibrating capacity domain features as inputs separately, where the latter achieves
that maintains steady performance against time-changing better performance.
without retraining, where the accuracy is 10.18% higher than DBN usually takes hand-crafted features as the input [73],
the uncalibrated classifier for 50 hand movements. Chen et al. [91]. Shim et al. [73 ] propose the split and Merge DBN,
[84] employ a 3D CNN for HD-EMG-based gesture which chooses the genetic algorithm to augment the
classification, and the result outperforms the instant EMG- performance of the DBN; the precision outperforms classical
based method. However, the computation cost is higher. DBN by 12.06%. Zhang et al. [91 ] recognized normal and

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 519

aggressive EMG signals, and each contained 10 actions using approach is a new way for severe amputees to control
DBN with time-domain feature sets. The best accuracy prosthetics without targeted muscle reinnervation (TMR) [93],
reached is 90.66 ± 1.47%. Additionally, Sun et al. [55] although convenience is a problem that is of concern.
proposed a novel method using a generative flow model Except for hand gestures, other movements can also be
(GFM), which belongs to a unsupervised learning methods recognized with the analogical procedure. Shao et al. [94]
similar to the DBN, for converting EMG data into factorized present a scheme for 12 upper limbs’ motion recognition with
features and applying the features for EMG classification single-channel EMG. The spectrum acquired by the FFT is
using a softmax classifier. first decomposed by the singular value decomposition (SVD)
The above methods apply a single type of deep learning method and then processed by wavelet deep belief networks
method for hand gesture prediction. There are mixed network (WDBN). Other tasks, such as gait stages classification [95],
structures, such as multi-stream networks, chosen for this task. wrist motion recognition [68], [96], and arm motion prediction
Ding et al. [40 ] handled the hand gesture classification task [97], [98], can also be addressed by deep learning, which has
using a parallel multiple-scale convolutional neural network. no obvious difference except for the label name compared
Wei et al. [42] decomposed EMG into multiple streams, and a with the hand gesture classification tasks. For detailed
CNN with multiple sub-streams is used for gesture information on typical movement classification, the relevant
classification. Both of the structures are as depicted in Fig. 5 (b). papers are summarized in Table I.
Mixed architectures that are connected as in Fig. 5 (a) have
been widely used for EMG-based gesture classification B. Joint Angle Estimation
[34]–[36], [39]. Gao et al. [34] proposed a dual-flow network Estimating human motion intentions continuously
that uses the CNN and LSTM individually to extract the EMG demonstrates good potential for human-robot interaction in
features simultaneously; then, the features are fed into fully scenarios such as exoskeleton robot control. The discrete
connected layers for classification. Wu et al. [35 ] propose a recognition of predefined gait stages can lead to disastrous
system based on CNN and LSTM with the attention results, such as falling, if the intention is badly decoded.
mechanism for hand gesture classification with CWT of EMG Another problem occurs with movement switching as most
as the input. Xie et al. [36] combine CNN with LSTM into a studies choose the stable section of the EMG signals and
unified structure, and an accuracy of 98.14% is achieved. neglect the switching section, which limits the application
Tong et al. [39 ] combine 3 layers of CNN with 3 layers of even with high decoding accuracy. These errors can be
RNN for hand gesture classification. avoided because the motion can be adjusted by the feedback
Tsinganos et al. [80 ] outperform state-of-the-art perform- of human vision or tactile sensation. Thus, it is a safer and
ance by 5% on Ninapro DB1 with temporal convolutional more advantageous method for man-machine interaction.
network (TCN). However, this technique chooses the whole The method for continuous movement recognition can vary.
section of the EMG data instead of the envelopes under 300 There have been approaches based on models such as the
ms as the input. Zanghieri et al. [81] developed a TCN-based polynomial model [101], state-space model [102], [103], and
network named TEMPONet, which runs on an embedded machine learning approaches, such as support vector
system. The performance reaches 49.6% on a Ninapro DB6, regression (SVR) [104], random forest regression [105], and
which outperforms the current state-of-the-art method by neural networks [106]. Deep learning-based joint angle
7.8%. prediction fits the EMG toward joint angles without
Some studies associate deep learning with machine learning background knowledge about muscle physiology. It can be
methods, in which the latter is used to elevate the decision selected for angle prediction of various body positions, such
performance [37], [38]. Shen et al. [37 ] proposed a scheme as the wrist, hand, upper limb, and lower limb. Performance
using a CNN and a stacking ensemble learning algorithm with measure regulations, such as mean square error (MSE), root
three types of inputs, including EMG data, DFT of EMG data, mean square error (RMSE), and coefficient of determination
and discrete wavelet packet transform (DWPT) of EMG data. (R2), are summarized in [77].
The CNN works as a low-level classifier, and the results are Most studies estimate hand gestures as predefined actions
optimized by an ensemble learning-based secondary classifier. because the human hand is dexterous, which makes it difficult
Chen et al. [38] propose a novel technique with typical CNN for continuous angle estimation. There are studies [107] that
networks whose output layer is replaced by machine learning estimate the continuous hand movements under the conditions
methods, including SVM, LDA, and K-nearest neighbor of both mobile and non-mobile wrists using the RNN with
(KNN). All three methods outperform the traditional feature- simple recurrent unit cells. Adversarial domain adaption is
based method under the conditions of inter-subject/inter- used to improve performance. Teban et al. [108 ] estimate
session, which shows that features obtained by the CNN are finger angle in the form of the flex angle of fingers using the
efficient for human intention recognition. RNN with LSTM cells to provide a flexion reference for a
EMG can be used for robot hand control for amputees. prosthetic hand.
However, it is difficult for high-level amputees whose Ameri et al. [51] decode 2 DOF (degree of freedom) wrist
muscles are not strong enough for EMG-based multi-action movements using the Fitts’ law test with a regression CNN
classification. Therefore, Lee et al. [92 ] recognize foot that has 8 convolution layers. This outperforms the support
postures based on EMG acquired from the lower limb and vector regression (SVR) based method with five EMG
map the foot postures to hand gestures by the CNN. This features as input. It shows that deep neural networks have

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
520 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

TABLE I
Typical Studies of EMG-based Movement Classification Using Deep Neural Networks
Participants Channel Electrodes
Task Input Category Networks Accuracy Reference
(default healthy) number position
EMG data Ninapro DB2 Part A – – Two stream CNN 75%–84% [40]
Ninapro DB1–7; Ninapro: 80%–90%
Feature image – – Multi-view CNN BioPatRec: 94% [41]
BioPatRec DB1–4
EMG data 1 8 32 Forearm Embedded CNN 98.15% [53]
EMG data 35 6 8 Forearm GRU 77.85% [57]
37 7 8 Forearm 98.81%
CWT Ninapro DB5: A, B, C, separately 16 Forearm Compact CNN 61%–70% [67]
Feature sets 28 5 2 Forearm DBN 88.00%–89.29% [73]
DB1: 66.59%
EMG data Ninapro DB1, DB2, DB3 – – CNN DB2: 67.27% [82]
DB3: 38.09%
DB2: 78.71%
Hand gesture FFT Ninapro DB2, DB3 – – CNN DB3: 73.31% [83]
classification (10 movement)
18 8 128 Forearm Over 95%
EMG data 3D CNN [84]
5 27 168 Forearm 61%–91%
EMG data; CNN: 90%+SSAE:
7 (15 days) 8 8 Forearm CNN;SSAE [90]
feature sets 75%–98%
3 (10 days) 9 8 Forearm 93.7%
EMG data TCN 49.6% [81]
Ninapro DB6 14 Forearm
Ninapro DB1 10 Forearm 66.9%
EMG data CNN+ML 68.23% [38]
Elonxi DB 18 Forearm
Ninapro: 82.2–97.6%
Six image Ninapro DB1–2,
Attention based BioPatRec: 90%+
representation BioPatRec26MOV, – – [99]
CNN-RNN CapMyo: 99.0%+
methods CapgMyo-DBa, CSL-HDEMG
CSL: 94%+
Forearm/
Foot posture EMG data 15+1 (amputee) 8 16 CNN 91.3% [92]
shank
Arm posture FFT 8 12 6 Right arm WDBN 97.7% [94]
Gait
subphase EMG data 3 (five conditions) 4 4 Thigh LSTM 87%–94% [95]
classification
Lower limb
EMG EMG data Three datasets 4 – – CNN Over 98% [100]
classification

better representation abilities compared with feature- to construct the deep neural networks. The proposed method
extraction method. Bao et al. [56] estimate 3-DOF movement has less computational cost compared with the LSTM or
of the wrist using a spectrum image of the EMG. Several GRU. Gautam et al. [111] propose a scheme that combines the
machine learning-based methods are compared with the CNN with the LSTM together to classify lower limb
proposed CNN, and the performance demonstrates the movement and to estimate the angle of the knee joint
superiority of the CNN-based method. simultaneously. Transfer learning was used to transfer the
Chen et al. [54] establish an LSTM-based model for upper parameters learned during the angle estimation for movement
limb angle prediction. The inputs are time-domain feature sets classification. Instead of predicting the angle of a single joint,
of EMG, while the participants perform two types of Chen et al. [60] estimate the angle of the hip, knee, and ankle
compound tasks. Ren et al. [70 ] predict the upper limb joint with regard to the right leg simultaneously while those
angle of both arms using the multi-stream LSTM dueling participants walk at different speeds. The DBN is used to
(MS-LSTM dueling) model, which selects the LSTM and diminish the dimensions of the EMG signals, and its
convolutional LSTM (ConvLSTM) as two individual streams performance outperforms PCA; then, MLP is used for angle
of the model. This model combines spatial information with prediction.
temporal information in parallel, as depicted in Fig. 6 (b). Currently, there is no definite distinction between EMG-
Unlike [70], Xia et al. [109] estimate the angle of the shoulder based motion prediction and motion estimation in most
and elbow using the CNN consecutively combined with the studies. The former is usually used to predict the angle in the
RNN. future, while the latter only estimates the angles
Huang et al. [110 ] predict the knee joint angle during simultaneously when motion occurs. Because it is widely
walking using EMG combined with IMU data. A fully acknowledged that the EMG occurs earlier before the physical
connected RNN, with the relu activation function, is employed actions, motion prediction is a more reasonable interpretation.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 521

TABLE II
Deep Learning-based Continuous Angle Prediction
Channel Electrodes
Estimation tasks Input Network Standard Performance Reference
number position
Wrist motions with
online Fitts’ law EMG data 8 Forearm CNN 0.99 [51]
style test
3D wrist motion FFT 6 Forearm CNN R2 0.565–0.904 [56]
Hand position in 3D RCNN: 0.903±0.045
FT 5 Upper limb CNN, RCNN [109]
space CNN: 0.776±0.056
The angle of the EMG features 7 Upper limb LSTM, MLP LSTM: 6.1833±0.6583 [54]
shoulder and elbow MLP: 7.1547±0.6168
The angle of the Hip: 3.58±0.67
lower limb (hip, EMG data 10 Lower limb DBN + BP Knee: 3.96±0.69 [60]
RMSE
knee, ankle) Ankle: 2.45±0.57
(degree)
The angle of the 3D: 11.30
EMG data 5 Upper limb MLP, TDNN, RNN [61]
shoulder and elbow 2D: 14.74
Finger and wrist RNN (SRU cell vs Immobile wrist: 22–23
EMG data 8 Forearm [107]
angle GRU cell) with ADA Mobile wrist: 20–21
Finger flexion with RMSE (flexion
EMG data 8 Forearm RNN 8%–9% [108]
a range of 0–100% percentage %)
The angle of the Angles and angular velocity RNN (fully connected MSE
3 Lower limb 8.6 [110]
knee in the past, EMG RNN units) (degree)
The angle of the MS-LSTM dueling
STFT 16 Upper limb 1.11 [70]
shoulder and elbow model
MAE
Long-term recurrent (degree)
The angle of the Healthy: 8.1±1.2
EMG data 4 Lower limb convolution network [111]
knee Pathology: 9.2±1.5
(LRCN)

Moreover, it can relieve the effects of time delays, which Deep learning has been used for EMG-based force/torque
affects online performance. estimation. In [120], [121], a framework estimates that the
In general, the deep learning-based method attempts to force of the elbow was developed using HD-EMG as the
portray the interrelationships between the EMG signals and input. In [120], the raw data is filtered, segmented, and
the limb angles without prior knowledge about muscle normalized. Then, the dimensions are reduced using principal
structure, feature engineering, regression models, etc. This component analysis (PCA), and finally, the output is fed into
technique is more intuitive than the gesture classification the DBN for force prediction. In [121], the raw data is
approach. All of the detailed information relevant to the preprocessed, spatially filtered by PCA and, then, dimension
papers mentioned before about this task is included in Table II. reduced by nonnegative matrix factorization (NMF) to remove
the redundancy of the electrode array. Finally, it is constructed
C. Force/Torque Estimation to train three types of deep neural networks, including the
Research that studies the relationship between the EMG CNN, LSTM, and C-LSTM, and their results are compared.
signals and the muscle force have a long history that can be Unlike [120], [121], which choose an electrodes array to
traced back to 1952 [112]. The core dilemma of the EMG- record the EMG signals, Li et al. [122 ] choose the Myo
based muscle force prediction include precision and armband as the EMG capture sensor. The data is filtered,
representativeness [113]. There are model-based methods segmented, and then feature extracted for further operations,
such as [114], [115] that estimate the EMG-Force relationship and the dimension of the features is reduced by PCA; finally,
based on the Hill model [116]. It bridges the gap between the two layer stacked autoencoder (SAE) networks are used to
EMG and the muscle force through an explainable technique. divide the force into eight levels. The predicted force is placed
However, the parameters of the model are complicated, and it into a fuzzy controller to control a prosthetic hand.
requires special knowledge about human muscle, which is Yang et al. [123], [124] develop a system to map the force
similar to the angle prediction tasks. Furthermore, the of 3-DOF wrist motion toward the position of the cursor on
parameters are difficult to optimize due to individual the screen. A deep CNN is imposed for the 3-DOF wrist force
differences, sensor noise, electrode shifts, and so on. In regression task, with the raw EMG data as the input. In [125],
addition to the model-based method, machine learning has a system is designed to recognize the direction and the
also been used for this question, which maps EMG signals magnitudes of the diverse forces acting on a designed facility
toward force by regression algorithms such as linear by a hand simultaneously. In [126], Yokoyam et al. predict
regression [117], polynomial regression [118], support vector handgrip forces using MLP with multiple hidden layers using
regression [119]. However, it is limited by the same electrodes placed on the back of the hand. In [127], Chen et al.
drawbacks of feature engineering. estimate the force of the multi-DOF finger continuously with

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
522 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

TABLE III
Deep Learning-based Force Estimation
Electrodes
Estimation task Input Channel number Network Standard Performance Reference
position
Elbow flexion force HD-EMG 128 Right arm PCA+DBN About 0.82–0.92 [120]
Baseline: 0.833
3D wrist force EMG data 8 Forearm CNN Electrode shift: 0.744 [123]
Cross arm: 0.519
Cross subject: 0.564
R2
3D wrist motion EMG data 8 Forearm CNN 0.528–0.632 [124]
DOF1: 0.85±0.014
3D wrist torque EMG data 16 Forearm SAE DOF2: 0.86±0.073 [128]
DOF3: 0.86±0.059
3D wrist torque EMG data 16 Forearm SAE 0.829 ± 0.050 [129]
NMF+CNN/LSTM/
Elbow flexion force HD-EMG 128 Biceps brachii About 5%–11% [121]
C-LSTM
RMSE (%) Intra-session: 16.2%
Back of the
Handgrip force EMG feature 4 MLP Intra-subject: 21.4% [126]
hand
Inter-subject: 26.5%
Grasp force at different
EMG feature 8 Forearm PCA+SAE Accuracy Over 95% [122]
levels
Grasp force and force Accuracy; 95.1 ± 1.50%;
FFT 8 Forearm CNN [125]
direction NRMSE 6.71 ± 2.41
CNN: P < 0.05
Finger force EMG data 160 Forearm CNN+RNN; CNN p [127]
CNN-RNN: p < 0.002

HD-EMG data as the input. This study compares the 92.55% is achieved. Morikawa et al. [133] choose lips EMG
performance of the CNN and CNN plus RNN with classical for identity authentication with CNN. Khowailed et al. [134]
methods that are based on linear regression with channel detect the timing EMG that occurs using an RNN. Wang et al.
merging methods such as common spatial pattern (CSP) and [135] predict EMG data of the future using historical EMG
so on, while the CNN combined with the LSTM achieves the signals. Nodera et al. [136] successfully classify six forms of
best performance. resting needle EMG using several deep neural networks,
In addition, there is a study that estimates joint torque as in including VGGNet, ResNet, and Inception v3. Data
[128], [129]. In [128], Yu et al. estimate the torque of the augmentation and transfer learning techniques are also used
wrist continuously using a five-layer stacked auto-encoder for optimizing the result. Nam et al. [137 ] classify needle
(SAE) based deep neural network. The SAE plays the role of EMG using Inception v4, and an accuracy of 93.8% is
data dimension reduction and then, the fully connected layers achieved.
are for torque regression. In [129], the wrist torque is In general, the EMG is used to fit predefined target labels
estimated using high-density EMG signals with an SAE-based using deep neural networks, and thus, it can be used on
method, and the performance outperforms several machine various tasks with the defined labels. Their processing
learning-based methods. procedures have no obvious difference with gesture
In general, deep learning for muscle force/torque estimation classification or angle prediction tasks theoretically. The
is almost the same as continuous limb angle estimation. It is relevant papers have been listed in Table IV.
more convenient to predict muscle force using a regression
deep network without complicated models. The performance IV. Recent Hot-spot Issues
is comparable with state-of-the-art strategies. Detailed This section will introduce several hot topics in EMG-based
information about the relevant research articles is described in human-machine collaboration, including multimodal sensing,
Table III. inter-subject/session, and robustness toward disturbances,
which can contribute to building practical and stable muscle
D. Other Tasks computer interfaces.
In addition to the tasks mentioned earlier, other HMI tasks,
such as disease diagnoses [130], [131], fall detection [132], or A. Multimodal Sensing
personal authentication [133], can also be solved with deep The EMG signals can be easily affected for various reasons,
learning methods. Qin et al. [130 ] predict tremor severity and thus, it could be difficult to develop a reliable HMI
levels of Parkinson’s disease by EMG with a lightweight system using only EMG signals. Other modal information,
CNN named S-Net. Sengur et al. [131 ] classify amyotrophic such as IMU data or video stream, can help to improve the
lateral sclerosis (ALS) patients from a normal person using reliability of the online performance. Thus, combining
EMG with time-frequency representations as input. Liu et al. multiple modal sensors can provide a novel path for HMI.
[132] detect falling using the dual parallel channels of CNN We choose multimodal data because different modal inputs
with EMG spectral features as input, and an accuracy of contain different information that can compensate for each

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 523

TABLE IV
Other Applications
Electrodes
Tasks Input Electrodes position Network Classes Performance Reference
number
Tremor severity of
Parkinson’s disease Feature set 2 Bicep CNN 5 90.55% [130]
(PD)
Spectrogram, CWT, and CNN with
Amyotrophic lateral smoothed pseudo reinforcement
1 Not mentioned 2 96.80% [131]
sclerosis disease Wigner-Ville sample learning
distribution (SPWVD) strategy
Improved dual
Fall detection FFT 4 Lower limb parallel 4 92.55% [132]
channels CNN
Personal authentication EMG data 3 Face CNN 5 47.661% [133]
Neural muscle Simulated EMG – – RNN – – [134]
activation detection
RMSE:
DNN with 4
Missing channel 2.53–11.11
EMG data 7 Lower limb fully-connected – [135]
prediction SAE:
layers, SAE
9.15–12.20
VGG16,
Biceps brachii, first dorsal
VGG19, Original: 86%
Waveform Mel-spectrogram of interosseous, vastus
30 ResNet50, 6 Augmented: [136]
identification of resting needle EMG medialis, or tibialis
ResNet152, 100% (best)
needle EMG anterior muscles
Inception v3
Needle EMG data 1 Not mentioned Inception v4 3 93.8% [137]

other for better performance [138]. Determining how to fuse hand gesture classification, as shown in [41], [143], [144].
various sensors’ data to make a better decision is a tough Other research like [145] classifies dynamic postures from
question due to the heterogeneity gap between different input- static gestures using features of EMG combined with IMU,
modals. The machine learning-based method usually solves which shows better results than IMU or EMG alone.
the feature gap of different input-modals by two methods In addition to EMG and IMU, pressure data and video data
[138]: 1) Eliminating the correlations between the inputs; 2) could also be used for hand gesture classification. Zhang et al.
Projecting these features into a common subspace. The [146] recognize hand gestures using the EMG and IMU data
method needs to extract features from every kind of input, of the Myo armband together with pressure data captured by a
which needs the expert knowledge of every modal of data. smart glove. LSTM is chosen for gesture recognition with
The deep learning-based method can learn high-level hand-crafted features as input. Gao et al. [147] combine EMG
representations from each modal of input data, and the feature images with the RGB images and the depth images of human
gap can be eliminated by constructing a fusion layer [138]. hands captured by Kinect to construct five-channel images
The whole scheme is usually end-to-end, which does not that are for hand gesture discrimination through the multiscale
require complicated feature extraction and selection/proje- parallel CNN. Li et al. [148 ] mingle EMG data with
ction, and as a result, better performance can usually be kinematics data, which is captured by the CyberGlove II
achieved. motion capture system together for the gesture classification
For sign language recognition (SLR) tasks, two problems of task. Huang et al. [110] choose the angle and angular velocity
the traditional algorithms are inconspicuous subsections and of the past, together with EMG signals to predict the joint
the diversity of input data. A multimodal deep learning-based angle in the future.
framework that merges the information of EMG and IMU is With the combination of different sensors, the performance
widely adopted to enhance the recognition accuracy of sign can be improved compared with single modal data. Deep
language recognition [139]–[142]. Yu et al. [139 ] fuse the multimodal learning makes it easy to perform gesture
EMG, accelerometer (ACC), and gyroscope (GYRO) at the classification by an end-to-end approach that is more
data level, feature level, and classification level using a DBN, convenient and efficient than the machine learning-based
and the best accuracy achieved is 95.1%. Wang et al. [140] method. Table V gives detailed information on the relevant
fused three types of data by a Siamese network that is papers.
designed based on CNN, and the accuracy is over 94%. Shin
et al. [141 ] also chose a CNN for data fusion, and the best B. The Inter-Subject/Session Problem
accuracy achieved is 99.13%. Zhang et al. [142 ] propose a The inter-subject/session problem can lead to a sharp
mixed architecture comprised of CNN, Bi-LSTM, and decline in the precision of the previously trained model. Inter-
connectionist temporal classification (CTC), and the whole session means that the data for training and the data for testing
network is trained with an end-to-end method. In addition, do not belong to the same session but the same person, while
EMG signals combined with IMU data could also be used for inter-subject means that the model is trained by one subject to

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
524 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

TABLE V
Multimodal Sensor Fusion Tasks
Task Input Category Participants Network Accuracy Reference
User dependent: 95.1%
150 8 DBN [139]
EMG, accelerometer, gyroscope User independent: 88.2%
Sign language 86 20 CNN Over 94% [140]
recognition EMG, accelerometer, gyroscope, Word: 70, CNN+Bi- Word level: 93.7%
15 [142]
orientation Sentence: 100 LSTM+CTC Sentence level: 93.1%
EMG, accelerometer 30 6 CNN Over 98% [141]
EMG, accelerometer, arm angle, CNN: 78.4±5.2%
50 5 CNN, RCNN [143]
palm angle RCNN: 87.3±4.9%
Hand gesture EMG, accelerometer 9 11 DBN+SVM 5-fold cross validation: 97.9% [144]
recognition EMG, IMU, pressure 10 10 LSTM 89.28% [146]
Grayscale EMG, depth image, Multiscale
10 6 92.45% [147]
RGB image parallel CNN
Subvocal
speech EMG, sound 20 10 CNN+RNN+CTC 91.56 % (best) [149]
recognition

another. EMG changes after even a few moments for the same crafted features, which give the facility to the hybrid feature-
action of a participant, which leads to the inter-session based classifier.
problem. The inter-subject problem is more complex In addition, deep transfer learning has also been used for
compared with the inter-session problem because of the augmenting the performance of within-session gesture
development degree of muscle, the thickness of fat, individual classification tasks solely for improving accuracy [159].
habits, and so on, of different people. Within session means a shorter time interval of the training
Deep transfer learning takes advantage of both deep dataset and the testing dataset, with no prolonged time rest
learning and transfer learning by combining the feature within the same session. EMG signals are unstable, and it
learning ability of deep learning with the distribution adaption changes even within the same session, which leads to the
ability of transfer learning. Fine-tuning is widely used for
distinction of their feature domain. Deep transfer learning can
deep transfer learning-based frameworks [150], [151]. To
help to learn a better representation between the two domains,
reduce the error of subject-transfer, Kim et al. [152] propose a
framework that decodes hand movements robustly using thus improving the evaluation accuracy. The results of [159]
supportive CNN classifiers. The classifiers are pre-trained by show that deep transfer learning can improve the
the data from several subjects, and then, they are fine-tuned by generalization ability on the test dataset.
part of the target data. Finally, the gesture is decided by the For the inter-session problem, a factor that affects long-term
voting of the supportive CNN classifiers. The results show performance is user adaption, which means user adapts to
improvement for both healthy and amputee subjects. HMI devices with time passing by. The way man adapts to the
Du et al. [49 ] handle the inter-session problem using the machine can involve two aspects: short-term adaption by
deep learning-based domain adaptation mechanism, which is a visual feedback [160] or long-term adaption even without
multi-stream extension of AdaBN [153]. It selects feedback [161]. EMG signals change its attribute after a
instantaneous EMG with majority voting instead of the period, but users can adapt to the changes to some extent.
classical sliding window method to make a decision. Côté- However, it can be time-consuming because of complicated
Allard et al. [154] proposed the self-calibrating asynchronous user recalibrations. To realize a more intuitive and efficient
domain adversarial neural network (SCADANN) to solve the human-machine interface, we should determine some common
inter-session problem, and the best accuracy improves by and invariant information from EMG signals directly despite
8.47%. Côté-Allard et al. [155], [156] propose a framework of the diversification of EMG signals and user adaption. Deep
CNN augmented by transfer learning. The architecture was transfer learning may be the right choice for solving this
inspired by progressive neural networks [157] together with a question by determining invariant information inside EMG
multi-stream AdaBatch scheme [49] to transfer stable and signals at different sessions.
general features to a new subject. In addition to deep transfer learning, the performance of
Sosin et al. [107] estimate continuous hand gestures using deep learning for inter-session/subject has also been evaluated
RNN and adversarial domain adaptation (ADA). The result in [121], [123], [126], [162]. The features learned by deep
shows improvement for inter-subject accuracy but a decline neural networks can share similar distributions that are
for inter-session accuracy. Côté-Allard et al. [158 ] improve constant across different subjects/sessions.
the inter-subject performance with the adaptive domain The representative papers are depicted in Table VI . In
adversarial neural network (ADANN), which increases the general, deep transfer learning can provide a chance for a
accuracy by 19.40% more when compared with a baseline more opportune human-machine interface in which the pre-
algorithm. Moreover, the topological structures of deep trained models can be adapted to the same user after a while,
learning-based features are analyzed in contrast with hand- or to new users with less time or even no re-calibration time.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 525

TABLE VI
Representative Studies of Inter-Subject/Session
Performance Performance
Task Input Dataset Method Reference
(Before transfer) (After transfer)
Inter-session: Inter-session:
CLS: 62.5% CLS: 82.3%
CSL,
DB-b: 47.9% DB-b: 63.3%
Session HD-EMG CapMyo Multi-stream AdaBN [49]
DB-c: 26.3% DB-c: 35.1%
transfer of DB-b, DB-c
Inter-subject: Inter-subject:
hand gesture
DB-b: 39.0% DB-b: 55.3%
recognition
Best improvement:
Long-term
Spectrogram SCADANN 8.47% by SCADANN [154]
3DC
10.81 by TSD DNN
RMSE: RMSE:
Continuous
Continuous Inter-session: Inter-session:
hand
hand gesture EMG data RNN with ADA 18.83–19.80 19.55–20.5 [107]
movement of
estimation Inter-subject: Inter-subject:
5 subject
21.5–23.45 20.89–22.29
Select supportive CNN from source CNN
Ninapro DB2: 49.76% DB2: 52.52%
FFT trained by source subject, then fine-tuned by [152]
DB2, DB3 DB3: about 29% DB3: about 34%
single-trial data of the target subject
Subject Raw EMG, Ninapro Training source network with multi-stream DB5: 66.32% DB5: 68.98%
transfer of spectrogram, DB5; Myo [155]
AdaBN and add another network with Myo: 97.95% Myo: 98.31%
hand gesture and CWT [155] progressive neural networks structure
recognition
Training source network with multi-stream
Spectrogram Myo [156] AdaBN and the addition of another network About 87–96.5% 93.36–97.81% [156]
with progressive neural networks structure

EMG data 3DC [163] ADANN (Adaptive domain adversarial neural 65.03±0.08% 84.43±0.05%. [158]
network)
10 physical 1) Deep feature extraction using AlexNet and
Hand gesture 1) 99.04%
STFT actions of 4 VGG16, feature fused and classified by SVM; Not mentioned [159]
recognition 2) 98.65%
people 2) Fine-tune AlexNet by EMG images

C. Robustness Under Non-ideal Conditions be used for pattern recognition. The algorithm can improve
It is generally accepted that the performance of EMG online performance if human-added noises occur in the online
pattern recognition is easily affected by many surrounding testing phase. Yang et al. [124 ] choose several data
noise sources, such as electrode shift, muscle fatigue, physical augmentation approaches, including reverse placement of
friction, sudation, and so on. These factors can be called non- electrodes, random switch of channels, cross-arm, electrode
ideal conditions [164], which often occur out of the shifts, and electrode breakdown, according to common errors
laboratory. Deep learning can pave the way for designing during EMG acquisition. The results show that data
robust and stable algorithms for these problems. This section augmentation can improve precision and durability under
focuses on four questions: electrode shifts, data augmentation, disturbances. Dantas et al. [169] develop a dataset aggregation
limb position, and muscle fatigue. approach named DAgger that can improve long term
Although the pattern of an EMG can be decoded accurately performance within 150 days. On one hand, deep learning is
in the laboratory, the performance is not strong against the data-dependent, and thus, more data means better
electrode shift [165], which occurs when the subject wears the performance. On the other hand, it is more stable, and
electrodes during daily life, and the consequences can be abstracted features can be obtained through deep neural
catastrophic. Even a 1 cm shift can lead to a sharp decline in networks with augmented datasets.
the performance [165]. Deep transfer learning, which has been Limb position can be another critical factor that often leads
used in the question of inter-subject/inter-session, can also be to poor testing performance if the limb position is different
used to relieve the effect of electrode shift. Ameri et al. [52] from the training stage while performing the same gesture. Yu
attempt to solve the problem using the deep transfer learning et al. [170 ] solve this problem by a mixed-LDA classifier,
method. A CNN-based deep neural network is pre-trained which reaches an accuracy of 93.6% over five upper limb
using EMG data acquired before electrode shift and then fine- positions for seven hand gestures. Mukhopadhyay et al. [74]
tuned using data after a roughly 2.5 cm shift. The performance choose fully connected DNN with multiple hidden layers for
is based on the outcomes of an SVM-based method and recognizing eight hand gestures under five arm positions. The
adaption approaches based on LDA and QDA [166]–[168]. accuracy is 98.88%, which outperforms four types of
Data augmentation can raise the amount of EMG data and traditional machine learning-based methods. The DNN-based
improve the durability of external disturbances. The method simplified the feature extraction/selection step, which
transformed data with added white noise or wrong placement can determine invariant features under different limb positions
of electrodes [124], in combination with the original data will in a unified scheme.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
526 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

Muscle fatigue ordinarily occurs after long periods of low-cost neuromorphic chip. It shows the potential of
strenuous exercise. It is usually divided into states that include designing new network structures that can run on an
fatigue, non-fatigue, and transition-to-fatigue [171]. Fatigue embedded system. However, the network is specially
can cause serious injury in the course of man-machine designed, and thus, it still has a long way to go before running
cooperation. The correct prediction of muscle fatigue is of ordinary deep neural networks such as CNN or LSTM on
great significance to the safe and stable human-machine platforms like this.
interface. Su et al. [172] address muscle fatigue of the upper Some studies estimate the joint angles in real-time, such as
limb using DBN with raw EMG as input. The result is [70], [110]. Ren et al. [70 ] control the exoskeleton robot
comparable to the SVM-based method. named NTUHII for upper limb rehabilitation training. A PID
Methods of multimodal sensing and inter-subject/session controller dominates every DOF of the robot. The deep
are also techniques that attempt to improve the robustness in learning-based model predicts the angle of the future with
real life. There are fewer studies that focus on these questions EMG signals and the angles of the past as input. Then, the
with deep learning techniques, and thus, further attention predicted angles are fed into the control system with the speed
should be given to it in the future. of the velocity that is calculated by the first order of difference
of the joint angle. The result of the experiment shows good
V. Applications stability and precision for the online movements of the four-
Deep learning has been widely used in EMG-based HMI degree manipulator. Huang et al. [110] prove the possibility of
systems. However, most studies focus on offline performance deep learning running in embedded systems with the
with multifarious datasets. Online performance in physical STM32F4 processor for online angle predicting. It chooses a
systems, such as with prosthetic hand control, exoskeleton simple RNN with the relu activation function to make
robot operation, and so on, should be seriously considered. decisions faster and at a lower cost. However, the
This section will discuss the online performance evaluations performance of online controlling devices, such as lower limb
of deep learning-based systems. exoskeletons is not evaluated.
EMG-based hand gesture classification can serve for A combination with other sensors, for example, a camera,
prosthetic hand control, robot arm control, and so on. can improve online reliability for grasping. The hand posture
Yamanoi et al. [85 ] propose a CNN-based framework to must stay unchanged if the subject wants to hold an object and
control a myoelectric hand, which was motivated by 13 place it somewhere else, with EMG signals as the input alone.
motors with wire-pulling methods. The STFT of the EMG is The object can fall to the ground if any error occurs. Gao et al.
reconstructed as an image to submit to the network. Once the [147] design a system for controlling a 7-DOF robot hand
posture is classified, the hand will move to the predefined using the fusion of EMG, RGB images, and depth images of
position if the posture remains unchanged. The whole system hand grasping. The three types of data are reconstructed into a
runs in a notebook PC using the pre-trained model. Similar 5-dimensional image and sent into a multiscale parallel CNN
studies as in [71], [173], [174] control robot hands using EMG for gesture classification. The amount of input data is
with deep learning methods. In addition to the robot hand, relatively large, but the performance is improved significantly.
Redrovan et al. [175 ] control several quadrotors with hand In this study, EMG, together with visual information is used
gestures that are recognized through CNNs with EMG signals. for gesture recognition. However, research as in [178] decides
Allard et al. [65 ] guide a 6-DOF robot arm named JACO the target hand gesture by mainly relying upon computer
using hand gestures captured by the Myo armband. The vision, while EMG only works as a trigger signal that reflects
spectrogram of the EMG is used for classification by the CNN whether a user wants to grasp or not. Although this method
network. Seven gestures are mapped to different actions of the could be affected by various environmental noise that is
robot arm. The whole system runs on a laptop with a GPU. inherently in computer vision, it can release human attention
The performance is in a class with state-of-the-art during grasping. It improves the flexibility of the system,
performance guided by joysticks. Côté-Allard et al. [156] also which means that the user does not need to pay all of their
guide JACO with EMG and IMU of Myo Armband, and the attention to the grasping task to avoid falling or any other
performance is similar to the joystick being controlled by a errors.
human. Nasri et al. [173 ] teleoperate a robot named Pepper Various applications in the online system are summarized
with hand gestures recognized by GRU network. Mendes and illustrated in Table VII. In general, deep learning has been
et al. [176] control a collaborative KUKA iiwa robot using used in some real systems, and the performance shows that the
EMG-based hand gestures recognized by CNN. decoding method is effective for real-life applications.
Most studies train deep neural networks using a high- However, there are still questions such as computational cost,
performance server with a powerful GPU. There are studies environmental disturbances, satisfaction, and so on.
such as [53] that provide an online system based on a low-cost Additional research should focus on these questions.
GPU named the Nvidia Jetson Nano. Zanghieri et al. [81]
train the offline data by TCN using an embedded platform VI. Discussion
based on an 8-core low power processor named GAP8. In EMG is fragile and can be easily affected by many factors,
addition, Donati et al. [177 ] present a neural processing which affect the reliability and precision of the recognition
system to classify EMG into two types of hand gestures. It performance. These reasons boost the productivity of deep
employs a recurrent spiking neural network that runs on a learning in EMG processing. In this part, we will discuss the

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 527

TABLE VII
Applications in Physical System
Action Application Input Network Platform Reference
Online simulation Instant HD-EMG embedded CNN NVIDIA Jetson Nano Platform [53]
Robot arm control by
FFT CNN Laptop with GPU [65]
hand gestures
Feature MLP Laptop [71]
Myoelectric hand control STFT, feature CNN Laptop [85]
EMG data CNN Laptop with GPU [174]
Hand EMG image, RGB image, Multiscale parallel
Robot hand control Platform with GTX1060 [147]
gesture and depth image CNN
Robot arm guidance FFT CNN Not mentioned [156]
Teleoperation of a
EMG data GRU Not mentioned [173]
humanoid robot
Ultra-low power
Online gesture classification EMG data RNN [177]
neuromorphic chip
Personal computer with a
Robot arm guidance EMG data CNN [176]
GTX 1080 Ti GPU
Upper limb exoskeleton MS-LSTM dueling
STFT of EMG, IMU Not mentioned [70]
robot control model
Joint angle
Online estimation of knee EMG, joint angle and angular velocity
RNN STM32F4 [110]
joint angle of the past

advantages, challenges, and opportunities that are brought amount of data and improving stability under noisy
about by deep learning for EMG-based HMI tasks. conditions. Furthermore, few-shot learning methods are
Deep learning has the advantage of learning a better worthwhile to try, and these require few samples to learn
representation from EMG data to obtain higher precision, faster and better.
which is better than hand-crafted features. It has shown better Deep learning brings us new chances to handle the
performance than machine learning methods with feature questions of EMG pattern recognition. It provides a new way
engineering, as shown for movement classification tasks in of determining how to enhance the robustness of the EMG
[79], [82], [89], [90], for angle regression in [51], [56], [70], recognition algorithms. A wide gap exists between laboratory
for force prediction in [123], and so on. By adjusting the EMG research and commercial myoelectric control systems
parameters of the deep neural networks and exploring new owing to the lack of robustness against various disturbances
network structures, more discriminant features can be [78], which is also an adverse factor for other systems.
extracted. In addition, deep learning can narrow the heterogen- Electrode shift, electrode drop, individual differences, muscle
eity gap of different sensors in high-level feature space, which fatigue, sweatiness, and so on, which often appear in our daily
is important for performance enhancement. Multiple sensor use, can lead to poor accuracy. The possible solution includes
fusion is an important method for improving the reliability of learning unchangeable features between normal EMG data
the system. The heterogeneity data gap between the different and disturbed EMG data with deep neural networks, and
sensors can lead to a decline in the performance, in which transferring knowledge from normal data to augmented data
each mode of data has a bad influence on the others. Deep with deep transfer learning, among other actions.
learning can narrow the heterogeneity gap of the different Deep learning brings an opportunity for more concise and
sensors in high-level abstraction space, ensuring that the efficient neural-machine interaction systems. It is suggested
performance is better than the single modal sensor method, by Farina et al. [78 ] that neural information extracted from
which is important for performance enhancement. EMG signals could help to design extremely accurate HMI
The primary challenges that exist for now are the systems. Neural information extracted by blind source
computational cost and the dependency of the data. For the separation or morphological matching could be a novel path to
first problem, although a high-performance computer with a rejecting uncorrelated information and separate information
powerful GPU can be easily accessed in most laboratories that is stable under various disturbances. This approach has
today, it is difficult to run the deep network online in portable been tested for motion intention estimation, as depicted in
systems, which influences applications in daily life. Possible [179]–[183]. Xiong et al. [181 ], [182] try to estimate
solutions include designing embedded neural networks that movement intention through sEMG decomposition, and the
contain fewer parameters to solve the original question, waveform information of MUAP is used for hand gesture
developing a system with an embedded GPU that is smaller classification. They are the first team to estimate movement
with a lower power cost, or sending the acquired data to intention by neural information decomposed from sEMG.
remote GPU servers and returning the results. For the second Farina et al. [183] choose the discharging time of the MU for
problem, the solution can also be varied. On the one hand, movement classification, and the performance is evaluated
publicly open datasets such as Ninapro provide sufficient data with three patients after TMR surgery. Chen et al. estimate
for performance evaluation. On the other hand, data hand gestures [179] and kinematics [180] using the
augmentation methods can be a good choice for increasing the discharging time information of MU, and the performance

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
528 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

outperforms hand-crafted features. Nevertheless, learning seldom considered, and they are worthy of future
features from MUAP is still a necessary procedure of their development.
method, which makes it more complicated than extracting
features from EMG signals directly. Deep learning has great C. EMG Decomposition
potential for simplifying this procedure in an end-to-end form. Neural coding acquired by decomposing EMG signals into
It can be used for mining information related to human motion motor unit spikes has been used for human intention
inside the MUAP automatically, which avoids feature recognition, which could help build a robust and accurate
extraction and selection. In addition, the MUAP can provide a neural-machine interface. Uncorrelated information can be
new way of thinking for analyzing the feature maps that are removed, and thus, high precision can be reached. This
extracted from the EMG signals by deep neural networks approach retains the nature of the EMG signals, which have
because similar topological structures could exist between physiological meaning. The relationship between the MUAP
them. Thus, feature visualization of a deep network could help and human motion intention can be estimated by deep neural
improve understanding of the neural code inside the EMG. networks, which is simple compared with extracting features
from the MUAP.
VII. Conclusion and Future Work
Deep learning has shown an expanding tendency in D. Portable Systems
biomedical signal pattern recognition over the past several Future research should show solicitude for physical systems
years. In this article, many papers that decode EMG using that could improve the living standards of post-stroke patients
deep learning methods are reviewed. Typical HMI tasks, such or amputees. The low equality of these patient’s EMG signals
as movement classification, angle/force prediction, and more, requires better decoding methods, and thus, a deep learning-
are introduced in detail. Several hot-spot issues, such as based scheme could help the application of EMG in these
multimodal sensor fusion, inter-subject/session, and scenarios. The designed system should be portable, and these
robustness, are also presented to convey the latest progress in embedded systems should be easy to carry. Methods such as
recent years. These new topics are built for convenient and parameter shrinking, embedded GPU, and remote servers
should be explored for solving the question of computational
reliable HMI systems. The applications are introduced to
cost.
show recent experimental progress. The merits, drawbacks,
and prospects are discussed to present a comprehensive
analysis of the current conditions and to pave the way for the References
coming stages. [1] M. Q. Chen and P. Zhou, “A novel framework based on FastICA for
In summary, deep learning-based methods are in their high density surface EMG decomposition,” IEEE Trans. Neural Syst.
Rehabil. Eng., vol. 24, no. 1, pp. 117–127, Jan. 2016.
infancy for now, and there is still a certain distance to go
[2] K. Kiatpanichagij and N. Afzulpurkar, “ Use of supervised
before their adoption in commercial systems, which means discretization with PCA in wavelet packet transformation-based
great prospects for the future. In the future, attention should be surface electromyogram classification,” Biomed. Signal Process.
paid not only to the performance improvements but also to the Control, vol. 4, no. 2, pp. 127–138, Apr. 2009.
system implementation. With the help of this technique, more [3] T. Matsubara and J. Morimoto, “Bilinear modeling of EMG signals to
advanced systems will be developed to raise the quality of life extract user-independent features for multiuser myoelectric interface,”
IEEE Trans. Biomed. Eng., vol. 60, no. 8, pp. 2205–2213, Aug. 2013.
of the user. Certain directions should be considered carefully,
[4] C. Jorgensen, D. D. Lee, and S. Agabont, “ Sub auditory speech
which are summarized as follows. recognition based on EMG signals,” in Proc. Int. Joint Conf. Neural
Networks, Portland, USA, 2003, pp. 3128−3133.
A. Feature Learning [5] M. Janke, M. Wand, and T. Schultz, “A spectral mapping method for
The automatic feature learning ability from EMG data is EMG-based recognition of silent speech,” in Proc. 1st Int. Workshop
quite appealing for improving the recognition performance. on Bio-inspired Human-Machine Interfaces and Healthcare
Applications, Valencia, Spain, 2010, pp. 22−31.
Many networks in other fields, such as natural language
[6] B. Potočnik, M. Divjak, F. Urh, A. Frančič, J. Kranjec, M. Šavc, I.
processing, computer vision, and newly emerged networks Cikajlo, Z. Matjačić, M. Zadravec, and A. Holobar, “ Estimation of
(capsule network [163], graph network [184], and so on) are muscle co-activations in wrist rehabilitation after stroke is sensitive to
unexplored, which shows the great potential for EMG motor unit distribution and action potential shapes,” IEEE Trans.
decoding as better features can be extracted, and thus, higher Neural Syst. Rehabil. Eng., vol. 28, no. 5, pp. 1208–1215, May 2020.
performance can be achieved. Additionally, combining the [7] J. C. Castiblanco, S. Ortmann, I. F. Mondragon, C. Alvarado-Rojas,
M. Jöbges, and J. D. Colorado, “ Myoelectric pattern recognition of
features learned by deep neural networks with machine hand motions for stroke rehabilitation,” Biomed. Signal Process.
learning as in [37], [38] could also be a good choice. Control, vol. 57, pp. 101737, Mar. 2020.
[8] P. K. Artemiadis and K. J. Kyriakopoulos, “ EMG-based control of a
B. Domain Adaption robot arm using low-dimensional embeddings,” IEEE Trans. Rob.,
Deep transfer learning has the ability of domain adaption, vol. 26, no. 2, pp. 393–398, Apr. 2010.
which is quite worthwhile for model generalization under non- [9] K. Kiguchi and Y. Hayashi, “An EMG-based control for an upper-limb
power-assist exoskeleton robot,” IEEE Trans. Syst., Man, Cybern. Part
ideal conditions. It has been used by some research to solve B–Cybern., vol. 42, no. 4, pp. 1064–1071, Aug. 2012.
the problem of inter-subject/session, as mentioned in Section IV. [10] P. Artemiadis, “EMG-based robot control interfaces: Past, present and
The performance might require further improvement for future,” Adv. Rob. Autom., vol. 1, no. 2, pp. 1000e107, Jan. 2012.
online use. Other problems, such as electrodes shift/drop, [11] M. Simão, N. Mendes, O. Gibaru, and P. Neto, “ A review on
noise, and muscle fatigue, in which the testing data has a electromyography decoding and pattern recognition for human-
distinctive distribution in contrast with the training data, are machine interaction,” IEEE Acc., vol. 7, pp. 39564–39582, Mar. 2019.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 529

[12] Y. LeCun, Y. Bengio, and G. Hinton, “ Deep learning,” Nature, Audio, Speech, Lang. Process., vol. 22, no. 4, pp. 778–784, Apr. 2014.
vol. 521, no. 7553, pp. 436–444, May 2015. [34] X. Gao, M. Iwase, J. Inoue, and E. Maeda, “ Poster: Gesture
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet recognition based on ConvLSTM-attention implementation of small
classification with deep convolutional neural networks,” in Proc. 25th data sEMG signals,” in Proc. ACM Int. Joint Conf. Pervasive and
Int. Conf. Neural Information Processing Systems, Lake Tahoe, USA, Ubiquitous Computing and Proc. ACM Int. Symp. Wearable
2012. Computers, London, UK, 2019, pp. 21−24.
[14] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. R. Mohamed, N. Jaitly, A. [35] Y. H. Wu, B. Zheng, and Y. T. Zhao, “ Dynamic gesture recognition
Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, based on LSTM-CNN,” in Proc. Chinese Autom. Congr., Xi’an,
“Deep neural networks for acoustic modeling in speech recognition: China, 2018, pp. 2446−2450.
The shared views of four research groups,” IEEE Signal Process. [36] B. A. Xie, H. B. Li, and A. Harland, “ Movement and gesture
Mag., vol. 29, no. 6, pp. 82–97, Nov. 2012. recognition using deep learning and wearable-sensor technology,”
[15] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning Proc. Int. Conf. Artificial Intelligence and Pattern Recognition,
with neural networks,” in Proc. 27th Int. Conf. Neural Information Beijing, China, 2018, pp. 26−31.
Processing Systems, Montreal, Canada, 2014. [37] S. Shen, K. Gu, X. R. Chen, M. Yang, and R. C. Wang, “Movements
[16] D. Buongiorno, G. D. Cascarano, A. Brunetti, I. De Feudis, and V. classification of multi-channel sEMG based on CNN and stacking
Bevilacqua, “ A survey on deep learning in electromyographic signal ensemble learning,” IEEE Acc., vol. 7, pp. 137489–137500, Sep. 2019.
analysis,” in Proc. 15th Int. Conf. Intelligent Computing [38] H. F. Chen, Y. Zhang, G. F. Li, Y. F. Fang, and H. H. Liu, “Surface
Methodologies, Nanchang, China, 2019, pp. 751−761. electromyography feature extraction via convolutional neural
[17] A. Phinyomark and E. Scheme, “EMG pattern recognition in the era of network,” Int. J. Mach. Learn. Cybern. , vol. 11, no. 1, pp. 185–196,
big data and deep learning,” Big Data Cogn. Comput. , vol. 2, no. 3, Jan. 2020.
pp. 21, Aug. 2018. [39] R. Z. Tong, Y. Zhang, H. F. Chen, and H. H. Liu, “Learn the temporal-
[18] O. Faust, Y. Hagiwara, T. J. Hong, O. S. Lih, and U. R. Acharya, spatial feature of sEMG via dual-flow network,” Int. J. Humanoid
“Deep learning for healthcare applications based on physiological Rob., vol. 16, no. 4, pp. 1941004, Aug. 2019.
signals: A review,” Comput. Methods Programs Biomed. , vol. 161, [40] Z. Ding, C. F. Yang, Z. H. Tian, C. Z. Yi, Y. S. Fu, and F. Jiang,
pp. 1–13, Jul. 2018.
“sEMG-based gesture recognition with convolution neural networks,”
[19] M. Mahmud, M. S. Kaiser, A. Hussain, and S. Vassanelli, Sustainability, vol. 10, no. 6, pp. 1865, Jun. 2018.
“Applications of deep learning and reinforcement learning to
[41] W. T. Wei, Q. F. Dai, Y. Wong, Y. Hu, M. Kankanhalli, and W. D.
biological data,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 6,
Geng, “Surface-electromyography-based gesture recognition by multi-
pp. 2063–2079, Jun. 2018.
view deep learning,” IEEE Trans. Biomed. Eng. , vol. 66, no. 10,
[20] G. E. Hinton, S. Osindero, and Y. W. Teh, “A fast learning algorithm pp. 2964–2973, Oct. 2019.
for deep belief nets,” Neural Comput. , vol. 18, no. 7, pp. 1527–1554,
[42] W. T. Wei, Y. Wong, Y. Du, Y. Hu, M. Kankanhalli, and W. D. Geng,
Jul. 2006.
“A multi-stream convolutional neural network for sEMG-based
[21] Y. LeCun, B. Boser, J. S. Denker, and D. Henderson, gesture recognition in muscle-computer interface,” Pattern Recogn.
“Backpropagation applied to handwritten zip code recognition,” Lett., vol. 119, pp. 131–138, Mar. 2019.
Neural Comput., vol. 1, no. 4, pp. 541–551, Dec. 1989.
[43] S. J. Pan and Q. Yang, “ A survey on transfer learning,” IEEE Trans.
[22] J. L. Elman, “ Finding structure in time,” Cogn. Sci. , vol. 14, no. 2, Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010.
pp. 179–211, Apr.–Jun. 1990.
[44] C. Q. Tan, F. C. Sun, T. Kong, W. C. Zhang, C. Yang, and C. F. Liu,
[23] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural “A survey on deep transfer learning,” in Proc. 27th Int. Conf. Artificial
Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997. Neural Networks and Machine Learning, Rhodes, Greece, 2018, pp.
[24] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation 270−279.
of gated recurrent neural networks on sequence modeling,” arXiv: [45] J. Hoffman, S. Guadarrama, E. Tzeng, R. H. Hu, J. Donahue, R.
1412.3555, Dec. 2014. Girshick, T. Darrell, and K. Saenko, “ LSDA: Large scale detection
[25] M. Schuster and K. K. Paliwal, “ Bidirectional recurrent neural through adaptation,” in Proc. 27th Int. Conf. Neural Information
networks,” IEEE Trans. Signal Process. , vol. 45, no. 11, pp. 2673– Processing Systems, Montreal, Canada, 2014,
2681, Nov. 1997. [46] Y. X. Tang, J. Wang, B. Y. Gao, E. Dellandréa, R. Gaizauskas, and L.
[26] D. H. Ballard, “ Modular learning in neural networks,” in Proc. 6th M. Chen, “ Large scale semi-supervised object detection using visual
Nat. Conf. Artificial Intelligence, Seattle, USA, 1987, pp. 279−284. and semantic knowledge transfer,” in Proc. IEEE Conf. Computer
[27] X. Wu, G. Q. Jiang, X. Wang, P. Xie, and X. L. Li, “ A multi-level- Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp.
denoising autoencoder approach for wind turbine fault detection,” 2119−2128.
IEEE Acc., vol. 7, pp. 59376–59387, May 2019. [47] E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko, “Simultaneous deep
[28] L. F. Li, “ Recognizing polyps in wireless endoscopy images using transfer across domains and tasks,” in Proc. IEEE Int. Conf. Computer
deep stacked auto encoder with constraint image model in flexible Vision, Santiago, Chile, 2015, pp. 4068−4076.
medical sensor platform,” IEEE Acc. , vol. 8, pp. 60653–60663, Mar. [48] M. S. Long, H. Zhu, J. M. Wang, and M. I. Jordan, “Unsupervised
2020. domain adaptation with residual transfer networks,” in Proc. 30th Int.
[29] M. A. Ranzato, C. S. Poultney, S. Chopra, and C. LeCun, “Efficient Conf. Neural Information Processing Systems, Barcelona, Spain, 2016,
learning of sparse representations with an energy-based model,” in pp. 136−144.
Proc. 20th Annu. Conf. Neural Information Processing Systems, [49] Y. Du, W. G. Jin, W. T. Wei, Y. Hu, and W. D. Geng, “Surface EMG-
Vancouver, Canada, 2007, pp. 1137−1144. based inter-session gesture recognition enhanced by deep domain
[30] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. A. Manzagol, adaptation,” Sensors, vol. 17, no. 3, pp. 458, Mar. 2017.
“Stacked denoising autoencoders: Learning useful representations in a [50] C. Amma, T. Krings, J. Böer, and T. Schultz, “ Advancing muscle-
deep network with a local denoising criterion,” J. Mach. Learn. Res., computer interfaces with high-density electromyography,” in Proc.
vol. 11, no. 12, pp. 3371–3408, Dec. 2010. 33rd Annu. ACM Conf. Human Factors in Computing Systems, Seoul,
[31] P. Tamilselvan and P. F. Wang, “ Failure diagnosis using deep belief Korea (South), 2015, pp. 929−938.
learning based health state classification,” Reliab. Eng. Syst. Saf., [51] A. Ameri, M. A. Akhaee, E. Scheme, and K. Englehart, “Regression
vol. 115, pp. 124–135, Jul. 2013. convolutional neural network for improved simultaneous EMG
[32] A. R. Mohamed, G. E. Dahl, and G. Hinton, “Acoustic modeling using control,” J. Neural Eng., vol. 16, no. 3, pp. 036015, Jun. 2019.
deep belief networks,” IEEE Trans. Audio, Speech, Lang. Process., [52] A. Ameri, M. A. Akhaee, E. Scheme, and K. Englehart, “ A deep
vol. 20, no. 1, pp. 14–22, Jan. 2012. transfer learning approach to reducing the effect of electrode shift in
[33] R. Sarikaya, G. E. Hinton, and A. Deoras, “Application of deep belief EMG pattern recognition-based control,” IEEE Trans. Neural Syst.
networks for natural language understanding,” IEEE/ACM Trans. Rehabil. Eng., vol. 28, no. 2, pp. 370–379, Feb. 2020.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
530 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

[53] S. Tam, M. Boukadoum, A. Campeau-Lecours, and B. Gosselin, “A [72] P. Koch, H. Phan, M. Maass, F. Katzberg, R. Mazur, and A. Mertins,
fully embedded adaptive real-time hand gesture classifier leveraging “Recurrent neural networks with weighting loss for early prediction of
HD-sEMG and deep learning,” IEEE Trans. Biomed. Circuits Syst., hand movements,” in Proc. 26th European Signal Processing Conf.,
vol. 14, no. 2, pp. 232–243, Apr. 2020. Rome, Italy, 2018, pp. 1152−1156.
[54] Y. Chen, S. Yu, K. Ma, S. Y. Huang, G. F. Li, S. Q. Cai, and L. H. [73] H. M. Shim, H. An, S. Lee, E. H. Lee, H. K. Min, and S. Lee, “EMG
Xie, “ A continuous estimation model of upper limb joint angles by pattern classification by split and merge deep belief network,”
using surface electromyography and deep learning method,” IEEE Symmetry, vol. 8, no. 12, pp. 148, Dec. 2016.
Acc., vol. 7, pp. 174940–174950, Dec. 2019. [74] A. K. Mukhopadhyay and S. Samui, “An experimental study on upper
[55] W. T. Sun, H. X. Liu, R. Y. Tang, Y. R. Lang, J. P. He, and Q. Huang, limb position invariant EMG signal classification based on deep neural
“sEMG-based hand-gesture classification using a generative flow network,” Biomed. Signal Process. Control , vol. 55, pp. 101669, Jan.
model,” Sensors, vol. 19, no. 8, pp. 1952, Apr. 2019. 2020.
[56] T. Z. Bao, A. Zaidi, S. G. Xie, and Z. Q. Zhang, “Surface-EMG based [75] A. K. Mukhopadhyay, I. Chakrabarti, and M. Sharad, “Classification
wrist kinematics estimation using convolutional neural network,” in of hand movements by surface myoelectric signal using artificial-
Proc. IEEE 16th Int. Conf. Wearable and Implantable Body Sensor spiking neural network model,” in Proc. IEEE SENSORS, New Delhi,
Networks, Chicago, USA, 2019, pp. 1−4. India, 2018, pp. 419−422.
[57] N. Nasri, S. Orts-Escolano, F. Gomez-Donoso, and M. Cazorla, [76] P. Tsinganos, B. Cornelis, J. Cornelis, B. Jansen, and A. Skodras, “A
“Inferring static hand poses from a low-cost non-intrusive sEMG hilbert curve based representation of sEMG signals for gesture
sensor,” Sensors, vol. 19, no. 2, pp. 371, Jan. 2019. recognition,” in Proc. Int. Conf. Systems, Signals and Image
[58] B. Hudgins, P. Parker, and R. N. Scott, “ A new strategy for Processing, Osijek, Croatia, 2019, pp. 201−206.
multifunction myoelectric control,” IEEE Trans. Biomed. Eng., vol. 40, [77] L. Z. Bi, A. G. Feleke, and C. T. Guan, “ A review on EMG-based
no. 1, pp. 82–94, Jan. 1993. motor intention prediction of continuous human upper limb motion for
[59] W. D. Geng, Y. Du, W. G. Jin, W. T. Wei, Y. Hu, and J. J. Li, human-robot collaboration,” Biomed. Signal Process. Control , vol. 51,
“Gesture recognition by instantaneous surface EMG images,” Sci. pp. 113–127, May 2019.
Rep., vol. 6, no. 1, pp. 36571, Nov. 2016. [78] D. Farina, N. Jiang, H. Rehbaum, A. Holobar, B. Graimann, H. Dietl,
[60] J. C. Chen, X. D. Zhang, Y. Cheng, and N. Xi, “Surface EMG based and O. C. Aszmann, “ The extraction of neural information from the
continuous estimation of human lower limb joint angles by using deep surface EMG for the control of upper-limb prostheses: Emerging
belief networks,” Biomed. Signal Process. Control , vol. 40, avenues and challenges,” IEEE Trans. Neural Syst. Rehabil. Eng.,
pp. 335–342, Feb. 2018. vol. 22, no. 4, pp. 797–809, Jul. 2014.
[61] C. Grech, T. Camilleri, and M. Bugeja, “ Using neural networks for [79] K. H. Park and S. W. Lee, “ Movement intention decoding based on
simultaneous and proportional estimation of upper arm kinematics,” in deep learning for multiuser myoelectric interfaces,” in Proc. 4th Int.
Proc. 25th Mediterranean Conf. Control and Autom., Valletta, Malta, Winter Conf. Brain-Computer Interface, Yongpyong, South Korea,
2017, pp. 247−252. 2016.
[62] A. E. Olsson, P. Sager, E. Andersson, A. Björkman, N. Malešević, and [80] P. Tsinganos, B. Cornelis, J. Cornelis, B. Jansen, and A. Skodras,
C. Antfolk, “Extraction of multi-labelled movement information from “Improved gesture recognition based on sEMG signals and TCN,” in
the raw HD-sEMG image with time-domain depth,” Sci. Rep. , vol. 9, Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing,
no. 1, pp. 7244, May 2019. Brighton, United Kingdom, 2019, pp. 1169−1173.
[63] R. M. Stephenson, R. Chai, and D. Eager, “ Isometric finger pose [81] M. Zanghieri, S. Benatti, A. Burrello, V. Kartsch, F. Conti, and L.
recognition with sparse channel spatio temporal EMG imaging,” in Benini, “ Robust real-time embedded EMG recognition framework
Proc. 40th Annu. Int. Conf. IEEE Engineering in Medicine and using temporal convolutional networks on a multicore IoT processor,”
Biology Society, Honolulu, USA, 2018, pp. 5232−5235. IEEE Trans. Biomed. Circuits Syst., vol. 14, no. 2, pp. 244–256, Apr.
2020.
[64] W. Yang, D. P. Yang, Y. Liu, and H. Liu, “EMG pattern recognition
using convolutional neural network with different scale signal/spectra [82] M. Atzori, M. Cognolato, and H. Müeller, “ Deep learning with
input,” Int. J. Humanoid Rob., vol. 16, no. 4, pp. 1950013, Aug. 2019. convolutional neural networks applied to electromyography data: A
resource for the classification of movements for prosthetic hands,”
[65] U. C. Allard, F. Nougarou, C. L. Fall, P. Giguère, C. Gosselin, F.
Front. Neurorobot., vol. 10, pp. 9, Sep. 2016.
Laviolette, and B. Gosselin, “ A convolutional neural network for
robotic arm guidance using sEMG based frequency-features,” in Proc. [83] X. L. Zhai, B. Jelfs, R. H. M. Chan, and C. Tin, “Self-recalibrating
IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Daejeon, South surface EMG pattern recognition for neuroprosthesis control based on
Korea, 2016, pp. 2464−2470. convolutional neural network,” Front. Neurosci. , vol. 11, pp. 379, Jul.
[66] Y. Yamanoi and R. Kato, “Control method for myoelectric hand using 2017.
convolutional neural network to simplify learning of EMG signals,” in [84] J. C. Chen, S. Bi, G. Zhang, and G. Z. Cao, “ High-density surface
Proc. IEEE Int. Conf. Cyborg and Bionic Systems, Beijing, China, EMG-based gesture recognition using a 3D convolutional neural
2017, pp. 114−118. network,” Sensors, vol. 20, no. 4, pp. 1201, Feb. 2020.
[67] L. Chen, J. T. Fu, Y. H. Wu, H. C. Li, and B. Zheng, “Hand gesture [85] Y. Yamanoi, Y. Ogiri, and R. Kato, “EMG-based posture classification
recognition using compact CNN via surface electromyography using a convolutional neural network for a myoelectric hand,” Biomed.
signals,” Sensors, vol. 20, no. 3, pp. 672, Jan. 2020. Signal Process. Control, vol. 55, pp. 101574, Jan. 2020.
[68] A. David Orjuela-Cañón, A. F. Ruíz-Olaya, and L. Forero, “Deep [86] M. Simão, P. Neto, and O. Gibaru, “EMG-based online classification
neural network for EMG signal classification of wrist position: of gestures with recurrent neural networks,” Pattern Recogn. Lett.,
Preliminary results,” in Proc. IEEE Latin American Conf. vol. 128, pp. 45–51, Dec. 2019.
Computational Intelligence, Arequipa, Peru, 2017, pp. 1−5. [87] A. Samadani, “ Gated recurrent neural networks for EMG-based hand
[69] K. Asai and N. Takase, “Finger motion estimation based on frequency gesture classification. A comparative study,” in Proc. 40th Annu. Int.
conversion of EMG signals and image recognition using convolutional Conf. IEEE Engineering in Medicine and Biology Society, Honolulu,
neural network,” in Proc. 17th Int. Conf. Control, Autom. and Systems, USA, 2018, pp. 1−4.
Jeju, South Korea, 2017, pp. 1366−1371. [88] M. Alfaro-Ponce and I. Chairez, “ Continuous and recurrent pattern
[70] J. L. Ren, Y. H. Chien, E. Y. Chia, L. C. Fu, and J. S. Lai, “Deep dynamic neural networks recognition of electrophysiological signals,”
learning based motion prediction for exoskeleton robot control in Biomed. Signal Process. Control, vol. 57, pp. 101783, Mar. 2020.
upper limb rehabilitation,” in Proc. IEEE Int. Conf. Robotics and [89] M. Z. U. Rehman, S. O. Gilani, A. Waris, I. K. Niazi, G. Slabaugh, D.
Autom., Montreal, Canada, 2019, pp. 5076−5082. Farina, and E. N. Kamavuako, “Stacked sparse autoencoders for EMG-
[71] N. Naseer, F. Ali, S. Ahmed, S. Iftikhar, R. A. Khan, and H. Nazeer, based classification of hand motions: A comparative multi day
“EMG based control of individual fingers of robotic hand,” in Proc. analyses between surface and intramuscular EMG,” Appl. Sci. , vol. 8,
3rd Int. Conf. Sustainable Information Engineering and Technology, no. 7, pp. 1126, Jul. 2018.
Malang, Indonesia, 2018, pp. 6−9. [90] M. Z. U. Rehman, A. Waris, S. O. Gilani, M. Jochumsen, I. K. Niazi,

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 531

M. Jamil, D. Farina, and E. N. Kamavuako, “ Multiday EMG-based and E. M. Petriu, “ Recurrent neural network models for myoelectric-
classification of hand motions with deep learning techniques,” Sensors, based control of a prosthetic hand,” in Proc. 22nd Int. Conf. System
vol. 18, no. 8, Aug. 2018. Theory, Control and Computing, 2018, pp. 603−608.
[91] J. H. Zhang, C. Ling, and S. N. Li, “EMG signals based human action [109] P. Xia, J. Hu, and Y. H. Peng, “ EMG-based estimation of limb
recognition via deep belief networks,” IFAC-PapersOnline , vol. 52, movement using deep learning with recurrent convolutional neural
no. 19, pp. 271–276, Sep. 2019. networks,” Artif. Organs, vol. 42, no. 5, pp. E67–E77, May 2018.
[92] S. Lee, M. Sung, and Y. Choi, “Wearable fabric sensor for controlling [110] Y. C. Huang, Z. X. He, Y. X. Liu, R. Y. Yang, X. F. Zhang, G. Cheng,
myoelectric hand prosthesis via classification of foot postures,” Smart J. G. Yi, J. P. Ferreira, and T. Liu, “ Real-time intended knee joint
Mater. Struct., vol. 29, no. 3, pp. 035004, Mar. 2020. motion prediction by deep-recurrent neural networks,” IEEE Sens. J.,
[93] T. A. Kuiken, L. A. Miller, R. D. Lipschutz, B. A. Lock, K. vol. 19, no. 23, pp. 11503–11509, Dec. 2019.
Stubblefield, P. D. Marasco, P. Zhou, and G. A. Dumanian, “Targeted [111] A. Gautam, M. Panwar, D. Biswas, and A. Acharyya, “ MyoNet: A
reinnervation for enhanced prosthetic arm function in a woman with a transfer-learning-based LRCN for lower limb movement recognition
proximal amputation: A case study,” Lancet , vol. 369, no. 9559, and knee joint angle prediction for remote monitoring of rehabilitation
pp. 371–380, Feb. 2007. progress from sEMG,” IEEE J. Trans. Eng. Health Med., vol. 8,
[94] J. K. Shao, Y. F. Niu, C. Q. Xue, Q. Wu, X. Z. Zhou, Y. Xie, and X. L. pp. 2100310, Feb. 2020.
Zhao, “Single-channel SEMG using wavelet deep belief networks for [112] V. T. Inman, H. J. Ralston, J. De C M Saunders, M. B. Feinstein, and
upper limb motion recognition,” Int. J. Ind. Ergon. , vol. 76, E. W. Wright Jr., “ Relation of human electromyogram to muscular
pp. 102905, Mar. 2020. tension,” Electroencephalogr. Clin. Neurophysiol. , vol. 4, no. 2,
[95] R. M. Luo, S. Q. Sun, X. F. Zhang, Z. C. Tang, and W. D. Wang, “A pp. 187–194, May 1952.
low-cost end-to-end sEMG-based gait sub-phase recognition system,” [113] D. Staudenmann, K. Roeleveld, D. F. Stegeman, and J. H. van Dieen,
IEEE Trans. Neural Syst. Rehabil. Eng., vol. 28, no. 1, pp. 267–276, “Methodological aspects of SEMG recordings for force estimation—A
Jan. 2020. tutorial and review,” J. Electromyogr. Kinesiol. , vol. 20, no. 3,
[96] A. Ameri, M. A. Akhaee, E. Scheme, and K. Englehart, “Real-time, pp. 375–387, Jun. 2010.
simultaneous myoelectric control using a convolutional neural [114] A. G. Noughaby and G. R. Vossoughi, “The control of an exoskeleton
network,” PLoS One, vol. 13, no. 9, pp. e0203835, Sep. 2018. and the reduction of interaction force using human intent detection by
[97] J. H. Wang, L. Qi, and X. Wang, “Surface EMG signals based motion EMG signals and torque estimation,” in Proc. 6th RSI Int. Conf.
intent recognition using multi-layer ELM,” in Proc. SPIE 10605, Robotics and Mechatronics, Tehran, Iran, 2018, pp. 536−541.
LIDAR Imaging Detection and Target Recognition, Changchun, China, [115] E. E. Cavallaro, J. Rosen, J. C. Perry, and S. Burns, “Real-time
2017. myoprocessors for a neural controlled powered exoskeleton arm,”
[98] A. Olsson, N. Malešević, A. Björkman, and C. Antfolk, “Exploiting IEEE Trans. Biomed. Eng., vol. 53, no. 11, pp. 2387–2396, Nov. 2006.
the intertemporal structure of the upper-limb sEMG: Comparisons [116] J. M. Winters, “ Hill-based muscle models: A systems engineering
between an LSTM network and cross-sectional myoelectric pattern perspective,” in Multiple Muscle Systems: Biomechanics and
recognition methods,” in Proc. 41st Annu. Int. Conf. IEEE Engineering Movement Organization, J. M. Winters and S. L. Y. Woo, Eds. New
in Medicine and Biology Society, Berlin, Germany, 2019, pp. York: Springer, 1990, pp. 69−93.
6611−6615. [117] X. Zhang, D. Q. Wang, Z. Y. Yu, X. Chen, S. Li, and P. Zhou, “EMG-
[99] Y. Hu, Y. Wong, W. T. Wei, Y. Du, M. Kankanhalli, and W. D. Geng, torque relation in chronic stroke: A novel EMG complexity
“A novel attention-based hybrid CNN-RNN architecture for sEMG- representation with a linear electrode array,” IEEE J. Biomed. Health
based gesture recognition,” PLoS One , vol. 13, no. 10, pp. e0206049, Inform., vol. 21, no. 6, pp. 1562–1572, Nov. 2017.
Oct. 2018. [118] P. Liu, L. K. Liu, and E. A. Clancy, “Influence of joint angle on EMG-
[100] R. Akhundov, D. J. Saxby, S. Edwards, S. Snodgrass, P. Clausen, and torque model during constant-posture, torque-varying contractions,”
L. E. Diamond, “Development of a deep neural network for automated IEEE Trans. Neural Syst. Rehabil. Eng., vol. 23, no. 6, pp. 1039–1046,
electromyographic pattern classification,” J. Exp. Biol. , vol. 222, Nov. 2015.
pp. jeb198101, Mar. 2019. [119] Q. J. Song, B. Y. Sun, J. H. Lei, Z. Gao, Y. Yu, M. Liu, and Y. J. Ge,
[101] E. A. Clancy, L. K. Liu, P. Liu, and D. V. Z. Moyer, “Identification of “Prediction of human elbow torque from EMG using SVM based on
constant-posture EMG – torque relationship about the elbow using AWR information acquisition platform,” in Proc. IEEE Int. Conf.
nonlinear dynamic models,” IEEE Trans. Biomed. Eng. , vol. 59, no. 1, Information Acquisition, Weihai, China, 2006, pp. 1274−1278.
pp. 205–212, Jan. 2012. [120] R. C. Hu, X. Chen, S. Cao, X. Zhang, and X. Chen, “Investigation on
[102] Q. C. Ding, J. D. Han, and X. G. Zhao, “ Continuous estimation of the contributions of different muscles to the generated force based on
human multi-joint angles from sEMG using a state-space model,” HD-sEMG and DBN,” in Proc. 41st Annu. Int. Conf. IEEE
IEEE Trans. Neural Syst. Rehabil. Eng., vol. 25, no. 9, pp. 1518–1528, Engineering in Medicine and Biology Society, Berlin, Germany, 2019,
Sep. 2017. pp. 2645−2648.
[103] J. D. Han, Q. C. Ding, A. B. Xiong, and X. G. Zhao, “ A state-space [121] L. F. Xu, X. Chen, S. Cao, X. Zhang, and X. Chen, “Feasibility study
EMG model for the estimation of continuous joint movements,” IEEE of advanced neural networks applied to sEMG-based force
Trans. Ind. Electron., vol. 62, no. 7, pp. 4267–4275, Jul. 2015. estimation,” Sensors, vol. 18, no. 10, pp. 3226, Sep. 2018.
[104] W. Meng, B. Ding, Z. D. Zhou, Q. Liu, and Q. S. Ai, “An EMG-based [122] C. J. Li, J. Ren, H. Q. Huang, B. Wang, Y. F. Zhu, and H. S. Hu,
force prediction and control approach for robot-assisted lower limb “PCA and deep learning based myoelectric grasping control of a
rehabilitation,” in Proc. IEEE Int. Conf. Systems, Man, and prosthetic hand,” Biomed. Eng. Online , vol. 17, no. 1, pp. 107, Aug.
Cybernetics, San Diego, USA, 2014, pp. 2198−2203. 2018.
[105] D. Xiong, D. Zhang, X. Zhao, and Y. Zhao, “Continuous human gait [123] W. Yang, D. P. Yang, Y. Liu, and H. Liu, “ Decoding simultaneous
tracking using sEMG signals,” in Proc. 42nd IEEE Annu. Int. Conf. multi-DOF wrist movements from raw EMG signals using a
Engineering in Medicine and Biology Society (EMBC), Montreal, QC, convolutional neural network,” IEEE Trans. Human-Mach. Syst.,
Canada, 2020, pp. 3094–3097 vol. 49, no. 5, pp. 411–420, Oct. 2019.
[106] Z. Li, D. H. Zhang, X. G. Zhao, F. Y. Wang, B. Zhang, D. Ye, and J. [124] W. Yang, D. P. Yang, J. M. Li, Y. Liu, and H. Liu, “ EMG dataset
D. Han, “ A temporally smoothed MLP regression scheme for augmentation approaches for improving the multi-DOF wrist
continuous knee/ankle angles estimation by using multi-channel movement regression accuracy and robustness,” in Proc. IEEE Int.
sEMG,” IEEE Acc., vol. 8, pp. 47433–47444, Mar. 2020. Conf. Robotics and Biomimetics, Kuala Lumpur, Malaysia, 2018, pp.
[107] I. Sosin, D. Kudenko, and A. Shpilman, “ Continuous gesture 1268−1273.
recognition from sEMG sensor data with recurrent neural networks [125] Y. Ban, “Estimating the direction of force applied to the grasped object
and adversarial domain adaptation,” in Proc. 15th Int. Conf. Control, using the surface EMG,” in Proc. 11th Int. Conf. Haptics: Science,
Autom., Robotics and Vision, Singapore, 2018, pp. 1436−1441. Technology, and Applications, Pisa, Italy, 2018, pp. 226−238.
[108] T. A. Teban, R. E. Precup, E. C. Lunca, A. Albu, C. A. Bojan-Dragos, [126] M. Yokoyama, R. Koyama, and M. Yanagisawa, “ An evaluation of

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
532 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 3, MARCH 2021

hand-force prediction using artificial neural-network regression models [145] J. Lopes, M. Simão, N. Mendes, M. Safeea, J. Afonso, and P. Neto,
of surface EMG signals for handwear devices,” J. Sens., vol. 2017, “Hand/arm gesture segmentation by motion using IMU and EMG
pp. 3980906, Oct. 2017. sensing,” Proced. Manuf., vol. 11, pp. 107–113, 2017.
[127] Y. Y. Chen, C. Y. Dai, and W. Chen, “Cross-comparison of EMG-to- [146] X. L. Zhang, Z. Q. Yang, T. Y. Chen, D. L. Chen, and M. C. Huang,
force methods for multi-DoF finger force prediction using one-DoF “Cooperative sensing and wearable computing for sequential hand
training,” IEEE Acc., vol. 8, pp. 13958–13968, Jan. 2020. gesture recognition,” IEEE Sens. J., vol. 19, no. 14, pp. 5775–5783, Jul.
[128] Y. Yu, C. Chen, X. J. Sheng, and X. Y. Zhu, “Continuous estimation 2019.
of wrist torques with stack-autoencoder based deep neural network: A [147] Q. Gao, J. G. Liu, and Z. J. Ju, “ Hand gesture recognition using
preliminary study,” in Proc. 9th Int. IEEE/EMBS Conf. Neural multimodal data fusion and multiscale parallel convolutional neural
Engineering, San Francisco, USA, 2019, pp. 473−476. network for human-robot interaction,” Expert Systems. DOI:
[129] Y. Yu, C. Chen, X. J. Sheng, and X. Y. Zhu, “Multi-DoF continuous 10.1111/exsy.12490, Jan. 2020.
estimation for wrist torques using stacked autoencoder,” Biomed. [148] Z. Y. Li, H. Zhou, D. D. Yang, and S. Q. Xie, “ Multimodal deep
Signal Process. Control, vol. 57, pp. 101733, Mar. 2020. learning network based hand ADLs tasks classification for prosthetics
[130] Z. Y. Qin, Z. Y. Jiang, J. S. Chen, C. H. Hu, and Y. Ma, “sEMG-based control,” in Proc. Int. Conf. Progress in Informatics and Computing,
tremor severity evaluation for Parkinson’s disease using a light-weight Nanjing, China, 2017, pp. 91−95.
CNN,” IEEE Signal Process. Lett. , vol. 26, no. 4, pp. 637–641, Apr. [149] M. S. Elmahdy and A. A. Morsy, “ Subvocal speech recognition via
2019. close-talk microphone and surface electromyogram using deep
[131] A. Sengur, Y. Akbulut, Y. Guo, and V. Bajaj, “ Classification of learning,” in Proc. Federated Conf. Computer Science and
amyotrophic lateral sclerosis disease based on convolutional neural Information Systems, Prague, Czech, 2017, pp. 165−168.
network and reinforcement sample learning algorithm,” Health Inf. [150] G. Mesnil, Y. Dauphin, X. Glorot, S. Rifai, Y. Bengio, I. Goodfellow,
Sci. Syst., vol. 5, no. 1, pp. 9, Oct. 2017. E. Lavoie, X. Muller, G. Desjardins, D. Warde-Farley, P. Vincent, A.
[132] X. G. Liu, H. L. Li, C. G. Lou, T. Liang, X. L. Liu, and H. R. Wang, Courville, and J. Bergstra, “ Unsupervised and transfer learning
“A new approach to fall detection based on improved dual parallel challenge: A deep learning approach,” in Proc. Int. Conf.
channels convolutional neural network,” Sensors , vol. 19, no. 12, Unsupervised and Transfer Learning Workshop, Bellevue, USA, 2012,
pp. 2814, Jun. 2019. pp. 97−110.
[133] S. Morikawa, S. I. Ito, M. Ito, and M. Fukumi, “Personal [151] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are
authentication by lips EMG using dry electrode and CNN,” in Proc. features in deep neural networks?” in Proc. 27th Int. Conf. Neural
IEEE Int. Conf. Internet of Things and Intelligence System, Bali, Information Processing Systems, Montreal, Canada, 2014, pp.
Indonesia, 2018, pp. 180−183. 3320−3328.
[134] I. A. Khowailed and A. Abotabl, “Neural muscle activation detection: [152] K. T. Kim, C. T. Guan, and S. W. Lee, “A subject-transfer framework
A deep learning approach using surface electromyography,” J. based on single-trial EMG analysis using convolutional neural
Biomech., vol. 95, pp. 109322, Oct. 2019. networks,” IEEE Trans. Neural Syst. Rehabil. Eng. , vol. 28, no. 1,
[135] P. Wang, E. L. Tan, Y. L. Jin, L. Li, and J. Wang, “Prediction of EMG pp. 94–103, Jan. 2020.
signal on missing channel from signal captured from other related [153] Y. H. Li, N. Y. Wang, J. P. Shi, X. D. Hou, and J. Y. Liu, “Adaptive
channels via deep neural network,” in Proc. IEEE Int. Conf. Robotics batch normalization for practical domain adaptation,” Pattern Recogn.,
and Biomimetics, Kuala Lumpur, Malaysia, 2018, pp. 1287−1291. vol. 80, pp. 109–117, Aug. 2008.
[136] H. Nodera, Y. Osaki, H. Yamazaki, A. Mori, Y. Izumi, and R. Kaji, [154] U. Côté-Allard, G. Gagnon-Turcotte, A. Phinyomark, K. Glette, E. J.
“Deep learning for waveform identification of resting needle Scheme, F. Laviolette, and B. Gosselin, “ Unsupervised domain
electromyography signals,” Clin. Neurophysiol. , vol. 130, no. 5, adversarial self-calibration for electromyography-based gesture
pp. 617–623, May 2019. recognition,” IEEE Acc., vol. 8, pp. 177941–177955, Sep. 2020.
[137] S. Nam, M. K. Sohn, H. A. Kim, H. J. Kong, and I. Y. Jung, [155] U. Côté-Allard, C. L. Fall, A. Drouin, A. Campeau-Lecours, C.
“Development of artificial intelligence to support needle Gosselin, K. Glette, F. Laviolette, and B. Gosselin, “Deep learning for
electromyography diagnostic analysis,” Healthc. Inform. Res. , vol. 25, electromyographic hand gesture signal classification using transfer
no. 2, pp. 131–138, Apr. 2019. learning,” IEEE Trans. Neural Syst. Rehabil. Eng. , vol. 27, no. 4,
[138] D. Ramachandram and G. W. Taylor, “ Deep multimodal learning: A pp. 760–771, Apr. 2019.
survey on recent advances and trends,” IEEE Signal Process. Mag., [156] U. Côté-Allard, C. L. Fall, A. Campeau-Lecours, C. Gosselin, F.
vol. 34, no. 6, pp. 96–108, Nov. 2017. Laviolette, and B. Gosselin, “ Transfer learning for sEMG hand
[139] Y. Yu, X. Chen, S. Cao, X. Zhang, and X. Chen, “ Exploration of gestures recognition using convolutional neural networks,” in Proc.
Chinese sign language recognition using wearable sensors based on IEEE Int. Conf. Systems, Man, and Cybernetics, Banff, AB, Canada,
deep belief net,” IEEE J. Biomedical and Health Informatics , vol. 24, 2017, pp. 1663−1668.
no. 5, pp. 1310–1320, May 2020. [157] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick,
[140] F. Wang, S. S. Zhao, X. Q. Zhou, C. Li, M. Y. Li, and Z. Zeng, “An K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “ Progressive neural
recognition-verification mechanism for real-time Chinese sign networks,” arXiv: 1606.04671, 2016. [Online]. Available: [Link]
language recognition based on multi-information fusion,” Sensors, org/abs/1606.04671
vol. 11, pp. 2495, May 2019. [158] U. Côté-Allard, E. Campbell, A. Phinyomark, F. Laviolette, B.
[141] S. Shin, Y. Baek, J. Lee, Y. Eun, and S. H. Son, “ Korean sign Gosselin, and E. Scheme, “ Interpreting deep learning features for
language recognition using EMG and IMU sensors based on group- myoelectric control: A comparison with handcrafted features,” Front.
dependent NN models,” in Proc. IEEE Symp. Series on Computational Bioeng. Biotechnol., vol. 8, pp. 158, Mar. 2020.
Intelligence, Honolulu, USA, 2017, pp. 1770−1776. [159] F. Demir, V. Bajaj, M. C. Ince, S. Taran, and A. Şengür, “Surface
[142] Q. Zhang, D. Wang, R. Zhao, and Y. G. Yu, “MyoSign: Enabling end- EMG signals and deep transfer learning-based physical action
to-end sign language recognition with wearables,” in Proc. 24th Int. classification,” Neural Comput. Appl. , vol. 31, no. 12, pp. 8455–8462,
Conf. Intelligent User Interfaces, Marina del Ray, USA, 2019, pp. Dec. 2019.
650−660. [160] M. A. Powell, R. R. Kaliki, and N. V. Thakor, “ User training for
[143] W. M. Wang, B. Chen, P. Xia, J. Hu, and Y. H. Peng, “Sensor fusion pattern recognition-based myoelectric prostheses: Improving phantom
for myoelectric control based on deep learning with recurrent limb movement consistency and distinguishability,” IEEE Trans.
convolutional neural networks,” Artif. Organs , vol. 42, no. 9, Neural Syst. Rehabil. Eng., vol. 22, no. 3, pp. 522−532, May 2014.
pp. E272–E282, Sep. 2018. [161] J. Y. He, D. G. Zhang, N. Jiang, X. J. Sheng, D. Farina, and X. Y. Zhu,
[144] C. W. Yeh, T. Y. Pan, and M. C. Hu, “ A sensor-based official “User adaptation in long-term, open-loop myoelectric training:
basketball referee signals recognition system using deep belief Implications for EMG pattern recognition in prosthesis control,” J.
networks,” in Proc. 23rd Int. Conf. Multimedia Modeling, Reykjavik, Neural Eng., vol. 12, no. 4, pp. 046005, Aug. 2015.
Iceland, 2017, pp. 565−575. [162] C. Maufroy and D. Bargmann, “ CNN-based detection and

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.
XIONG et al.: DEEP LEARNING FOR EMG-BASED HMI: A REVIEW 533

classification of grasps relevant for worker support scenarios using Process. Control, vol. 55, pp. 101637, Jan. 2020.
sEMG signals of forearm muscles,” in Proc. IEEE Int. Conf. Systems, [180] C. Chen, G. H. Chai, W. C. Guo, X. J. Sheng, D. Farina, and X. Y.
Man, and Cybernetics, Miyazaki, Japan, 2018, pp. 141−146. Zhu, “Prediction of finger kinematics from discharge timings of motor
[163] M. Kwabena Patrick, A. Felix Adekoya, A. Abra Mighty, and B. Y. units: Implications for intuitive control of myoelectric prostheses,” J.
Edward, “Capsule networks—A survey,” J. King Saud Univ.–Comput. Neural Eng., vol. 16, no. 2, pp. 026005, Apr. 2019.
Inf. Sci. DOI: 10.1016/[Link].2019.09.014, Sep. 2019.
[181] A. B. Xiong, D. H. Zhang, X. G. Zhao, J. D. Han, and G. J. Liu,
[164] Q. C. Ding, Z. Y. Li, X. G. Zhao, Y. F. Xiao, and J. D. Han, “Real- “Classification of gesture based on sEMG decomposition: A
time myoelectric prosthetic-hand control to reject outlier motion preliminary study,” IFAC Proc. Vol. , vol. 47, no. 3, pp. 2969–2974,
interference using one-class classifier,” in Proc. 32nd Youth Academic Aug. 2014.
Annu. Conf. Chinese Association of Autom., Hefei, China, 2017, pp.
96−101. [182] A. B. Xiong, X. G. Zhao, J. Han, G. J. Liu, and Q. C. Ding, “An user-
independent gesture recognition method based on sEMG
[165] L. Hargrove, K. Englehart, and B. Hudgins, “ The effect of electrode decomposition,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and
displacements on pattern recognition based myoelectric control,” in Systems, Hamburg, Germany, 2015, pp. 4185−4190.
Proc. Annu. Int. Conf. IEEE Engineering in Medicine and Biology
Society, New York, USA, 2016, pp. 2203−2206. [183] D. Farina, I. Vujaklija, M. Sartori, T. Kapelner, F. Negro, N. Jiang, K.
Bergmeister, A. Andalib, J. Principe, and O. C. Aszmann,
[166] M. M. C. Vidovic, H. J. Hwang, S. Amsüss, J. M. Hahne, D. Farina,
“Man/machine interface based on the discharge timings of spinal
and K. R. Müller, “ Improving the robustness of myoelectric pattern
recognition for upper limb prostheses by covariate shift adaptation,” motor neurons after targeted muscle reinnervation,” Nat. Biomed.
IEEE Trans. Neural Syst. Rehabil. Eng., vol. 24, no. 9, pp. 961–970, Eng., vol. 1, no. 2, pp. 0025, Feb. 2017.
Sep. 2016. [184] Z. H. Wu, S. R. Pan, F. W. Chen, G. D. Long, C. Q. Zhang, and P. S.
[167] J. W. Liu, X. J. Sheng, D. G. Zhang, J. Y. He, and X. Y. Zhu, Yu, “A comprehensive survey on graph neural networks,” IEEE Trans.
“Reduced daily recalibration of myoelectric prosthesis classifiers Neural Networks and Learning Systems. DOI: 10.1109/TNNLS.2020.
based on domain adaptation,” IEEE J. Biomed. Health Inform., vol. 20, 2978386, Mar. 2020.
no. 1, pp. 166–176, Jan. 2016.
[168] X. Y. Zhu, J. W. Liu, D. G. Zhang, X. J. Sheng, and N. Jiang, Dezhen Xiong received the B.E. degree in
“Cascaded adaptation framework for fast calibration of myoelectric automation from North University of China, in 2018.
control,” IEEE Trans. Neural Syst. Rehabil. Eng. , vol. 25, no. 3, He is currently pursuing the Ph.D. degree with the
pp. 254–264, Mar. 2017. State Key Laboratory of Robotics, Shenyang Institute
[169] H. Dantas, D. J. Warren, S. M. Wendelken, T. S. Davis, G. A. Clark, of Automation, Chinese Academy of Sciences. He is
and V. J. Mathews, “Deep learning movement intent decoders trained also currently with the University of Chinese
with dataset aggregation for prosthetic limb control,” IEEE Trans. Academy of Sciences. His research interests include
Biomed. Eng., vol. 66, no. 11, pp. 3192–3203, Nov. 2019. biomedical signal processing, blind source
separation, pattern recognition, and deep learning.
[170] Y. Yu, X. J. Sheng, W. C. Guo, and X. Y. Zhu, “ Attenuating the
impact of limb position on surface EMG pattern recognition using a
mixed-LDA classifier,” in Proc. IEEE Int. Conf. Robotics and
Biomimetics, Macau, China, 2017, pp. 1497−1502. Daohui Zhang (M’ 19) received the B.E. degree in
[171] M. R. Al-Mulla, F. Sepulveda, and M. Colley, “ A review of non- mechanical engineering and automation from
invasive techniques to detect and predict localised muscle fatigue,” Northeastern University, in 2010, and the Ph.D.
Sensors, vol. 11, no. 4, pp. 3545–3594, Mar. 2011. degree in pattern recognition and intelligent system
[172] Y. Su, S. L. Sun, Y. Ozturk, and M. Tian, “Measurement of upper limb from Shenyang Institute of Automation, Chinese
muscle fatigue using deep belief networks,” J. Mechanics in Medicine Academy of Sciences, in 2018. He is currently an
and Biology, vol. 16, no. 8, pp. 1640032, Sep. 2016. Associate Professor with the State Key Laboratory of
Robotics, Shenyang Institute of Automation, Chinese
[173] N. Nasri, F. Gomez-Donoso, S. Orts-Escolano, and M. Cazorla, Academy of Sciences. His research interests include
“Using inferred gestures from sEMG signal to teleoperate a domestic nonlinear estimation and control, robotics, and
robot for the disabled,” in Proc. 15th Int. Work-Conf. Artificial Neural pattern recognition.
Networks Computational Intelligence, Gran Canaria, Spain, 2019, pp.
198−207.
[174] Y. F. Wan, Z. S. Han, J. Zhong, and G. H. Chen, “Pattern recognition Xingang Zhao (M’ 12) received the B.E. and M.E.
and bionic manipulator driving by surface electromyography signals degrees in mechanics from Jilin University, in 2000
using convolutional neural network,” Int. J. Adv. Rob. Syst., vol. 15, and 2004, respectively, and the Ph.D. degree in
no. 5, Oct. 2018. pattern recognition and intelligent systems from
Shenyang Institute of Automation, Chinese Academy
[175] D. V. Redrovan and D. Kim, “ Hand gestures recognition using
of Sciences, in 2008. From 2015 to 2016, he was a
machine learning for control of multiple quadrotors,” in Proc. IEEE
Visiting Scientist at the Rehabilitation Institute of
Sensors Applications Symp., Seoul, South Korea, 2018, pp. 394−399.
Chicago, Chicago, USA. He is currently a Professor
[176] N. Mendes, M. Simão, and P. Neto, “ Segmentation of electro- at Shenyang Institute of Automation, Chinese
myography signals for pattern recognition,” in Proc. 45th Annu. Conf. Academy of Sciences. His research interests include
IEEE Industrial Electronics Society, Lisbon, Portugal, 2019, pp. 732− medical robots, rehabilitation robots, robot control, and pattern recognition.
737.
[177] E. Donati, M. Payvand, N. Risi, R. Krause, K. Burelo, G. Indiveri, T.
Dalgaty, and E. Vianello, “ Processing EMG signals using reservoir Yiwen Zhao received the [Link]. degree in control
computing on an event-based neuromorphic system,” in Proc. IEEE science and engineering and the [Link]. degree in
Biomedical Circuits and Systems Conf., Cleveland, USA, 2018, pp. mechanical and electrical engineering from Harbin
455−458. Institute of Technology, in 1995 and 1997,
respectively, and the Ph.D. degree in mechanical and
[178] V. Gregori, M. Cognolato, G. Saetta, M. Atzori, The MeganePro electrical engineering from Shenyang Institute of
Consortium, and A. Gijsberts, “ On the visuomotor behavior of Automation, Chinese Academy of Science in 2000.
amputees and able-bodied people during grasping,” Front. Bioeng. Since 2000, he has been with the State Key
Biotechnol., vol. 7, pp. 316, Nov. 2019. Laboratory of Robotics, Shenyang Institute of
[179] C. Chen, Y. Yu, S. H. Ma, X. J. Sheng. C. Lin, D. Farina, and X. Y. Automation, Chinese Academy of Sciences, where
Zhu, “ Hand gesture recognition based on motor unit spike trains he is currently a Professor. His research interests include medical robots,
decoded from high-density electromyography,” Biomed. Signal autonomous mobile robots, and intelligent system control.

Authorized licensed use limited to: Shenyang Institute of Automation. Downloaded on February 10,2021 at 18:23:21 UTC from IEEE Xplore. Restrictions apply.