Received October 9, 2019, accepted November 11, 2019, date of publication November 14, 2019,

date of current version November 25, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2953535

Independent Decision Path Fusion for Bimodal

Asynchronous Brain–Computer Interface to
Discriminate Multiclass Mental States
XINYU JIANG 1 , XIAO GU 2 , KE XU1 , HAORAN REN 1,

AND WEI CHEN 1 , (Senior Member, IEEE)

1 Center for Intelligent Medical Electronics, School of Information Science and Technology, Fudan University, Shanghai 200433, China
2 Hamlyn Centre, Imperial College London, London SW7 2AZ, U.K.
Corresponding author: Wei Chen (w_chen@ [Link])
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFE0112000, and in
part by the Shanghai Municipal Science and Technology Major Project under Grant 2017SHZDZX01.

ABSTRACT With the increasing development of brain imaging and sensing technologies, a wide vari-
ety of medical signals in multiple modalities have facilitated a better understanding of mental health.
The brain-computer interface (BCI) is a technology capable of detecting mental states automatically by
employing various brain sensing and machine learning methods. This contributes to efforts involving
neurological disease management and restoration of cognitive function. In this study, we present an
independent decision path fusion (IDPF) method by developing a bimodal asynchronous BCI based on
electroencephalographs (EEGs) and functional near-infrared spectroscopy (fNIRS) to discriminate multiple
mental states. The proposed IDPF method generates several independent decision paths, each of which
is capable of analyzing cerebral information with respect to a specific aspect, thus interpreting the brain
state from multiple points of view. Moreover, in one particular decision path for the fNIRS analysis of
the IDPF method, we modified the EEG-based common spatial pattern (CSP) algorithm according to the
characteristics of fNIRS. The results show that the modified common spatial pattern (MCSP) significantly
outperforms CSP in fNIRS-based BCIs. Through validation on an open-access EEG-fNIRS dataset and
comparison with recent studies, we found that our IDPF method achieves a high accuracy of 70.32% for
a four-class classification problem (left hand motor imagery, right hand motor imagery, mental arithmetic,
and resting state).

INDEX TERMS Bimodal BCIs, mental states, EEG, fNIRS, machine learning.
I. INTRODUCTION and provide compensations for loss of normal neuromotor
Brain–computer interface (BCI) is used extensively in inter- mechanisms [1]. Recent research has proven the effectiveness
preting different mental states, which can trigger alerts auto- of BCIs in post-stroke rehabilitation [2]. Specifically, with
matically when detecting abnormal brain activities in patients respect to the attention deficit hyperactivity disorder (ADHD)
with mental illnesses, and assist paralyzed patients to con- in children, a BCI-based attention-training program was
trol external equipment without voluntary muscular inter- launched [3] for cognitive rehabilitation. In motor neuron dis-
vention. For example, through imaginary of left or right ease (MND) patients, BCI-based wheelchairs [4] and robotic
hand movement, the amputees can control the movement of arms [5] are expected to provide convenience in daily life.
corresponding artificial prosthesis as long as their mental Besides, the BCI for automated task load monitoring in
states can be accurately detected. Therefore, this powerful robotic surgery was developed [6].
tool is nowadays closely associated with numerous healthcare Compared with invasive BCIs, which require sensors
related applications [1]–[6]. In particular, BCI technology is implanted directly into the brain, noninvasive devices are
practiced in neurological rehabilitation, where it is applied preferred due to safety and convenience. Mental states can
to help restore motor or cognition functions of patients, be detected through several noninvasive sensing and imag-
ing technologies, such as the electroencephalograph (EEG)
The associate editor coordinating the review of this manuscript and [7]–[13], magnetoencephalogram (MEG) [14], functional
approving it for publication was Jenny Mahoney. magnetic resonance imaging (fMRI) [15], and functional

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
VOLUME 7, 2019 165303
 X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

near-infrared spectroscopy (fNIRS) [16]–[22]. However, not signals are widely applied in the BCI field [31], [32]. For
all of these noninvasive interfaces are suitable for mental example, a variety of EEG features can be extracted by
state detection in real-life situations. Although MEG and various algorithms for detection of the mental state, such as
fMRI are both brain imaging techniques with high spatial adaptive auto-regressive parameters [33], time-frequency fea-
resolution, they are too cumbersome to be used in free-living tures [34], and inverse model-based features [35]. Moreover,
environments. The multi-channel EEG signal is the total spatial filter algorithms are applied to EEG signals to improve
sum of the potential difference vectors measured in differ- the signal-to-noise ratio (SNR) [32]. With respect to the
ent positions of cerebral cortex or scalp surface, reflect- fNIRS, since the technology is only a few years old, the algo-
ing cerebral electrophysiological characteristics. Owing to rithms are far less complete than those in the EEG field [36].
the convenience of EEG signal acquisition, wearable EEG As EEG and fNIRS share the similarity of monitoring brain
devices are widely applied in medical and engineering areas. activity from multiple channels, it seems promising to trans-
Furthermore, fNIRS is an increasingly popular technique fer algorithms, such as spatial filter algorithms, from EEG
for cerebral function imaging, employing optical radiation to fNIRS-related researches. Common spatial pattern (CSP)
within the near-infrared range to monitor cerebral hemo- [32] is a well-known spatial filter algorithm and has been
dynamic changes. Radiation absorption changes in these widely applied in EEG signal processing field. By applying
measurements reflect the local concentration of oxygenated CSP, a lower dimensional but more separable feature vector
hemoglobin (HbO) and deoxygenated hemoglobin (HbR), can be extracted from EEG signals for classification. Previous
which are in close correlation with mental states. As a conse- work has proved the effectiveness of CSP algorithm on fNIRS
quence of their unique advantages in wearability and portabil- signals [37]. However, in spite of all these similarities, EEG
ity, both EEG and fNIRS devices are feasible for 24/7 mental and fNIRS also characterize their own unique properties to
state monitoring. Moreover, many clinical EEG-fNIRS stud- each other. Therefore, simply transferring the original CSP
ies [23]–[27] have reported that cerebral electrophysiological algorithm from EEG to fNIRS field without modification can
characteristics and the hemodynamic response are correlated not contribute to the best results. In our previous study [38],
with each other and coupling of the two is an important we modified CSP algorithm to create a modified common
indicator for brain monitoring. Therefore, algorithms for dis- spatial pattern (MCSP) algorithm based on unique properties
criminating different mental states based on the EEG and of fNIRS. MCSP can extract highly separable features by
fNIRS have the potential to speed up the exploration of daily projecting fNIRS signals from original signal space onto a
mental health monitoring. signal subspace. As an important part of the present work,
The BCI can be divided into two categories, namely we further generalized MCSP from binary classification to
synchronous and asynchronous. For synchronous BCI (also multiclass classification problems. The results in this work
known as stimulus-driven BCI), the arousal of a specific men- further proved the effectiveness of MCSP on more complex
tal state depends on its corresponding external stimuli. The classification problems using more data. Considering the
most common ones are the P300 BCI [28] and stable-state scarcity of fNIRS signal processing methods, this work can
visual evoked potential (SSVEP) BCI [29]. In SSVEP BCI, broaden the horizons in this lately emerging fields.
as an example, subjects are asked to stare at flashes at a fixed By now, weak classification accuracy and limited infor-
frequency, and consequently a frequency doubling response mation obtained by traditional sensing with single modal
component in their EEG signals is detected. Although the brain signals remain a challenge. To address it, an increasing
synchronous BCI can achieve a high transmission rate, number of studies have attempted employing multimodal
the shortcomings are evident. Gazing at the light source data to detect mental states, the most popular combination
requires subjects focusing their attention, preventing them of which is the bimodal method based on the EEG and
from receiving feedback from the controlled equipment and fNIRS [23]–[27], [39]–[41]. The EEG is a brain sensing
external emergencies. Besides, such external stimulation is method with high temporal resolution, however the relatively
usually bothersome and may lead to fatigue. In contrast, asyn- low spatial resolution and sensitivity to motion artifacts limit
chronous BCI (also known as intention-driven BCI) is inde- its application. In contrast, fNIRS sensing has a high spatial
pendent of external stimuli, where subjects are free to perform resolution and robustness to motion artifacts, which provides
different voluntary and self-driven mental tasks. A particular complimentary brain sensing data to the EEG. EEG-fNIRS
example is motor imaginary. Through the imagining of the monitoring has been applied in many areas, such as stress
movement of the left or right hands, brain signals present assessment [39], drivers’ fatigue assessment [40], sleep mon-
different characteristics. The asynchronous BCI is more ideal itoring [23], epilepsy studies [24]–[26], as well as presurgical
for daily practice, even though the low classification accuracy investigation [27]. These studies employed EEG-fNIRS to
needs to be improved, especially for multiclass classification investigate the correlation between electrophysiological and
problems [30]. hemodynamic mechanisms. In addition, better classification
To date, many single-modal studies have proposed numer- performance was achieved by the bimodal method com-
ous classical methods to discriminate between mental states pared with the single modal one. Nevertheless, the analysis
based on the EEG or fNIRS. With respect to the EEG, of complementarity between the EEG and fNIRS from the
advanced algorithms based on the characteristics of EEG perspective of information theory, with the aim to explain

165304 VOLUME 7, 2019

X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

the enhanced performance obtained by their fusion, has not II. MATERIALS
been given much attention in the research to date. In the A. DATASET
present study, we analyze the mutual information obtained An open-access EEG and fNIRS dataset [42] was utilized in
by the two modes to investigate modal fusion in informa- our work with four mental states, namely left hand motor
tion theory. Moreover, we provide the correlation of elec- imagery (LI), right hand motor imagery (RI), mental arith-
trophysiological and hemodynamic response under normal metic (MA), and the baseline resting state (BL). As LI and
self-driven brain activities, which yields similar results to the RI can be considered as a single category, mental states under
EEG-fNIRS epilepsy studies [25]–[27]. EEG-fNIRS has like- motor imagery (MI) can also be assessed. The dataset con-
wise demonstrated its high potential in areas of the BCI [41]. tains one left-handed and twenty-eight right-handed healthy
Nevertheless, to date, few methodologies have been explored subjects (14 males and 15 females) with an average age
for EEG-fNIRS-based BCIs to investigate brain informa- of 28.5±3.7 years. For the EEG, 30-channel signals of all
tion in different aspects and improve the classification subjects were recorded at a 200 Hz sampling rate. For fNIRS,
accuracy to the maximum extent. Moreover, most existing 14 sources and 16 detectors represented 30 channels with a
EEG-fNIRS BCI studies focus on the two-class classification 10 Hz sampling rate. The electrode placement is displayed
problem, which cannot be adapted to a challenging daily in Fig. 1.
scenario.
In the BCI, it has been emphasized that analyzing signals
in terms of a single aspect, or looking at a particular fea-
ture set in a neural recording, may omit other relevant
information. Because a single-decision path usually con-
tains mental information concerning a certain aspect, only
weak accuracy can be achieved. In the computer science
area, decision-level fusion using combined classifiers has
been widely investigated and proved to be an effective
modal fusion method. Therefore, we developed an indepen-
dent decision path fusion (IDPF) method for bimodal asyn-
chronous BCI, where different kinds of brain information
are taken into consideration, including electrophysiological
spatial patterns, electrophysiological temporal and frequency FIGURE 1. Electrodes placement in the dataset.

properties, hemodynamic spatial patterns, as well as hemo-

dynamic state transitions. All of these measurable aspects of For each subject, 120 trials (30 for each mental task) were
brain activity constitute multiple points of view of mental executed. The task paradigm is shown in Fig. 2, and the task
states. Several decision paths are trained separately using onset time is marked as 0 s. The data can be downloaded via
specific machine learning methods, such as support vector [Link]
machine (SVM), linear discriminant analysis (LDA), and
hidden Markov model (HMM), and each can characterize
a particular attribute of mental state information. The low
level of shared information used for predictions obtained
by these decision paths individually proves their mutual
independence. Accordingly, these independent decision paths
can be complemented by each other, leading to a better
performance. Subsequently, a reliability-based decision-level
fusion strategy is introduced to effectively integrate these
independent decision paths. By validation on an open-access
dataset [42] and comparison with recent studies, our method FIGURE 2. Data structure of a trial.
proved that it can achieve higher accuracy for a multiclass
classification problem.
The remainder of this paper is structured as follows. B. PREPROCESSING OF EEG SIGNALS
In section II, the dataset, as well as the preprocessing methods EEG data were first bandpass filtered at 0.5–50 Hz. Elec-
utilized in our work, are described. In section III, the sig- trooculograph (EOG) is the main artifact of EEG signals.
nal processing pipelines of each independent decision path The EOG removal methods are based on the assumption
characterizing different aspects of the mental state are intro- that the measured EEG signal is the linear combination of
duced. In section IV, our fusion strategy of decision paths is clean EEG and independent EOG component. Due to the
demonstrated. In section V, we provide a discussion based on weak correlation of EOG and EEG, the EOG artifact can
the results. Finally, section VI provides the conclusion of this be removed through independent component analysis (ICA)
research. [43]. In this work, ICA-based EOG removal was performed

VOLUME 7, 2019 165305

 X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

FIGURE 3. Framework of IDPF-based bimodal BCIs.

using EEGLAB toolbox [44]. The EEG signals with EOG signal is the total sum of the potential difference vectors
removed were normalized by Z-score and then divided into measured on the surface of the cerebral cortex or scalp.
several single task segments (Fig. 2), denoted as Xitask ∈ The main frequency range of the EEG is 0.5–30 Hz, which
RN1 ×T1 . contains information about mental states and can be divided
into four main bands, δ waves (0.5–3 Hz), θ waves (4–7 Hz),
C. PREPROCESSING OF fNIRS SIGNALS α waves (8–13 Hz) and β waves (14–30 Hz). For different
The raw signals of fNIRS were first bandpass filtered brain activities, the power changes of the four bands comply
at 0.01–0.2 Hz to remove physiological noise caused with different patterns, which implies that the powers of the
by respiration, heartbeat, and Mayer waves [45]. The four waves are key indicators of the mental states. As a
bandpass-filtered signals were then converted to concentra- result, the power spectrum has been widely employed for
tion changes of HbO and HbR by the modified Beer-Lambert characterization of electrophysiological frequency properties
law [46]. The whole process was performed using nirsLAB when detecting specific brain activity [48].
toolbox [47]. HbO and HbR data were normalized per chan- The flow chart of the power spectrum-based EEG method
nel by the Z-score. The preprocessed data were divided into is presented in Fig. 3. Because of the subject-specific
segments, with signals of 2 s offset extracted as Yitask ∈ electrophysiological responses, a specific 10 s task of differ-
RN2 ×T2 , in accordance to the slow hemodynamic response ent subjects may arouse various patterns of power changes,
of fNIRS (Fig. 2). Yirest represents signals of the 5 s inter- leading to different optimal classification time periods,
val before instruction, the mean value of which can reflect channels and frequency bands. In order to determine the most
the hemoglobin concentration level before onset. Therefore, sensitive time of a specific task, the preprocessed EEG signals
Ẏitask (Yitask - mean value of Yirest ) is utilized to characterize are first passed through a sliding window (3 s window length,
concentration changes of HbO and HbR. 1 s sliding step, and a total of eight windows for a 10 s task)
In our annotations, {N1 , T1 } and {N2 , T2 } represent the to find the most discriminative time period [42]. Signals in
channel number and sampling number of the EEG and fNIRS, each single window are then passed through four bandpass
respectively, and the subscript i ∈ {1, 2, 3, 4} refers to LI, RI, filters (0.5–3 Hz, 4–7 Hz, 8–13 Hz, and 14–30 Hz) to obtain
MA, and BL, respectively. The preprocessed data were then δ, θ , α and β waves. The powers of each frequency band in
applied to the IDPF method we propose in this paper. each channel are calculated, formulating a 120-dimensional
feature vector (30 channels × 4 frequency bands) for signals
III. METHODS in each sliding window. A single factor ANOVA (analysis of
In this section, the framework of IDPF will be introduced in variance) is performed to select the most effective features,
detail. As shown in Fig. 3, several mental state predictions thus reducing feature dimensionality. The selected features
are given individually, and a reliability-based weight fusion are normalized before being classified by SVM. For each
is performed to provide the final predicted class. In each subject, we applied a leave-one-out cross-validation strategy,
single-decision path, one particular aspect of mental infor- and the segment in the window of the highest performance
mation is taken into consideration. The detailed pipeline of for the validation set is selected.
each independent decision path is given below.
B. CSP BASED EEG DECISION PATH
A. POWER SPECTRUM BASED EEG DECISION PATH The CSP algorithm is aimed at achieving the optimal spatial
During specific brain activity, the potential of a large number filter ω ∈ RN1 ×1 realizing the maximum ratio between
of neuronal cells changes. As mentioned before, the EEG the variance of spatially filtered signals during two mental

165306 VOLUME 7, 2019

X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

tasks. The optimal spatial filter transforms raw data into (4 classes × 4 filter pairs × 2 filters each pair) feature vector.
the underlying electrophysiological spatial pattern. Subse- Finally, the LDA classifier is applied for classification.
quently, the variance features of spatially filtered signals are
extracted for classification, which can reflect the large-scale C. CHANNEL-WISE BASED fNIRS DECISION PATH
fluctuations of EEG signals, indicating electrophysiological The channel-wise based decision path is aimed at extracting
temporal evolution. The objective function of CSP takes the features from each channel of HbO and HbR concentra-
following form, tions. Due to the relatively high spatial resolution of fNIRS,
1 P T task k2 the hemodynamic spatial pattern is hidden in channel-wise
n1 i∈k kω X̃i
J1 (X , ω) = 1 P 1 feature vectors.
T task k2
n2 i∈k2 kω X̃i Fig. 3 shows the flow chart of the channel-wise based
method. Similar to the power spectrum-based EEG method,
i∈k ω Ri ω
1 P T
n ωT R̄1 ω
= 11 P 1 = T . (1) the preprocessed signals are passed through a sliding win-
n i∈k2 ω Ri ω
T ω R̄2 ω dow [42] with 3 s window length and 1 s sliding step, to find
2
the most sensitive time period for each subject. For signals
X̃itaskrefers to the mean-removed Xitask , where the mean value
in the sliding window, mean value features of each channel
of each channel is zero. Besides, k1 , k2 refer to the two
are extracted. For both HbO and HbR, a 36-dimensional
tasks to be classified, n1 , n2 refer to the number of each
feature vector (36 channels) is derived separately. All fea-
type of task, and Ri is the autocorrelation matrix of X̃itask .
tures are normalized, and subsequently, the LDA classifier
The optimal filter is applied to merge all channels of each
is applied for classification. HbO and HbR concentration
trial into a single time series, and according to the EEG’s
signals assume two individual decision paths using the same
high temporal resolution and frequent fluctuations, variance
signal processing flow chart.
of spatially filtered signal is extracted as an important feature.
This algorithm has proven to be a powerful tool in EEG-based
D. HMM BASED fNIRS DECISION PATH
BCIs, which reduces the high feature dimensionality for clas-
sification assignments [49]. During a particular mental state of our brain, the hemoglobin
The form of objective function (1) as well as its opti- concentration will change following a specific but unknown
mization procedure is the same as many other well-known rule due to cerebral hemodynamics. As a consequence,
subspace analysis method, such as principle component anal- the hemodynamic state transition over time is of the essence
ysis (PCA) [50] and LDA [51]. The optimal solution ω that to detect mental states. In our work, HMM is adopted to
maximizes the objective function (1) is the eigenvector of classify fNIRS signals under different mental states due to
R̄−1 its advantages in modelling time-dependent data [53]. Since
2 R̄1 [52], which can be calculated through the singular
value decomposition as Equation (2). In Equation (2), λj is fNIRS data is in the form of a multichannel time series,
the jth largest eigenvalue of matrix R̄−1 signals in each channel, denoted as yki (t) (k is the channel
2 R̄1 , with a descending
number and i refers to the trial index), construct the observa-
order (j = 1, 2, · · · , ξ and ξ is the rank of R̄−1
2 R̄1 ), and the tion matrix Ẏitask . For a 10 s task signal at 10 Hz sampling
solution ωj is the eigenvector corresponding to eigenvalue λj ,
rate, the 36×100-dimensional Ẏitask can be formulated as the
which contributes to the jth largest value of objective func-
vector sequence in (4), where µi (t) refers to a 36×1 dimen-
tion (1). Then we choose the 1st − k th filter pairs (the k th pair:
j sional vector constructed by sampling values of each channel
ωk and ωξ +1−k ), which are applied to transform X̃itask into zi
at time t.
as in (3). Ultimately, variance features of each filtered signal
are extracted for classification. For the binary classification 
y1i (t)

problem, we select the first three filter pairs to establish a  y2 (t) 
 i
six-dimensional feature vector. Ẏitask =  .  = [µi (1) , µi (2) , · · · , µi (100)] (4)

   .  .
R̄−1 R̄1 ωj = λj ωj (2)
i (t)
y36
2
j
zi = ωjT X̃itask (3)
HMM is a statistical model used to describe a Markov pro-
From the formula form of objective function (1), CSP cess with implicit unknown parameters and indirectly observ-
is mainly applied to the binary classification problem. For able states. It has been shown that HMM outperforms SVM
the multiple classification problem in this study, we use the in discriminating fNIRS signals under LI and RI tasks [54],
one-versus-rest strategy. For each type of task, the other indicating that HMM is an effective tool to analyze cerebral
three tasks are viewed as a single category, and the CSP hemodynamic changes. In recent years, HMM has also been
algorithm is performed as a binary classification problem applied to classify the MA task and music imagery task [55].
to obtain the optimal spatial filters. Each type of task is For a HMM training problem, several parameters are of
handled in the same way and for the four-class classification great concern, including the number of states Q, the state
problem, this procedure is repeated four times. Each time, transition probability matrix A (A = {aij } and aij refers to
we select the first four filter pairs to extract a 32-dimensional the transition probabilities between state i and j), the initial

VOLUME 7, 2019 165307

 X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

state distribution vector π , and observation probability distri- activate our brain to a larger extent. As a contrast, in Fig. 4b,
butions b in state j (b = {bj (µ)}) and j ∈ {1, 2, · · · , Q}). the variance of fNIRS signal under the MA task is lower than
In this research, since we are dealing with a continuous that under the BL task while the absolute mean value is the
HMM training problem, continuous observation densities opposite. This indicates that, for fNIRS signal, the activation
given by the Gaussian mixture models (GMM) are applied. degree is mainly reflected in the overall concentration during
For each of the four tasks, an HMM is trained using a period of time rather than the instantaneous fluctuations.
the expectation-maximization algorithm [53], denoted by This is also consistent with the low frequency characteristics
HMMLI , HMMRI , HMMMA and HMMBL , respectively. For of fNIRS. Therefore, for fNIRS, the CSP algorithm should be
each test trial, the vector sequence signal is passed through modified based on its particular properties.
each of the four HMMs. Four log-likelihood (LL) results Differently from the basic CSP, which focuses on variance
are generated using the forward-backward algorithm [53], and is more applicable to the EEG, in fNIRS we adopt the
denoted as LLLI , LLRI , LLMA , and LLBL , respectively, and mean concentration level as an indicator. Firstly, the pre-
subsequently, the test trial is classified as the class cor- processed fNIRS signal Ẏitask , indicating the concentration
responding to the largest LL. Subject-specific HMMs are changes after the task onset, proceeds through a sigmoid
trained due to the subject-dependent hemodynamic response function (5) that converts the whole signal to the range (0,1).
pattern, and for the test set of each subject, we tried several   1
combinations of Q and M (M refers to the number of Gaus- Ÿitask = sigmoid Ẏitask = task
(5)
sian components in the GMM) to find the best parameters 1 + e−Ẏi
based on the training and validation set of the same sub- Then, Ÿitask is employed as the input of the objective func-
ject. The need for a subject-specific HMM model has been tion in the MCSP algorithm, aimed at finding the largest ratio
researched by [54] before. Fig. 3 presents the flow chart of of the square averages of two signal sets. It is formulated as
the HMM-based fNIRS method. below,
1 P T task k2
n1 i∈k kω Ÿi
E. MODIFIED COMMON SPATIAL PATTERN ALGORITHM J2 (Y , ω) = 1 P 1
T task k2
n2 i∈k2 kω Ÿi
In our pilot study, we modified the CSP algorithm based
i∈k ω Ri ω
1 P T
on the characteristics of fNIRS, which is applied as another n1 ωT R̄1 ω
= 1 P 1 = T . (6)
decision path to further complement mental information on
n i∈k2 ω Ri ω
T ω R̄2 ω
2
the hemodynamic spatial pattern under a specific mental
The filter can be calculated through a singular value
state. As introduced in section III-A, EEG is a kind of
decomposition of the matrix R̄−1 2 R̄1 , in the same manner as
brain signal reflecting cerebral electrophysiology with rich
in CSP. Similarly, we choose the 1st − k th filter pairs, which
high-frequency components and large fluctuations. Hence, j
variance is an important indicator. Accordingly, the objec- are applied to transform Ÿitask into zi as in (7). Afterwards,
tive function (1) of the CSP algorithm in EEG-based BCIs mean values, instead of variance features, are extracted for
is aimed at maximizing the ratio between the variance of classification.
j
spatially filtered signals under two mental tasks. However, zi = ωjT Ÿitask (7)
fNIRS signals present different temporal features based on
the hemodynamic mechanism, within a very low-frequency In the CSP algorithm, since X̃itask is the mean subtraction of
band. Due to low-frequency characteristics, for fNIRS, Xitask , kωT X̃itask k2corresponds to the variance of filtered sig-
the temporal mean value has proved to be a more effective nals, which mostly reflects the large-scale fluctuation. In our
feature compared with others, such as variance and slope [56]. MCSP algorithm, the mean value of Yirest is subtracted from
As shown in Fig. 4a, the variance of EEG signal under the Yitask to obtain Ẏitask , and therefore, Ẏitask can better represent
MA task is obviously higher than that under the BL task, the relative concentration level changes during a specific task.
which is in line with intuition because doing arithmetic will We employ Ÿitask , a constant-positive signal, to find the best
spatial filter. kωT Ÿitask k2 corresponds to the square average
of the mapped HbO and HbR concentrations, which is more
related to the average concentration level during a specific
period. The main goal of the algorithm modification is to
change the optimization objective, where the variance of HbO
and HbR concentration signal is replaced by the square aver-
age of the sigmoid-mapped concentration signal, in addition
to the ultimate feature switched to mean value, thus better
reflecting the overall hemoglobin concentration level instead
of the fluctuation level.
The MCSP algorithm has been proposed in our pilot work
[38] in which HbO and HbR signals were processed individ-
FIGURE 4. Waveform comparison of EEG and fNIRS. ually. In this study, in contrast, we take both HbO and HbR

165308 VOLUME 7, 2019

X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

into consideration simultaneously. For binary classification various factors [57]. Besides, the experiment establish-
problems, we employ six filter pairs to form a 24-dimensional ment and the task execution is an extremely cumber-
(2 signal types HbO and HbR × 6 filter pairs × 2 filters each some and annoying process, imposing quite a burden
pair) feature vector. For the four-class classification problem, to subjects. Therefore, the training dataset is usually
we employ three filter pairs and apply the one-versus-rest small, and our strategy can deal with the common
strategy to construct a 48-dimensional feature vector (2 signal small-dataset problem.
types HbO and HbR × 4 classes × 3 filter pairs × 2 filters 3) The feature extraction part does not exist in all classifi-
each pair). Detailed descriptions of the algorithm principles cation paths. The HMM-based decision path, for exam-
can be found in in [38]. ple, analyzes the preprocessed signals directly to gen-
erate the classification result, where the feature-level
IV. FUSION OF INDEPENDENT DECISION PATHS fusion strategy is not appropriate.
In section III, several decision paths indicating different The algorithm framework is shown in Fig. 3, where
aspects of mental information are introduced respectively, si = {p1 , p2 , p3 , p4 } refers to the probability of the segment
all of which together constitute multiangle views of men- belonging to class LI, RI, MA or BL. In this research, three
tal states. To realize the complementary optimization, our different approaches for calculating weights are discussed.
strategy on the integration of all these decision paths will be Here, we denote the accuracy, reliability, and weight of the
demonstrated. ith path as ai , ri , and wi respectively. p0 is the accuracy
For multimodal problems, modal fusion is essential as well expectation of random classification, and for the four-class
as necessary. The common methods adopted in most multi- classification problem, p0 is equal to 0.25.
modal studies can be divided into three categories, namely (a)
data-level, feature-level, and decision-level. With respect to 1) distributing weights on average:
data-level fusion, which directly concatenates all initial data,
1
no shallow-level information is lost. However, the data size wi = (8)
will increase, thus adding to the computational burden. More- 6
over, the raw data from different modes sometimes cannot be 2) distributing weights based on accuracy:
directly combined due to different sampling rates or differ- ai
ences in the effective frequency band. wi = P6 (9)
k=1 ak
In the feature-level fusion, features of each mode are
extracted separately from the raw data, and all low- 3) distributing weights based on reliability:
dimensional feature vectors are integrated into a high- 
π ai − p0

dimensional vector used for classification. In this strategy, ri = tan · (10)
2 1 − p0
large amounts of training data are required to avoid ri
over-fitting. wi = P6 (11)
For decision-level fusion, data from each mode are k=1 rk
processed and learned separately to derive a path-specific In strategy (a), all paths are assigned the same weight.
primary decision, and the weighted average is calculated. In strategy (b), paths are assigned different weights based
In each single-decision path, only low-dimensional features on their respective accuracy. In strategy (c), the reliabilities
need to be handled so as to reduce the extreme requirement of each path are evaluated first by the formula (10), and
of data volume. However, one single-decision path some- then weights are assigned by reliabilities of each path as in
times exhibits weakness. Therefore, an effective decision path formula (11). To be more specific, if the accuracy of a specific
fusion strategy for the final decision is of vital importance. path is 100%, the reliability should be +∞, so that the path is
In our study, decision-level fusion is preferred for the given a full value. However, if a specific path yields a random
following reasons: classification result, its reliability should be zero, leading to a
1) The sampling rates of the EEG and fNIRS are consid- zero weight. The weight is pretrained on the training set, and
erably different due to their unique frequency charac- the results of the three path fusion strategies are presented in
teristics. Accordingly, they are not suitable for direct section V.
fusion at the data level.
2) The scale of the dataset utilized is relatively small, V. RESULTS AND DISCUSSION
which is not sufficient to support the constitu- A. VALIDATION METHODOLOGIES
tion of high-dimensional feature vectors. As a con- Due to the relatively small size of the dataset (only 30 trials
sequence, the decision-level fusion method, where for each task of each subject), to make the best use of the
low-dimensional feature vector in each single-decision data, we applied leave-one-out cross-validation strategy to
path is handled separately, is applicable. evaluate the proposed IDPF method in this research. The
More generally, in BCIs, subjects are asked to perform whole model is built specific to each subject. For each test
different mental tasks to acquire signals. However, trial, all remaining trials of the same subject were used for
brain sensing experiments can be easily disturbed by model training (LDA, SVM, and HMM). Then, we further

VOLUME 7, 2019 165309

 X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

divided training data into training set (21 trials for each task)
and validation set (the remaining 9 or 8 trials for each task),
to select the best parameters (e.g. 3 s windows, weights
of decision paths) for each single test trial based on the
performance of validation set. For CSP and MCSP decision
paths, the training of optimal spatial filters is required. Five
trials of each mental task in the training set were randomly
selected to achieve the optimal spatial filters by optimizing
the objective function (1) and (6). Machine learning tech-
niques in all decision paths require a training phase but the
left-out-test trial each time was completely independent of the FIGURE 6. Classification accuracy of different filter pairs.
*MCSP-MEAN: MCSP algorithm with mean value as feature.
training process, including classifier training and parameter CSP-MEAN: CSP algorithm with mean value as feature.
selection. Moreover, we also take Kappa coefficient as the CSP-Var: CSP algorithm with variance as feature.
metric for classification consistency. All experimental data
were analyzed offline using MATLAB (MathWorks Inc.).
study [38]. And this is consistent with the qualitative results
B. MCSP VS. CSP IN fNIRS shown in Fig. 5.
1) QUALITATIVE RESULTS
3) DISCUSSION ON MCSP
To show the effectiveness of MCSP more intuitively,
we first give some qualitative results. According to For a specific pattern recognition problem, extracting features
Fig. 5a and Fig. 5b, the distance between signals under MA with high separability is of significance and necessity. Nor-
and BL tasks filtered by MCSP is visibly larger than that mally, a high-dimensional feature vector can better represents
filtered by CSP. The same is true for HbR signals in original signals. However, in some scenarios, the feature
Fig. 5c and Fig. 5d. Because ω1 and ω36 contributes to the vector with extremely high dimensionality may lead to over-
maximum and minimum of the objective function (6) respec- fitting, especially when the size of training data set is small,
tively for the given 36-channel fNIRS signals, the value of which is usually the case with fNIRS data set. Therefore, fea-
signal under MA task is extremely high after filtered by ω1 ture dimensionality reduction is required. MCSP can merge
and extremely low after filtered by ω36 . In general, signals fNIRS in all channels and extract more separable features
filtered through MCSP are much more easier to classify. only in those merged channels. Different biomedical signals
usually share some similarities but still have their own char-
acteristics. It is of vital importance to transfer and modify
processing methods between different signals based on their
similarities and unique characteristics respectively.

C. CLASSIFICATION RESULTS OF EACH DECISION PATH

The classification result of each decision path is shown
in Fig. 9. For each single-decision path, classification accu-
racy and classification consistency are not satisfactory.
Especially for the four-class classification problem, the high-
est accuracy of one decision path is merely 60.49%, which
cannot support a BCI to work continuously and steadily
in practice. Thus, a significant accuracy improvement is
required.

D. CLASSIFICATION RESULTS OF DECISION-LEVEL FUSION

Before applying decision-level fusion, we calculated the
FIGURE 5. HbO and HbR Signals of MA and BL tasks filtered by the first mutual information of classification results between each
filter pair (a representative sample).
two decision paths for the four-class classification problem,
shown in Table 1. When applying multimodal fusion, it is of
vital importance to evaluate the similarity metric of different
2) QUANTITATIVE RESULTS modes or paths to estimate if they are suitable for fusion.
In Fig. 6, the classification accuracies of MA vs. BL and Some commonly used similarity metrics include covariance,
LI vs. RI across different filter pair numbers are presented. correlation, and mutual information. In this work, mutual
As shown in Fig. 6, our MCSP algorithm performs better information was selected. In Table 1, the highest mutual
than CSP with both the mean value and variance as classi- information of the classification result between each two
fication features, which shares similar results with our pilot decision paths is merely 0.2592 bit (HbO Channel-wise path

165310 VOLUME 7, 2019

X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

TABLE 1. Pair-wise mutual information of classification results between decision paths for four-class classification problem.

TABLE 2. Confusion matrix for three-class classification problem.

decision path with quite poor performance will be assigned

a near zero or even negative weight, further illustrating the
rationality of weight assignment based on reliability. The
performance of different fusion strategies for each single
subject is shown in Fig. 7. We can see that all fusion strategy
can achieve good results but performance improvement of
strategy (c) tends to be significant if validated on more data
volumes.
By employing fusion strategy (c), accuracies of 85.60%
and 70.32% can be achieved for ‘‘MA vs. BL vs. MI’’
FIGURE 7. Comparison of different fusion strategies. and ‘‘MA vs. BL vs. LI vs. RI’’ tasks, respectively. Com-
*The p-values quantify the significance of improvement.
pared with each single decision path (following the order
shown in Fig. 3), the final classification accuracy increased
by 12.99%, 7.18%, 16.35%, 18.96%, 25.74%, and 20.57%
(p < 0.01, for all one-tailed paired t-tests) separately for
three-class classification problem, and increased by 11.01%,
9.83%, 14.08%, 17.19%, 25.46%, 18.25% (p < 0.01, for
all one-tailed paired t-tests) separately for four-class clas-
sification problem. The fusion results of decision paths in
each mode are shown in Fig. 9. For the EEG mode, results
for three-class classification problem and four-class classi-
fication problem are 80.06% and 62.50%, respectively. For
the fNIRS mode, results for three-class classification prob-
lem and four-class classification problem are 74.17% and
60.23%, respectively. Further, bimodal fusion performs sig-
FIGURE 8. Weight distribution intra- and inter-subject.
nificantly better than both single-modal methods (p < 0.01,
for both one-tailed paired t-tests). The detailed accuracy and
and MCSP path), indicating that in many cases, two individ- Kappa coefficients of all subjects are shown in Table 4,
ual decision paths give different predictions, which further and the confusion matrix of multiclass classification results
suggests that these decision paths can complement each other. are shown in Table 2 and Table 3. Obviously, in Table 3,
The fusion result of each aforementioned strategy is the sensitivities of LI and RI are lower than those of MA and
shown in Fig. 10. The fusion strategy (c), namely the BL. However, if we consider LI and RI as a single task, MI can
reliability-based decision-level fusion strategy, can achieve achieve the highest sensitivity in Table 2. This indicates that
a better performance for all classification tasks. The rea- LI and RI are difficult to distinguish, but easily distinguish-
son might be that weights are assigned more precisely and able with respect to other tasks, further demonstrating that
properly based on the reliability of decision paths. By intro- motor imagery of the left and right hand share a similar
ducing a nonlinear growth of reliability with classification brain pattern that differs significantly from MA and BL tasks.
accuracy in a specific decision path, the weights are tuned Therefore, when designing an asynchronous BCI, selecting
more sensitively when the accuracy improves. Besides, one tasks with diverse patterns is of vital importance.

VOLUME 7, 2019 165311

 X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

TABLE 3. Confusion matrix for four-class classification problem.

FIGURE 9. Classification results of each decision path and modality.

TABLE 4. Classification results of each subject.

FIGURE 10. Classification results of each fusion strategy.

Since a leave-one-out cross-validation strategy was

applied, for the mental state prediction of every single trial,
weights of these independent decision paths were calculated
by the training and the validation set. The weight distribution
of all trials of one representative subject (subject 26) is
displayed in Fig. 8a. The low variability of the weight dis-
tribution within each decision path indicates their respective
stable effects in a specific subject. Moreover, we calcu-
lated the mean weights of all trials for each subject. The
weight distribution across all subjects is shown in Fig. 8b.
Although the inter-subject variability of weights is larger
than intra-subject variability, the weight distribution across
all subjects in Fig. 8b shares some similar patterns with that
within one subject. For example, the weights of the CSP
decision path are significantly higher than others (p < 0.01,
compared with the power spectrum decision path), and the
weights of HMM decision path are significantly lower than
others (p < 0.01, compared with the MCSP decision path).
In general, decision paths of the EEG mode contribute to a
larger extent to the final prediction in comparison to those of
the fNIRS mode.
average mutual information of each modality listed in the
E. COMPARISON AND CORRELATION BETWEEN previous columns. The mutual information of intra-modality
EEG AND fNIRS is significantly higher than that of inter-modality (one-tailed
The basis for the fusion of two modalities is that the infor- paired t-test: p < 0.01). As introduced before, the EEG
mation they provide is complementary. To see this more and fNIRS characterize cerebral electrophysiological activ-
clearly, in the last two columns of Table 1, we calculate the ities and hemodynamic activities, respectively. Accordingly,

165312 VOLUME 7, 2019

X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

TABLE 5. Centers of the optimal 3s window. F. COMPARISON WITH PREVIOUS STUDIES

The review of asynchronous BCI studies (after 2011) focus-
ing on 3-class and 4-class classification problems is shown
in Table 6 and Table 7 respectively. In all of these studies,
models are built specific to each subject and the reported
the amount of mutual information shared between methods results are based on intra-subject performance evaluation.
based on different modes is relatively small, due to the dif- The proposed IDPF method can achieve an excellent per-
ferent information that the EEG and fNIRS carry. Therefore, formance for both problems. Because the dataset we used is
the combination of multiple sensing modes can contribute to the first EEG-fNIRS BCI dataset, lately established in 2017,
a more accurate brain analysis. limited studies have been performed based on this dataset.
Furthermore, we compared the most separable time seg- The studies using the same dataset mainly focus on the
ments of the EEG and fNIRS. Since the leave-one-out binary-classification problem, results of which are also com-
cross-validation was applied for each test sample, the optimal pared in this study and shown in Table 8. In [58], an 84.94%
sliding window was selected based on the training and valida- accuracy of ‘‘MA vs. BL’’ is achieved through fNIRS data in
tion sets. On average, more than 86% of samples of a single the dataset. In [59], likewise for the ‘‘MA vs. BL’’ problem,
subject share the same optimal sliding window parameters an accuracy lower than 80% is achieved through EEG data
(both for the EEG and fNIRS). We define this window as in the dataset using multi-channel tensor subspace learning
the optimal one for that subject. The center of the optimal and regression to reduce the channel number. Our method
window is defined as the optimal time point for classification. outperforms both of them. Moreover, the dataset builder also
The optimal classification time (after task onset) of EEG, reports their classification accuracy based on bimodal data in
HbO and HbR is shown in Table 5. For EEG, the optimal clas- the dataset, utilizing some widely applied methods such as
sification time is almost halfway through the tasks. However, CSP-based features for EEG classification and mean value
for HbO and HbR, the hemodynamic response is strongest and average slope features for fNIRS classification [42].
near the end of the tasks, which is significantly hysteretic Also in [42], a meta-classifier was employed to achieve an
compared with electrophysiological response (p < 0.01 for accuracy of 74.2% for ‘‘LI vs. RI’’ and an accuracy of 88.1%
both one-tailed paired t-tests). Other EEG-fNIRS epilepsy for ‘‘MA vs. BL’’. Our results are better than both of them
studies [25]–[27] show similar results. Therefore, the hystere- (78.56% for ‘‘LI vs. RI’’ and 91.15% for ‘‘MA vs. BL’’).
sis quality of the hemodynamic response can help to choose We hope that, with more bimodal BCI studies utilizing this
the optimal time window for classification. public dataset, a broader method comparison based on the
TABLE 6. Review of previous studies (3-class classification problem).

VOLUME 7, 2019 165313

 X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

TABLE 7. Review of previous studies (4-class classification problem).

TABLE 8. Comparison of studies using the same dataset.

same dataset can be possible in the future. Besides, for each of the IDPF framework does not depend on a large amount of
subject, no trials were removed due to signal quality. The data in the training set, because in each decision path, a rela-
dataset builder also declared that no channel or subject selec- tively low-dimensional feature vector that indicates a specific
tion was performed based on signal quality [42]. Since the aspect of mental state is handled. Besides, as one of the
data volume of 120 trials for each of the 29 subjects can contributions to EEG-fNIRS BCI technique, mutual informa-
greatly reduce contingency of result and no data selection tion is employed to evaluate the information shared between
was performed, our classification accuracy should have high different paths and modalities, which is of great significance
credibility. for modality fusion. Based on our flexible framework of the
The present work has proved the significantly modular structure, the algorithm in a specific decision path
improved performance of multiple decision path fusion. can be replaced by another one according to the actual situa-
Several independent decision paths were generated and inte- tion, as long as it is independent of other paths, in other words,
grated to jointly give predictions of mental states. The effect as long as it shares little mutual information with other paths.

165314 VOLUME 7, 2019

X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

Although the prediction given by each decision path is usually [2] A. Chowdhury, Y. K. Meena, H. Raza, B. Bhushan, A. K. Uttam, N. Pandey,
weak, their fusion yields a high accuracy through IDPF. The A. A. Hashmi, A. Bajpai, A. Dutta, and G. Prasad, ‘‘Active physical
practice followed by mental practice using BCI-driven hand exoskeleton:
decision paths with low accuracy but highly independent A pilot trial for clinical effectiveness and usability,’’ IEEE J. Biomed.
contribution (low mutual information) are usually ignored by Health Inform., vol. 22, no. 6, pp. 1786–1795, Nov. 2018.
researchers. For future work of BCI community, more studies [3] C. G. Lim, T. S. Lee, C. Guan, D. S. S. Fung, Y. Zhao, S. S. W. Teng,
H. Zhang, and K. R. R. Krishnan, ‘‘A brain-computer interface based atten-
focusing on finding new EEG-based and fNIRS-based algo- tion training program for treating attention deficit hyperactivity disorder,’’
rithms which provide irreplaceable information can further PLoS ONE, vol. 7, no. 10, Oct. 2012, Art. no. e46692.
promote BCI technology. Moreover, the present work is one [4] T. Carlson and J. Del R Millan, ‘‘Brain-controlled wheelchairs: A robotic
architecture,’’ IEEE Robot. Autom. Mag., vol. 20, no. 1, pp. 65–73,
of the very few studies focusing on the bimodal and multiclass Mar. 2013.
classification BCI, which can fill in the gaps in this field. [5] J. Meng, S. Zhang, A. Bekyo, J. Olsoe, B. Baxter, and B. He, ‘‘Noninvasive
The MCSP algorithm will also further inspire more studies to electroencephalogram based control of a robotic arm for reach and grasp
tasks,’’ Sci. Rep., vol. 6, p. 38565, Dec. 2016.
develop fNIRS-based signal processing methods. All of these
[6] T. O. Zander, K. Shetty, R. Lorenz, D. R. Leff, L. R. Krol, A. W. Darzi,
detailed results will contribute to the body of knowledge on K. Gramann, and G.-Z. Yang, ‘‘Automated task load detection with
BCI technique, especially EEG-fNIRS based BCIs. electroencephalography: Towards passive brain-computer interfacing in
Nevertheless, one limitation is that, the present work used robotic surgery,’’ J. Med. Robot. Res., vol. 2, no. 1, Dec. 2016,
Art. no. 1750003.
offline signal processing and machine learning approaches to [7] G. Lin, W. Cheng, J. Zhang, and J. Wang, ‘‘EEG classification for motor
discriminate different mental states. A practical BCI relies imagery and resting state in BCI applications using multi-class Adaboost
heavily on the development of on-line and real-time algo- extreme learning machine,’’ Rev. Sci. Instrum., vol. 87, no. 8, 2016,
Art. no. 085110.
rithms. For the on-line BCI, the computing speed, comput- [8] P. Xu, P. Yang, X. Lei, and D. Yao, ‘‘An enhanced probabilistic LDA for
ing resource consumption and system robustness all need multi-class brain computer interface,’’ PLoS ONE, vol. 6, no. 1, Jan. 2011,
to be taken into account, which is a more complicated Art. no. e14634.
[9] D. Wang, D. Miao, and G. Blohm, ‘‘Multi-class motor imagery EEG
case. We hope this work can provide theoretical support on decoding for brain-computer interfaces,’’ Frontiers Neurosci., vol. 6,
modality fusion for future BCI based on double or multiple p. 151, Oct. 2012.
modalities. [10] H.-J. Hwang, J.-H. Lim, and C.-H. Im, ‘‘Classification of mental states
based on spatiospectral patterns of brain electrical activity,’’ J. Biomed.
VI. CONCLUSION Eng. Res., vol. 33, no. 1, pp. 15–24, 2012.
[11] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten, ‘‘Multiclass brain–
We have proposed the IDPF algorithm for bimodal and mul- computer interface classification by Riemannian geometry,’’ IEEE Trans.
ticlass classification asynchronous BCI in this work. Sev- Biomed. Eng., vol. 59, no. 4, pp. 920–928, Apr. 2012.
eral decision paths were trained separately and provided [12] S. H. F. Syam, H. Lakany, and B. A. Conway, ‘‘Single trial classification of
EEG in predicting intention and direction of wrist movement: Translation
mental state predictions independently. A reliability-based toward development of four-class brain computer interface system based
decision-level fusion strategy was performed to achieve the on a single limb,’’ in Proc. 8th Int. Conf. Adv. Cogn. Technol. Appl. New
final decision on mental state prediction. The accuracy can York, NY, USA, Curran Associates, 2016, pp. 90–95.
[13] S. Shahtalebi and A. Mohammadi, ‘‘Ternary ECOC classifiers coupled
reach up to 85.60% for the three-class classification problem,
with optimized spatio-spectral patterns for multiclass motor imagery clas-
and 70.32% for the four-class classification problem. This sification,’’ in Proc. IEEE Int. Conf. Syst. Man, Cybern. (SMC), Oct. 2017,
study is an example proving the effectiveness of modal fusion, pp. 2231–2236.
i.e., the combination of EEG and fNIRS in particular. Besides, [14] J. Mellinger, G. Schalk, C. Braun, H. Preissl, W. Rosenstiel, N. Birbaumer,
and A. Kübler, ‘‘An MEG-based brain-computer interface (BCI),’’ Neu-
the MCSP proposed in our previous work has also been roImage, vol. 36, no. 3, pp. 581–593, Jul. 2007.
employed in one decision path for fNIRS signal analysis. [15] R. Sitaram, A. Caria, R. Veit, T. Gaber, G. Rota, A. Kuebler, and
We further prove the effectiveness of MCSP in the specific N. Birbaumer, ‘‘FMRI brain-computer interface: A tool for neuroscien-
tific research and treatment,’’ Comput. Intell. Neurosci., vol. 2007, p. 1,
task of the present work using a larger data volume, which Apr. 2007.
we believe can contribute to the body of knowledge in fNIRS [16] X. Yin, B. Xu, C. Jiang, Y. Fu, Z. Wang, H. Li, and G. Shi, ‘‘Clas-
signal processing field. sification of hemodynamic responses associated with force and speed
imagery for a brain-computer interface,’’ J. Med. Syst., vol. 39, no. 5, p. 53,
Furthermore, since this framework is capable of discrim- Mar. 2015.
inating voluntary mental states with relatively obscure pat- [17] K.-S. Hong, N. Naseer, and Y.-H. Kim, ‘‘Classification of prefrontal and
terns, it has a high potential of being applied to detect mental motor cortex signals for three-class fNIRS–BCI,’’ Neurosci. Lett., vol. 587,
pp. 87–92, Feb. 2015.
illnesses like epilepsy, depression, and Alzheimer’s disease,
[18] C. S. Larissa and C. Tom, ‘‘Towards a ternary NIRS-BCI: Single-trial
all of which possess more specific characteristics. The pro- classification of verbal fluency task, stroop task and unconstrained rest,’’
posed IDPF method represents actually a common approach J. Neural Eng., vol. 12, no. 6, Oct. 2015, Art. no. 066008.
for the detection of mental states. Similarly to some existing [19] L. C. Schudlo and T. Chau, ‘‘Development of a ternary near-infrared spec-
troscopy brain-computer interface: Online classification of verbal fluency
clinical studies focusing on the combination of the EEG and task, stroop task and rest,’’ Int. J. Neural Syst., vol. 28, no. 4, Oct. 2017,
fNIRS to monitor mental states [24]–[26], we aim to adapt Art. no. 1750052.
our framework to a more specific clinical application in a later [20] J. Shin, J. Kwon, J. Choi, and C.-H. Im, ‘‘Performance enhancement of
a brain-computer interface using high-density multi-distance NIRS,’’ Sci.
study. Rep., vol. 7, p. 16545, Nov. 2017.
[21] S. D. Power, A. Kushki, and T. Chau, ‘‘Automatic single-trial discrimi-
REFERENCES nation of mental arithmetic, mental singing and the no-control state from
[1] J. J. Daly and J. R. Wolpaw, ‘‘Brain–computer interfaces in neurological prefrontal activity: Toward a three-state NIRS-BCI,’’ BMC Res. Notes,
rehabilitation,’’ Lancet Neurol., vol. 7, no. 11, pp. 1032–1043, 2008. vol. 5, no. 1, p. 141, 2012.

VOLUME 7, 2019 165315

 X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

[22] J. Shin, J. Kwon, J. Choi, and C.-H. Im, ‘‘Ternary near-infrared spec- [42] J. Shin, A. V. Lühmann, B. Blankertz, D.-W. Kim, J. Jeong, H.-J. Hwang,
troscopy brain-computer interface with increased information transfer rate and K.-R. Müller, ‘‘Open access dataset for EEG+NIRS single-trial clas-
using prefrontal hemodynamic changes during mental arithmetic, breath- sification,’’ IEEE Trans. Neural Syst. Rehabil. Eng., vol. 25, no. 10,
holding, and idle state,’’ IEEE Access, vol. 6, pp. 19491–19498, 2018. pp. 1735–1745, Oct. 2017.
[23] N. Roche-Labarbe, F. Wallois, E. Ponchel, G. Kongolo, and R. Grebe, [43] R. N. Vigário, ‘‘Extraction of ocular artefacts from EEG using independent
‘‘Coupled oxygenation oscillation measured by NIRS and intermittent component analysis,’’ Electroencephalogr. Clin. Neurophysiol., vol. 103,
cerebral activation on EEG in premature infants,’’ NeuroImage, vol. 36, no. 3, pp. 395–404, Sep. 1997.
no. 3, pp. 718–727, Jul. 2007. [44] A. Delorme and S. Makeig, ‘‘EEGLAB: An open source toolbox for
[24] K. Peng, P. Pouliot, F. Lesage, and D. K. Nguyen, ‘‘Multichannel analysis of single-trial EEG dynamics including independent component
continuous electroencephalography-functional near-infrared spectroscopy analysis,’’ J. Neurosci. Methods, vol. 134, no. 1, pp. 9–21, Mar. 2004.
recording of focal seizures and interictal epileptiform discharges in [45] C. Julien, ‘‘The enigma of Mayer waves: Facts and models,’’ Cardiovas-
human epilepsy: A review,’’ Neurophotonics, vol. 3, no. 3, Feb. 2016, cular Res., vol. 70, no. 1, pp. 12–21, Apr. 2006.
Art. no. 031402. [46] L. Kocsis, P. Herman, and A. Eke, ‘‘The modified beer-Lambert law
[25] Y. Sato, M. Fukuda, M. Oishi, A. Shirasawa, and Y. Fujii, ‘‘Ictal revisited,’’ Phys. Med. Biol., vol. 51, no. 5, pp. 91–98, Mar. 2006.
near-infrared spectroscopy and electrocorticography study of supplemen- [47] Y. Xu, H. L. Graber, and R. L. Barbour, ‘‘nirsLAB: A computing envi-
tary motor area seizures,’’ J. Biomed. Opt., vol. 18, no. 7, Jul. 2013, ronment for fNIRS neuroimaging data analysis,’’ in Proc. Biomed. Opt.,
Art. no. 076022. Apr. 2014, p. BM3A.1.
[26] D. K. Nguyen, J. Tremblay, P. Pouliot, P. Vannasing, O. Florea, L. Carmant, [48] G. Pfurtscheller, C. Neuper, D. Flotzinger, and M. Pregenzerb,
F. Lepore, M. Sawan, F. Lesage, and M. Lassonde, ‘‘Noninvasive con- ‘‘EEG-based discrimination between imagination of right and left
tinuous functional near-infrared spectroscopy combined with electroen- hand movement,’’ Electroencephalogr. Clin. Neurophysiol., vol. 103,
cephalography recording of frontal lobe seizures,’’ Epilepsia, vol. 54, no. 2, no. 6, pp. 642–651, Dec. 1997.
pp. 331–340, Feb. 2013. [49] B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K. R. Müller,
[27] A. Gallagher, M. Lassonde, D. Bastien, P. Vannasing, F. Lesage, C. Grova, ‘‘Optimizing spatial filters for robust EEG single-trial analysis,’’ IEEE
A. Bouthillier, L. Carmant, F. Lepore, R. Béland, and D. K. Nguyen, ‘‘Non- Signal Process. Mag., vol. 25, no. 1, pp. 41–56, Jan. 2008.
invasive pre-surgical investigation of a 10 year-old epileptic boy using [50] S. Wold, K. Esbensen, and P. Geladi, ‘‘Principal component analysis,’’
simultaneous EEG-NIRS,’’ Seizure, vol. 17, no. 6, pp. 576–582, Sep. 2008. Chemometrics Intell. Lab. Syst., vol. 2, nos. 1–3, pp. 37–52, 1987.
[28] L. A. Farwell and E. Donchin, ‘‘Talking off the top of your head: Toward [51] F. Z. Chelali, A. Djeradi, and R. Djeradi, ‘‘Linear discriminant analysis for
a mental prosthesis utilizing event-related brain potentials,’’ Electroen- face recognition,’’ in Proc. Int. Conf. Multimedia Comput. Syst., Apr. 2009,
cephalograph. Clin. Neurophysiol., vol. 70, no. 6, pp. 510–523, Dec. 1988. pp. 1–10.
[29] V. P. Oikonomou, S. Nikolopoulos, and I. Kompatsiaris, ‘‘A Bayesian [52] K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed.
multiple kernel learning algorithm for SSVEP BCI detection,’’ IEEE J. San Diego, CA, USA: Academic, 1990.
Biomed. Health Inform., vol. 23, no. 5, pp. 1990–2001, Sep. 2018. [53] L. Rabiner, ‘‘A tutorial on hidden Markov models and selected applications
[30] L. Cao, J. Li, H. Ji, and C. Jiang, ‘‘A hybrid brain computer interface system in speech recognition,’’ Proc. IEEE, vol. 77, no. 2, pp. 257–286, Feb. 1989.
based on the neurophysiological protocol and brain-actuated switch for [54] R. Sitaram, H. Zhang, C. Guan, M. Thulasidas, Y. Hoshi, A. Ishikawa,
wheelchair control,’’ J. Neurosci. Methods, vol. 229, pp. 33–43, May 2014. K. Shimizu, and N. Birbaumer, ‘‘Temporal classification of multichannel
[31] F. Lotte, M. Congedo, A. Lécuyer, F. Lamarche, and B. Arnaldi, ‘‘A review near-infrared spectroscopy signals of motor imagery for developing a
of classification algorithms for EEG-based brain-computer interfaces,’’ brain-computer interface,’’ NeuroImage, vol. 34, no. 4, pp. 1416–1427,
J. Neural Eng., vol. 4, no. 2, pp. R1–R13, Jun. 2007. 2007.
[32] D. Wu, J.-T. King, C.-H. Chuang, C.-T. Lin, and T.-P. Jung, ‘‘Spatial [55] S. D. Power, T. H. Falk, and T. Chau, ‘‘Classification of prefrontal activity
filtering for EEG-based regression problems in brain–computer interface due to mental arithmetic and music imagery using hidden Markov models
(BCI),’’ IEEE Trans. Fuzzy Syst., vol. 26, no. 2, pp. 771–781, Apr. 2018. and frequency domain near-infrared spectroscopy,’’ J. Neural Eng., vol. 7,
[33] G. Pfurtscheller, C. Neuper, A. Schlogl, and K. Lugger, ‘‘Separability of no. 2, p. 26002, Apr. 2010.
EEG signals recorded during right and left motor imagery using adaptive [56] J. Shin, J. Kwon, and C.-H. Im, ‘‘A ternary hybrid EEG-NIRS brain-
autoregressive parameters,’’ IEEE Trans. Rehabil. Eng., vol. 6, no. 3, computer interface for the classification of brain activation patterns during
pp. 316–325, Sep. 1998. mental arithmetic, motor imagery, and idle state,’’ Frontiers Neuroinform.,
[34] S. Ge, Y.-H. Shi, R.-M. Wang, P. Lin, J.-F. Gao, G.-P. Sun, K. Iramina, vol. 12, p. 5, Feb. 2018.
Y.-K. Yang, Y. Leng, H.-X. Wang, and W.-M. Zheng, ‘‘Sinusoidal signal [57] F. Orihuela-Espina, D. R. Leff, D. R. C. James, A. W. Darzi, and
assisted multivariate empirical mode decomposition for brain-computer G. Z. Yang, ‘‘Quality control and assurance in functional near infrared
interfaces,’’ IEEE J. Biomed. Health Inform., vol. 22, no. 5, pp. 1373–1384, spectroscopy (fNIRS) experimentation,’’ Phys. Med. Biol., vol. 55, no. 13,
Sep. 2018. p. 3701, Jun. 2010.
[35] M. Congedo, F. Lotte, and A. Lécuyer, ‘‘Classification of movement [58] E. Ergün and O. Aydemir, ‘‘Decoding of binary mental arithmetic based
intention by spatially filtered electromagnetic inverse solutions,’’ Phys. near-infrared spectroscopy signals,’’ in Proc. 3rd Int. Conf. Comput. Sci.
Med. Biol., vol. 51, no. 8, p. 1971, Mar. 2006. Eng. (UBMK), Sep. 2018, pp. 201–204.
[36] M. D. Pfeifer, F. Scholkmann, and R. Labruyère, ‘‘Signal processing [59] S. V. Eyndhoven, M. Boussé, B. Hunyadi, L. D. Lathauwer, and
in functional near-infrared spectroscopy (fNIRS): Methodological dif- S. V. Huffel, ‘‘Single-channel EEG classification by multi-channel tensor
ferences lead to different statistical results,’’ Frontiers Hum. Neurosci., subspace learning and regression,’’ in Proc. IEEE 28th Int. Workshop
vol. 11, p. 641, Jan. 2018. Mach. Learn. Signal Process. (MLSP), Sep. 2018, pp. 1–6.
[37] S. Zhang, Y. Zheng, D. Wang, L. Wang, J. Ma, J. Zhang, W. Xu, D. Li, and
D. Zhang, ‘‘Application of a common spatial pattern-based algorithm for
an fNIRS-based motor imagery brain-computer interface,’’ Neurosci. Lett.,
vol. 655, pp. 35–40, Aug. 2017.
[38] X. Jiang, X. Gu, Z. Mei, H. Ren, and W. Chen, ‘‘A modified common
spatial pattern algorithm customized for feature dimensionality reduction XINYU JIANG received the [Link]. degree (Hons.)
in fNIRS-based BCIs,’’ in Proc. 40th Annu. Int. Conf. IEEE Eng. Med. Biol.
in electronic engineering from Fudan University,
Soc. (EMBC), Jul. 2018, pp. 5073–5076.
China, in 2018, where he is currently pursuing
[39] F. Al-Shargie, T. B. Tang, and M. Kiguchi, ‘‘Stress assessment based
on decision fusion of EEG and fNIRS signals,’’ IEEE Access, vol. 5,
the [Link]. degree with the Center for Intelligent
pp. 19889–19896, 2017. Medical Electronics, Department of Electronic
[40] C.-H. Chuang, Z. Cao, J.-T. King, B.-S. Wu, Y.-K. Wang, and C.-T. Lin, Engineering, School of Information Science and
‘‘Brain electrodynamic and hemodynamic signatures against fatigue dur- Technology.
ing driving,’’ Frontiers Neurosci., vol. 12, p. 181, Mar. 2018. His research interests include biomedical signal
[41] A. P. Buccino, H. O. Keles, and A. Omurtag, ‘‘Hybrid EEG-fNIRS asyn- processing, EEG-fNIRS-based brain monitoring,
chronous brain-computer interface for multiple motor tasks,’’ PLoS ONE, brain–computer interface, and automated epileptic
vol. 11, no. 1, 2016, Art. no. e0146610. seizure detection.

165316 VOLUME 7, 2019

X. Jiang et al.: IDPF for Bimodal Asynchronous BCI to Discriminate Multiclass Mental States

XIAO GU received the [Link]. degree (Hons.) HAORAN REN received the B.S. and M.S.
in electronic engineering from Fudan University, degrees in biomedical engineering from Yanshan
China, in 2018, and the [Link]. degree (with dis- University, China, in 2013 and 2017, respectively.
tinction) in medical robotics and image guided She is currently pursuing the Ph.D. degree in
intervention from Imperial College London, U.K., biomedical engineering with Fudan University.
in 2019, where he is currently pursuing the Ph.D. Her current research interest includes func-
degree with the Hamlyn Centre. tional brain imaging using near-infrared spec-
His research interests include biomedical sig- troscopy (fNIRS) combined with EEG for disease
nal processing, healthcare systems, and computer detection and cognition.
vision.
WEI CHEN received the [Link]. and [Link].
degrees in telecommunication systems and smart
sensor systems from the School of Electrics and
Information Engineering, Xian Jiaotong Univer-
sity, China, in 1999 and 2002, respectively, and
the Ph.D. degree in performance monitoring and
impairment mitigation for optical communication
systems from the Department of Electrical and
Electronics Engineering, The University of Mel-
bourne, Australia, in 2007.
She was an Intern with Bell Laboratories, Germany, Alcatel-Lucent,
Stuttgart, Germany, in 2005. She was a Research Assistant with the
Department of Electrical and Electronics Engineering, The University of
Melbourne, Australia, in 2007. From 2007 to 2015, she was an Assistant
Professor with the Eindhoven University of Technology, The Netherlands.
From 2009 to 2013, she was the Chair of theme health care with the
KE XU received the [Link]. degree (Hons.) in Department of Industrial Design, Eindhoven University of Technology, The
electronic engineering from Fudan University, Netherlands. Since October 2015, she has been a Full Professor and the
China, in 2018, where he is currently pursuing Director of the Center for Intelligent Medical Electronics, Department of
the [Link]. degree with the Center for Intelligent Electronic Engineering, School of Information Science and Technology,
Medical Electronics, Department of Electronic Fudan University. Her research interests include patient health monitoring,
Engineering, School of Information Science and medical monitoring system design using wearable sensors, sleep monitoring,
Technology. brain activity monitoring, wireless body area networks, ambient intelligence,
His research interests include biomedical personalized and smart environment, smart sensor systems, and signal pro-
sensing technology, biomedical signal processing, cessing.
artificial intelligence for health care, wireless body Dr. Chen is an Associate Editor of the IEEE JOURNAL ON BIOMEDICAL HEALTH
sensing networks, ultra-low power consumption, and wearable embedded INFORMATICS.
system design.

VOLUME 7, 2019 165317