ORIGINAL RESEARCH

published: 25 April 2022

doi: 10.3389/fnins.2022.869522

EEG-TNet: An End-To-End Brain

Computer Interface Framework for
Mental Workload Estimation
Chaojie Fan 1,2 , Jin Hu 3 , Shufang Huang 4 , Yong Peng 1* and Sam Kwong 2
1
Key Laboratory of Traffic Safety on Track of Ministry of Education, School of Traffic and Transportation Engineering, Central
South University, Changsha, China, 2 Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong
SAR, China, 3 Hunan Communications Research Institute Co., Ltd., Hunan Communication & Water Conservancy Group
Ltd., Changsha, China, 4 School of Business and Trade, Hunan Industry Polytechnic, Changsha, China

The mental workload (MWL) of different occupational groups’ workers is the main
and direct factor of unsafe behavior, which may cause serious accidents. One of the
new and useful technologies to estimate MWL is the Brain computer interface (BCI)
based on EEG signals, which is regarded as the gold standard of cognitive status.
However, estimation systems involving handcrafted EEG features are time-consuming
and unsuitable to apply in real-time. The purpose of this study was to propose an end-
Edited by: to-end BCI framework for MWL estimation. First, a new automated data preprocessing
Gui Ye,
Chongqing University, China
method was proposed to remove the artifact without human interference. Then a
Reviewed by:
new neural network structure named EEG-TNet was designed to extract both the
Lizhuang Ma, temporal and frequency information from the original EEG. Furthermore, two types of
Shanghai Jiao Tong University, China
experiments and ablation studies were performed to prove the effectiveness of this
Yufei Chen,
Xi’an Jiaotong University, China model. In the subject-dependent experiment, the estimation accuracy of dual-task
Jianbo Zhu, estimation (No task vs. TASK) and triple-task estimation (Lo vs. Mi vs. Hi) reached
Southeast University, China
99.82 and 99.21%, respectively. In contrast, the accuracy of different tasks reached
*Correspondence:
Yong Peng
82.78 and 66.83% in subject-independent experiments. Additionally, the ablation studies
yong_peng@[Link] proved that preprocessing method and network structure had significant contributions
to estimation MWL. The proposed method is convenient without any human intervention
Specialty section:
This article was submitted to
and outperforms other related studies, which becomes an effective way to reduce human
Front. Neurosci, factor risks.
a section of the journal
Keywords: mental workload, brain computer interface, deep neural network, occupational safety, ergonomics
Frontiers in Neuroscience

Received: 04 February 2022

Accepted: 18 March 2022
Published: 25 April 2022
1. INTRODUCTION
Citation: Information systems are increasingly approaching the boundaries of human competence due
Fan C, Hu J, Huang S, Peng Y and to their increasing complexity and autonomy. A dynamic and automated adaptation of the
Kwong S (2022) EEG-TNet: An
system to the user state is required to minimize user overload in high-demand scenarios
End-To-End Brain Computer Interface
Framework for Mental Workload
(Mühl et al., 2014). A reliable understanding of the user’s current status, particularly the
Estimation. workload, is essential for timely and appropriate system adaptation (van Erp et al., 2010).
Front. Neurosci. 16:869522. The workload is a direct factor in unsafe operations. Workers of special occupational groups
doi: 10.3389/fnins.2022.869522 such as construction workers, car drivers, pilots are prone to physical exhaustion and lack of

Frontiers in Neuroscience | [Link] 1 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

consciousness under high workloads for a long time, leading 1.1. Related Study
to numbness of safety conditions and causing great insecurity. 1.1.1. Handcrafted Features-Based BCI Framework
Therefore, it is extremely important to effectively assess and Brain-computer interface, as a new human-computer interaction
reduce the workload of operators in preventing unsafe behaviors technology, provides a new method of communication with the
and reducing dangerous accidents. Thus, workload estimation outside world and enables direct human control of machines. In
is an actively growing research field, for it possesses numerous recent years, with deep cross-fertilization of artificial intelligence
human factor applications in many occupational groups to technology in neuroscience, cybernetics, computer science, and
reduce safety risks (Roy et al., 2016; Yin et al., 2019). other related fields, research on BCI cognitive status computing
The workload is mainly divided into physical workload systems based on EEG. There are a large number of BCI
and mental workload (MWL). When the human body is frameworks for MWL assessment have been proposed in recent
under different physical workloads, various physiological years, and most of the research has used handcrafted features.
parameters such as oxygen consumption, heart rate, pulmonary Lim et al. (2018) assessed the MWL induced by the single-session
ventilation, energy expenditure rate, and various chemical simultaneous capacity (SIMKAP) experiment. They collected
enzymes related to energy conversion show changes. the 14-channels of EEG and extracted different bands’ power
(Roscoe, 1992; Abdelhamid and Everett, 2002). However, spectral density (PSD). The Neighborhood Component Analysis
the estimation of MWL is more complicated than physical (NCA) was used to select critical features and the Support
workload, while the former is more closely associated Vector Regression (SVR) model was trained to assess the MWL.
with safety. A helmet with EEG sensors was designed by Wang et al.
There are two types of MWL estimation methods, subjective (2017) to meet the requirements of the construction industry,
and objective estimation methods (Hogervorst et al., 2014; and they designed different construction activities to induce
Charles and Nixon, 2019). The subjective test is a self-recorded different levels of MWL. The results showed that Gamma waves
and a questionnaire-based test in which the subject’s workload is and Fp1 and Tp10 channels are good candidates for MWL
scored. Among a large number of subjective estimation methods, estimation in the frequency domain. However, the unavoidable
the National Aeronautics and Space Administration’s Task Load limitations of current BCI frameworks which use handcrafted
Index (NASA TLX) (Hart, 2006) and Subjective Workload features should not be ignored. The extraction of EEG signal
Assessment Technique (SWAT) (Reid and Nygren, 1988) are the features requires researchers to master interdisciplinary theories
most popular subjective estimation methods. Additionally, the and research results in stochastic signal analysis and cognitive
objective estimation methods are used to estimate their workload neuroscience, raising the threshold for researching this field
by collecting the object’s physiological signals. (Cheng et al., 2022). Thus, the incomprehensibility of domain
The objective test has developed rapidly in recent years knowledge can limit the extracted features that cannot effectively
due to developments in sensor technology. The rationality for represent the implicit MWL-related information in the original
the objective test based on physiological signals is that when signal. In addition, restricted by the performance of computing
people are under MWL, the parameters of each physiological units of wearable devices, algorithms with high computational
condition deviate from the normal state. Thus, it is possible complexity cannot be applied on brain-computer interface
to detect changes in the body’s physiological signals to systems. The computation of EEG features, especially non-
estimate [Link] changed physiological parameters include linear features such as entropy value and complexity, requires
cardiac activity, electrical brain activity, eye movements, and much time and, thus, cannot meet the needs of brain-computer
metabolic changes (Fairclough and Houston, 2004). Therefore, interface systems.
many physiological indicators have been used to estimate
MWL, such as electrocardiograms (ECG), eye movements, 1.1.2. Deep Learning-Based BCI Framework
electroencephalography (EEG) measurements, respiration, and To solve the feature extraction problem, inspired by the success
electromyography (EMG). Among these physiological indicators, of the feature extraction ability of convolutional neural network,
EEG is widely used because MWL changes are closely linked decoding EEG according to CNN, which constructing the end-
to brain cortical activity and because it is non-smooth, non- to-end BCI framework are receiving increasing attention. There
invasive, and highly discriminative (Wilson et al., 1994; Dehais are some related end-to-end studies in the field of EEG-based
et al., 2020; Pieper et al., 2021; Liu et al., 2022; Yu et al., 2022). This BCI frameworks for other tasks, such as emotion recognition,
is why EEG is also known as the gold standard. In conclusion, word imagined, and epileptic seizure recognition (Xu et al., 2020;
EEG had the best and most reliable estimation performance of Datta and Boulgouris, 2021; Hu et al., 2021). Furthermore, unlike
MWL. the images, EEG signals are typically time-series signals, and the
To sum up, in terms of accuracy and practicality, EEG is evolutionary trends in neural activity during complex or simple
optimal for estimating the MWL. The entire framework also can cognitive processing are of equal interest. Therefore, combined
be referred to as a brain-computer interface (BCI) by means of models by merging CNN and Long short-term memory (LSTM)
computer algorithms that decode information from the brain network was proposed and attempted to extract features by CNN
and thus access the state of the human. In this study, an end-to- and obtain the temporal information by LSTM layers. However,
end BCI framework using EEG is proposed to estimate workers’ most CNN-LSTM studies use 1-D or 2-D convolutional kernels
MWL continuously, which can directly decode EEG without and full connected layers to process EEG data. The original
feature extraction. EEG was transformed into 1-D or 2-D tensors, which were then

Frontiers in Neuroscience | [Link] 2 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

FIGURE 1 | The detailed flowchart of this study.

fed into the LSTM layers. The above algorithms disrupt the First, our proposed end-to-end BCI framework for workers’
temporal information and the transformed data does not have a MWL estimation, which decodes mental workload related
real time sequence in the “time step” dimension (Xu et al., 2020). relevant information directly from raw EEG, is able to avoid the
Therefore, the effect of LSTM layers is weakened because of the time consumption associated with complex feature extraction
wrong temporal information. and thus meet the hardware requirements of brain-computer
interface systems.
1.2. Contribution Second, this method uses a combination of filters, ASR,
To fill the research gap mentioned above, in this study, a and ICA with ADJUST to obtain relatively pure EEG signals
convenient and efficient end-to-end BCI framework for MWL without manual involvement. Additionally, the MWL related
estimation was proposed. The contributions of the article can be neural information is decoded smoothly from the original EEG
summarized as follows: by the designed time fixed 3-D-CNN layers while the temporal

Frontiers in Neuroscience | [Link] 3 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

dimension is unchanged. Then the following bi-LSTM layers can are calculated. Second, update memory cells Ct by combining
be used to extract temporal features. forgetting gates ft and input gates it .
Third, according to two types of comparison experiments
and ablation studies, the estimation effectiveness of EEG-TNet
can be proved. it = σ (Wi [ht−1 , xt ] + bi ) (3)

2. NEURAL NETWORK PRELIMINARY gt = tanh(Wc [ht−1 , xt ] + bC ) (4)

In this section, some preliminary knowledge about the
neural network including convolutional layer, LSTM, and fully Ct = ft ∗ Ct−1 + it ∗ gt (5)
connected layers were introduced, which are the basis of our
EEG-TNet method. Ultimately, we need to determine the output, which is based
on the state of our memory cells Ct . First, a sigmoid layer is
2.1. Convolutional Layer used to determine which parts of the memory cell state will be
In this study, convolutional layers include four normal output. Second, the memory cell is processed through tanh and
convolutional layers, a depthwise convolutional, and a pointwise multiplied by ot .
convolutional. For the normal convolutional, the input of the
convolutional layer is Xin (Cin , Din , Hin , Win ), and the output
yout (Cout , Dout , Hout , Wout ). The formula of the convolutional ot = σ (Wo [ht−1 , xt ] + bo ) (6)
layer can be described as follows:

ht = ot ∗ tanh(Ct ) (7)
CX
in −1
yout = b + w ⋆ xin (1)
2.3. Fully Connected Layer
k=0
The fully connected layer serves as an “estimator” in the entire
Where the ⋆ is the valid 3D cross-correlation operation. The neural network structure. The procedures such as convolutional
shape of yout (Cout , Dout , Hout , Wout ) can be calculated according layers, pooling, LSTM, and activation function translate the
to the kernel size (KD , KH , KW ) and the kernel number original data to the hidden feature space. The fully connected
Cout . Specifically, the depthwise separable convolution (Chollet, layer transfers them to the sample labeling space. As Equation
2017) which consists of depthwise convolutional and pointwise (8) shows, the fully connected layer multiplies the weight matrix
convolutional was used in our research to extract spatial with the input vector and then adds the bias.
information from EEG with a lower number of convolutional
parameters (Chollet, 2017).
y = xAT + b (8)
2.2. LSTM Layer
By designing time-fixed 3D convolutional layers, we retain where AT is the learnable parameter and b is the bias. In addition,
the EEG information in each time step and further analyze a softmax activation function may be used to calculate the likely
the temporal information using LSTM networks (Hochreiter distribution of the output classes. In the final FC layer, the
and Schmidhuber, 1997). Recurrent neural networks (RNN) softmax function is utilized, which is defined as follows:
have an excellent memory capability owing to their distinctive
self-connected structure, which has an absolute advantage in
processing temporal data (Mikolov et al., 2010). The LSTM ei
Si = Pk for i = 1, ...k. (9)
j
network is a popular expansion of RNN to address the gradient j=1 e
disappearance problem while RNN only processes long-term
data. The LSTM introduces a gating mechanism to control where i is the input vector, the output Si is between o to 1, and
P
the rate of accumulation of information, including adding new
i Si = 1
information and forgetting previous information by using the
gates. There are three gates including input gate it , forget gate
ft , cell gate gt , and output gate ot , respectively. Specifically, the 3. METHODS
forget gate ft controls the rates of previous information required
The detailed procedure of this study can be summarized in
to be forgotten about the internal state ct−1 at the last moment.
several steps, which are described by the detailed flowchart
shown in Figure 1. In this study, first, we preprocessed the data
ft = σ (Wf [ht−1 , xt ] + bf ) (2) from the STEW database (Lim et al., 2018) by our designed
automated methods. Then the processed EEG was directly
The input gate it determines the rates of new information which imported to the proposed EEG-TNet to estimate the MWL. The
is allowed to be added to the current ct . Two steps are required comparison studies and ablation studies were performed to prove
to achieve this. First, calculate the input gate it and cell gate gt the effectiveness of the proposed end-to-end EEG-TNet model.

Frontiers in Neuroscience | [Link] 4 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

FIGURE 2 | The framework of the EEG-TNet model. The EEG-TNet model consists of Data segmentation, dimension expansion, time fixed 3-D-CNN layers, Bi-LSTM
layer, fully connected layers, and softmax operation.

3.1. MWL EEG Database 1. High-pass filter raw data at 1 Hz and low-pass filter raw data
The database used in this study is STEW (Lim et al., 2018), which at 40 Hz.
contains EEG data of 48 subjects under different MWL levels. 2. Notch filter raw data at 50 Hz to avoid power line interference.
Specifically, the subjects performed the Simultaneous Capacity 3. Perform Artifact Subspace Reconstruction (ASR) (Chang
(SIMKAP) test to induce MWL. After the test, all the subjects et al., 2018).
were required to finish the subjective questionnaire to report 4. Perform Independent Component Analysis (ICA).
their MWL, which is a 9-point rating scale. During the whole 5. ADJUST (Mognon et al., 2011) was performed to automated
experiments, the EEG signals were recorded using an Emotiv inspect the artifact component from ICA.
EPOC EEG headset with 14 electrodes (AF3, F7, F3, FC5, T7, 6. Average re-reference the data channels.
P7, O1, O2, P8, T8, FC6, F4, F8, AF4) and two reference
channels (CMS, DRL). The sampling frequency was 128 Hz and 3.3. EEG-TNet Architecture
the resolution was 16-bit A/D. In this study, the classifier was The architecture of our EEG-TNet framework is inspired by
proposed to finish two tasks, the first one is classified “No Task” the network architecture EEGNet of Lawhern et al. (2018),
vs. “SIMKAP Task”, which was a binary classification. The second which is a widely used end-to-end EEG BCI framework. The
task was classifying Low vs. Moderate vs. High MWL, which was detailed framework of our proposed EEG-TNet model can be
divided by a rating scale. A detailed definition of the label can be summarized in three steps, which are shown in Figure 2. Step
found in the article (Lim et al., 2018). 1 is to segment the raw EEG to the required size and expand a
new dimension for the model need, which is used to keep the
temporal dimension stable. Step 2 is to extract the temporal and
3.2. Data Preprocessing spatial information from each EEG fragment without between-
To meet the requirements of the automation process, we fragments temporal information loss according to the designed
eliminated parts of the preprocessing process that require manual temporal fixed 3-D-CNN layers. Step 3 is to extract the temporal
intervention, such as manual artifact removal and manual information between each EEG fragment by using the LSTM
judgment of ICA components to remove artifacts, especially layer. The output of the last time step in the last layer is used
eye movement artifacts (Fan et al., 2021; Peng et al., 2021). to compute the final status according to the fully connected layer
This undoubtedly reduces the quality of the data and, therefore, and the softmax function.
the accuracy of the recognition, but it makes sense for real-
world applications (Rosanne et al., 2021). Table 1 shows the 3.3.1. Data Conversion
comparison of traditional preprocessing steps and ours. The The original EEG signals are defined as D = (d1 , d2 , ...dS ) ∈
whole preprocessing steps are RS ×C, where S is the time- series length of the original EEG,

Frontiers in Neuroscience | [Link] 5 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

TABLE 1 | Comparison of traditional preprocessing steps and ours.

Traditional steps Human interference Ours Human interference Goal

Filters NO Filters NO Remove low-frequency drifts and

power line interference
Manual artifacts remove YES ASR NO Remove drifts like muscle activity,
sensor motion
ICA with manual judgment YES ICA with ADJUST NO Remove artifacts especially eye
movement artifact

and C denotes the channel number. Similar to the previous BCI The final output is obtained at each moment by combining the
task, the original EEG was segmented and constructed by using output of the forward and backward layers at the corresponding
the overlapping sliding window and non-overlapping sliding moment. In this study, as shown in Figure 2, the output sample
window. The input dataset X̂ = (x̂1 , x̂2 , ...x̂M ) ∈ RM×T×C×L , shape of the time-fixed convolutional layers is T × 1 × 1 × 8, so
where M is the number of samples. The sample size of each that both the forward layer and backward of Bi-LSTM have 8 cells
sample x̂i ∈ X̂ ∈ (1, 2, ..., M) was (T × C × L), the C × L is the size to fill the data shape.
of per EEG fragments, which is set as 64 × 14 in this study, which
means the 0.5s EEG signals of two forehead channels. In addition,
the number of EEG fragments was T, the larger T represents 4. RESULTS
the longer EEG data considered per sample. Most of the related
In this section, two types of experiments were conducted to
studies analyze the input sample as a graph, where C × L is the
evaluate the MWL estimation performance using the proposed
height and width of the graph, and the dimension T is the channel
EEG-TNet BCI framework. The first type of experiment is
size of EEG, such as the RGB. However, after 2-d convolutional
subject-dependent while the second one is subject-independent.
layers, the temporal information between each fragment might
be lost. In this step, we expand a new dimension whose size
is 1 to meet the requirement of channel size. Furthermore, 4.1. Subject-Dependent Experiment
dimension T was considered the depth of the sample, which is In the subject-dependent experiment, we adopt a similar
stale during the whole convolutional process. Finally, the dataset experimental protocol as that of Chakladar et al. (2020), Kingphai
X = (x1 .x2 ..., xN ) ∈ RN×T×1×C×L . and Moshfeghi (2021), Zhu et al. (2021). The five-fold cross-
validation method was applied to evaluate the performance
3.3.2. Time-Fixed Convolutional Layer of the framework. As shown in Figure 2, specifically, the first
Four convolutional layers were used in the EEG-TNet model. 80% of all samples were selected as the training set and the
To ensure the temporal dimension is unchanged, the KD of remaining 20% of the samples were kept aside as the test
the kernel size (KD , KH , KD ) is set as 1 during the whole dataset. Then the other 20% of the samples were selected as
convolutional processing. First, eight 3-d normal convolutional the second test set, the last 80% of samples were the second
filters of size (1,1,2/L) were used to extract frequency features training set. Divide all samples like this five times and the average
from the EEG signal (Lawhern et al., 2018). Then 16 accuracy of the five experiments was taken as the final result.
convolutional filers of size (1,C,1) are fitted for the channel In addition, five-fold cross-validation was also applied to find
information aggregation. Subsequently, an average pooling optimal hyperparameters.
operation (kernel size = 1 × 1 × 4) is performed to aggregate The proposed EEG-TNet framework was compared with
information and reduce the data dimension. Then, 16 depthwise the other four baseline methods for MWL estimation on the
separable convolutions are constructed, which consists of 16 STEW dataset under the subject-dependent experiments setting,
depthwise convolutional filters (1 × 1 × 16) and pointwise as shown in Table 2. The results showed that the proposed
convolutional filters (1 × 1 × 1). EEG-TNet framework achieved higher estimation accuracies
than the other four methods. Although most of the recent
3.3.3. Bi-LSTM Layer studies use many kinds of features like frequency features
As we introduced before, the traditional LSTM layer receives [PSD (Chakladar et al., 2020; Kingphai and Moshfeghi, 2021)],
the inputs solely in the forward direction through hidden states, non-linear features [Approximate Entropy (ApEn) (Chakladar
which only retains the past information. Bidirectional LSTM et al., 2020; Kingphai and Moshfeghi, 2021)], linear features
(BLSTM) has been proposed to solve the problem, which has [autoregressive coefficient (AR) (Chakladar et al., 2020)], and
two layers named forward layer and backward layer. The forward the graph-based features (clustering coefficient, mean degree)
layer is computed forward from moment 1 to moment t, and (Zhu et al., 2021). However, traditional machine learning models
the output of the forward hidden layer is obtained and saved (SVM and random forest cannot learn the full EEG information.
at each moment. In the backward layer, the output of the Moreover, some combined deep neural networks (CNN-LSTM,
backward implicit layer is obtained and stored at each moment BLSTM-LSTM) show better performance than machine learning
by computing the backward layer from moment t to moment 1. models, it is still well below our proposed model.

Frontiers in Neuroscience | [Link] 6 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

TABLE 2 | Comparisons of the estimation accuracy (%) of subject-dependent experiments among the various methods.

References Method Features No task vs. task Lo vs. Mi vs. Hi

Zhu et al. (2021) SVM Graph features 89.60 79.50

Chakladar et al. (2020) Random Forest Frequency features Linear features 83.00 78.46
Non-Linear features
Chakladar et al. (2020) CNN-LSTM Frequency features Linear features 85.21 76.76
Non-Linear features
Kingphai and Moshfeghi (2021) BLSTM-LSTM Frequency features Non-Linear features 91.15 89.44
EEG-TNet None 99.82 99.21

The bold values indicate the best performance accuracies.

4.2. Subject-Independent Experiment random results (33.33%). In contrast, our estimation accuracies
The leave-subject-out (LSO) cross-validation method was used range from 52.00 to 74.72%. On the most difficult estimation task
to evaluate the performance of the proposed framework in the with high MWL, our results were nearly twice as good as theirs.
subject-independent experiments. As shown in Figure 3, in the
LSO cross-validation experimental protocol, the EEG samples 5. DISCUSSION
of 36 subjects (80% of a total of 45 subjects) were selected for
training the model and the last EEG samples of 8 subjects (20% 5.1. Practicability
of a total of 45 subjects) were used to test the model performance. The main objective of this study is to propose a practical
The whole process was repeated five times so that all the subjects’ and effective MWL estimation method for workers of special
samples were taken as the test set. The average accuracy of the occupational groups, which can be used to ensure safety during
five experiments was taken as the final result. Similarly, the the course of their work. EEG signals are regarded as the
hyperparameters were found according to the five-fold cross- gold standard and BCI systems based on EEG signals have
validation in the training set. natural advantages. However, most BCI systems cannot meet
Compared with the subject-dependent experiments, there are the requirements of online evaluation due to the high manual
fewer studies that perform the subject-independent experiments involvement in signal noise reduction, complex and time-
because of their difficulty. The proposed EEG-TNet framework consuming feature extraction, and other disadvantages. In order
was compared with the other three baseline methods under the to solve the disadvantage of manual involvement in signal noise
subject-independent experiments setting, as shown in Table 3. It reduction, this system uses a combination of filters, ASR, and
is worth noting that no studies were conducted with either dual- ICA with ADJUST to obtain relatively pure EEG signals without
task estimation (No Task vs. Task) or triple task estimation (Lo vs. manual involvement.
Mi vs. Hi) subject-independent experiments simultaneously until The classification model is at the heart of the BCI system,
now (Lim et al., 2018; Pandey et al., 2020). because of the system’s computing time, deployment method, and
Table 3 summarized the comparative results regarding the evaluation accuracy depends on it. In most related BCI systems,
average estimation accuracy under the subject-independent the handcrafted features were used to estimate the MWL, which
experiments setting. Pandey et al. (2020) realized that may be computationally demanding and not suitable for a real-
handcrafted features would slow down the speed of computing time system. This study exploited deep neural networks’ powerful
feature extraction and classification capabilities to design the
and significantly increased the evaluation time, making it
EEG-TNet network as the computational core of an end-to-
challenging to apply them in practical scenarios. Therefore,
end BCI framework. This study improves the traditional neural
he used an end-to-end structure similar to ours to finish the
network model named EEG-Net and extracts features through
dual-task estimation(No Task vs. Task). However, the results
the processes of data segmentation, dimension expansion, time
were unsatisfactory, with his best assessment only reaching
fixed 3-D-CNN layers, Bi-LSTM layer, fully connected layer,
61.08%, which is only a tiny improvement over the random
and softmax operation. The time-fixed method was designed to
classification (50%).
ensure the temporal segment order, and a Bi-LSTM layer was
For practical reasons, we need to focus more on triple-task
added at the end for temporal information analysis. Moreover,
estimation (Lo vs. Mi vs. Hi) than on the dual-task estimation the total time cost of this model is only 386.74 ms in our machine
(No Task vs. Task). Lim et al. (2018) contributed the STEW [System: Ubuntu 20.04, CPU: Intel(R) Core(TM) i7-8700K CPU
dataset, where he extracted the PSD of different bans as features @ 3.70GHz, Memory: 32 GB, GPU: GeForce RTX 2080Ti]. The
and used SVM as a classifier. Although their recognition results low time cost proves that the proposed EEG-TNet can meet the
were slightly higher than our results, comparing the confusion requirements of real-time application.
matrices shows that our estimation results are more balanced
and valid. As Figure 4 shows, the accuracy of their results was 5.2. Estimation Performance
99.54% for low MWL, down to 46.15% for medium workloads The most important metric for evaluating a model is estimation
and only 31.07% for high MWL, which was even lower than the accuracy. Unlike other research areas, both subject-dependent

Frontiers in Neuroscience | [Link] 7 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

FIGURE 3 | The divided methods of different experiments.

TABLE 3 | Comparisons of the estimation accuracy (%) of subject-independent experiments among the various methods.

References Method Features No task vs. task Lo vs. Mi vs. Hi

Pandey et al. (2020) KNN None 61.08 \

Pandey et al. (2020) MLP None 58.68 \
Pandey et al. (2020) LSTM None 57.30 \
Lim et al. (2018) SVM Frequency features \ 69.00
EEG-TNet None 82.78 66.83

The bold values indicate the best performance accuracies.

FIGURE 4 | Confusion matrix of EEG-TNet and Lim’s work.

experiments and subject-independent experiments need to be cases, subject-dependent experiments are more accurate because
considered in the field of human factors engineering. In most such experimental methods allow the model to obtain EEG data

Frontiers in Neuroscience | [Link] 8 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

FIGURE 5 | Ablation studies.

for each individual during the training phase. Information on able to achieve 82.78%, which is sufficient for use in
individual differences can be extracted. As shown in Table 2, realistic scenarios.
most studies have achieved more than 80% or even more than The triple task estimation(Lo vs. Mi vs. Hi) is the most urgent
90% recognition accuracy, while our recognition accuracy is close from the point of view of ensuring safety in different fields like
to 100%. transportation and construction. Accurate assessment of the high
However, the subject-dependent experimental approach or moderate MWL of workers helps managers to allocate tasks
is often not applicable to practical scenarios. Training a rationally and to avoid overloading workers with work that could
unique classification model for each worker would be time- lead to human factor accidents. Although the estimation accuracy
consuming and costly, so subject-independent experiments use of the method proposed by Lim et al. (2018) appears to be slightly
unseen subjects’ data as the test set, satisfying the need for higher than our proposed EEG-TNet method. By analyzing
“Plug-and-Play”. However, head shape, scalp impedance, and and comparing the confusion matrices of the two methods
psychological state can all affect the EEG data, resulting in in Figure 4, the method of Lim et al. (2018) may not apply
large variations in EEG data among subjects. The accuracy of to worker workload estimation. According to their confusion
the method is poor. Therefore, as shown in Table 3, almost matrix, we can find that the low MWL was assessed at 99.54%,
all of the previous methods do not apply to estimating the which means that almost all low load situations were successfully
MWL of workers. In the dual-task estimation, the method identified. However, the estimation accuracy for moderate MWL
proposed by Pandey et al. (2020) was only marginally dropped to 46.15%, and only 31.07% of the samples with high
more accurate than random. In contrast, our method was MWL were correctly estimated, with an accuracy rate even lower

Frontiers in Neuroscience | [Link] 9 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

than the random results (33.33%). More notably, 37.57% of tasks still reaches 82.78 and 66.83% respectively. Especially, in the
the medium workload samples and 38.40% of the high MWL subject-independent experiments, compared to previous study,
samples were misclassified as low MWL. In many occupational the likelihood of underestimation was 7 times and 2 times
fields, underestimating workers’ workload by managers can lead lower respectively, which means that our proposed EEG-TNet
to their scheduling of excessive workloads, leading to workers model can fill the requirement of real-time application. In the
being overloaded, making the chances of unsafe behavior much future, we will extend the research by designing new network
higher. In our assessment results, just 5.25% of the moderately structures such as graph neural networks (GNN) to improve
loaded and 19.78% of the high loaded sample were incorrectly the estimation accuracy of high MWL and designing a closed-
underestimated as low workloads. Compared to Lim et al. (2018), loop system that includes real-time estimation and feedback
the likelihood of underestimation was 7 times and 2 times systems. Additionally, building a new database that includes
lower, respectively. more occupational groups will also be our future direction.
To verify the effectiveness of our model, ablation experiments
are performed on the STEW database. There are two kinds DATA AVAILABILITY STATEMENT
of ablation experiments: (1) Ablation experiments on the
effectiveness of designed automated preprocessing method (2) The datasets presented in this study can be found in online
Ablation experiments on the effectiveness of Bi-LSTM. As repositories. The names of the repository/repositories
Figure 5 shows, estimation accuracy significantly decreased and accession number(s) can be found below:
when the data pre-processing process or LSTM layer was [Link]
removed, not only in subject-dependent experiments but also in eeg-workload-dataset.
subject-independent experiments.
However, there were some limitations to this study. First, this AUTHOR CONTRIBUTIONS
study used the multi-channel EEG to build the EEG-TNet model.
However, collecting multi-channel EEG needs gel-based EEG CF responsible for the conceptualization, performed the majority
caps or clumsy dry electrode EEG caps, which is too troublesome of the experiments and analyzes, made the figures, and wrote
to fill the practical usage. Additionally, the increase in the number the first draft of the manuscript. JH and SH performed some
of channels also brings a significant increase in computational experiments, updated the figures, performed the statistics, and
complexity. Furthermore, although the estimation accuracy for edited the manuscript. YP responsible for the methodology,
high MWL is much larger than the previous study, it is still not project administration, resources, funding acquisition, and
enough for application scenarios. Finally, the STEW database validation. SK responsible for the conceptualization and
only contains 48 students’ EEG data under different levels supervision. All authors contributed to the article and approved
of MWL, which were not selected to be representative. It is the submitted version.
important to analyze the EEG signals of different occupational
groups, ages, and work experiences. FUNDING

6. CONCLUSION This study was supported by the National Natural Science

Foundation of China [Grant Numbers: 52075553]; the Natural
This study proposed an end-to-end BCI framework named Science Foundation of Hunan [Grant Numbers: 2020JJ7030]; the
EEG-TNet for the estimation of worker MWL using EEG Hunan Science Foundation for Distinguished Young Scholars
signals and conducted different types of experiments to assess of China [Grant Numbers: 2021JJ10059]; the Postgraduate
the effectiveness of the EEG-TNet framework. In the subject- Scientific Research Innovation Project of Hunan Province [Grant
dependent experiments, the estimation accuracy of dual-task Numbers: CX20210099].
estimation (No task vs. TASK) and that of triple-task estimation
(Lo vs. Mi vs. Hi) reach 99.82 and 99.21% respectively. Compared ACKNOWLEDGMENTS
with the state-of-the-art methods proposed in previous studies,
the accuracy is improved by 8.67 and 9.77%, respectively. Our sincere thanks to W. L. Lim, O. Sourina, and L. P.
Although there is a substantial decrease in estimation accuracy Wang at Nanyang Technological University for providing the
in subject-independent experiments, the accuracy of different STEW dataset.

REFERENCES and evolutionary algorithm. Biomed. Signal Process. Control 60, 101989.
doi: 10.1016/[Link].2020.101989
Abdelhamid, T. S., and Everett, J. G. (2002). Physiological demands Chang, C.-Y., Hsu, S.-H., Pion-Tonachini, L., and Jung, T.-P. (2018). “Evaluation
during construction work. J. Construct. Eng. Manage. 128, 427–437. of artifact subspace reconstruction for automatic EEG artifact removal,”
doi: 10.1061/(ASCE)0733-9364(2002)128:5(427) in 2018 40th Annual International Conference of the IEEE Engineering in
Chakladar, D. D., Dey, S., Roy, P. P., and Dogra, D. P. (2020). EEG- Medicine and Biology Society (EMBC) (Honolulu, HI: IEEE), 1242–1245.
based mental workload estimation using deep BLSTM-LSTM network doi: 10.1109/EMBC.2018.8512547

Frontiers in Neuroscience | [Link] 10 April 2022 | Volume 16 | Article 869522

Fan et al. Mental Workload Estimation Based EEG

Charles, R. L., and Nixon, J. (2019). Measuring mental workload using assessment and degradation mechanism. Build. Environ. 2021, 108711.
physiological measures: a systematic review. Appl. Ergon. 74, 221–232. doi: 10.1016/[Link].2021.108711
doi: 10.1016/[Link].2018.08.028 Pieper, K., Spang, R. P., Prietz, P., Möller, S., Paajanen, E., Vaalgamaa, M., et al.
Cheng, B., Fan, C., Fu, H., Huang, J., Chen, H., and Luo, X. (2022). (2021). Working with environmental noise and noise-cancelation: a workload
Measuring and computing cognitive statuses of construction workers based on assessment with EEG and subjective measures. Front. Neurosci. 15, 771533.
electroencephalogram: a critical review. IEEE Trans. Comput. Soc. Syst. 1–16. doi: 10.3389/fnins.2021.771533
doi: 10.1109/TCSS.2022.3158585 Reid, G. B., and Nygren, T. E. (1988). “The subjective workload assessment
Chollet, F. (2017). “Xception: deep learning with depthwise separable technique: a scaling procedure for measuring mental workload,” in Advances
convolutions,” in Proceedings of the IEEE Conference on Computer in Psychology, eds P. A. Hancock and N. Meshkati (Elsevier), 185–218.
Vision and Pattern Recognition, (Honolulu, HI: IEEE), 1251–1258. doi: 10.1016/S0166-4115(08)62387-0
doi: 10.1109/CVPR.2017.195 Rosanne, O., Albuquerque, I., Cassani, R., Gagnon, J.-F., Tremblay, S.,
Datta, S., and Boulgouris, N. V. (2021). Recognition of grammatical class and Falk, T. H. (2021). Adaptive filtering for improved EEG-based
of imagined words from EEG signals using convolutional neural network. mental workload assessment of ambulant users. Front. Neurosci. 15:341.
Neurocomputing 465, 301–309. doi: 10.1016/[Link].2021.08.035 doi: 10.3389/fnins.2021.611962
Dehais, F., Lafont, A., Roy, R., and Fairclough, S. (2020). A neuroergonomics Roscoe, A. H. (1992). Assessing pilot workload. Why measure heart rate, hrv and
approach to mental workload, engagement and human performance. Front. respiration? Biol. Psychol. 34, 259–287. doi: 10.1016/0301-0511(92)90018-P
Neurosci. 14, 268. doi: 10.3389/fnins.2020.00268 Roy, R. N., Charbonnier, S., Campagne, A., and Bonnet, S. (2016). Efficient mental
Fairclough, S. H., and Houston, K. (2004). A metabolic measure of mental effort. workload estimation using task-independent EEG features. J. Neural Eng. 13,
Biol. Psychol. 66, 177–190. doi: 10.1016/[Link].2003.10.001 026019. doi: 10.1088/1741-2560/13/2/026019
Fan, C., Peng, Y., Peng, S., Zhang, H., Wu, Y., and Sam, K. (2021). Detection van Erp, J. B., Veltman, H. J., and Grootjen, M. (2010). “Brain-based indices for
of train driver fatigue and distraction based on forehead EEG: A time- user system symbiosis,” in Brain-Computer Interfaces, eds J. Vanderdonckt and
series ensemble learning method. IEEE Trans. Intell. Transport. Syst. 1–1. Q. V. Liao (Springer), 201–219. doi: 10.1007/978-1-84996-272-8_12
doi: 10.1109/TITS.2021.3125737 Wang, D., Chen, J., Zhao, D., Dai, F., Zheng, C., and Wu, X. (2017).
Hart, S. G. (2006). “Nasa-task load index (NASA-TLX); 20 years later,” Monitoring workers’ attention and vigilance in construction activities through
in Proceedings of the Human Factors and Ergonomics Society Annual a wireless and wearable electroencephalography system. Automat. Construct.
Meeting (Sage, CA; Los Angeles, CA: Sage Publications), 904–908. 82, 122–137. doi: 10.1016/[Link].2017.02.001
doi: 10.1177/154193120605000909 Wilson, G. F., Fullenkamp, P., and Davis, I. (1994). Evoked potential, cardiac, blink,
Hochreiter, S., and Schmidhuber, J. (1997). Long short-term memory. Neural and respiration measures of pilot workload in air-to-ground missions. Aviat.
Comput. 9, 1735–1780. doi: 10.1162/neco.1997.9.8.1735 Space Environ. Med. 65, 100–105.
Hogervorst, M. A., Brouwer, A.-M., and Van Erp, J. B. (2014). Combining Xu, G., Ren, T., Chen, Y., and Che, W. (2020). A one-dimensional cnn-lstm model
and comparing EEG, peripheral physiology and eye-related measures for epileptic seizure recognition using EEG signal analysis. Front. Neurosci. 14,
for the assessment of mental workload. Front. Neurosci. 8, 322. 1253. doi: 10.3389/fnins.2020.578126
doi: 10.3389/fnins.2014.00322 Yin, Z., Zhao, M., Zhang, W., Wang, Y., Wang, Y., and Zhang, J. (2019).
Hu, J., Wang, C., Jia, Q., Bu, Q., Sutcliffe, R., and Feng, J. (2021). Physiological-signal-based mental workload estimation via transfer dynamical
ScalingNet: extracting features from raw EEG data for emotion recognition. autoencoders in a deep learning framework. Neurocomputing 347, 212–229.
Neurocomputing 463, 177–184. doi: 10.1016/[Link].2021.08.018 doi: 10.1016/[Link].2019.02.061
Kingphai, K., and Moshfeghi, Y. (2021). “On EEG preprocessing role in deep Yu, X., Liu, T., He, L., and Yajie, L. (2022). Micro-foundations of strategic decision-
learning effectiveness for mental workload classification,” in International making in family business organisations: a cognitive neuroscience perspective.
Symposium on Human Mental Workload: Models and Applications (Springer), Long Range Plann. 2022, 102198. doi: 10.1016/[Link].2022.102198
81–98. doi: 10.1007/978-3-030-91408-0_6 Zhu, G., Zong, F., Zhang, H., Wei, B., and Liu, F. (2021). Cognitive load
Lawhern, V. J., Solon, A. J., Waytowich, N. R., Gordon, S. M., Hung, C. during multitasking can be accurately assessed based on single channel
P., and Lance, B. J. (2018). EEGNet: a compact convolutional neural electroencephalography using graph methods. IEEE Access 9, 33102–33109.
network for EEG-based brain-computer interfaces. J. Neural Eng. 15, 056013. doi: 10.1109/ACCESS.2021.3058271
doi: 10.1088/1741-2552/aace8c
Lim, W., Sourina, O., and Wang, L. (2018). Stew: simultaneous task EEG Conflict of Interest: JH was employed by the Hunan Communications Research
workload data set. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 2106–2114. Institute Co., Ltd., Hunan Communication & Water Conservancy Group Ltd.
doi: 10.1109/TNSRE.2018.2872924
Liu, X., Chen, S., Guo, X., and Fu, H. (2022). Can social norms promote recycled The remaining authors declare that the research was conducted in the absence of
water use on campus? The evidence from event-related potentials. Front. any commercial or financial relationships that could be construed as a potential
Psychol. 13, 818292. doi: 10.3389/fpsyg.2022.818292 conflict of interest.
Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. (2010).
“Recurrent neural network based language model,” in Interspeech (Makuhari), Publisher’s Note: All claims expressed in this article are solely those of the authors
1045–1048. doi: 10.21437/Interspeech.2010-343 and do not necessarily represent those of their affiliated organizations, or those of
Mognon, A., Jovicich, J., Bruzzone, L., and Buiatti, M. (2011). Adjust: An automatic
the publisher, the editors and the reviewers. Any product that may be evaluated in
EEG artifact detector based on the joint use of spatial and temporal features.
this article, or claim that may be made by its manufacturer, is not guaranteed or
Psychophysiology 48, 229–240. doi: 10.1111/j.1469-8986.2010.01061.x
Mühl, C., Jeunet, C., and Lotte, F. (2014). EEG-based workload estimation across endorsed by the publisher.
affective contexts. Front. Neurosci. 8, 114. doi: 10.3389/fnins.2014.00114
Pandey, V., Choudhary, D. K., Verma, V., Sharma, G., Singh, R., and Copyright © 2022 Fan, Hu, Huang, Peng and Kwong. This is an open-access article
Chandra, S. (2020). “Mental workload estimation using EEG,” in 2020 distributed under the terms of the Creative Commons Attribution License (CC BY).
Fifth International Conference on Research in Computational Intelligence The use, distribution or reproduction in other forums is permitted, provided the
and Communication Networks (ICRCICN) (Bangalore: IEEE), 83–86. original author(s) and the copyright owner(s) are credited and that the original
doi: 10.1109/ICRCICN50933.2020.9296150 publication in this journal is cited, in accordance with accepted academic practice.
Peng, Y., Lin, Y., Fan, C., Xu, Q., Xu, D., Yi, S., et al. (2021). Passenger No use, distribution or reproduction is permitted which does not comply with these
overallcomfort in high-speed railway environments based on EEG: terms.

Frontiers in Neuroscience | [Link] 11 April 2022 | Volume 16 | Article 869522