E-13, UPSIDC Site-IV, Kasna Road, Greater Noida - 201308, India

Phone: +91.120.4296878 | Website: [Link] [Link]

Email: info@[Link] | iiprd@[Link]
Indian Offices: Noida | New Delhi | Bangalore | Mumbai | Pune | Mumbai | Hyderabad | Jalandhar | Chennai
International Offices: US |Bangladesh | Sri Lanka | Malaysia | Vietnam | Myanmar | Nepal | Malaysia

INVENTIVE DISCLOSURE - CONFIDENTIAL

1. Proposed Title of the Invention

Multimodal Brain-Signal-Based System for Cognitive and Emotional Processing with
Biometric Authentication Using EEG and fNIRS

2. Proposed Abstract of the Invention

The present invention proposes a novel multimodal brain-computer interface (BCI)
system for accurate detection of emotional states and cognitive load using fused EEG
(Electroencephalography) and fNIRS (functional Near Infrared Spectroscopy) data. The
invention integrates two separate emotion recognition models (EEG-based and fNIRS-
based) with a Stroop-based cognitive model (EEG + fNIRS fusion). By leveraging the
temporal precision of EEG and the spatial specificity of fNIRS, the proposed system
enhances the performance of human state decoding. The emotion models are trained
independently on labeled EEG and fNIRS datasets, while the Stroop model combines
both modalities for classifying task-induced cognitive states. The models are fused using
deep learning architectures and evaluated on real experimental datasets, showing
significant improvement in classification accuracy over unimodal approaches. This
innovation enables robust and real-time monitoring of mental states, offering applications
in mental health, human-computer interaction, and neuroadaptive technologies.

3. Key Words:
Brain-Computer Interface (BCI), EEG, fNIRS, Emotion Recognition, Cognitive State,
Multimodal Fusion, Stroop Task, Deep Learning

4. Background of the Invention:

 Current state-of-the-art systems for emotion or cognitive state recognition
predominantly rely on unimodal approaches such as EEG or fNIRS individually. EEG
offers high temporal resolution but is susceptible to artifacts and limited in spatial
coverage. On the other hand, fNIRS provides better spatial localization of brain activity
but lacks temporal resolution. Moreover, traditional emotion recognition or cognitive task
models fail to generalize across tasks and subjects due to limited data representation. The
absence of real-time, robust systems that can handle emotion and cognition together
forms a significant gap in the field.

5. What problems does the invention address and how your Invention is able to
overcome the limitations/ problems of the existing technologies?
This invention addresses the limitations of unimodal systems by proposing a multimodal
fusion of EEG and fNIRS signals. It overcomes challenges like poor spatial or temporal
resolution, noisy signals, and poor model generalizability. By training separate models
for emotion (EEG and fNIRS-based) and a combined model for Stroop-based cognitive
inference, it creates a more comprehensive understanding of user mental states. The
fusion leads to higher classification accuracy and robustness in real-time applications.

6. Detailed Explanation of the Invention along with working examples.

We have developed a novel authentication model that utilizes a hybrid approach by
integrating three major models: EEG, fNIRS, and Stroop task. This multi-stage system
aims to provide a robust and secure authentication mechanism by leveraging the
complementary strengths of these different brain-sensing modalities.
Stage 1: EEG-based Authentication
The EEG-based authentication model comprises two parallel sub-models:
1. Emotion Prediction: The EEG signals are processed using techniques like
bandpass filtering and artifact removal. Time-frequency features, such as Power
Spectral Density (PSD), are extracted and fed into a deep learning model (e.g.,
Convolutional Neural Network or Long Short-Term Memory) to classify the
user's emotional states (e.g., happy, sad, neutral).
2. Subject-wise Authentication: The same preprocessed EEG data is also used to
train a separate model for subject-wise authentication. This model learns to
identify unique brain activity patterns associated with each individual user.
The final output of the EEG model is the combined result of these two sub-models,
providing both emotion prediction and subject-wise authentication capabilities.
Stage 2: fNIRS-based Authentication
The fNIRS-based authentication model also follows a two-tier approach:
1. Emotion Prediction (Tier 1): The raw fNIRS data is processed to extract
hemodynamic features, such as oxygenated (HbO) and deoxygenated (HbR)
hemoglobin concentrations, using techniques like the Modified Beer–Lambert
Law. These features are then input into a neural network to classify the user's
emotional states.
 2. Subject-wise Authentication (Tier 2): If the emotion prediction in the first tier is
successful, the system proceeds to the second tier, which focuses on subject-wise
authentication. The fNIRS features are fed into a separate model that is trained to
identify unique brain activity patterns associated with each individual user.
This tiered approach ensures that the authentication process first verifies the user's
emotional state before proceeding to the subject-wise authentication, providing an
additional layer of security.
Stage 3: Stroop Task-based Authentication
In the final stage, the user is asked to perform a Stroop task, which is a standard cognitive
load paradigm. Both EEG and fNIRS data are recorded during this task. The fusion
model takes features from both modalities as input, with EEG contributing frequency-
domain features and fNIRS providing hemodynamic features. These features are
combined and input into a hybrid CNN-LSTM network that can model both spatial and
temporal patterns to classify the user's cognitive workload levels.
Final Decision-Making
The results from the individual EEG and fNIRS emotion models, as well as the fused
Stroop model, are combined using a late fusion strategy. This allows the system to track
the evolution of the user's emotional and cognitive states in a temporally aligned manner,
leading to a more comprehensive and robust authentication decision.
Performance and Accuracy
The overall authentication system has achieved an impressive accuracy of around 90%...
Working Example of the Multimodal Authentication Model
To illustrate the working of the multimodal authentication system, consider the following
example:
1. A user is presented with a series of emotionally evocative videos while their EEG
and fNIRS data are recorded simultaneously. The user's emotional responses are
also captured through self-reports or facial expression analysis.
2. The user then proceeds to perform a Stroop task, during which their EEG and
fNIRS data are again collected.
3. The recorded data is used to train the three models: the EEG-based emotion
prediction and subject-wise authentication, the fNIRS-based tiered emotion
prediction and subject-wise authentication, and the Stroop task-based fusion
model.
4. During authentication, the system combines the outputs from these three models
to make a final decision, leveraging the complementary information from the
different brain-sensing modalities.
5. The system has demonstrated an overall accuracy of around 90%, with the fNIRS
model achieving up to 95% accuracy in subject-wise authentication.
This multimodal approach provides a robust and secure authentication solution that can
be applied in various real-world applications, such as access control, mental health
monitoring, and neuroadaptive interfaces
7. Kindly attach drawings, reports, papers, charts or other materials that may aid in
your description.
Emotion Model:
EEG model:
fNIRS Model:
Stroop Model:
Final Integration:
8. What are the aspects of your disclosure that you want to claim/monopolize?
Proposed Claims:
1. A brain-computer interface (BCI) system for biometric authentication, comprising:

 a dual-mode brain signal acquisition module configured to simultaneously acquire

electroencephalography (EEG) signals and functional near-infrared spectroscopy
(fNIRS) signals from a subject;
 an emotion recognition module configured to analyze the acquired EEG and
fNIRS signals and detect emotional states using a fused signal approach;
 a cognitive assessment module configured to classify cognitive load based on a
Stroop task using combined EEG and fNIRS features;
 a two-tier authentication module wherein a first tier uses the detected emotional
state and a second tier uses subject-specific biometric features derived from the
EEG and fNIRS signals for identity verification; and
 a processing pipeline configured to perform real-time monitoring and decision-
level fusion of emotional and cognitive states for authentication purposes.

2. The system of claim 1, wherein the EEG and fNIRS signals are fused at the data level
or decision level to enhance classification accuracy of the emotional and cognitive states.

3. The system of claim 1, wherein the emotion recognition module is trained using
labeled EEG and fNIRS datasets corresponding to distinct emotional classes.

4. The system of claim 1, wherein the cognitive assessment module utilizes responses
from a Stroop task to detect variations in cognitive workload.

5. The system of claim 1, wherein the authentication module performs user verification
based on deep learning classification of fused EEG-fNIRS features corresponding to
emotion and cognitive states.

6. The system of claim 1, wherein the two-tier authentication process first verifies the
emotional state as a filter criterion and then applies subject-wise biometric identification
for enhanced specificity.

7. The system of claim 1, wherein the real-time processing pipeline includes modules for
signal preprocessing, feature extraction, model inference, and user state monitoring.

8. The system of claim 1, wherein the system achieves a classification accuracy of

approximately 90% for user authentication across multiple experimental subjects.

9. Have you conducted novelty/inventiveness search for your invention? If yes, what
are the databases /references used by you? What are the search results?
 Yes.
Databases Searched: Google Patents, IEEE Xplore, PubMed, WIPO
Findings: While EEG-based and fNIRS-based emotion detection systems exist, and
cognitive load assessments with EEG/fNIRS exist independently, no system was found
that:
 Simultaneously fuses both modalities for both emotion and cognitive assessment
 Uses Stroop task plus an emotion-labeling phase in one integrated architecture.

10. Do you feel that a person of “average” skill (not-extraordinary skill) in your area of
technology would have arrived at your invention with existing knowledge in public
domain? If no, what could be the reasons for the same?
No. A person of average skill in neuroscience or signal processing would not intuitively
fuse EEG and fNIRS for both emotion and cognitive classification in a single real-time
architecture. Challenges in synchronizing data, labeling, and model fusion require
multidisciplinary expertise.

11. Kindly provide broad workable ranges for all the parameters involved in your
invention.
(i) EEG Emotion Recognition Model (CNN-based):

 Input channels: 14 to 32
 Sampling frequency: 128 Hz to 1000 Hz
 Epoch length: 1 to 2 seconds (corresponding to 128 to 256 samples per epoch)
 Number of convolutional layers: 2 to 4
 Filter sizes: 3×3 to 5×5
 Number of filters per layer: 16 to 64
 Pooling type: MaxPooling
 Dropout rate: 0.3 to 0.5
 Activation function: ReLU
 Loss function: Categorical CrossEntropy
 Optimizer: Adam
 Learning rate: 0.0001 to 0.001
 Batch size: 16 to 64
 Epochs for training: 50 to 300

(ii) fNIRS Emotion Recognition Model (Transformer-based):

 Input features: 20 to 40 channels (HbO features)

 Sampling frequency: 10 Hz to 100 Hz
 Epoch duration: 2 to 4 seconds
 Transformer layers: 1 to 4
 Attention heads: 2 to 8
  Embedding dimension: 32 to 128
 Dropout rate: 0.1 to 0.4
 Learning rate: 0.0001 to 0.001
 Optimizer: Adam
 Batch size: 16 to 64
 Epochs for training: 50 to 300

(iii) Stroop-Based Biometric Identification Model (MLP/Residual/Transformer):

 Input feature vector size: 460 features (340 EEG + 120 fNIRS)
 Model types: Transformer, Residual MLP, Plain MLP
 Hidden layer sizes: 128 to 512 neurons
 Dropout rate: 0.2 to 0.5
 Activation function: ReLU
 Optimizer: Adam
 Learning rate: 0.0001 to 0.001
 Loss function: CrossEntropy
 Batch size: 16 to 64
 Epochs: 50 to 300
 Evaluation strategy: 5-fold cross-validation

12. References:
A. Datasets Used

1. Mahajan, A. (2021). Emotion Recognition using EEG and Computer Games.

Dataset available on Kaggle.
[Link]
2. Spapé, M., Mäkelä, K., & Ruotsalo, T. (2024). NEMO: A Database for
Emotion Analysis Using Functional Near-Infrared Spectroscopy. IEEE
Transactions on Affective Computing, 15(3), 1166–1177.
[Link]
3. Aspinall, D., Coelho, M., & Arteaga-Falconi, J. S. (2021). Open Access
Dataset Integrating Electroencephalography and Functional Near-Infrared
Spectroscopy During Stroop Tasks. Data in Brief.
[Link]

B. Supporting Literature

1. Ren, H., Zhou, S., Zhang, L., Zhao, F., & Qiao, L. (2022). Identifying
Individuals by fNIRS-Based Brain Functional Network Fingerprints. Frontiers in
Neuroscience, 16, 837052. [Link]
2. Li, R., Yang, D., Fang, F., Hong, K.-S., Reiss, A. L., & Zhang, Y. (2022).
Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic,
 Methodology-Focused Review. Sensors, 22(15), 5857.
[Link]
3. Uchitel, J., Vidal-Rosas, E. E., Cooper, R. J., & Zhao, H. (2021). Wearable,
Integrated EEG–fNIRS Technologies: A Review. Sensors, 21(18), 6106.
[Link]
4. Rahman, A., Chowdhury, M. E. H., Khandakar, A., Kiranyaz, S., Zaman, K.
S., Reaz, M. B. I., Islam, M. T., Ezedddin, M., & Kadir, M. A. (2021).
Multimodal EEG and Keystroke Dynamics-Based Biometric System Using
Machine Learning Algorithms. IEEE Access, 9, 128131–128147.
[Link]
5. Yükselen, G., Öztürk, O. C., Canlı, G. D., & Erdoğan, S. B. Investigating the
Neural Correlates of Processing Basic Emotions: A Functional Near-Infrared
Spectroscopy (fNIRS) Study. Acıbadem Mehmet Ali Aydınlar University,
Istanbul, Turkey.

13. Inventors Details:

Name: Disha M
Nationality: Indian
Address: VIT Chennai Vandalur-Kelambakkam Road, Chennai-600127, Tamil Nadu,
India.

Name: Chandrakala K
Nationality: Indian
Address: VIT Chennai Vandalur-Kelambakkam Road, Chennai-600127, Tamil Nadu,
India.

Name: Krishnan E M
Nationality: Indian
Address: VIT Chennai Vandalur-Kelambakkam Road, Chennai-600127, Tamil Nadu,
India.

14. Applicant Details:

Name: Disha M
Nationality: Indian
Address: VIT Chennai Vandalur-Kelambakkam Road, Chennai-600127, Tamil Nadu,
India.

Name: Chandrakala K
Nationality: Indian
Address: VIT Chennai Vandalur-Kelambakkam Road, Chennai-600127, Tamil Nadu,
India.
 Name: Krishnan E M
Nationality: Indian
Address: VIT Chennai Vandalur-Kelambakkam Road, Chennai-600127, Tamil Nadu,
India.

15. Remarks.
This system can be applied to educational monitoring, psychological evaluation, virtual
reality emotional modeling, defense training simulations, and workplace fatigue
assessment. Future extensions may include integration with real-time feedback systems
or brain-computer interface applications.