Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition

Yihang Dong, Xuhang Chen, Yanyan Shen, Michael Kwok Po Ng, Tao Qian, Shuqiang Wang*

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

1 Citation (Scopus)

Abstract

Emotion recognition based on Electroencephalography (EEG) has gained significant attention and diversified development in fields such as neural signal processing and affective computing. However, the unique brain anatomy of individuals leads to non-negligible natural differences in EEG signals across subjects, posing challenges for cross-subject emotion recognition. While recent studies have attempted to address these issues, they still face limitations in practical effectiveness and model framework unity. Current methods often struggle to capture the complex spatial-temporal dynamics of EEG signals and fail to effectively integrate multimodal information, resulting in suboptimal performance and limited generalizability across subjects. To overcome these limitations, we develop a Pre-trained model based Multimodal Mood Reader for cross-subject emotion recognition that utilizes masked brain signal modeling and interlinked spatial-temporal attention mechanism. The model learns universal latent representations of EEG signals through pre-training on large scale dataset, and employs Interlinked spatial-temporal attention mechanism to process Differential Entropy(DE) features extracted from EEG data. Subsequently, a multi-level fusion layer is proposed to integrate the discriminative features, maximizing the advantages of features across different dimensions and modalities. Extensive experiments on public datasets demonstrate Mood Reader’s superior performance in cross-subject emotion recognition tasks, outperforming state-of-the-art methods. Additionally, the model is dissected from attention perspective, providing qualitative analysis of emotion-related brain areas, offering valuable insights for affective research in neural signal processing.

Original languageEnglish
Title of host publicationNeural Computing for Advanced Applications - 5th International Conference, NCAA 2024, Proceedings
EditorsHaijun Zhang, Xianxian Li, Tianyong Hao, Weizhi Meng, Zhou Wu, Qian He
PublisherSpringer Science and Business Media Deutschland GmbH
Pages178-192
Number of pages15
ISBN (Electronic) 9789819770076
ISBN (Print)9789819770069
DOIs
Publication statusPublished - 22 Sept 2024
Event5th International Conference on Neural Computing for Advanced Applications, NCAA 2024 - Guilin, China
Duration: 5 Jul 20247 Jul 2024

Publication series

NameCommunications in Computer and Information Science
Volume2183 CCIS
ISSN (Print)1865-0929
ISSN (Electronic)1865-0937

Conference

Conference5th International Conference on Neural Computing for Advanced Applications, NCAA 2024
Country/TerritoryChina
CityGuilin
Period5/07/247/07/24

Scopus Subject Areas

  • General Computer Science
  • General Mathematics

User-Defined Keywords

  • EEG-based emotion recognition
  • masked brain signal modeling
  • Pre-trained Model
  • spatial-temporal attention

Fingerprint

Dive into the research topics of 'Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition'. Together they form a unique fingerprint.

Cite this