南京信息工程大学学报2026，Vol.18Issue(2)：183-191,9.DOI:10.13878/j.cnki.jnuist.20250213001
基于面部图像和心率变异性的多模态融合情感识别算法
Multimodal fusion emotion recognition based on face image and heart rate variability
摘要
Abstract
To address the challenges of ineffective emotion recognition in virtual sports,this study proposes an al-gorithm of Multimodal Fusion Emotion Recognition based on Face Image and Heart Rate Variability(MFER-FI-HRV).By capturing and analyzing data from both facial image and heart rate variability modalities,the proposed approach enables a fine-grained perception of users'emotional changes,thereby facilitating a personalized Human-Computer Interaction(HCI)experience tailored to their current emotional state.The algorithm first employs a multi-modal fusion Transformer to achieve complementary learning between facial images and HRV data.Then,a multimo-dal feature fusion strategy is utilized to concatenate fused features with the original feature representations.Addition-ally,a lightweight self-attention mechanism is introduced to capture high-level semantic representations within the multimodal data.Extensive experiments on two public datasets demonstrate the superior performance of the proposed method,confirming its effectiveness and robustness.These findings provide valuable insights for the design and de-velopment of user experience systems.关键词
多模态情感识别/Transformer/卷积神经网络/双向长短期记忆网络Key words
multimodal emotion recognition/Transformer/convolutional neural network(CNN)/bi-directional long short-term memory(BiLSTM)分类
信息技术与安全科学引用本文复制引用
张宇轩,林贤煊,王爽,潘志庚..基于面部图像和心率变异性的多模态融合情感识别算法[J].南京信息工程大学学报,2026,18(2):183-191,9.基金项目
国家自然科学基金(62072150,6257076952) （62072150,6257076952）
