计算机应用研究2026，Vol.43Issue(2)：361-368,8.DOI:10.19734/j.issn.1001-3695.2025.07.0233
基于超图学习与成对跨模态融合的多模态对话情绪识别
Multimodal emotion recognition in conversation based on hypergraph learning and pairwise cross-modal fusion
摘要
Abstract
To address issues such as insufficient utilization of interaction information between modalities and multivariate dia-logue relations in current multimodal emotion recognition in conversation models,this paper proposed a multimodal emotion recognition in conversation model based on hypergraph learning and pairwise cross-modal fusion.In the hypergraph learning module of the model,it took discourse representations as nodes,and designed two different types of hyperedges containing multimodal and temporal information to form a hypergraph.It used hypergraph convolution to capture multivariate dialogue re-lations between speakers.Meanwhile,this paper proposed a dual-stream gated attention network to dynamically adjust node features and reduce information redundancy.In the pairwise cross-modal fusion module,it used each modality as a baseline feature.And based on the cross-modal attention mechanism,it was repeatedly reinforced with other modal features to excavate deep interaction information between pairwise modalities and enhance cross-modal feature representation.Experimental results show that on the IEMOCAP and CMU-MOSEI datasets,the accuracy and weighted average F1 score of the proposed model are better than those of multiple comparison models,fully verifying the effectiveness of the model.关键词
对话情绪识别/超图/跨模态融合/双流门控注意力网络/TransformerKey words
emotion recognition in conversation/hypergraph/cross-modal fusion/dual-stream gated attention network/Transformer分类
信息技术与安全科学引用本文复制引用
李尚往,缪裕青,刘同来,张万桢,周明..基于超图学习与成对跨模态融合的多模态对话情绪识别[J].计算机应用研究,2026,43(2):361-368,8.基金项目
国家自然科学基金资助项目(62366010,62366011) （62366010,62366011）
广东省自然科学基金资助项目(2023A1515011230) （2023A1515011230）
广东省哲学社会科学规划专项项目(GD25CW04) （GD25CW04）
桂林电子科技大学研究生教育创新计划资助项目(2025YCXS076) （2025YCXS076）
