计算机科学与探索2024，Vol.18Issue(5)：1318-1327,10.DOI:10.3778/j.issn.1673-9418.2311004
基于复合跨模态交互网络的时序多模态情感分析
Temporal Multimodal Sentiment Analysis with Composite Cross Modal Interaction Network
摘要
Abstract
To address the issues of insufficient modal fusion and weak interactivity caused by semantic feature dif-ferences between different modalities in multimodal emotion analysis,a temporal multimodal sentiment analysis model for composite cross modal interaction network(CCIN-SA)is constructed by studying and analyzing the po-tential correlations between different modalities.The model first uses a bidirectional gated loop unit and a multi-head attention mechanism to extract temporal features of text,visual,and speech modalities with contextual seman-tic information.Then,a cross modal attention interaction layer is designed to continuously strengthen the target mode using low order signals from auxiliary modes,enabling the target mode to learn information from auxiliary modes and capture potential adaptability between modes.Then it inputs the enhanced features into the composite feature fusion layer,further captures the similarity between different modalities through condition vectors,enhances the correlation degree of important features,and mines deeper level interactivity between modalities.Finally,using a multi-head attention mechanism,the composite cross modal enhanced features are concatenated and fused with low order signals to increase the weight of important features within the modality,preserve the unique feature infor-mation of the initial modality,and perform the final emotion classification task on the obtained multimodal fused features.The model evaluation is conducted on the CMU-MOSI and CMU-MOSEI datasets,and the results show that the model is improved in accuracy and F1 metrics compared with other existing models.It can be seen that the CCIN-SA model can effectively explore the correlation between different modalities and make more accurate emo-tional judgments.关键词
跨模态交互/注意力机制/特征融合/复合融合层/多模态情感分析Key words
cross modal interaction/attention mechanism/feature fusion/composite fusion layer/multimodal emo-tional analysis分类
信息技术与安全科学引用本文复制引用
杨力,钟俊弘,张赟,宋欣渝..基于复合跨模态交互网络的时序多模态情感分析[J].计算机科学与探索,2024,18(5):1318-1327,10.基金项目
国家自然科学基金(61175122) （61175122）
四川省科技计划项目(2022NSFSC0555). This work was supported by the National Natural Science Foundation of China(61175122),and the Science and Technology Program of Sichuan Province(2022NSFSC0555). （2022NSFSC0555）
