中北大学学报（自然科学版）2025，Vol.46Issue(1)：1-9,9.DOI:10.62756/jnuc.issn.1673-3193.2024.07.0005
自适应门控解耦特征融合的多模态情感分析
Multimodal Sentiment Analysis of Adaptive Gated Decoupling Feature Fusion
摘要
Abstract
In the existing research on multi-modal sentiment analysis,the fusion different modal information is mainly through the overall interaction of different modal features,but it doesn't consider the relationship between unique features and common features contained in different modes,so the complex emotions can't be analyzed effectively.To solve this problem,a multimodal sentiment analysis model based on adaptive gated decoupling feature fusion(AGDF)was proposed.Firstly,the pre-trained BERT model and Transformer model were used for feature extraction of different modes.Secondly,according to the principle that the common features of different modes were similar but the unique features were not similar,the contrast pair was con-structed.By contrastive learning,the features of different modes were decomposed into unique features and common features.Thirdly,according to the principle that the image and speech modes were offset in the text mode,a new adaptive gating mechanism was designed to fuse the features and integrate other modal information into the text mode.At the same time,the relation graph of unique features and common features was designed,and the fusion of the graph attention neural network was used to balance the unique information and common information among the modes.Finally,the fusion features were classified.Experiments on the datasets CMU-MOSI and CMU-MOSEI show that the accuracy and F1 score of the proposed method are improved by about 1 percentage point compared with the baseline method.In addition,compared with other feature decomposition methods,the proposed method improves accuracy by 1.23 percentage point,F1 score by 1.37 percentage point,Corr by 2.13 percentage point,and reduces MAE by 4.83 percentage point.Consequently,the proposed method can make full use of the heterogeneous information of different modes and effectively improve the effect of sentiment analysis.关键词
情感分析/对比学习/图神经网络/多模态信息融合/自适应门控Key words
sentiment analysis/contrastive learning/graph neural network/multimodal information fusion/adaptive gating分类
计算机与自动化引用本文复制引用
李烽源,蔺素珍,王彦博,李大威,顾梦瑶..自适应门控解耦特征融合的多模态情感分析[J].中北大学学报（自然科学版）,2025,46(1):1-9,9.基金项目
国家自然科学基金项目(62271453) （62271453）
山西省自然科学基金项目(202303021211147) （202303021211147）
山西省应用基础研究计划(20210302123025) （20210302123025）
山西省知识产权局专利转化专项计划(202302001) （202302001）
