计算机工程与应用2024，Vol.60Issue(1)：165-173,9.DOI:10.3778/j.issn.1002-8331.2207-0498
跨模态语义时空动态交互情感分析研究
Cross-Modal Emotion Analysis of Semantic and Spatio-Temporal Dynamic Interaction
摘要
Abstract
Considering the problems of poor interaction between multimodal and low fusion of spatial and temporal features in traditional sentiment analysis,a semantic and spatio-temporal dynamic interaction network of cross-modal is proposed.By introducing bi-directional long short-term memory,the time series features of each modality are mined.Meanwhile,a self-attention mechanism is added to strengthen the weight distribution of features within the modality,and the automatically screened feature matrix is sent to the graph convolutional neural networks for semantic interaction.Then,based on the timestamp,the feature aggregation is carried out,the correlation coefficient of the aggregation layer is calculated,and the fused features are obtained to realize cross-modal space interaction.Finally the classification and prediction of emotional polarity are performed.The proposed model is evaluated and verified using public datasets.The experimental results show that multi-modal time series extraction and cross-modal semantic space interaction mechanism can achieve full dynamic fusion of intra-modal and inter-modal features,and effectively improve the accuracy and F1 value of sentiment classification.On the CMU-MOSEI dataset they have increased by 1.7%～13.5%and 2.1%～14.0%respectively,showing good robustness and advancement.关键词
跨模态情感分析/语义交互/时空交互/双向长短期记忆网络/图卷积网络Key words
cross modal sentiment analysis/semantic interaction/spatio-temporal interaction/bi-directional long short-term memory/graph convolutional network分类
信息技术与安全科学引用本文复制引用
屈立成,郤丽媛,刘紫君,魏思,董哲为..跨模态语义时空动态交互情感分析研究[J].计算机工程与应用,2024,60(1):165-173,9.基金项目
陕西省哲学社会科学重大理论与现实问题研究项目(2022HZ1387) （2022HZ1387）
国家重点研发计划(2019YFE0108300) （2019YFE0108300）
国家自然科学基金(62001058) （62001058）
陕西省自然科学基础研究计划(2020JM-258). （2020JM-258）
