计算机工程与应用2024，Vol.60Issue(13)：124-135,12.DOI:10.3778/j.issn.1002-8331.2302-0238
情感分析的跨模态Transformer组合模型
Cross-Modal Transformer Combination Model for Sentiment Analysis
摘要
Abstract
The Transformer-based end-to-end combination deep learning model is the mainstream model of multimodal sentiment analysis.In view of the lack of sentiment feature extraction ability of low-resource modal data,the difference of feature scales of non-aligned data in different modals,which lead to the loss of key feature information in the alignment and fusion process,and the unreliable multimodal long-term dependency mechanism caused by the parallel processing of multimodal data by the traditional attention model,this paper proposes an sentiment analysis model LAACMT based on lightweight attention aggregation module and cross-modal Transformer,which can use multimodal non-aligned data to perform binary classification and multiclass classification tasks.The model proposes to extract low-resource modal infor-mation using gated recurrent unit(GRU)and improved feature extraction algorithm,proposes positional encoding and convolution scaling methods for aligning multimodal contexts,proposes a multimodal multi-head attention mechanism to fuse aligned multimodal data and establishes a reliable cross-modal long-term dependency mechanism.The experimental results of the model on CMU-MOSI,which contains three modals of non-aligned dataset including text,voice and video,show that the performance evaluation index of the model has been steadily improved compared with SOTA,in which Acc7 has been improved by 3.96%,Acc2 has been improved by 4.08%,and F1 score has been improved by 3.35% .The results of ablation study show that the model proposed in this paper solves the problems in multimodal sentiment analysis,reduces the complexity of the multimodal sentiment analysis model based on Transformer,improves the performance of the model,and avoids over-fitting problems.关键词
多模态情感分析/轻量级注意力聚合模块/跨模态Transformer/门控循环单元/跨模态多头注意力机制Key words
multimodal sentiment analysis/lightweight attentive aggregation module/cross-modal Transformer/gated recurrent unit/cross-modal multi-head attention mechanism分类
信息技术与安全科学引用本文复制引用
王亮,王屹,王军..情感分析的跨模态Transformer组合模型[J].计算机工程与应用,2024,60(13):124-135,12.基金项目
国家外国专家项目(G2022006008L) （G2022006008L）
中国高校产学研创新基金(2021LD06009) （2021LD06009）
辽宁省自然科学基金(2022-MS-291) （2022-MS-291）
辽宁省教育厅科研项目(LJ2020024) （LJ2020024）
辽宁省教育厅基本科研项目(LJKMZ20220781). （LJKMZ20220781）
