基于CLIP和交叉注意力的多模态情感分析模型OACSTPCD

In response to the issues of limited annotated data,insufficient fusion between modalities,and informa-tion redundancy in multimodal sentiment analysis,a multimodal sentiment analysis model called CLIP-CA-MSA based on contrastive language-image pretraining(CLIP)and cross-attention mechanism was proposed in this study.This model employed models such as BERT which was pre-trained by CLIP,and PIFT to extract feature vectors from videos and textual content.Subsequently,a cross-attention mechanism was applied to facilitate interaction be-tween image feature vectors and text feature vectors,enhancing information exchange across different modalities.Finally,the uncertainty loss was utilized to compute the fused features,and the ultimate sentiment classification re-sults were generated from the outputs.The experimental results showed that the model could increase accuracyrate by 5 percentage points to 14 percentage points and the F1 value by 3 percentage point to 12 percentage point over other multimodal models,which verifieed the superiority of the model in this study.And uses of ablation experi-ments to verified the validity of each module of the model.This model could effectively utilize the complementarity and correlation of multimodal data,and utilize uncertainty loss to improve the robustness and generalization ability of the model.