计算机工程与应用2025，Vol.61Issue(6)：171-182,12.DOI:10.3778/j.issn.1002-8331.2310-0392
多模态分级特征映射与融合表征方法研究
Research on Multimodal Hierarchical Feature Mapping and Fusion Representation Method
摘要
Abstract
Multimodal feature representation serves as the foundation for multimodal tasks.To address the issue of a single-level fusion in existing multimodal feature representation methods,which fails to adequately capture the inter-modal relationships,a novel approach for multimodal hierarchical feature mapping and fusion representation is proposed.This method,built upon the text model RoBERTa and the image model DenseNet,extracts features from intermediate layers of both models spanning from low to high levels.Leveraging the concept of feature reuse,it maps and fuses features at different levels of the text and image modalities,capturing the internal relationships between text and image modalities and effec-tively integrating features between the two modalities.The hierarchical feature mapping and fusion representation is then fed into a classifier for sentiment classification in the context of multimodal sentiment analysis.A comparative analysis is also conducted between the constructed representation method and baseline representation methods.The experimental results indicate that the proposed representation method surpasses all baseline models in terms of sentiment classification performance on both the Weibo sentiment and MVSA-Multiple datasets.Specifically,it achieves a 0.013 7 increase in F1 score on the Weibo dataset and a 0.022 2 increase on the MVSA-Multiple dataset.Image features enhance sentiment classification accuracy under the single modality of text,but the degree of improvement is closely tied to the fusion strategy.The multimodal hierarchical feature mapping and fusion representation method effectively maps the relationship between text and image features,ultimately improving the effectiveness of sentiment classification in multimodal sentiment analysis.关键词
多模态特征融合/分级特征/映射与融合/情感分类/特征表示Key words
multimodal feature fusion/hierarchical feature/map and fusion/sentiment classification,feature representation分类
信息技术与安全科学引用本文复制引用
郭小宇,马静,陈杰..多模态分级特征映射与融合表征方法研究[J].计算机工程与应用,2025,61(6):171-182,12.基金项目
国家自然科学基金面上项目(72174086). （72174086）
