计算机工程2023，Vol.49Issue(12)：88-95,8.DOI:10.19678/j.issn.1000-3428.0066938
基于多模态知识图谱的中文跨模态实体对齐方法
Chinese Cross-modal Entity Alignment Method Based on Multi-modal Knowledge Graph
摘要
Abstract
Interactive tasks involving multi-modal data present advanced requirements for the comprehensive utilization of knowledge from different modalities,leading to the emergence of multi-modal knowledge graphs.When constructing these graphs,accurately determining whether image and text entities refer to the same object is particularly important for entity alignment of Chinese cross-modal entities.To address this problem,a Chinese cross-modal entity alignment method based on a multi-modal knowledge graph is proposed.Image information is introduced into the entity alignment task,and a single and dual-stream interactive pre-trained language model,namely CCMEA,is designed for domain-specific,fine-grained images and Chinese text.Utilizing a self-supervised learning method,Text-Visual features are extracted using Text-Visual Encoder,and fine-grained modeling is performed using cross-coders.Finally,a comparison learning method is employed to evaluate the degree of alignment between image and text entities.The experimental results show that the Mean Recall(MR)of the CCMEA model improved by 3.20 and 11.96 percentage points compared to that of the WukongViT-B baseline model on the MUGE and Flickr30k-CN datasets,respectively.Furthermore,the model achieved a remarkable MR of 94.3%on the self-built TEXTILE dataset.These results demonstrate that the proposed method can effectively align Chinese cross-modal entities with high accuracy in practical applications.关键词
多模态/知识图谱/实体对齐/自监督/纺织行业Key words
multi-modal/knowledge graph/entity alignment/self-supervision/textile industry分类
信息技术与安全科学引用本文复制引用
王欢,宋丽娟,杜方..基于多模态知识图谱的中文跨模态实体对齐方法[J].计算机工程,2023,49(12):88-95,8.基金项目
国家自然科学基金(62062058) （62062058）
宁夏回族自治区重点研发计划(2021BEE03013). （2021BEE03013）
