多尺度视觉语义增强的多模态命名实体识别方法OA北大核心CSTPCD

To address the issues of semantic loss in image features and weak semantic constraints in multimodal representations encountered in the research of multimodal named entity recognition(MNER)methods,multi-scale visual semantic enhancement for multimodal named entity recognition method(MSVSE)is proposed.After supple-menting image semantics by extracting multiple visual features,the semantic interaction and feature fusion between text features and various visual features are explored through a multimodal feature fusion module.This process out-puts multi-scale visual semantic-enhanced multimodal text representations.The visual entity classifier is used to de-code multi-scale visual semantic features to learn the semantic consistency between various visual features.The multi-task decoder is invoked to mine the fine-grained semantic representation in multimodal text repre-sentation and text features,and carry out joint decoding to solve the semantic bias problem,thereby further improving the accuracy of named entity recognition.To verify the effectiveness of the method,experiments were carried out on Twitter-2015 and Twitter-2017 respectively,and compared with other 10 methods.The average F1 values of the MSVSE on the two datasets have increased.