现代情报2025，Vol.45Issue(6)：34-45,12.DOI:10.3969/j.issn.1008-0821.2025.06.004
基于增强异构图融合的多模态医学实体识别研究
Multimodal Medical Named Entity Recognition Based on Augmented Heterogeneous Graph Fusion
摘要
Abstract
[Purpose/Significance]To fully explore the semantic association between medical images and text,this paper employs an augmented heterogeneous graph fusion method to improve image feature representation effect and interac-tive image and text information to achieve feature fusion,thereby enhancing the performance of multimodal medical named entity recognition.[Method/Process]Firstly,RoBERTa and ResNet extracted features from medical text and images,re-spectively.Subsequently,a visual augment module extracted key information from images and filtered out irrelevant noise.After extracting the features,a heterogeneous graph was constructed by using text and image nodes along with their corre-sponding edges to capture fine-grained semantic associations between the two modalities.The fusion of medical multimodal features occurred through a self-attention mechanism,a cross-modal gating mechanism,and a position-wise feed-forward network.Finally,the experiment validated the effectiveness of entity recognition on a Chinese multimodal medical dataset.[Result/Conclusion]The RMGFM model constructed in this study achieves an F1 value of 88.99％on the Chinese multi-modal medical dataset,which is an improvement of 5.52％,5.28％,and 5.08％compared to the F1 values of the multi-modal baseline models of UMT,AGBAN,and UMGF,respectively.Experiments show that the Ro-UMGF∗+Manifold(RMGFM)model proposed in this study can effectively mine semantic associations between medical images and texts,and performs well in the task of recognizing multimodal medical entities in Chinese.关键词
异构图融合/视觉增强/多模态命名实体识别/语义融合/医疗健康Key words
heterogeneous graph fusion/visual augmentation/multimodal named entity recognition/semantic fu-sion/medical and health引用本文复制引用
韩普,李雄..基于增强异构图融合的多模态医学实体识别研究[J].现代情报,2025,45(6):34-45,12.基金项目
国家社会科学基金项目"面向多模态医疗健康数据的知识组织模式研究"(项目编号:22BTQ096) （项目编号:22BTQ096）
江苏省研究生科研创新计划基金项目"面向医疗健康数据的多模态信息抽取研究"(项目编号:KYCX24_1094). （项目编号:KYCX24_1094）
