含能材料2026,Vol.34Issue(10):1111-1121,11.DOI:10.11943/CJEM2026166
面向含能材料知识图谱构建的大语言模型零样本信息抽取基准研究
A Benchmark for Zero-shot Information Extraction of Large Language Models in Energetic Materials
摘要
Abstract
Accurate extraction of material entities,attributes,and entity-attribute associations from unstructured scientific litera-ture is a critical step in constructing high-quality datasets for energetic materials(EMs)and providing a structured knowledge foundation for subsequent knowledge graph construction.In this study,we developed EM-Bench 1.0,a full-text information ex-traction benchmark for Chinese language literature on EMs.The benchmark comprises 40 articles published in Chinese core jour-nals from 2021 to 2026 and includes annotations for their material names,chemical formulas,polymorphs,crystal systems,space groups,unit-cell parameters,and densities.Simplified Molecular Input Line Entry System(SMILES)representations were treated separately as an exploratory knowledge-reasoning task.The benchmark ground truth was independently annotated by two domain researchers,followed by cross-validation,with disputed cases adjudicated by a domain expert.Under a task-level zero-shot setting,each article was converted into a complete Markdown document and evaluated using a unified prompt and JSON output format.Four large language model(LLM)platforms,ChatGPT,DeepSeek,Gemini,and ChemELLM,were evaluat-ed at the full-text level.Each model-document pair was independently tested three times under identical conditions,and the val-id output with the highest overall F1 score was retained as the final result to characterize the relatively strong performance attain-able under the current test setting.Predictions were semantically matched and comparatively analyzed at the material-field in-stance level.Gemini achieved the highest overall F1 score(70.1%),whereas ChatGPT achieved the highest recall(82.2%).ChemELLM performed well in material-name extraction(F1=70.5%)and polymorph extraction(F1=13.8%),produced no missing units for unit-cell parameters,and generated the fewest unsupported entities(51).The results indicate that current LLMs already possess a certain capability for full-text information extraction from Chinese literature on EMs;however,omissions,un-supported generations,and entity-attribute mismatches remain common.Therefore,model outputs should not be directly used for high-confidence knowledge ingestion without expert verification.The reported results reflect model performance under the specific test dates,web-interface configurations,and best-of-three selection strategy adopted in this study,rather than stable model performance or persistent cross-version rankings.EM-Bench 1.0 is intended to provide a standardized benchmark for large language model selection and fine-tuning,EM high-quality dataset construction,and automated knowledge acquisition for knowledge graph construction.关键词
含能材料/大语言模型/信息抽取/零样本学习/知识图谱/评测基准Key words
energetic materials/large language models/information extraction/zero-shot learning/knowledge graph/benchmark分类
军事科技引用本文复制引用
臧小为,钱志翔,葛一帆,朱朋,许建兵,张伟,叶迎华,沈瑞琪..面向含能材料知识图谱构建的大语言模型零样本信息抽取基准研究[J].含能材料,2026,34(10):1111-1121,11.基金项目
瞬态化学效应与控制全国重点实验室稳定运行费资助项目(WDYX25614260207) (WDYX25614260207)
中央高校基本科研业务费专项资金资助(30926010601) State Key Laboratory of Transient Chemical Effects and Control,China(No.WDYX25614260207) (30926010601)
The Funda-mental Research Funds for the Central Universities of Ministry(No.30926010601) (No.30926010601)