中医杂志2026，Vol.67Issue(17)：1845-1852,8.DOI:10.13288/j.11-2166/r.2026.17.008
面向中医辨证任务的大语言模型证型幻觉自动评测方法比较研究
Comparison of Automatic Evaluation Methods for Pattern Hallucinations of Large Language Models in Traditional Chinese Medicine Syndrome Differentiation
摘要
Abstract
Objective To compare the performance of different automatic evaluation methods for detecting pattern hallucinations generated by large language models(LLMs)in traditional Chinese medicine(TCM)syndrome differentiation and to identify a strategy with higher overall discriminative performance against expert manual judg-ment.Methods A standardized TCM syndrome-differentiation dataset containing 598 cases was constructed from case reports published in Chinese core journals indexed by Peking University Core Journals or Chinese Science Citation Database.Each of the 598 cases was submitted to five Chinese LLMs,with each model generating one syndrome-pattern output per case,yielding 2990 outputs in total.The 598 outputs generated by Qwen3-Next-80B-A3B-Thinking were independently annotated for syndrome-pattern hallucinations by two licensed TCM physicians with intermediate or higher professional titles.Disagreements were adjudicated by a third licensed TCM physician with a senior associate professional title and more than 10 years of clinical experience.The resulting consensus annotations served as the reference labels for training and validating the automated evaluation method.Based on these reference labels,three categories of automated evaluation methods were developed and compared following a progressive strategy from text-level semantic similarity,to syndrome-element structural consistency,and finally multi-feature fusion.Method A employed sentence-embedding models based on semantic similarity between pattern texts.Method B adopted a rule-based threshold method using weighted Jaccard coefficients of disease-location and disease-nature syndrome elements.Method C utilized a machine-learning classifier integrating syndrome-element matching scores,missing and redundant syndrome-element counts,and multiple semantic-similarity features.Five-fold cross-validation was used to evaluate the discriminative performance of each method against manual judgment.The optimal method was then applied to uniformly assess the pattern hallucination rates of five LLMs.Results In method A,text2vec-large-chinese showed best overall performance,with an area under the receiver operating characteristic curve(AUROC),area under the precision-recall curve(PR-AUC),F1-score,sensitivity,and specificity of 0.849,0.884,0.794,0.801,and 0.691,respectively.In method B,the corresponding metrics of the rule-based syndrome-element Jaccard threshold method were 0.848,0.851,0.817,0.845,and 0.684,respectively;those of the LLM-based syndrome-element Jaccard method were 0.872,0.906,0.795,0.764,and 0.780,respectively.In method C,the multilayer perceptron(MLP)classifier showed the best overall performance,with AUROC,PR-AUC,F1-score,sensitivity,and specificity of 0.933,0.951,0.871,0.857,and 0.846,respectively.Under the unified evaluation framework combining Deep-Seek-R1 syndrome-element extraction and MLP classifier,the estimated pattern hallucination rates of the five LLMs ranged from 56.0％to 77.4％.Conclusion Among the three automatic evaluation methods,Method C integrating syndrome-element discrepancy features and text semantic features using an MLP classifier showed the highest overall discriminative performance against manual judgment,and is more suitable for evaluating syndrome hallucinations in LLM-based TCM syndrome differentiation tasks.关键词
中医辨证/大语言模型/证素/机器学习/多层感知机/文本相似度/Jaccard系数Key words
traditional Chinese medicine syndrome differentiation/large language models/syndrome elements/machine learning/multilayer perceptron/text similarity/Jaccard coefficient引用本文复制引用
吴沁蔚,高雨竹,苟兴悦,姚俊宇,周川淦,薛正淳,徐志荣,陈新林,曹东..面向中医辨证任务的大语言模型证型幻觉自动评测方法比较研究[J].中医杂志,2026,67(17):1845-1852,8.基金项目
国家重点研发计划(2023YFC3503002) （2023YFC3503002）
广东省高等教育教学改革项目(2024-556) （2024-556）
广州中医药大学中医药大数据研究中心平台项目(A1-2601-25-439-127Z102) （A1-2601-25-439-127Z102）
广州中医药大学2026年医学信息工程学院"揭榜挂帅"项目(2026-7) （2026-7）
