信息安全研究2024，Vol.10Issue(9)：795-803,9.DOI:10.12379/j.issn.2096-1057.2024.09.02
一种面向特殊领域隐语的大语言模型检测系统
A Large Language Model Detection System for Domain-specific Jargon
摘要
Abstract
Large language model(LLM)retrieve knowledge from their own structures and reasoning processes to generate responses to user queries,thus many researchers begin to evaluate the reasoning capabilities of large language models.However,while these models have demonstrated strong reasoning and comprehension skills in generic language tasks,there remains a need to evaluate their proficiency in addressing specific domain-related problems,such as those found in telecommunications fraud.In response to this challenge,this paper presents the first evaluation system for assessing the reasoning abilities of Domain-Specific Jargon and proposes the first domain specific jargon dataset.To address issues related to cross matching problem and complex data calculation problem,we propose the collaborative harmony algorithm and the data aware algorithm based on indicator functions.These algorithms provide a multi-dimensional assessment of the performance of large language models.Our experimental results demonstrate that our system is adaptable in evaluating the accuracy of question-answering by large language models within specialized domains.Moreover,our findings reveal,for the first time,variations in recognition accuracy based on question style and contextual cues utilized by the models.In conclusion,our system serves as an objective auditing tool to enhance the reliability and security of large language models,particularly when applied to specialized domains.关键词
大语言模型/特殊领域隐语/隐语检测/评估系统/黑话/推理Key words
large language model/Domain-Specific Jargon/cant language detection/evaluation system/slang/reasoning分类
信息技术与安全科学引用本文复制引用
姬旭,张健毅,赵张驰,周子寅,李毅龙,孙泽正..一种面向特殊领域隐语的大语言模型检测系统[J].信息安全研究,2024,10(9):795-803,9.基金项目
国家重点研发计划项目(2018YFB1004100) （2018YFB1004100）
