一种面向特殊领域隐语的大语言模型检测系统OA北大核心CSTPCD

Large language model(LLM)retrieve knowledge from their own structures and reasoning processes to generate responses to user queries,thus many researchers begin to evaluate the reasoning capabilities of large language models.However,while these models have demonstrated strong reasoning and comprehension skills in generic language tasks,there remains a need to evaluate their proficiency in addressing specific domain-related problems,such as those found in telecommunications fraud.In response to this challenge,this paper presents the first evaluation system for assessing the reasoning abilities of Domain-Specific Jargon and proposes the first domain specific jargon dataset.To address issues related to cross matching problem and complex data calculation problem,we propose the collaborative harmony algorithm and the data aware algorithm based on indicator functions.These algorithms provide a multi-dimensional assessment of the performance of large language models.Our experimental results demonstrate that our system is adaptable in evaluating the accuracy of question-answering by large language models within specialized domains.Moreover,our findings reveal,for the first time,variations in recognition accuracy based on question style and contextual cues utilized by the models.In conclusion,our system serves as an objective auditing tool to enhance the reliability and security of large language models,particularly when applied to specialized domains.