计算机科学与探索2024，Vol.18Issue(8)：2156-2168,13.DOI:10.3778/j.issn.1673-9418.2306073
采用低秩编码优化大语言模型的高校基础知识问答研究
Research on University Basic Knowledge Question-Answering Using Low-Rank Encoding to Optimize Large Language Model
摘要
Abstract
In the field of higher education,foundational knowledge question-answering(QA)systems play a crucial role in enhancing students'academic performance and facilitating equitable distribution of educational resources.In recent years,question-answering techniques based on machine reading comprehension and text similarity matching have been developed atop pre-trained language models.However,when addressing complex natural language prob-lems,these techniques still face challenges in answer quality and accuracy due to limitations like insufficient training data and restricted model generalization capabilities.This research aims to address the dual objective of reducing re-source consumption while simultaneously enhancing the performance and accuracy of basic knowledge question-answering systems in university settings.To achieve this goal,this paper proposes an efficient fine-tuning approach for large language model with low-rank encoding in the domain of fundamental knowledge.This method uses low-rank encoding to minimize computational costs during the training and prediction stages of large language models.It en-hances research and analysis in our university's basic knowledge question-answering field by employing the genera-tive capabilities of these models,improving the quality,accuracy,and response speed of everyday queries.By freezing the weights of the pre-trained model and integrating university-specific knowledge into the Transformer architec-ture,along with a question-answering optimization module,this approach preserves generative abilities and achieves superior performance and accuracy in the university knowledge domain while reducing trainable parameters for downstream tasks.关键词
生成式语言模型/基础知识问答/大语言模型/Transformer/冻结模型权重Key words
generative language model/fundamental knowledge question-answering/large language model/Trans-former/freezing model weights分类
信息技术与安全科学引用本文复制引用
骆仕杰,金日泽,韩抒真..采用低秩编码优化大语言模型的高校基础知识问答研究[J].计算机科学与探索,2024,18(8):2156-2168,13.基金项目
国家自然科学基金(61806142) （61806142）
天津市科学技术局项目(19PTZWHZ00020) （19PTZWHZ00020）
中国学位与研究生教育学会项目(2020MSA50) （2020MSA50）
产学合作协同育人项目(202102084059). This work was supported by the National Natural Science Foundation of China(61806142),the Program of Tianjin Municipal Science and Technology Bureau(19PTZWHZ00020),the Project of the Association of Chinese Graduate Education(2020MSA50),and the Uni-versity Industry Collaborative Education Program(202102084059). （202102084059）
