基于动态Power迭代的大语言模型微调算法
Algorithm for Fine-Tuning Large Language Models Based on Dynamic Power Iteration
摘要
Abstract
With the widespread application of large language models(LLMs)across various domains,fine-tuning has become a critical method for adapting these models to specific tasks.Current mainstream fine-tuning approaches can be categorized into two types:partial fine-tuning and full fine-tuning.Although partial fine-tuning reduces computational overhead,it updates only a small subset of model parameters,which limits the performance of the fine-tuned model in complex task scenarios.In contrast,full fine-tuning overcomes this limitation by updating all model parameters,but it comes at the cost of significantly higher computational requirements and longer fine-tuning time.To address the challenges associated with full fine-tuning,this paper proposes a dynamic power iteration gradient low-rank projection algorithm(DPI-GLRP).Specifically,this method is based on the idea of rank-one matrix approximation,where the gradient matrix generated from the weight matrix of a large model during backpropagation is decomposed into multiple rank-one matrices.The top r eigenvectors are then computed using the Power iteration algorithm to construct the projection matrix.This approach not only overcomes the limitation of traditional Power iteration,which can extract only the dominant eigenvector,but also addresses the high time complexity and long fine-tuning duration associated with constructing projection matrices using singular value decomposition in previous studies.Furthermore,this paper analyzes the traditional Power iteration method and observes that it converges slowly when eigenvalues are closely distributed.To tackle this issue,this paper introduces a dynamic Power iteration algorithm that adaptively adjusts its iteration parameters to accelerate the computation of eigenvectors.This paper also provides theoretical proof that the proposed dynamic Power iteration achieves faster convergence compared with the traditional variant.Finally,experiments conducted on large models such as LLaMA and Qwen demonstrate that,compared with mainstream approaches like LoRA,the DPI-GLRP algorithm not only maintains or improves model performance,but also significantly reduces fine-tuning time,achieving up to an 80%reduction on average.关键词
大语言模型/微调/梯度低秩投影/Power迭代Key words
large language models/fine-tuning/gradient low-rank projection/Power iteration分类
信息技术与安全科学引用本文复制引用
匡豪,刘波,李辉越,曾闰,段围..基于动态Power迭代的大语言模型微调算法[J].计算机科学与探索,2026,20(3):785-800,16.基金项目
重庆市自然科学基金创新发展联合基金项目(CSTB2025NSCQ-LZX0134) （CSTB2025NSCQ-LZX0134）
重庆市自然科学基金重点项目(CSTB2023NSCQ-LZX0029) （CSTB2023NSCQ-LZX0029）
重庆市教育委员会科学技术研究项目重点项目(KJZD-K202400809) （KJZD-K202400809）
教育部科学研究发展中心资助项目(2023ZY024) （2023ZY024）
重庆工商大学开放基金项目(1752004).This work was supported by the Chongqing Natural Science Foundation Joint Fund for Innovation and Development(CSTB2025NSCQ-LZX0134),the Key Project of the Natural Science Foundation of Chongqing(CSTB2023NSCQ-LZX0029),the Key Project of the Science and Technology Research Program of the Chongqing Municipal Education Commission(KJZD-K202400809),the Project of Science and Research Development Center of the Ministry of Education of China(2023ZY024),and the Open Fund of Chongqing Technology and Business University(1752004). （1752004）
