华南理工大学学报（自然科学版）2026，Vol.54Issue(8)：1-13,13.DOI:10.12141/j.issn.1000-565X.250406
基于昇腾NPU的矩阵向量乘优化策略
Optimization Strategies for Matrix-Vector Multiplication Based on Ascend NPU
摘要
Abstract
General Matrix-Vector Multiplication(GEMV)is a core operation in the decoding phase of large lan-guage models,and its performance directly determines the inference efficiency of the models.However,existing Neural Processing Units(NPUs)mostly implement GEMV operations by invoking General Matrix Multiplication(GEMM)interfaces,which suffer from poor adaptability and low hardware resource utilization in small-scale dimen-sion scenarios.This paper conducts research on hardware adaptation and performance optimization of GEMV opera-tors based on the Ascend 910 B NPU.By leveraging the hardware features of the Ascend 910 B AI Core compute kernels,this paper designs fundamental GEMV implementation schemes adapted to the corresponding compute units,respectively based on the matrix operation unit(AIC)and the vector operation unit(AIV).To meet the hard-ware data layout requirements of AIC,this paper equivalently transforms the GEMV operation into matrix-transpose-vector multiplication to improve result write-back efficiency.To address the memory access characteristics of the AIV,differentiated computation flows are designed according to different data layouts to ensure the continuity of memory access and compute instructions.On this basis,this paper further proposes general optimization strategies:adopting a rectangular tiling mechanism to maintain memory access continuity and reduce instruction dispatch over-head,and employing secondary task partitioning to alleviate computational load imbalance.Additionally,this paper designs an operator fusion interface that supports the fusion of GEMV with elementwise operators,with performance overhead kept within 2%when fusion computation is completed on the AIV unit.Experimental results show that under three data precision scenarios—single-precision floating-point,half-precision floating-point,and 8-bit integer,the performance of the GEMV implemented and optimized in this study reaches 3.38 times,2.94 times,and 23.95 times(including dequantization)that of the default GEMM interface of the Ascend CANN platform,respectively.This provides strong support for efficient deployment and inference of large models on NPU platforms.关键词
通用矩阵向量乘法/昇腾NPU/算子融合Key words
general matrix-vector multiplication/Ascend NPU/operator fusion分类
信息技术与安全科学引用本文复制引用
陆璐,古钟书..基于昇腾NPU的矩阵向量乘优化策略[J].华南理工大学学报（自然科学版）,2026,54(8):1-13,13.基金项目
广东省自然科学基金项目(2024A1515010204) （2024A1515010204）
南方电网科学研究院项目(1500002024030103XA00063) Supported by the Natural Science Foundation of Guangdong Province(2024A1515010204) （1500002024030103XA00063）
