高技术通讯2024，Vol.34Issue(6)：567-577,11.DOI:10.3772/j.issn.1002-0470.2024.06.002
一种基于自适应PoT量化的无乘法神经网络训练方法
Multiplication-free neural network training based on adaptive PoT quantization
摘要
Abstract
The current deep neural network training process needs a large number of full-precision multiply-accumulate(MAC)operations,resulting in a situation that the energy consumption of the linear layers(including the convolu-tional layer and the fully connected layer)accounts for the vast majority of the overall energy consumption,reac-hing more than 90%.This work proposes an adaptive layer-wise scaling quantization training method,which can support the replacement of full-precision multiplication in all linear layers with 4-bit fixed-point addition and 1-bit XOR operation.The experimental results show that the above method is superior to the existing methods in terms of energy consumption and accuracy,and can reduce the energy consumption of linear layers by 95.8%in the train-ing process.The convolutional neural networks on ImageNet and the Transformer networks on WMT En-De achieve less than 1%accuracy loss.关键词
神经网络/量化/训练加速/低能耗Key words
neural network/quantization/training acceleration/low energy consumption引用本文复制引用
刘畅,张蕊,支天..一种基于自适应PoT量化的无乘法神经网络训练方法[J].高技术通讯,2024,34(6):567-577,11.基金项目
国家重点研发计划(2018AAA0103300),国家自然科学基金(62102399,U22A2028,U20A20227)和中国科学院稳定支持基础研究领域青年团队计划(YSBR-029)资助项目. （2018AAA0103300）
