电子学报2025，Vol.53Issue(2)：568-580,13.DOI:10.12263/DZXB.20240640
高效还原式二值神经网络
Efficient Restoration for Binary Neural Networks
摘要
Abstract
Restoring the weight distribution,activation distribution,and gradient to the original full precision network data as much as possible can greatly improve the inference ability of the binary network.However,existing methods direct-ly apply the restoration operation in forward propagation to binary data,and the gradient approximation functions for back-propagation are fixed or manually determined,resulting in the need for improvement in the restoration efficiency of binary networks.To address this problem,the efficient restoration method is investigated for binary neural networks.Firstly,a dis-tribution recovery method for maximizing information entropy is proposed.By shifting the original full precision weight mean and scaling the modulus,the quantized binary weight directly has the characteristic of maximum distribution restora-tion.At the same time,a simple statistical translation and scaling factor is used to greatly improve the restoration efficiency of weight and activation.Furthermore,it is proposed a gradient function based on adaptive distribution approximation,which dynamically determines the update range of the current gradient in the P-percentile according to the actual distribu-tion of the current full precision data.It adaptively changes the shape of the approximation function to efficiently update the gradient during the training process,thereby improving the convergence ability of the model.On the premise of ensuring the improvement of execution efficiency,theoretical analysis has confirmed that the method proposed in this paper can achieve maximum restoration of binary data.Compared with the existing advanced binary network models,the experimental results of our method show excellent performance,with a 60%and 67%reduction in computational time for the distribution resto-ration operation quantization of ResNet-18 and ResNet-20,respectively.An accuracy of 93.0%is achieved for VGG-Small binary quantization on the CIFAR-10 dataset,and 61.9%is achieved for ResNet-18 binary quantization on the ImageNet da-taset,both of which are the best performance of the current binary neural network.The relevant code is available inhttps://github.com/sjmp525/IA/tree/ER-BNN.关键词
二值神经网络/信息还原/信息熵最大/自适应梯度Key words
binary neural network/information restoration/maximum information entropy/adaptive gradient分类
信息技术与安全科学引用本文复制引用
曾凯,万子鑫,王铭涛,沈韬..高效还原式二值神经网络[J].电子学报,2025,53(2):568-580,13.基金项目
云南省杰出青年人才项目(No.202301AV070003) （No.202301AV070003）
云南省重大科技专项(No.202302AG050009) （No.202302AG050009）
云南省重大科技专项(No.202202AD080013) Yunnan Outstanding Young Talent Project(No.202301AV070003) （No.202202AD080013）
Yunnan Major Science and Technology Project(No.202302AG050009) （No.202302AG050009）
Yunnan Major Science and Technology Project(No.202202AD080013) （No.202202AD080013）
