Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations


The problem of training a deep convolutional neural network with both low-bitwidth weights and activations is investigated in this paper. The non-differentiability of the quantizer makes it very difficult to optimize a network with low precision, which can lead to a significant loss of accuracy. This can make the optimization process very difficult. In order to solve this problem, we have come up with three useful solutions: I progressive quantization; (ii) stochastic precision; and (iii) joint knowledge distillation, all of which aim to improve the training of the network. First, for the purpose of progressive quantization, we suggest two different strategies for progressively arriving at good local minima. To be more specific, we propose first optimizing a network with quantized weights, and then, after that, quantizing activations in the network. In contrast to the conventional methods, which optimize both of them simultaneously, this approach optimizes neither. In addition, we propose a second progressive quantization scheme that, during training, makes the bitwidth transition gradually from high-precision to low-precision. Second, in order to alleviate the excessive training burden that is brought on by the multi-round training stages, we further propose a one-stage stochastic precision strategy that will randomly sample and quantize sub-networks while maintaining other parts in full-precision. Last but not least, we use an innovative learning strategy to train a full-precision model alongside a low-precision model at the same time. The performance of the low-precision network is significantly improved as a result of the full-precision model's use of this method, which involves providing hints to guide the training of the low-precision model. Extensive testing of the proposed methods on a variety of datasets (such as CIFAR-100 and ImageNet, for example) demonstrates that they are effective.

Did you like this research project?

To get this research project Guidelines, Training and Code... Click Here

PROJECT TITLE : The Devil Is in the Details An Efficient Convolutional Neural Network for Transport Mode Detection ABSTRACT: The objective of the classification problem known as transport mode detection is to devise an algorithm
PROJECT TITLE : Real-Time Tracking Algorithm for Aerial Vehicles Using Improved Convolutional Neural Network and Transfer Learning ABSTRACT: A real-time tracking algorithm that makes use of an improved convolutional neural network
PROJECT TITLE : On Smart Gaze based Annotation of Histopathology Images for Training of Deep Convolutional Neural Networks ABSTRACT: To fully realize the potential of deep learning in histopathology applications, a bottleneck
PROJECT TITLE : Train Time Delay Prediction for High-Speed Train Dispatching Based on Spatio-Temporal Graph Convolutional Network ABSTRACT: Train delay prediction has the potential to improve the quality of train dispatching,
PROJECT TITLE : Spatio-Temporal-Spectral Hierarchical Graph Convolutional Network With Semisupervised Active Learning for Patient-Specific Seizure Prediction ABSTRACT: At the moment, one of the most cutting-edge approaches for

Ready to Complete Your Academic MTech Project Work In Affordable Price ?

Project Enquiry