# Compress network by fixed point quantization¶

This tutorial introduces how to compress your network by fixed point quantization.

## Introduction¶

Neural networks show reliable results on AI fields, such as object recognition and detections are useful in real applications. Concurrent to the the progress in recognition, the increase of IoT devices at the edge of the network is producing a massive amount of data to be computed to data centers, pushing network bandwidth requirements to the limit. Despite the improvements of network technology, data centers cannot guarantee acceptable transfer rates and response times, which could be a critical requirement for many applications. But CNN-based recognition systems need large amounts of memory and computational power, which perform well on expensive GPU-based machines, for example, AlexNet has 61M parameters (249MB of memory) and performs 1.5B high precision operations to classify one image. These numbers are even higher for deeper CNNs e.g.,VGG. They are often unsuitable for such edge devices, like cell phones, small devices and embedded electronics. Hence, reducing the storage of network and computation complexity is the task of optimization.

Currently, quantizing the weights and (or) input is the way which can dramatically reduce the memory and computation time and even keep original accuracy [1] (*).

This tutorial tends to share a few of experiments using nnabla for such optimitions.

## Two approaches¶

There are mainly 2 approaches to design a fixed-point deep convolution network.

• Train a network with fix-point constraint
• Convert a pretrain float-point network to its fixed-point version

Binary Connect series functions, such as BinaryConnectAffine, BinaryConnectConvolution and Binary Weight series functions, such BinaryWeightAffine and BinaryWeightConvolution, fixed-point quantized series, such as FixedPointQuantizedAffine, FixedPointQuantizedConvolution acts as an important role for the first approach. These functions have different data paths for forward and backward. When forward, the float-point weights are converted to fixed-point weights or even binary weights, and the output is calculated by multiple such binary(quantized) weights with input. When backward, only float-point(or learnable) weights participate in the calculation of updating weights. In fact, binarizing weight is too extreme, fixed point quantization seems to be more commonly used.

Due to the second approach, a few steps need to be performed to keep minimal loss of accuracy. [2] provides a comprehensive analysis of such quantization for the relationship between quantization size and performance. Here, we shall do some experiments to illustrate the trade-off between accuracy and storage size.

We compared the ResNet23’s original version and the version using binary_weight_convolution(), we can observe the accuracy loss after using binary_weight_convolution(), which is worse than [3] mentioned. It should be possible to improve after more careful fine-tuning.

This tutorial only shows a basic way to compress network. There are still a lot of new approach not presented in this tutorial. For example, a new way is proposed as in [4], a quantization network is designed as a student network, a float-point high precision network is designed as a teacher network, the distillation technique is used to train to obtain the-state-of-art performance and accuracy.

## The benchmark¶

We chose a simple network ResNet23 and CIFAR-10 dataset as the benchmark. First, let us obtain base metrics of this benchmark, such as accuracy, model size, and so on. The following shows the network structure:

Layer name Shape Required buffer size
conv1 (1, 3, 32, 32) -> (1, 64, 32, 32) 281344
conv2 (1, 64, 32, 32)->(1, 32, 32, 32)->(1, 32, 32, 32) -> (1, 64, 32, 32) 786432
conv3 (1, 64, 16, 16)->(1, 32, 16, 16)->(1, 32, 16, 16) -> (1, 64, 16, 16) 196608
conv4 (1, 64, 16, 16)->(1, 32, 16, 16)->(1, 32, 16, 16) -> (1, 64, 16, 16) 196608
conv5 (1, 64, 8, 8)->(1, 32, 8, 8)->(1, 32, 8, 8) -> (1, 64, 8, 8) 49152
conv6 (1, 64, 8, 8)->(1, 32, 8, 8)->(1, 32, 8, 8) -> (1, 64, 8, 8) 49152
conv7 (1, 64, 4, 4)->(1, 32, 4, 4)->(1, 32, 4, 4) -> (1, 64, 4, 4) 12288
conv8 (1, 64, 4, 4)->(1, 32, 4, 4)->(1, 32, 4, 4) -> (1, 64, 4, 4) 12288

To downsize the footprint, we should consider to downsize both the parameter size and variable buffer size, and especially prior for the maximal buffer size.

The network is created by the following code:

The top-1 error reaches to 0.16 as the following diagram:

We compared the accuracy between nnabla_cli infer and nnablart infer in CIFAR10 test dataset. The comparison code is as the following:

In this code, nnablart is an executable implemented based on nnabla-c-runtime. nnablart is a simple command-line interface, which can infer the network defined by *.nnb file. As we known, nnabla-c-runtime is a c implementation aims to small device with constraint memory, it contains carefully designed memory policy, and the code for training purpose is removed for saving memory. This test program iterates 100 samples, comparing with ground truth, figure out the accuracy.

...
NNabla command line interface (Version:1.0.18, Build:190619071959)
0: input.bin
1: output_1
Input[0] size:3072
Input[0] data type:NN_DATA_TYPE_FLOAT, fp:0
Input[0] Shape ( 1 3 32 32 )
Output[0] size:10
Output[0] filename output_1_0.bin
Output[0] Shape ( 1 10 )
Output[0] data type:NN_DATA_TYPE_FLOAT, fp:0
99:  all same!
nnp accuracy: 0.81, nnb accuracy: 0.81


## binary_weight_convolution¶

We replaced PF.convolution() with PF.binary_weight_convolution() as the following:

The training become a bit slower and the accuracy loss can be detected. As the following:

We saved the model and parameters as *.nnp file. Then, we shall convert it to *.nnb so that it can fit the memory-constraint device.

### Reduce parameter size of *.nnb model¶

We need to set the data type of corresponding parameters, so that the binarized weights can be represented by SIGN data type.

Export a buffer setting file from our trained model

$/> nnabla_cli nnb_template bin_class.nnp setting.yaml  The output setting.yaml looks like: functions: ... variables: ... input: FLOAT32 <-- buffer conv1/bwn_conv/W: FLOAT32 <-- parameter conv1/bwn_conv/Wb: FLOAT32 <-- parameter conv1/bwn_conv/alpha: FLOAT32 <-- parameter BinaryWeightConvolution_Output: FLOAT32 conv1/bn/beta: FLOAT32 conv1/bn/gamma: FLOAT32 conv1/bn/mean: FLOAT32 conv1/bn/var: FLOAT32 BatchNormalization_Output: FLOAT32 ReLU_Output: FLOAT32 affine/W: FLOAT32 affine/b: FLOAT32 output: FLOAT32 ...  We annotated buffer and parameter type based on its name character. The different between buffer and parameter is that: buffer value is undetermined at this time, while parameter values is determined. The quantization policy is different. As we known, conv1/bwn_conv/W is the float version, will not be used, just omit this. We need identify conv1/bwn_conv/Wb to be “SIGN” type, it looks like: functions: ... variables: ... input: FLOAT32 conv1/bwn_conv/W: FLOAT32 <-- omit conv1/bwn_conv/Wb: SIGN <-- identified as SIGN ... output: FLOAT32 ...  We tested the top-1 error in test dataset as the following: nnp accuracy: 0.76, nnb accuracy: 0.73  As we can see, accuracy loss is trivial compared with its float version. And the *.nnb size is reduced from 830KB to 219KB. ## binary_connect_convolution¶ We replaced PF.convolution() with PF.binary_connect_convolution() and do same training as above. The training become a bit slower and the accuracy loss can be detected. As the following: We tested the top-1 error in test dataset as the following: nnp accuracy: 0.68, nnb accuracy: 0.71  As we can see, accuracy loss can be observed, but nnabla_cli got worse result than nnablart. From this test result, we found float32 version worse than binary quantized version. That seems a problem. The reason might be since the training process passes the data through 2 data paths, the binary weight data path has lower loss that the float32 data path at the same time. ## Quantization functions¶ The mainly difference between binary_weight series functions and binary_connect series functions is the quantizing formular: For binary_weight_convolution or binary_weight_affine: $\alpha^*=\frac{\sum{|W_i|}}{n}=\frac{1}{n}\|\mathbf{W}\|_{\ell_1}$ $\begin{split}B=\left\{ \begin{array}{ll} +1 & if \ \ W \ge 0 \\ -1 & if \ \ W < 0 \end{array} \right.\end{split}$ $W \approx \alpha B$ For binary_connect_convolution or binary_connect_affine, there are 2 alternative binarization operations, one is: $\begin{split}W_b= \left\{ \begin{array}{ll} +1 & if\ w \ge 0 \\ -1 & otherwise \end{array} \right.\end{split}$ Another way is: $\begin{split}W_b= \left\{ \begin{array}{ll} +1 & if\ with\ probability\ p=\sigma (\omega ) \\ -1 & if\ with\ probability\ 1 - p \end{array} \right. \\\end{split}$ where sigma is the “hard sigmoid” function, $\sigma (x) = clip(\frac{x + 1}{2}, 0, 1) = max (0, min(1, \frac{x+1}{2}))$ In nnabla implementation, binary_connect_xxxx() implements the following formular: $\begin{split}W_b=sign(W) = \left\{ \begin{array}{ll} +1 & if \ \ W > 0 \\ 0 & if \ \ W = 0 \\ -1 & if \ \ W < 0 \end{array} \right.\end{split}$ According to this experiment, the accuracy is a bit different: Accuracy inferred by nnabla Accuracy inferred by nnablart Model size float point 0.81 0.81 449.5 KB using binary weight convolution 0.76 0.75 52.1 KB using binary connect convolution 0.68 0.71 47.3 KB The model size has already been cut-down about 10x dramatically. ## Further reduce footprint¶ In order to keep maximal accuracy and reduce footprint as much as possible, let us try the second method as mentioned before. This method tends to work based on a pretrained network. Quantization process is done after training. Here, we chose the float-point trained model and start our experiment. As previous analysis, we knew that the big share of footprint in this benchmark network is buffer size. As the following diagram, a circle represents a variable buffer, a rectangle represents a function. When perform function 1, buffer 1, buffer 2 and buffer 3 are occupied. After performing function 1, when function2 is performed, buffer 1 and buffer 2 are released, and this function’s output reuse buffer 1 if the size of buffer 1 can hold the data of function 2’s output. This buffer-reuse policy has been implemented during converting from *.nnp to *.nnb. The maximal of the occupying memory for each function represents the maximal footprint memory for inferring this network. In order to reduce this size, we may use quantization data type for variable buffer. As previous setting.yaml, if the following buffer type is changed, when converting from *.nnp to *.nnb, the buffer size will be calculated based on this new buffer type definition. functions: ... variables: ... input: FLOAT32 ==> input: FIXED8 conv1/bwn_conv/W: FIXED8 conv1/bwn_conv/Wb: FIXED8 ... output: FLOAT32 ...  As we known, this quantization process introduce quantization noise to the network, which sometimes cause obviously loss of the accuracy. How to choose best quantization step size, you may refer to [2]. ## Determine the fixed-point position¶ In converting from *.nnp to *.nnb, the fixed-point position can be determined automatically when quantizing parameter type, since the histogram(or distribution) is known at that time. Due to the fixed-point position of the variable’s buffer, in order to keep distortion as little as possible, we should determine its fixed-point according to its histogram (or distribution). But it is hard to know exactly the distribution of each of variable buffer, even though we statistics during training time. We supposed future test dataset has same distribution as current known dataset, make fixed-point decision based on the collection of current known dataset. Manually tuning the fixed-point position is a work like an art. We shared some experience here. But an intelligent and automatic method seems be necessary. As the following diagram, we collected the variable buffer’s distribution in a small known dataset. (Not all variable distribution are listed here.) Of course, the simplest way to determine fixed-point position is only to dump the minimal and maximal value that occurs in variable buffer, but doing so might cause the value range is enlarged for values that some seldom occurs, then, cause precision loss of decimal part. According to this distribution, we calculated the FP_POS as the following: (new_setting.yaml) variables: ... Convolution_3_Output: FIXED16_12 <-- Change data type according to value distribution conv2/conv2/bn/beta: FLOAT32 conv2/conv2/bn/gamma: FLOAT32 conv2/conv2/bn/mean: FLOAT32 conv2/conv2/bn/var: FLOAT32 BatchNormalization_3_Output: FIXED16_12 ReLU_3_Output: FIXED16_12 conv2/conv3/conv/W: FIXED8 Convolution_4_Output: FIXED16_12 conv2/conv3/bn/beta: FLOAT32 conv2/conv3/bn/gamma: FLOAT32 conv2/conv3/bn/mean: FLOAT32 conv2/conv3/bn/var: FLOAT32 BatchNormalization_4_Output: FIXED8_4 Add2_Output: FIXED8_4 ReLU_4_Output: FIXED8_4 ...  After modifying new_setting.yaml, the following command is used to convert from *.nnp to *.nnb: $> nnabla_cli convert -b 1 -d nnb_3 models/bin_class_float.nnp models/bin_class_f_fq.nnb -s setting/setting_f_fq.yaml


-d nnb_3 is necessary to enable memory saving policy. By tuning, we got the the-state-of-art result:

nnp accuracy: 0.81, nnb accuracy: 0.79


## Summary¶

After the quantization of variable buffer, the footprint is reduced noticeably from 1.2M to 495.2K, and accuracy is almost the same, as shown in the following table:

model size footprint accuracy
float-point model 449.5 KB 1.2 M 0.81
parameter-quantized 126.0 KB 1.0 M 0.81
parameter-and-buffer-quantized 126.0 KB 495.2 KB 0.79

Comparing these two ways, the second way shows better result on current nnabla’s implementation. The reason is currently nnabla does not support SIGN parameter mode, so binarization weights lose accuracy without gaining the best benefit of saving memory. Future work is needed such that float-point weight parameters are removed from *.nnb for binary series functions.

• Notice: Currently, all experiments focus on classification problem, and softmax operation has higher tolerance to quantization error. The testing against regression problem has not yet been performed, so the accuracy loss is still unconfirmed.