Analysis of quantization and normalization effects in deep neural networks
Abstract/Contents
- Abstract
- There is great interest in the machine learning community to reduce Deep Neural Network (DNN) model sizes. Decreasing the memory and compute requirements expands the range of resource-constrained mobile applications where DNNs can be deployed. By far, the most popular method of compressing model size is uniform quantization. In this work, we illustrate how quantization performance was fortuitously advanced by Batch Normalization (BatchNorm), a technique originally developed to aid training convergence. This improvement is due to BatchNorm's reshaping of the network's activation distributions. Additionally, due to the limited consensus on why BatchNorm is effective, this work uses concepts from the traditional adaptive filter domain to provide insights into its dynamics and inner workings. First, we show that the convolution weight updates have natural modes whose stability and convergence speed are tied to the eigenvalues of the input autocorrelation matrices. Furthermore, our experiments demonstrate that the speed and stability benefits are distinct effects. At low learning rates, it is BatchNorm's amplification of the smallest eigenvalues that improves convergence speed. In contrast, at high learning rates, it is BatchNorm's suppression of the largest eigenvalues that ensures stability. Next, we prove that in the first training step, when normalization is needed most, BatchNorm satisfies the same optimization as Normalized Least Mean Square (NLMS), while it continues to approximate this condition in subsequent steps. The analyses provided lay the groundwork for gaining further insight into the operation of modern neural network structures using adaptive filter theory. Finally, we highlight contributions made to a real-world application of DNNs in the Smart Hospital space.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2021; ©2021 |
| Publication date | 2021; 2021 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Chai, Elaina Teresa |
|---|---|
| Degree supervisor | Murmann, Boris |
| Thesis advisor | Murmann, Boris |
| Thesis advisor | Mujica, Fernando |
| Thesis advisor | Pilanci, Mert |
| Degree committee member | Mujica, Fernando |
| Degree committee member | Pilanci, Mert |
| Associated with | Stanford University, Department of Electrical Engineering |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Elaina Teresa Chai. |
|---|---|
| Note | Submitted to the Department of Electrical Engineering. |
| Thesis | Thesis Ph.D. Stanford University 2021. |
| Location | https://purl.stanford.edu/gb995rt5179 |
Access conditions
- Copyright
- © 2021 by Elaina Teresa Chai
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...