Algorithmic techniques for neural network training on memory-constrained hardware
Abstract/Contents
- Abstract
- Deep learning has enabled accurate inference for many real world problems, making it an attractive tool for embedded devices at the edge. Unfortunately, these devices are often limited by tight energy and memory budgets, prompting past research to optimize for neural network inference efficiency and size. In our work, we take some of the first steps towards enabling gradient descent-based neural network training on these inference-optimized devices. For example, power-of-two quantization is commonly used to reduce weight size, but training on quantized weights is known to be difficult. We discuss some potential causes of this difficulty and propose improved quantization techniques to enable training. More recently, emerging memory promises to obviate off-chip data movement, but is expensive to update. We present an algorithmic compression technique to enable training without incurring overwhelming write costs of emerging memory technology. When applying these techniques to perform transfer learning on CHIMERA, a chip with large scale resistive RAM (RRAM) macros, we observe a 101x reduction in the number of weight update steps and a 340x reduction in energy-delay product compared to a stochastic gradient descent baseline that achieves the same accuracy.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2021; ©2021 |
| Publication date | 2021; 2021 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Gural, Albert |
|---|---|
| Degree supervisor | Murmann, Boris |
| Thesis advisor | Murmann, Boris |
| Thesis advisor | Mitra, Subhasish |
| Thesis advisor | Pilanci, Mert |
| Degree committee member | Mitra, Subhasish |
| Degree committee member | Pilanci, Mert |
| Associated with | Stanford University, Department of Electrical Engineering |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Albert Gural. |
|---|---|
| Note | Submitted to the Department of Electrical Engineering. |
| Thesis | Thesis Ph.D. Stanford University 2021. |
| Location | https://purl.stanford.edu/gq639xh0614 |
Access conditions
- Copyright
- © 2021 by Albert Gural
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...