Approximate memory for energy-efficient machine learning
Abstract/Contents
- Abstract
- The success of convolutional neural networks (ConvNets) has led to impressive performance in a wide range of cloud-centric applications including image classification, speech recognition, and text analysis. To reduce latency and the high energy cost of communication with the cloud, our work focuses on the development of highly energy-efficient, edge computing ConvNet ASICs, with the additional benefit that local computation provides better privacy guarantees per user device. Unfortunately, the performance of ConvNets is often directly linked to the massive number of parameters needed to encode the network and the availability of representative datasets for training. Deployment of ConvNets in resource-constrained Internet of Everything (IoE) systems remains a challenge due to the high memory energy consumption caused by network storage requirements and substantial data movement. Recently, there has been an emergence of interest in the field of approximate computing, which explores trade-offs between the performance of an algorithm and hardware energy consumption with reduced precision. ConvNets are one class of algorithms which have been shown to be inherently error resilient, motivating extensive studies on the effect of noise in ConvNets with the goal of decreasing compute energy through methods of approximate computing. Similarly, we can leverage the error resilience of ConvNets by accepting bit errors at reduced voltages for memory energy savings (approximate memory), but few implementations utilize this due to the limited understanding of how bit errors affect the classification performance of ConvNets. Motivated by the need to reduce memory energy consumption in hardware ConvNets, and the current lack of understanding of ConvNet tolerance to bit errors, this thesis presents the first siliconvalidated study on the efficacy of memory voltage scaling in SRAMs on the MNIST and CIFAR-10 datasets. Using a hardware-software co-design approach, we demonstrate that supply voltage in SRAMs for MNIST ConvNets can be scaled well below the Vmin and furthermore, with re-training to account for these SRAM bit errors, we demonstrate additional improvements in classification accuracy and energy savings.We further show that a uniform bit error model is sufficient to achieve classification accuracies very close to training with the physical SRAM in the loop. Using this framework, we extend these methods to a multi-layer binarized ConvNet performing a more complex image classification task (CIFAR-10), demonstrating that significant errors can accumulate in the network with little to no degradation in classification accuracy. Furthermore, we show that additional energy savings are possible by leveraging the different bit error tolerances between weights and activations, and over the different layers of the network. Finally, we compare our required bit error tolerances between our MNIST and CIFAR-10 implementations, demonstrating that the CIFAR-10 network is less error resilient but still tolerates bit error rates significantly higher than conventional memory applications. Our findings and proposed methods serve as a framework which can be applied to the design of custom memory (e.g. hybrid 8T/6T, larger bitcells) and emerging memory technologies (e.g. RRAM, PCM) for ConvNet applications.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2018; ©2018 |
| Publication date | 2018; 2018 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Yang, Lita |
|---|---|
| Degree supervisor | Murmann, Boris |
| Thesis advisor | Murmann, Boris |
| Thesis advisor | Arbabian, Amin |
| Thesis advisor | Wong, S. Simon |
| Degree committee member | Arbabian, Amin |
| Degree committee member | Wong, S. Simon |
| Associated with | Stanford University, Department of Electrical Engineering. |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Lita Yang. |
|---|---|
| Note | Submitted to the Department of Electrical Engineering. |
| Thesis | Thesis Ph.D. Stanford University 2018. |
| Location | electronic resource |
Access conditions
- Copyright
- © 2018 by Lita Yang
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...