Understanding and optimizing the statistical performance of machine learning models under memory budgets
Abstract/Contents
- Abstract
- Machine learning models are trending larger to attain state-of-the-art performance across various application domains. These state-of-the-art models can consume large amounts of memory for training and inference. However, memory is often an expensive and limited hardware resource. Thus, designing machine learning models which still attain strong performance with less memory is critical for settings with hardware constraints. In many machine learning models, a major memory-consuming component is feature representations, such as word embeddings in natural language processing tasks and kernel approximation features for kernel methods. Because of this, to attain models with strong performance under a memory budget, it is important to study the trade-off between the memory footprint of feature representations and the statistical performance (e.g. accuracy, stability) of machine learning models trained with these feature representations. In this thesis, we focus on understanding what determines the statistical performance of models trained on different feature representations and then using this understanding to optimize the statistical performance under memory budgets. We show that we can use fixed design regression---a theoretical framework for analyzing the statistical performance of machine learning models---as a unified tool to understand and optimize the performance of models trained with different feature representations under memory budgets. This tool helps us understand what properties of feature representations are crucial for strong statistical performance. Although our analysis is specific to this fixed design regression setting, we show that it yields insights for empirically understanding and optimizing the statistical performance of a wide range of models trained on these representations. In particular, we use this unified tool to study 1) the accuracy of natural language processing models attained by word embeddings, 2) the instability of natural language processing models trained on different word embeddings where the model instability is defined as the percentage of predictions which disagree when a model is trained on different feature representations respectively and 3) the accuracy of kernel models trained with kernel approximation features. For each of the above three settings, we leverage fixed design regression as the unified tool to develop a new way to measure the quality of feature representations, and use the proposed quality measure to understand and optimize the statistical performance under memory budgets in the following consistent way. Theoretically, we propose a new way to measure the quality of feature representations and use the new quality measure to bound the statistical performance in the shared context of fixed design regression. Empirically, we show that the proposed quality measure attains stronger correlation with the statistical performance than existing measures across different models and memory budgets. To demonstrate the utility of our proposed measure in optimizing statistical performance, we show that the measure can guide the design or selection of feature representations to achieve improved statistical performance under memory budgets across numerous benchmark tasks: specifically, we show that our proposed quality measures can 1) be a criterion to select between compressed word embeddings for better model accuracy with up to 2X lower selection error than existing measures; 2) guide the selection of word embeddings to attain competitive or better model stability across memory budgets than existing criteria and 3) guide the design of new kernel approximation features to achieve matching model accuracy with up to 10X less memory than existing methods.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2019; ©2019 |
| Publication date | 2019; 2020 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Zhang, Jian |
|---|---|
| Degree supervisor | Ré, Christopher |
| Thesis advisor | Ré, Christopher |
| Thesis advisor | Liang, Percy |
| Thesis advisor | Olukotun, Oyekunle Ayinde |
| Degree committee member | Liang, Percy |
| Degree committee member | Olukotun, Oyekunle Ayinde |
| Associated with | Stanford University, Computer Science Department. |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Jian Zhang. |
|---|---|
| Note | Submitted to the Computer Science Department. |
| Thesis | Thesis Ph.D. Stanford University 2020. |
| Location | electronic resource |
Access conditions
- Copyright
- © 2019 by Jian Zhang
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...