Theoretical insights on generalization in supervised and self-supervised deep learning
Abstract/Contents
- Abstract
- Deep learning has been extremely impactful empirically, but theoretical understanding lags behind. Neural networks are much more complex than more classical machine learning models in terms of both architecture and training algorithms, so traditional theoretical intuitions may not apply. This thesis seeks to gain a better theoretical understanding of generalization in deep learning. First, we study factors influencing generalization in supervised settings where all data are labeled, obtaining improved generalization bounds for neural networks by considering additional data-dependent properties of the model. Second, we study generalization in a setting with unlabeled data. In the vision setting, we present a theoretical framework for understanding recent self-training and self-supervised contrastive learning algorithms by leveraging a realistic assumption on the data. In the NLP setting, we analyze why pretraining can help with downstream tasks in the setting where data is generated according to an underlying latent variable model.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2022; ©2022 |
| Publication date | 2022; 2022 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Wei, Colin |
|---|---|
| Degree supervisor | Ma, Tengyu |
| Thesis advisor | Ma, Tengyu |
| Thesis advisor | Hashimoto, Tatsunori |
| Thesis advisor | Liang, Percy |
| Degree committee member | Hashimoto, Tatsunori |
| Degree committee member | Liang, Percy |
| Associated with | Stanford University, Computer Science Department |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Colin Wei. |
|---|---|
| Note | Submitted to the Computer Science Department. |
| Thesis | Thesis Ph.D. Stanford University 2022. |
| Location | https://purl.stanford.edu/yw554bd7619 |
Access conditions
- Copyright
- © 2022 by Colin Wei
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...