Analysis of wide neural networks : insights from linearized models
Abstract/Contents
- Abstract
- In this thesis, we study two linearized approximations to neural networks: Random Feature model (RF) and Neural Tangent Model (NT). For two-layer neural networks (NN), we prove the existence of a performance gap between NN and these approximations under two simple data distribution models. We show that this gap in performance stems from the fact that, unlike NT and RF, NN learns efficient representations of the target function. In the second part of this thesis, we study the approximation power of NT and RF in high dimensions. We show that when the features are uniformly distributed on the sphere, these models can only fit surprisingly simple polynomial functions to the data. In the end, we examine the generalization behavior of these approximations in high dimensions. We show that in order to learn anything beyond low-degree polynomial functions, these approximate networks require extremely large number of training data points. Our results suggest that, while these approximations might perform well in various applications, they do not sufficiently capture the full power of neural networks
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2020; ©2020 |
| Publication date | 2020; 2020 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Ghorbani, Behrooz |
|---|---|
| Degree supervisor | Donoho, David Leigh |
| Thesis advisor | Donoho, David Leigh |
| Thesis advisor | Johnstone, Iain |
| Thesis advisor | Montanari, Andrea |
| Thesis advisor | Weissman, Tsachy |
| Degree committee member | Johnstone, Iain |
| Degree committee member | Montanari, Andrea |
| Degree committee member | Weissman, Tsachy |
| Associated with | Stanford University, Department of Electrical Engineering. |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Behrooz Ghorbani |
|---|---|
| Note | Submitted to the Department of Electrical Engineering |
| Thesis | Thesis Ph.D. Stanford University 2020 |
| Location | electronic resource |
Access conditions
- Copyright
- © 2020 by Behrooz Ghorbani
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...