Reliable machine learning in the wild
Abstract/Contents
- Abstract
- Machine learning systems today are widely deployed but unreliable: they can work well in some contexts but fail in others, with severe consequences. For instance, facial recognition models can do well on average but fail on particular demographic subpopulations; and medical models can do well in the hospitals they are trained on, but fail in different hospitals. This dissertation studies three aspects of building reliable machine learning systems that are robust to these failures. First, we introduce a tool for inspecting models and understanding how and why they fail. Since training data is the key ingredient in any machine learning model, we build on the classical notion of influence functions to quantify the influence of each training example on a model's predictions, allowing us to trace errors back to the training data and better understand how to correct them. Second, we develop a distributionally robust optimization (DRO) method for directly training models to perform well across different subpopulations. These subpopulation shifts arise in many important applications, from the facial recognition example above, to medical applications (with different patient subpopulations) and to remote sensing applications (with different geographic subregions). Third, we discuss benchmarks for distribution shifts more generally; this includes subpopulation shifts, as well as shifts across different environments, such as different hospitals. We introduce WILDS---a benchmark of in-the-wild distribution shifts spanning applications such as pathology, conservation, remote sensing, and drug discovery---and show how state-of-the-art methods, which perform well on synthetic shifts, fail on these real-world shifts. Our results underscore the urgent need to ground method development and evaluation in the application settings that arise in the wild.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2022; ©2022 |
| Publication date | 2022; 2022 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Koh, Pang Wei |
|---|---|
| Degree supervisor | Liang, Percy |
| Thesis advisor | Liang, Percy |
| Thesis advisor | Leskovec, Jurij |
| Thesis advisor | Ma, Tengyu |
| Degree committee member | Leskovec, Jurij |
| Degree committee member | Ma, Tengyu |
| Associated with | Stanford University, Computer Science Department |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Pang Wei Koh. |
|---|---|
| Note | Submitted to the Computer Science Department. |
| Thesis | Thesis Ph.D. Stanford University 2022. |
| Location | https://purl.stanford.edu/hm880nk8837 |
Access conditions
- Copyright
- © 2022 by Pang Wei Koh
Also listed in
Loading usage metrics...