Uncertainty and information for machine learning powered decision-making
Abstract/Contents
- Abstract
- Machine learning (ML) is undergoing a paradigm shift --- ML models are increasingly being provided as a service to automate a variety of downstream decisions rather than being trained and deployed end-to-end on specific tasks by ML experts. Examples include image or text classification APIs provided by large tech companies and used by a wide range of third-party app developers, and various forecasts (e.g., weather, COVID, traffic, etc.) provided through websites to millions of users to help with their planning. Although this new paradigm democratizes ML by making it more widely accessible, it raises concerns about trustworthiness (users have no visibility on how they were trained and their failure modes) and performance (prediction models are no longer tailored for a specific downstream task). This dissertation tackles these issues by: Contribution 1. proposing a new approach to convey confidence to downstream decision makers who will use the predictions for (high stakes) decisions by accurate uncertainty quantification. Accurate uncertainty quantification can be achieved by predicting the true probability of the outcome of interest (such as the true probability of a patient's illness given the symptoms). While outputting these probabilities exactly is impossible in most cases, I show that it is surprisingly possible to learn probabilities that are indistinguishable from the true probabilities for large classes of decision making tasks. Indistinguishability ensures reliability for decision makers, because they should not be able to tell the difference between the predicted probability and the true probability in their decision tasks. As an application, I develop predictions models in domains such as medical diagnosis, flight delay prediction, and poverty prediction. I show that by using my methods, decision makers can confidently make decisions leading good outcomes. % where the Contribution 2. developing a new theory of information to rigorously reason about and optimize the ``usefulness'' of ML predictions in a wide range of decision tasks. Shannon information theory has wide applications in machine learning, but suffers several limitations when applied to complex learning and decision tasks. For example, consider a dataset of securely encrypted messages intercepted from an opponent. According to information theory, these encrypted messages have high mutual information with the opponent's plans, yet any computationally bounded decision maker cannot utilize this information. To address these limitations, I put forward a new framework called ``utilitarian information theory'' that generalizes Shannon's entropy, information and divergence to account for how information will be used by a decision maker with limited knowledge or modeling power. As an application, I apply the new information to Bayesian optimization problems and show an order of magnitude improvements in sample efficiency compared to current methods that use Shannon information.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2022; ©2022 |
| Publication date | 2022; 2022 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Zhao, Shengjia |
|---|---|
| Degree supervisor | Ermon, Stefano |
| Thesis advisor | Ermon, Stefano |
| Thesis advisor | Finn, Chelsea |
| Thesis advisor | Ma, Tengyu |
| Degree committee member | Finn, Chelsea |
| Degree committee member | Ma, Tengyu |
| Associated with | Stanford University, Computer Science Department |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Shengjia Zhao. |
|---|---|
| Note | Submitted to the Computer Science Department. |
| Thesis | Thesis Ph.D. Stanford University 2022. |
| Location | https://purl.stanford.edu/bm546bb8038 |
Access conditions
- Copyright
- © 2022 by Shengjia Zhao
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...