Human-AI interaction under societal disagreement
Abstract/Contents
- Abstract
- Whose voices - whose labels - should a machine learning algorithm learn to emulate? For AI tasks ranging from online comment toxicity detection to poster design to medical treatment, different groups in society may have irreconcilable disagreements about what constitutes ground truth. Today's supervised machine learning pipeline typically resolves these disagreements implicitly by majority vote over annotators' opinions. This majoritarian procedure abstracts individual people out of the pipeline and collapses their labels into an aggregate pseudo-human, ignoring minority groups' labels. In this dissertation, I will present Jury Learning: an interactive AI architecture that enables developers to explicitly reason over whose voice a model ought to emulate through the metaphor of a jury. Through my exploratory interface, practitioners can declaratively define which people or groups, in what proportion, determine the classifier's prediction. To evaluate models under societal disagreement, I will also present The Disagreement Deconvolution: a metric transformation showing how, in abstracting away the individual people that models impact, current metrics dramatically overstate the performance of many user-facing tasks. These components become building blocks of a new pipeline for encoding our goals and values in human-AI systems, which strives to bridge principles of HCI with the realities of machine learning.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2023; ©2023 |
| Publication date | 2023; 2023 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Gordon, Mitchell Louis |
|---|---|
| Degree supervisor | Bernstein, Michael |
| Degree supervisor | Landay, James |
| Thesis advisor | Bernstein, Michael |
| Thesis advisor | Landay, James |
| Thesis advisor | Hashimoto, Tatsunori |
| Degree committee member | Hashimoto, Tatsunori |
| Associated with | Stanford University, School of Engineering |
| Associated with | Stanford University, Computer Science Department |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Mitchell L. Gordon. |
|---|---|
| Note | Submitted to the Computer Science Department. |
| Thesis | Thesis Ph.D. Stanford University 2023. |
| Location | https://purl.stanford.edu/xf168rn2553 |
Access conditions
- Copyright
- © 2023 by Mitchell Louis Gordon
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...