Designing and evaluating language models for human interaction
Abstract/Contents
- Abstract
- Despite the ubiquity of language models (LMs) in real-world applications (e.g., web search, text autocomplete, and content generation), most LMs are not optimized for, nor are they evaluated on, real-world usage where human users interact with LMs. To address this gap, this dissertation focuses on designing and evaluating LMs for human interaction. We first start by focusing on one specific need that writers encounter in the revision process: coming up with content given surrounding context. To support this need, we propose a training method to enable any pre-trained LMs to accomplish the task of filling in the blanks, helping to better facilitate human-LM interaction. Second, we build a platform, CoAuthor, to capture human-LM interaction as interaction traces. With CoAuthor, we demonstrate how collecting a large interaction dataset and analyzing the traces provide unique insights into LM capabilities regarding language, ideation, and collaboration in human-LM interaction. Lastly, we propose a new evaluation framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and metrics for human-LM interaction tasks beyond writing. Finally, we discuss open challenges and future directions in this field.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2023; ©2023 |
| Publication date | 2023; 2023 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Lee, Mina |
|---|---|
| Degree supervisor | Liang, Percy |
| Thesis advisor | Liang, Percy |
| Thesis advisor | Bernstein, Michael S, 1984- |
| Thesis advisor | Yang, Qian |
| Degree committee member | Bernstein, Michael S, 1984- |
| Degree committee member | Yang, Qian |
| Associated with | Stanford University, School of Engineering |
| Associated with | Stanford University, Computer Science Department |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Mina Lee. |
|---|---|
| Note | Submitted to the Computer Science Department. |
| Thesis | Thesis Ph.D. Stanford University 2023. |
| Location | https://purl.stanford.edu/rj262yf1528 |
Access conditions
- Copyright
- © 2023 by Mina Lee
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...