Designing and evaluating language models for human interaction

Placeholder Show Content


Despite the ubiquity of language models (LMs) in real-world applications (e.g., web search, text autocomplete, and content generation), most LMs are not optimized for, nor are they evaluated on, real-world usage where human users interact with LMs. To address this gap, this dissertation focuses on designing and evaluating LMs for human interaction. We first start by focusing on one specific need that writers encounter in the revision process: coming up with content given surrounding context. To support this need, we propose a training method to enable any pre-trained LMs to accomplish the task of filling in the blanks, helping to better facilitate human-LM interaction. Second, we build a platform, CoAuthor, to capture human-LM interaction as interaction traces. With CoAuthor, we demonstrate how collecting a large interaction dataset and analyzing the traces provide unique insights into LM capabilities regarding language, ideation, and collaboration in human-LM interaction. Lastly, we propose a new evaluation framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and metrics for human-LM interaction tasks beyond writing. Finally, we discuss open challenges and future directions in this field.


Type of resource text
Form electronic resource; remote; computer; online resource
Extent 1 online resource.
Place California
Place [Stanford, California]
Publisher [Stanford University]
Copyright date 2023; ©2023
Publication date 2023; 2023
Issuance monographic
Language English


Author Lee, Mina
Degree supervisor Liang, Percy
Thesis advisor Liang, Percy
Thesis advisor Bernstein, Michael S, 1984-
Thesis advisor Yang, Qian
Degree committee member Bernstein, Michael S, 1984-
Degree committee member Yang, Qian
Associated with Stanford University, School of Engineering
Associated with Stanford University, Computer Science Department


Genre Theses
Genre Text

Bibliographic information

Statement of responsibility Mina Lee.
Note Submitted to the Computer Science Department.
Thesis Thesis Ph.D. Stanford University 2023.

Access conditions

© 2023 by Mina Lee
This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).

Also listed in

Loading usage metrics...