Long-horizon prediction for human-robot collaboration

Placeholder Show Content


Robots are mainly seen as tools that offer passive assistance to humans or operate independently in close proximity to them. However, we are still far from witnessing robots that can collaborate with humans on tasks wherein both agents' actions and dynamics are interdependent and achieve synergistic results. As an example, and one that serves as the central motivation of my work, robots carrying a table towards a goal location and avoiding obstacles must be able to predict their partners' actions and mutually adapt to changing strategies during collaboration. Key challenges have been accounting for diverse, multimodal human behaviors over both short and long horizons, and producing joint actions that are temporally consistent. To address these issues, I leverage generative modeling for accurate sequence prediction of human-like motion and behaviors. I focus on two approaches to model interaction: first, by predicting team subgoals and executing cooperative actions; and second, by predicting partner actions and executing collaborative actions that would account for the partner's efforts. In the first part of this thesis, I develop a Variational Recurrent Neural Network-based planner trained with human-human demonstrations that learns to sample sequences of future waypoints autoregressively to achieve successful synergy with a human-in-the-loop on the table-carrying task. Next, I show that mixture density models can capture multimodality in human-human demonstrations, but fail to predict action sequences accurately for the cooperative carrying task. Finally, I show that a co-policy developed with a Transformer-based diffusion model conditioning on past human actions can not only plan action sequences with real humans-in-the-loop to achieve high success rates, but also display compelling collaborative behaviors in novel, out-of-training-distribution settings.


Type of resource text
Form electronic resource; remote; computer; online resource
Extent 1 online resource.
Place California
Place [Stanford, California]
Publisher [Stanford University]
Copyright date 2023; ©2023
Publication date 2023; 2023
Issuance monographic
Language English


Author Ng, Eley
Degree supervisor Kennedy, Monroe
Thesis advisor Kennedy, Monroe
Thesis advisor Sadigh, Dorsa
Thesis advisor Schwager, Mac
Degree committee member Sadigh, Dorsa
Degree committee member Schwager, Mac
Associated with Stanford University, School of Engineering
Associated with Stanford University, Department of Mechanical Engineering


Genre Theses
Genre Text

Bibliographic information

Statement of responsibility Eley Ng.
Note Submitted to the Department of Mechanical Engineering.
Thesis Thesis Ph.D. Stanford University 2023.
Location https://purl.stanford.edu/nm281rs5069

Access conditions

© 2023 by Eley Ng
This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).

Also listed in

Loading usage metrics...