Learning Multimodal Rewards from Rankings

<a href="https://embed.stanford.edu/iframe/?url=https%3A%2F%2Fpurl.stanford.edu%2Fpt103ty8267" class="su-underline">Show Content</a>

Abstract/Contents

Abstract: Learning from human feedback has been shown to be a useful approach in acquiring robot reward functions. However, expert feedback is often assumed to be drawn from an underlying unimodal reward function. This assumption does not always hold, including in settings where multiple experts provide data or when a single expert provides data for different tasks. We thus have to go beyond learning a unimodal reward and focus on learning a multimodal reward function. We formulate the multimodal reward learning as a mixture learning problem and develop a novel ranking-based learning approach, in which the experts are only required to rank a given set of trajectories. Furthermore, as access to interaction data is often expensive in robotics, we develop an active querying approach to accelerate the learning process. We conduct experiments and user studies using a multi-task variant of OpenAI's LunarLander and a real Fetch robot, where we collect data from multiple users with different preferences. The results suggest that our approach can efficiently learn multimodal reward functions, and improve data-efficiency over benchmark methods that we adapt to our learning problem.

Author	Myers, Vivek
Author	Bıyık, Erdem
Author	Anari, Nima
Author	Sadigh, Dorsa
Degree granting institution	Stanford University
Department	Department of Computer Science

Subject	HRI, reward learning, multi-modality, rankings, active learning
Genre	Text
Genre	Thesis

Use and reproduction: User agrees that, where applicable, content will not be used to identify or to otherwise infringe the privacy or confidentiality rights of individuals. Content distributed via the Stanford Digital Repository may be subject to additional license and use restrictions applied by the depositor.
License: This work is licensed under a Creative Commons Attribution Non Commercial 4.0 International license (CC BY-NC).

Preferred citation: Myers, V., Bıyık, E., Anari, N., and Sadigh, D. (2022). Learning Multimodal Rewards from Rankings. Stanford Digital Repository. Available at https://purl.stanford.edu/pt103ty8267

Undergraduate Theses, School of Engineering

Loading usage metrics...