Structured cooperative multi-agent coordination
Abstract/Contents
- Abstract
- Reinforcement learning (RL) involves solving decision-making tasks by learning from the experience of agents without prior knowledge of the dynamics of the problems. The advances in deep neural networks and reinforcement learning have seen success in several application domains including game playing, autonomous driving, and finance. In many real-world problems, multiple entities make decisions and interact with each other within a shared environment. It is hard to model the decision-maker as a single agent in such problems. Instead, it is more natural and reasonable to model each participating entity as an agent and frame the problem as multi-agent reinforcement learning (MARL). MARL is more challenging to solve than single-agent RL due to the complexity of inter- agent interaction and limited information for each agent. The major challenges in MARL are: 1. scalability, the complexity of MARL problems grows with the number of agents; 2. partial observability, each agent lacks sufficient information to coordinate optimally due to limited local observation; 3. non-stationarity, each agent sees other agents as a part of the environment, whose evolving behavioral patterns create non-stationary environment dynamics, breaking the Markov property assumption of RL algorithms. This dissertation explores approaches to address the aforementioned challenges. We exploit the structural nature of inter-agent interaction to achieve effective coordination between agents. We use techniques from reinforcement learning and deep learning to develop efficient MARL algorithms. In the first contribution, we demonstrate solving a multi-agent problem from a single agent's perspective using utility decomposition and fusion in an approximate and decentralized way that relies on the spatial structure between agents. On top of it, we use deep Q-learning to optimize the approximate solution. In the second contribution, we use a graph structure to model the interaction between agents and use the attention mechanism to dynamically learn the graph representation. We further conduct communication of the graph to achieve multi-agent coordination. Finally, in the last contribution, we explore and analyze learning emergent discrete message communication and its interpretability. We demonstrate an explicit broadcast-based communication model and study a human-agent interaction method using discrete message communication. In summary, in this dissertation, we try to find a balanced mid-ground between the optimality and the efficiency in solving MARL problems.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2022; ©2022 |
| Publication date | 2022; 2022 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Li, Sheng, active 2022 | |
|---|---|---|
| Degree supervisor | Kochenderfer, Mykel J, 1980- | |
| Thesis advisor | Kochenderfer, Mykel J, 1980- | |
| Thesis advisor | Ermon, Stefano | |
| Thesis advisor | Schwager, Mac | |
| Degree committee member | Ermon, Stefano | |
| Degree committee member | Schwager, Mac | |
| Associated with | Stanford University, Department of Aeronautics and Astronautics |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Sheng Li. |
|---|---|
| Note | Submitted to the Department of Aeronautics and Astronautics. |
| Thesis | Thesis Ph.D. Stanford University 2022. |
| Location | https://purl.stanford.edu/cr844qy0850 |
Access conditions
- Copyright
- © 2022 by Sheng Li
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...