Mark Crowley | AAMAS talk, PhD Dissertation Award, ChemGymRL Updates

There are a few news items to share from the lab this month.

AAMAS 2023 Conference

Next week Sriram will be in London for the AAMAS 2023 conference where he’s presenting his paper “Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning” (Ganapathi Subramanian et al., 2023) which looks at the problem of simultaneously learning from multiple independent advisors in a multi-agent reinforcement learning setting. The approach leverages a two-level Q-learning architecture extended to multi-agent settings and includes algorithms for incorporating a set of advisors by both evaluating the advisors at each state and subsequently using the advisors to guide action selection. There are also great theoretical convergence and sample complexity guarantees as well as empirical validation through simulation experiments.

PhD Thesis Award for Sriram!

Every year the Canadian Society for the Computational Studies of Intelligence (or Société Canadienne pour l’étude de l’intelligence par ordinateur)(CAIAC) calls out for nominations for the best PhD dissertation in Canada on the topic of Artificial Intelligence. Supervisors and Department Chairs nominate graduates from the previous year, then the thesis and support letters are evaluated by an independent committee of Professors from across Canada. This year, our lab’s most recent PhD graduate, Sriram Ganapathi Subramanian, has been chosen for the best dissertation (Subramanian, 2022). It’s a huge honour and we’re all very proud of him, congratulations Sriram!

Sriram will give an acceptance talk on his thesis research at the Canadian AI 2023 conference this year in Montreal at McGill.

Sriram is currently a postdoc position at the Vector Institute working with Prof. Sheila Mcllraith at UofT and Prof. Pascal Poupart at UWaterloo.

ChemGymRL Arxiv Paper is Up

Related to our ongoing NRC funded project on Digital Chemistry and Material Design we have recently posted a paper preprint on Arxiv titled “ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistry” (missing reference). The paper is a detailed introduction to the open-source simulation framework and experimental results on standard RL algorithms.

References:

Multi-Advisor-MARL

Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning

Sriram Ganapathi Subramanian, Matthew Taylor, Kate Larson, and Mark Crowley

In Proceedings of the 22nd International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), London, United Kingdom. Sep, 2023.

Abs PDF

Multi-agent reinforcement learning typically suffers from the problem of sample inefficiency, where learning suitable policies involves the use of many data samples. Learning from external demonstrators is a possible solution that mitigates this problem. However, most prior approaches in this area assume the presence of a single demonstrator. Leveraging multiple knowledge sources (i.e., advisors) with expertise in distinct aspects of the environment could substantially speed up learning in complex environments. This paper considers the problem of simultaneously learning from multiple independent advisors in multi-agent reinforcement learning. The approach leverages a two-level Q-learning architecture, and extends this framework from single-agent to multi-agent settings. We provide principled algorithms that incorporate a set of advisors by both evaluating the advisors at each state and subsequently using the advisors to guide action selection. Also, we provide theoretical convergence and sample complexity guarantees. Experimentally, we validate our approach in three different test-beds and show that our algorithms give better performances than baselines, can effectively integrate the combined expertise of different advisors, and learn to ignore bad advice.

Phd Thesis

Multi-Agent Reinforcement Learning in Large Complex Environments

sriramganapathisubramanian.

UWSpace, Jun, 2022.

Abs PDF URL

Multi-agent reinforcement learning (MARL) has seen much success in the past decade. However, these methods are yet to find wide application in large-scale real world problems due to two important reasons. First, MARL algorithms have poor sample efficiency, where many data samples need to be obtained through interactions with the environment to learn meaningful policies, even in small environments. Second, MARL algorithms are not scalable to environments with many agents since, typically, these algorithms are exponential in the number of agents in the environment. This dissertation aims to address both of these challenges with the goal of making MARL applicable to a variety of real world environments. Towards improving sample efficiency, an important observation is that many real world environments already, in practice, deploy sub-optimal or heuristic approaches for generating policies. A useful possibility that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this dissertation, we provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings. To this end, we propose a general model for learning from external advisors in MARL and show that desirable theoretical properties such as convergence to a unique solution concept, and reasonable finite sample complexity bounds exist, under a set of common assumptions. Furthermore, extensive experiments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors. Towards scaling MARL, we explore the use of mean field theory. Mean field theory provides an effective way of scaling multi-agent reinforcement learning algorithms to environments with many agents, where other agents can be abstracted by a virtual mean agent. Prior work has used mean field theory in MARL, however, they suffer from several stringent assumptions such as requiring fully homogeneous agents, full observability of the environment, and centralized learning settings, that prevent their wide application in practical environments. In this dissertation, we extend mean field methods to environments having heterogeneous agents, and partially observable settings. Further, we extend mean field methods to include decentralized approaches. We provide novel mean field based MARL algorithms that outperform previous methods on a set of large games with many agents. Theoretically, we provide bounds on the information loss experienced as a result of using the mean field and further provide fixed point guarantees for Q-learning-based algorithms in each of these environments. Subsequently, we combine our work in mean field learning and learning from advisors to show that we can achieve powerful MARL algorithms that are more suitable for real world environments as compared to prior approaches. This method uses the recently introduced attention mechanism to perform per-agent modelling of others in the locality, in addition to using the mean field for global responses. Notably, in this dissertation, we show applications in several real world multi-agent environments such as the Ising model, the ride-pool matching problem, and the massively multi-player online (MMO) game setting (which is currently a multi-billion dollar market).