Mark Crowley | Lab News - PhD Defense, papers, jobs

Members of the UWECEML lab have had a great couple months, with some nice a publications and lots of career news.

Publications

A New Approach to Scaling Decision Making to Many Agents

Ganapathi Subramanian, S., Larson, K., Taylor, M., & Crowley, M. (2022). Multi-Agent Advisor Q-Learning. Journal of Artificial Intelligence Research (JAIR), 74, 1–74. https://doi.org/https://doi.org/10.1613/jair.1.13445

In situations where the number of agents is very large, we need to make some assumptions about structure to make RL feasible. Mean Field Theory is one such approach where each agent considers the impact to its interactions via another “cloud” agent which is actually an aggregation of all other agents, via a mean field calculation. This work was a core topic in the final stages of my PhD student Sriram Ganapathi Subramanian into Multi-Agent Reinforcement Learning (MARL).

In this paper, we build on a previous work in the lab to take advantage of the fact that many real-world environments already deploy sub-optimal or heuristic approaches for generating policies and show how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this paper, we provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings.

Empirical Study of Reinforcement Learning Algorithms in Multi-Agent Settings

Lee, K. M., Ganapathi Subramanian, S., & Crowley, M. (2022). Investigation of Independent Reinforcement Learning Algorithms in Multi-Agent Environments. Frontiers in Artificial Intelligence, 27. https://doi.org/10.3389/frai.2022.805823

This Frontiers AI Paper was led by fourth-year undergraduate student Ken Ming Lee with lots of guidance and help from my PhD student Sriram Ganapathi Subramanian (Lee et al., 2022). It’s a great empirical comparison of a number of single-agent and multi-agent RL algorithms on a standard set of MARL problems. Often single-agent algorithms are quickly hacked in order to to decision making on multi-agent domains and seem to work fairly well. Our motivating question was how often is this true and what kinds of problems require approaches that consider more dedicated multi-agent interaction.

This work expands on a previous study (Lee et al., 2021) from a NeurIPS Workshop in 2021.

Lab Member News

Sriram’s Defence

On June 27, 2022, long-time lab member Sriram Ganapathi Subramanian successfully defended his PhD thesis: “Multi-Agent Reinforcement Learning in Large Complex Environments”

Subramanian, S. G. (2022). Multi-Agent Reinforcement Learning in Large Complex Environments [PhD thesis, UWSpace]. https://doi.org/http://hdl.handle.net/10012/18442

Congratulations Sriram!

Sriram started a postdoc position at the Vector Institute in September 2022 working with Prof. Sheila Mcllraith at UofT and Prof. Pascal Poupart at UWaterloo.

Benyamin Ghojogh

Former PhD student Benyamin Ghojogh has taken up a new position at Research Associate at KisoJi Biotechnology working on exciting research on

We are working on developing AI tools for paratope-epitope interaction prediction, such as Machine Learning methods to describe VHH Antibodies for semantic clustering, selection, and matching. We aim to expedite the drug discovery process through the in-silico methodologies.

Can’t wait to hear more Benyamin!

Promotion and Tenure

As of July 1, 2022, Dr. Mark Crowley is now a tenured, Associate Professor in the ECE department! Phew.

Independent reinforcement learning algorithms have no theoretical guarantees for finding the best policy in multi-agent settings. However, in practice, prior works have reported good performance with independent algorithms in some domains and bad performance in others. Moreover, a comprehensive study of the strengths and weaknesses of independent algorithms is lacking in the literature. In this paper, we carry out an empirical comparison of the performance of independent algorithms on seven PettingZoo environments that span the three main categories of multi-agent environments, i.e., cooperative, competitive, and mixed. For the cooperative setting, we show that independent algorithms can perform on par with multi-agent algorithms in fully-observable environments, while adding recurrence improves the learning of independent algorithms in partially-observable environments. In the competitive setting, independent algorithms can perform on par or better than multi-agent algorithms, even in more challenging environments. We also show that agents trained via independent algorithms learn to perform well individually, but fail to learn to cooperate with allies and compete with enemies in mixed environments.

Independent reinforcement learning algorithms have no theoretical guarantees for finding the best policy in multi-agent settings. However, in practice, prior works have reported good performance with independent algorithms in some domains and bad performance in others. Moreover, a comprehensive study of the strengths and weaknesses of independent algorithms is lacking in the literature. In this paper, we carry out an empirical comparison of the performance of independent algorithms on four PettingZoo environments that span the three main categories of multi-agent environments, i.e., cooperative, competitive, and mixed. We show that in fully-observable environments, independent algorithms can perform on par with multi-agent algorithms in cooperative and competitive settings. For the mixed environments, we show that agents trained via independent algorithms learn to perform well individually, but fail to learn to cooperate with allies and compete with enemies. We also show that adding recurrence improves the learning of independent algorithms in cooperative partially observable environments.