Mark Crowley | New TMLR Journal Publication on Revisiting Neighbourhoods in Mean Field Reinforcement Learning

We are thrilled to announce that a distinguished former lab member’s research on scaling multi-agent reinforcement learning (MARL) in large agent populations has been published in Transactions on Machine Learning Research (TMLR) !

Subramanian, S. G., Taylor, M. E., Larson, K., Crowley, M., & Poupart, P. (2026). Revisiting Neighbourhoods in Mean Field Reinforcement Learning. Transactions on Machine Learning Research (TMLR), 36. https://openreview.net/forum?id=PQ5R7K0WDc

In this paper, Sriram summarized and expanded on results addressing a fundamental limitation in traditional Mean Field RL, the assumptions that agents within a neighborhood are homogeneous and that agents outside have no influence (Subramanian et al., 2026).

Dr. Sriram Ganapathi Subramanian is a former PhD student in the lab, now an Assistant Professor at Carleton University. This publication is a culmination, and extension, of the fantastic work Sriram carried out in and since his PhD thesis (Subramanian, 2022), developing Mean Field Attention (MFA), a novel framework combining attention mechanisms for fine-grained local interactions with mean field approximations for global population dynamics. Sriram showed that MFA consistently outperforms prior mean field baselines across both competitive and cooperative multi-agent benchmarks.

References:

PhD Thesis

Multi-Agent Reinforcement Learning in Large Complex Environments

Sriram Ganapathi Subramanian.

UWSpace Thesis Repository, University of Waterloo, Jun, 2022.

Abs URL

Multi-agent reinforcement learning (MARL) has seen much success in the past decade. However, these methods are yet to find wide application in large-scale real world problems due to two important reasons. First, MARL algorithms have poor sample efficiency, where many data samples need to be obtained through interactions with the environment to learn meaningful policies, even in small environments. Second, MARL algorithms are not scalable to environments with many agents since, typically, these algorithms are exponential in the number of agents in the environment. This dissertation aims to address both of these challenges with the goal of making MARL applicable to a variety of real world environments. Towards improving sample efficiency, an important observation is that many real world environments already, in practice, deploy sub-optimal or heuristic approaches for generating policies. A useful possibility that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this dissertation, we provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings. To this end, we propose a general model for learning from external advisors in MARL and show that desirable theoretical properties such as convergence to a unique solution concept, and reasonable finite sample complexity bounds exist, under a set of common assumptions. Furthermore, extensive experiments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors. Towards scaling MARL, we explore the use of mean field theory. Mean field theory provides an effective way of scaling multi-agent reinforcement learning algorithms to environments with many agents, where other agents can be abstracted by a virtual mean agent. Prior work has used mean field theory in MARL, however, they suffer from several stringent assumptions such as requiring fully homogeneous agents, full observability of the environment, and centralized learning settings, that prevent their wide application in practical environments. In this dissertation, we extend mean field methods to environments having heterogeneous agents, and partially observable settings. Further, we extend mean field methods to include decentralized approaches. We provide novel mean field based MARL algorithms that outperform previous methods on a set of large games with many agents. Theoretically, we provide bounds on the information loss experienced as a result of using the mean field and further provide fixed point guarantees for Q-learning-based algorithms in each of these environments. Subsequently, we combine our work in mean field learning and learning from advisors to show that we can achieve powerful MARL algorithms that are more suitable for real world environments as compared to prior approaches. This method uses the recently introduced attention mechanism to perform per-agent modelling of others in the locality, in addition to using the mean field for global responses. Notably, in this dissertation, we show applications in several real world multi-agent environments such as the Ising model, the ride-pool matching problem, and the massively multi-player online (MMO) game setting (which is currently a multi-billion dollar market).

TMLR

Revisiting Neighbourhoods in Mean Field Reinforcement Learning

Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, Mark Crowley, and Pascal Poupart.

Transactions on Machine Learning Research (TMLR). Jul, 2026.

Abs PDF Code URL

Many multi-agent reinforcement learning (MARL) algorithms do not scale well as the number of agents increases due to an exponential time and space complexity dependency on the number of agents in the environment. Mean field theory has been used to address this problem by approximating the effect of neighbourhoods of agents by a single representative agent. While this approximation allows MARL algorithms to scale to environments with many agents, approaches typically assumed that agents 1) inside a neighbourhood are homogeneous, and 2) outside a neighbourhood have no influence (and can therefore be ignored). This paper relaxes these assumptions and proposes a novel framework, mean field attention (MFA), which uses an attention mechanism for local responses and the mean field approximation for global responses. We implement MFA with two new algorithms leveraging Q-learning and actor-critic. These novel MFA algorithms consistently outperform other MARL algorithms, including prior mean field-based algorithms, across multiple metrics and benchmarks.