Mark Crowley

Research Lab (UWECEML)
Blog: Computationally Thinking
ORC-ID
Google Scholar
Contact
Twitter / Mastodon
BOOK! : "Elem. of Dim. Reduction and Manifold Learning"

My research seeks dependable and transparent ways to augment human decision making in complex domains in the presence of many agents, spatial structure, or uncertainty. My focus is on developing new algorithms within the fields of Reinforcement Learning, Deep Learning and Ensemble Methods, and Manifold Learning (Dimensionality Reduction) as well as other Research Topics. I often work in collaboration with researchers in applied fields such as Computational Sustainability (including Sustainable Forest Management, Autonomous Driving, AI for Science and Medical Imaging. Read more about me in my bio.

Mark Crowley is an Associate Professor in the Department of Electrical and Computer Engineering at the University of Waterloo (also cross-appointed in the Cheriton School of Computer Science). He is a member of the Waterloo Artificial Intelligence Institute (WAII), the Waterloo Institute for Complexity and Innovation (WICI) and is National Secretary of the Canadian Artificial Intelligence Association (CAIAC) which coordinates the yearly organization of the Canadian Conference on AI. He and his students carry out research into single-agent and multi-agent Reinforcement Learning large-scale 2D/3D image-like processing, and manifold learning/dimensionality reduction. Some of this research is motivated by theoretical opportunities, particularly in manifold learning and multi-agent reinforcement learning. But most of the work flows out of challenges raised by real-world domains including forest fire management, the automotive domain, medical imaging, and digital chemistry/material design.

I’m always happy to talk to people about my research, or the impact of AI/ML/RL on our world and its role in our society in the future, but I do not take on many incoming students, so potential students should read this note about joining my lab.

Besides my publications, you can find my thoughts on Artificial Intelligence, Machine Learning and how technology and science are advancing on my blog Computationally Thinking (updated intermittently) or through social media as @compthink on Twitter(at least for now) or increasingly, on Mastodon. I’m always happy to talk to people about my research, or the impact of AI/ML/RL on our world and its role in our society in the future.

recent news

May 25, 2023	AAMAS talk, PhD Dissertation Award, ChemGymRL Updates

recent highlighted publications

ChemGymRL

ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistry

Chris Beeler, Sriram Ganapathi Subramanian, Kyle Sprague, Nouha Chatti, Colin Bellinger, Mitchell Shahen, Nicholas Paquin, Mark Baula, Amanuel Dawit, Zihan Yang, Xinkai Li, Mark Crowley, and Isaac Tamblyn.

Digital Discovery. 3, Feb, 2024.

Abs PDF

This paper provides a simulated laboratory for making use of reinforcement learning (RL) for material design, synthesis, and discovery. Since RL is fairly data intensive, training agents ‘on-the-fly’ by taking actions in the real world is infeasible and possibly dangerous. Moreover, chemical processing and discovery involves challenges which are not commonly found in RL benchmarks and therefore offer a rich space to work in. We introduce a set of highly customizable and open-source RL environments, ChemGymRL, implementing the standard gymnasium API. ChemGymRL supports a series of interconnected virtual chemical benches where RL agents can operate and train. The paper introduces and details each of these benches using well-known chemical reactions as illustrative examples, and trains a set of standard RL algorithms in each of these benches. Finally, discussion and comparison of the performances of several standard RL methods are provided in addition to a list of directions for future work as a vision for the further development and usage of ChemGymRL.
Multi-Advisor-MARL

Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning

Sriram Ganapathi Subramanian, Matthew Taylor, Kate Larson, and Mark Crowley

In Proceedings of the 22nd International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), London, United Kingdom. Sep, 2023.

Abs PDF

Multi-agent reinforcement learning typically suffers from the problem of sample inefficiency, where learning suitable policies involves the use of many data samples. Learning from external demonstrators is a possible solution that mitigates this problem. However, most prior approaches in this area assume the presence of a single demonstrator. Leveraging multiple knowledge sources (i.e., advisors) with expertise in distinct aspects of the environment could substantially speed up learning in complex environments. This paper considers the problem of simultaneously learning from multiple independent advisors in multi-agent reinforcement learning. The approach leverages a two-level Q-learning architecture, and extends this framework from single-agent to multi-agent settings. We provide principled algorithms that incorporate a set of advisors by both evaluating the advisors at each state and subsequently using the advisors to guide action selection. Also, we provide theoretical convergence and sample complexity guarantees. Experimentally, we validate our approach in three different test-beds and show that our algorithms give better performances than baselines, can effectively integrate the combined expertise of different advisors, and learn to ignore bad advice.
Machine-learning Assisted Swallowing Assessment: a deep learning-based quality improvement tool to screen for post-stroke dysphagia

Rami Saab, Arjun Balachandar, Hamza Mahdi, Eptehal Nashnoush, Lucas Perri, Ashley Waldron, Alireza Sadeghian, Gordon Rubenfeld, Mark Crowley, Mark I. Boulos, Brian Murray, and Houman Khosravani.

Frontiers in Neuroscience. 17, Nov, 2023.

Abs Code URL

Post-stroke dysphagia is common and associated with significant morbidity and mortality, rendering bedside screening of significant clinical importance. Using voice as a biomarker coupled with deep learning has the potential to improve patient access to screening and mitigate the subjectivity associated with detecting voice change, a component of several validated screening protocols. In this single-center study, we developed a proof-of-concept model for automated dysphagia screening and evaluated the performance of this model on training and testing cohorts. Patients were admitted to a comprehensive stroke center, where primary English speakers could follow commands without significant aphasia and participated on a rolling basis. The primary outcome was classification either as a pass or fail equivalent using a dysphagia screening test as a label. Voice data was recorded from patients who spoke a standardized set of vowels, words, and sentences from the National Institute of Health Stroke Scale. Seventy patients were recruited and 68 were included in the analysis, with 40 in training and 28 in testing cohorts, respectively. Speech from patients was segmented into 1,579 audio clips, from which 6,655 Mel-spectrogram images were computed and used as inputs for deep-learning models (DenseNet and ConvNext, separately and together). Clip-level and participant-level swallowing status predictions were obtained through a voting method. Model performance on the individual clip-level demonstrated dysphagia screening sensitivity of 71% and specificity of 77% (F1=0.73, AUC=0.80 [95% CI: 0.78-0.82]. On the participant-level the sensitivity and specificity were 89% and 79% respectively (F1=0.81, AUC=0.91 [95% CI: 0.77–1.05]). Our study is the first to demonstrate the feasibility of applying deep learning to classify vocalizations to detect post-stroke dysphagia.
ECML

Learning when to observe: A frugal reinforcement learning framework for a high-cost world

Colin Bellinger, Mark Crowley, and Isaac Tamblyn.

In Workshop on Simplification, Compression, Efficiency and Frugality for Artificial intelligence at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD). Turin, Italy. Sep, 2023.

Abs arXiv PDF URL

Reinforcement learning (RL) has been shown to learn so- phisticated control policies for complex tasks including games, robotics, heating and cooling systems and text generation. The action-perception cycle in RL, however, generally assumes that a measurement of the state of the environment is available at each time step without a cost. In applications such as materials design, deep-sea and planetary robot exploration and medicine, however, there can be a high cost associated with measuring, or even approximating, the state of the environment. In this paper, we survey the recently growing literature that adopts the perspective that an RL agent might not need, or even want, a costly mea- surement at each time step. Within this context, we propose the Deep Dynamic Multi-Step Observationless Agent (DMSOA), contrast it with the literature and empirically evaluate it on OpenAI gym and Atari Pong environments. Our results, show that DMSOA learns a better policy with fewer decision steps and measurements than the considered alternative from the literature. The corresponding code is available at: https: //github.com/cbellinger27/Learning-when-to-observe-in-RL
Multi-Advisor-QL

Multi-Agent Advisor Q-Learning

Sriram Ganapathi Subramanian, Kate Larson, Matthew Taylor, and Mark Crowley

In International Joint Conference on Artificial Intelligence (IJCAI) : Journal Track. Macao, China. Aug, 2023.

Abs PDF Code URL Video

In the last decade, there have been significant advances in multi-agent reinforcement learn- ing (MARL) but there are still numerous challenges, such as high sample complexity and slow convergence to stable policies, that need to be overcome before wide-spread deployment is possi- ble. However, many real-world environments already, in practice, deploy sub-optimal or heuristic approaches for generating policies. An interesting question that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this paper, we provide a principled framework for incorporating action recommendations from online sub- optimal advisors in multi-agent settings. We describe the problem of ADvising Multiple Intelligent Reinforcement Agents (ADMIRAL) in nonrestrictive general-sum stochastic game environments and present two novel Q-learning based algorithms: ADMIRAL - Decision Making (ADMIRAL-DM) and ADMIRAL - Advisor Evaluation (ADMIRAL-AE), which allow us to improve learning by appropriately incorporating advice from an advisor (ADMIRAL-DM), and evaluate the effectiveness of an advisor (ADMIRAL-AE). We analyze the algorithms theoretically and provide fixed point guarantees regarding their learning in general-sum stochastic games. Furthermore, extensive experi- ments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors.