Mark Crowley

Research Lab (UWECEML)
Blog: Computationally Thinking
ORC-ID
Google Scholar
Contact
Twitter / Mastodon
BOOK! : "Elem. of Dim. Reduction and Manifold Learning"

My research seeks dependable and transparent ways to augment human decision making in complex domains in the presence of many agents, spatial structure, or uncertainty. My focus is on developing new algorithms within the fields of Reinforcement Learning, Deep Learning and Ensemble Methods, and Manifold Learning (Dimensionality Reduction) as well as other Research Topics. I often work in collaboration with researchers in applied fields such as Computational Sustainability (including Sustainable Forest Management, Autonomous Driving, AI for Science and Medical Imaging. Read more about me in my bio.

Mark Crowley is an Associate Professor in the Department of Electrical and Computer Engineering at the University of Waterloo (also cross-appointed in the Cheriton School of Computer Science). He is a member of the Waterloo Artificial Intelligence Institute (WAII), the Waterloo Institute for Complexity and Innovation (WICI) and is National Secretary of the Canadian Artificial Intelligence Association (CAIAC) which coordinates the yearly organization of the Canadian Conference on AI. He and his students carry out research into single-agent and multi-agent Reinforcement Learning large-scale 2D/3D image-like processing, and manifold learning/dimensionality reduction. Some of this research is motivated by theoretical opportunities, particularly in manifold learning and multi-agent reinforcement learning. But most of the work flows out of challenges raised by real-world domains including forest fire management, the automotive domain, medical imaging, and digital chemistry/material design.

I’m always happy to talk to people about my research, or the impact of AI/ML/RL on our world and its role in our society in the future, but I do not take on many incoming students, so potential students should read this note about joining my lab.

Besides my publications, you can find my thoughts on Artificial Intelligence, Machine Learning and how technology and science are advancing on my blog Computationally Thinking (updated intermittently) or through social media as @compthink on Twitter(at least for now) or increasingly, on Mastodon. I’m always happy to talk to people about my research, or the impact of AI/ML/RL on our world and its role in our society in the future.

recent news

May 25, 2023	AAMAS talk, PhD Dissertation Award, ChemGymRL Updates

recent highlighted publications

Seeing the Forest Despite the Trees : Large Scale Spatial-Temporal Decision Making

Mark Crowley, John Nelson, and David Poole.

In Conference on Uncertainty in Artificial Intelligence (UAI09). Montreal, Canada. 2009.

Abs URL

We introduce a challenging real world planning problem where actions must be taken at each location in a spatial area at each point in time. We use forestry planning as the motivating application. In Large Scale Spatial-Temporal (LSST) planning problems, the state and action spaces are defined as the cross-products of many local state and action spaces spread over a large spatial area such as a city or forest. These problems possess state uncertainty, have complex utility functions involving spatial constraints and we generally must rely on simulations rather than an explicit transition model. We define LSST problems as reinforcement learning prob- lems and present a solution using policy gradients. We compare two different policy formulations: an explicit policy that identifies each location in space and the action to take there, and an abstract policy that defines the proportion of actions to take across all locations in space. We show that the abstract policy is more robust and achieves higher rewards with far fewer parameters than the elementary policy. This abstract policy is also a better fit to the properties that practitioners in LSST problem domains require for such methods to be widely useful
Policy gradient planning for environmental decision making with existing simulators

Mark Crowley, and David Poole.

In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence (AAAI). San Francisco. 2011.

Abs URL

In environmental and natural resource planning domains actions are taken at a large number of locations over multiple time periods. These problems have enormous state and action spaces, spatial correlation between actions, uncertainty and complex utility models. We present an approach for modeling these planning problems as factored Markov decision processes. The reward model can contain local and global components as well as spatial constraints between locations. The transition dynamics can be provided by existing simulators developed by domain experts. We propose a landscape policy defined as the equilibrium distribution of a Markov chain built from many locally-parameterized policies. This policy is optimized using a policy gradient algorithm. Experiments using a forestry simulator demonstrate the algorithm’s ability to devise policies for sustainable harvest planning of a forest. Copyright \textcopyright 2011, Association for the Advancement of Artificial Intelligence. All rights reserved.
Policy Gradient Optimization Using Equilibrium Policies for Spatial Planning Domains

markcrowley.

In 13th INFORMS Computing Society Conference. Santa Fe, NM, United States. 2013.
Cyclic causal models with discrete variables: Markov chain equilibrium semantics and sample ordering

David Poole, and Mark Crowley

In International Joint Conference on Artificial Intelligence (IJCAI). Beijing, China. 2013.

Abs URL

We analyze the foundations of cyclic causal models for discrete variables, and compare structural equation models (SEMs) to an alternative semantics as the equilibrium (stationary) distribution of a Markov chain. We show under general conditions, discrete cyclic SEMs cannot have independent noise, even in the simplest case, cyclic structural equation models imply constraints on the noise. We give a formalization of an alternative Markov chain equilibrium semantics which requires not only the causal graph, but also a sample order. We show how the resulting equilibrium is a function of the sample ordering, both theoretically and empirically.
PAC Optimal Planning for Invasive Species Management: Improved Exploration for Reinforcement Learning from Simulator-Defined MDPs

Thomas G Dietterich, Majid Alkaee Taleghan, and Mark Crowley

In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2013). Bellevue, WA, USA. 2013.

Abs URL

Often the most practical way to define a Markov Decision Process (MDP) is as a simulator that, given a state and an action, produces a resulting state and immediate reward sampled from the corresponding distributions. Simulators in natural resource management can be very expensive to execute, so that the time required to solve such MDPs is dominated by the number of calls to the simulator. This paper presents an algorithm, DDV, that combines improved confidence intervals on the Q values (as in interval estimation) with a novel upper bound on the discounted state occupancy probabilities to intelligently choose state-action pairs to explore. We prove that this algorithm terminates with a policy whose value is within εof the optimal policy (with probability 1−δ) after making only polynomially-many calls to the simulator. Experiments on one benchmark MDP and on an MDP for invasive species management show very large reductions in the number of simulator calls required.