showcased publications

2026

1. CausalJournal

   Modeling Soft Intervention Effects for Implicit Causal Representation Learning

   Shayan Shirahmad Gale Bagi, Zahra Gharaee, Oliver Schulte, and Mark Crowley

   Machine Learning. 115, (199). Springer Nature, Aug, 2026.

   Abs arXiv PDF URL Notes

   Note: The arxiv preprint version is a shorter version of this paper before the editing and revisions for the Nature ML journal, but contains most of the same major conclusions and insights.

   Abstract: Learning causal representations from data without access to ground-truth causal graphs remains a central challenge in representation learning. Implicit approaches—where the model learns causal dependencies without explicitly parameterizing the causal graph—offer a compelling advantage over explicit methods by avoiding optimization difficulties such as local minima associated with adjacency matrix estimation. However, existing implicit methods often assume access to hard interventions, which are rarely feasible in real-world scenarios. In contrast, soft interventions—more prevalent in practice—modify causal mechanisms without severing parental dependencies, introducing subtle, ambiguous effects that confound learning. To address this, we propose ICRL-SM, a novel method for Implicit Causal Representation Learning from soft interventions using a causal mechanism switch variable. This variable captures unwanted changes induced by soft interventions, enabling the model to focus on the necessary variations that reflect underlying causal structure. Our framework leverages a variational autoencoder trained on paired pre- and post-intervention samples, and is theoretically grounded under a set of assumptions that ensure identifiability of causal representations. Although some assumptions (e.g., Gaussianity of latent variables, diffeomorphic decoders) are strict, we show empirically that our method performs robustly even when these are violated—demonstrating strong results on both synthetic benchmarks and real-world image datasets. These findings highlight the potential of ICRL-SM to bridge the gap between theoretical identifiability and practical applicability, advancing causal representation learning under realistic conditions. The source code for this paper is available at: https://github.com/sshirahmad/ICRL.

2. AIES26

   The Forager’s Dilemma: Peer Enforcement Alone is Not Enough for Stable Multi-Agent Alignment

   Majid Ghasemi, Prabhjyot Singh, and Mark Crowley

   In AAAI Conference on AI, Ethics, and Society. Malmo, Sweden. Oct, 2026.
3. V^2-VLNCE

   View Invariant Learning for Vision-Language Navigation in Continuous Environments

   Josh Qixuan Sun, Huaiyuan Weng, Xiaoying Xing, Chul Min Yeum, and Mark Crowley

   IEEE Robotics and Automation Letters (IEEE RA-L). 11, (5). 2026.

   Abs PDF Code URL

   Abstract: Vision-Language Navigation in Continuous Environments (VLNCE), where an agent follows instructions and moves freely to reach a destination, is a key research problem in embodied AI. However, most existing approaches are sensitive to viewpoint changes, i.e. variations in camera height and viewing angle. Here we introduce a more general scenario, V2-VLNCE (VLNCE with Varied Viewpoints) and propose a view-invariant post-training framework, called VIL (View Invariant Learning), that makes existing navigation policies more robust to changes in camera viewpoint. VIL employs a contrastive learning framework to learn sparse and view-invariant features. We also introduce a teacher-student framework for the Waypoint Predictor Module, a standard part of VLNCE baselines, where a view-dependent teacher model distills knowledge into a view-invariant student model. We employ an end-to-end training paradigm to jointly optimize these components. Empirical results show that our method outperforms state-of-the-art approaches on V2-VLNCE by 8-15% measured on Success Rate for two standard benchmark datasets R2R-CE and RxR-CE. Evaluation of VIL in standard VLNCE settings shows that despite being trained for varied viewpoints, VIL often still improves performance. On the harder RxR-CE dataset, our method also achieved state-of-the-art performance across all metrics. This suggests that adding VIL does not diminish the standard viewpoint performance and can serve as a plug-and-play posttraining method. We further evaluate VIL for simulated camera placements derived from real robot configurations (e.g. Stretch RE1, LoCoBot), showing consistent improvements of performance. Finally, we present a proof-of-concept real-robot evaluation in two physical environments using a panoramic RGB sensor combined with LiDAR. These results show that VIL improves robustness not only in simulation but also in real-world navigation scenarios, making it a practical strategy for embodied agents. The code is available at https://github.com/realjoshqsun/V2-VLNCE.

4. ICML

   Preference-based Antibody Expression Ranking: Scaling with Large-scale Weak Supervision

   Josh Qixuan Sun, Morteza Babaie, Wenyang Hou, Mark Crowley, and David Young.

   In Forty-third International Conference on Machine Learning (ICML). PMLR, Seoul, South Korea. Jul, 2026.

   Abs PDF URL Notes

   Note: This work has just recently been accepted to the one of the top level, extremely competitive, international Machine Learning conferences ICML with myself, Josh and the company scientists as coauthors. Josh will present the results at ICML in South Korea over the summer.

   Abstract: Antibody expression ranking is a critical task in antibody design, yet its modeling is severely hindered by the scarcity of labeled experimental data. To address this, we propose a unified preference-based learning framework that integrates scarce quantitative expression data with large-scale weak positive supervision from immunization data. We adapt Direct Preference Optimization (DPO) to protein language models by introducing a union-masked log-likelihood approximation and IMGT-based alignment, enabling efficient training on variable-length sequences. Evaluating on a diverse internal dataset of 1254 labeled sequences and 4 million unlabeled camelid-derived antibodies, we show that our method consistently outperforms baselines on most metrics. Our results demonstrate that preference learning can effectively learn from weak supervision, providing a scalable solution for antibody expressibility optimization in data-constrained settings.

5. EWRL26

   Learning When to Trust in Contextual Social Bandits

   Majid Ghasemi, and Mark Crowley

   In European Workshop on RL (EWRL2026). Lille, France. Oct, 2026.
6. ISPRS

   A Generative Upsampling Framework for Reconstructing High-Density Tree Structures from Low-Density Airborne Lidar

   Erfan Hasanpour Zaryabi, Liam Bennett, Josh Qixuan Sun, Mark Crowley, Laura Chasmer, Christopher Hopkinson, and Jeff Boisvert.

   In Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.. Copernicus Publications, Toronto, Canada. Jul, 2026.

   Abs

   Abstract: Airborne Laser Scanning (ALS) enables efficient large-scale mapping of three-dimensional forest structure due to its extensive spatial coverage, but its low point density often limits the accurate representation of forest structural attributes required for applications such as wildfire management and forest health monitoring. This study proposes a deep generative upsampling framework to enhance low- density ALS tree point clouds and generate dense representations for improved structural analysis. To improve tree reconstruction, the framework incorporates a synthetic pretraining strategy that exposes the model to complete tree geometries before fine-tuning on real data. Experimental results demonstrate that the proposed framework generates high-density tree point clouds with strong spatial similarity to ground-truth point clouds. Synthetic pretraining further improves reconstruction performance, achieving lower Chamfer Distance and higher F1-scores across multiple upsampling rates. In addition, the reconstructed point clouds increase estimation accuracy of key structural attributes, including tree height, crown area, and tree volume. Qualitative experiments on real ALS data confirms the ability of the proposed framework to recover realistic tree structures from sparse point clouds. Overall, the proposed method provides a practical approach for bridging the wide spatial coverage of ALS with the detailed structural information typically obtained from high-density lidar data.

7. Toward Virtuous Reinforcement Learning: A Critique and Roadmap

   Majid Ghasemi, and Mark Crowley

   In Workshop on Machine Ethics: From Formal Methods To Emergent Machine Ethics at AAAI 2026.. Singapore. Jan, 2026.

   Abs arXiv PDF

   Abstract: This paper critiques common patterns in machine ethics for Reinforcement Learning (RL) and argues for a virtue focused alternative. We highlight two recurring limitations in much of the current literature: (i) rule based (deontological) methods that encode duties as constraints or shields often struggle under ambiguity and nonstationarity and do not cultivate lasting habits, and (ii) many reward based approaches, especially single objective RL, implicitly compress diverse moral considerations into a single scalar signal, which can obscure trade offs and invite proxy gaming in practice. We instead treat ethics as policy level dispositions, that is, relatively stable habits that hold up when incentives, partners, or contexts change. This shifts evaluation beyond rule checks or scalar returns toward trait summaries, durability under interventions, and explicit reporting of moral trade offs. Our roadmap combines four components: (1) social learning in multi agent RL to acquire virtue like patterns from imperfect but normatively informed exemplars; (2) multi objective and constrained formulations that preserve value conflicts and incorporate risk aware criteria to guard against harm; (3) affinity based regularization toward updateable virtue priors that support trait like stability under distribution shift while allowing norms to evolve; and (4) operationalizing diverse ethical traditions as practical control signals, making explicit the value and cultural assumptions that shape ethical RL benchmarks.

8. AIES26

   AI Alignment without Representation: From Scalar Rewards to Social Deliberation

   Majid Ghasemi, and Mark Crowley

   In AAAI Conference on AI, Ethics, and Society. Malmo, Sweden. Oct, 2026.

2025

1. Learning When to Observe: A Frugal Reinforcement Learning Framework for a High-Cost World

   Colin Bellinger, Isaac Tamblyn, and Mark Crowley

   Springer, Switzerland. 2025.

   Abs PDF URL Notes

   Note: From the Workshop "Machine Learning And Principles And Practice Of Knowledge Discovery In Databases", ECML-PKDD Turin, Italy, Sep 2023

   Abstract: Reinforcement learning (RL) has been shown to learn sophisticated control policies for complex tasks including games, robotics, heating and cooling systems and text generation. The action-perception cycle in RL, however, generally assumes that a measurement of the state of the environment is available at each time step without a cost. In applications such as materials design, deep-sea and planetary robot exploration and medicine, however, there can be a high cost associated with measuring, or even approximating, the state of the environment. In this paper, we survey the recently growing literature that adopts the perspective that an RL agent might not need, or even want, a costly measurement at each time step. Within this context, we propose the Deep Dynamic Multi-Step Observationless Agent (DMSOA), contrast it with the literature and empirically evaluate it on OpenAI gym and Atari Pong environments. Our results, show that DMSOA learns a better policy with fewer decision steps and measurements than the considered alternative from the literature. The corresponding code is available at: https://github.com/cbellinger27/Learning-when-to-observe-in-RL.

2024

1. Dynamic Programming with Incomplete Information to Overcome Navigational Uncertainty in POMDPs

   Chris Beeler, Xinkai Li, Colin Bellinger, Mark Crowley, Maia Fraser, and Isaac Tamblyn.

   In Proceedings of the Canadian Conference on Artificial Intelligence. Canadian Artificial Intelligence Association (CAIAC), Guelph, Ontario, Canada. May, 2024.

   Abs PDF URL

   Abstract: Using a generalizable novel nautical navigation environment, we show how dynamic programming can be used when only incomplete information about a partially observed Markov decision process (POMDP) is known. By incorporating uncertainty into our model, we show that navigation policies can be constructed that maintain safety, outperforming the baseline performance of traditional dynamic programming for Markov decision processes (MDPs). Adding in controlled sensing methods, we show that these policies can also lower measurement costs at the same time.

2. ChemGymRL

   ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistry

   Chris Beeler, Sriram Ganapathi Subramanian, Kyle Sprague, Nouha Chatti, Colin Bellinger, Mitchell Shahen, Nicholas Paquin, Mark Baula, Amanuel Dawit, Zihan Yang, Xinkai Li, Mark Crowley, and Isaac Tamblyn.

   Digital Discovery. 3, Feb, 2024.

   Abs PDF Code

   Abstract: This paper provides a simulated laboratory for making use of reinforcement learning (RL) for material design, synthesis, and discovery. Since RL is fairly data intensive, training agents ‘on-the-fly’ by taking actions in the real world is infeasible and possibly dangerous. Moreover, chemical processing and discovery involves challenges which are not commonly found in RL benchmarks and therefore offer a rich space to work in. We introduce a set of highly customizable and open-source RL environments, ChemGymRL, implementing the standard gymnasium API. ChemGymRL supports a series of interconnected virtual chemical benches where RL agents can operate and train. The paper introduces and details each of these benches using well-known chemical reactions as illustrative examples, and trains a set of standard RL algorithms in each of these benches. Finally, discussion and comparison of the performances of several standard RL methods are provided in addition to a list of directions for future work as a vision for the further development and usage of ChemGymRL.

2023

1. GCRL

   Generative Causal Representation Learning for Out-of-Distribution Motion Forecasting

   Shayan Shirahmad Gale Bagi, Zahra Gharaee, Oliver Schulte, and Mark Crowley

   In Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, Honolulu, Hawaii, USA. Jul, 2023. [13 citations]

   Abs arXiv PDF Code Poster URL Hypoth Notes

   Note: In this paper we propose a new framework for leveraging causal information to improve robustness of learning in the presence of distribution shift. Empirical results on the motion forecasting domain support the theoretical findings.

   Abstract: Conventional supervised learning methods typically assume i.i.d samples and are found to be sensitive to out-of-distribution (OOD) data. We propose Generative Causal Representation Learning (GCRL) which leverages causality to facilitate knowledge transfer under distribution shifts. While we evaluate the effectiveness of our proposed method in human trajectory prediction models, GCRL can be applied to other domains as well. First, we propose a novel causal model that explains the generative factors in motion forecasting datasets using features that are common across all environments and with features that are specific to each environment. Selection variables are used to determine which parts of the model can be directly transferred to a new environment without fine-tuning. Second, we propose an end-to-end variational learning paradigm to learn the causal mechanisms that generate observations from features. GCRL is supported by strong theoretical results that imply identifiability of the causal model under certain assumptions. Experimental results on synthetic and real-world motion forecasting datasets show the robustness and effectiveness of our proposed method for knowledge transfer under zero-shot and low-shot settings by substantially outperforming the prior motion forecasting models on out-of-distribution prediction.

2. Textbook

   Elements of Dimensionality Reduction and Manifold Learning

   Benyamin Ghojogh, Mark Crowley, Fakhri Karray, and Ali Ghodsi.

   Springer Nature, Feb, 2023. [63 citations]

   Abs URL Notes

   Note: Thanks mostly to the enourmous efforts of former PhD student Dr. Benyamin Ghojogh on this text based on the research and background reviews for his thesis.

   Abstract: Dimensionality reduction, also known as manifold learning, is an area of machine learning used for extracting informative features from data, for better representation of data or separation between classes. This book presents a cohesive review of linear and nonlinear dimensionality reduction and manifold learning. Three main aspects of dimensionality reduction are covered – spectral dimensionality reduction, probabilistic dimensionality reduction, and neural network-based dimensionality reduction, which have geometric, probabilistic, and information-theoretic points of view to dimensionality reduction, respectively. This book delves into basic concepts and recent developments in the field of dimensionality reduction and manifold learning, providing the reader with a comprehensive understanding. The necessary background and preliminaries, on linear algebra, optimization, and kernels, are also explained in the book to ensure a comprehensive understanding of the algorithms.

3. Multi-Advisor-MARL

   Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning

   Sriram Ganapathi Subramanian, Matthew Taylor, Kate Larson, and Mark Crowley

   In Proceedings of the 22nd International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), London, United Kingdom. Sep, 2023.

   Abs PDF URL

   Abstract: Multi-agent reinforcement learning typically suffers from the problem of sample inefficiency, where learning suitable policies involves the use of many data samples. Learning from external demonstrators is a possible solution that mitigates this problem. However, most prior approaches in this area assume the presence of a single demonstrator. Leveraging multiple knowledge sources (i.e., advisors) with expertise in distinct aspects of the environment could substantially speed up learning in complex environments. This paper considers the problem of simultaneously learning from multiple independent advisors in multi-agent reinforcement learning. The approach leverages a two-level Q-learning architecture, and extends this framework from single-agent to multi-agent settings. We provide principled algorithms that incorporate a set of advisors by both evaluating the advisors at each state and subsequently using the advisors to guide action selection. Also, we provide theoretical convergence and sample complexity guarantees. Experimentally, we validate our approach in three different test-beds and show that our algorithms give better performances than baselines, can effectively integrate the combined expertise of different advisors, and learn to ignore bad advice.

2022

1. Mean Field MARL

   Decentralized Mean Field Games

   Sriram Ganapathi Subramanian, Matthew Taylor, Mark Crowley, and Pascal Poupart.

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2022). Virtual. Feb, 2022. [11 citations]

   Abs arXiv PDF URL Hypoth

   Abstract: Multiagent reinforcement learning algorithms have not been widely adopted in large scale environments with many agents as they often scale poorly with the number of agents. Using mean field theory to aggregate agents has been proposed as a solution to this problem. However, almost all previous methods in this area make a strong assumption of a centralized system where all the agents in the environment learn the same policy and are effectively indistinguishable from each other. In this paper, we relax this assumption about indistinguishable agents and propose a new mean field system known as Decentralized Mean Field Games, where each agent can be quite different from others. All agents learn independent policies in a decentralized fashion, based on their local observations. We define a theoretical solution concept for this system and provide a fixed point guarantee for a Q-learning based algorithm in this system. A practical consequence of our approach is that we can address a ‘chicken-and-egg’ problem in empirical mean field reinforcement learning algorithms. Further, we provide Q-learning and actor-critic algorithms that use the decentralized mean field learning approach and give stronger performances compared to common baselines in this area. In our setting, agents do not need to be clones of each other and learn in a fully decentralized fashion. Hence, for the first time, we show the application of mean field learning methods in fully competitive environments, large-scale continuous action space environments, and other environments with heterogeneous agents. Importantly, we also apply the mean field method in a ride-sharing problem using a real-world dataset. We propose a decentralized solution to this problem, which is more practical than existing centralized training methods.

2. Multi-Advisor-QL

   Multi-Agent Advisor Q-Learning

   Sriram Ganapathi Subramanian, Kate Larson, Matthew Taylor, and Mark Crowley

   Journal of Artificial Intelligence Research (JAIR). 74, May, 2022. [12 citations]

   Abs PDF Code URL Notes

   Note: This paper uses forest widlfire as one of its motivational examples to demonstrate the difficulty of multi-agent planning in real-world domains with many agents acting at once.

   Abstract: In the last decade, there have been significant advances in multi-agent reinforcement learning (MARL) but there are still numerous challenges, such as high sample complexity and slow convergence to stable policies, that need to be overcome before wide-spread deployment is possible. However, many real-world environments already, in practice, deploy sub-optimal or heuristic approaches for generating policies. An interesting question that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this paper, we provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings. We describe the problem of ADvising Multiple Intelligent Reinforcement Agents (ADMIRAL) in nonrestrictive general-sum stochastic game environments and present two novel Q-learning based algorithms: ADMIRAL - Decision Making (ADMIRAL-DM) and ADMIRAL - Advisor Evaluation (ADMIRAL-AE), which allow us to improve learning by appropriately incorporating advice from an advisor (ADMIRAL-DM), and evaluate the effectiveness of an advisor (ADMIRAL-AE). We analyze the algorithms theoretically and provide fixed point guarantees regarding their learning in general-sum stochastic games. Furthermore, extensive experi- ments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors.

2021

1. MARLEmpircal

   Investigation of Independent Reinforcement Learning Algorithms in Multi-Agent Environments

   Ken Ming Lee, Sriram Ganapathi Subramanian, and Mark Crowley

   In NeurIPS 2021 Deep Reinforcement Learning Workshop. Dec, 2021.

   Abs arXiv

   Abstract: Independent reinforcement learning algorithms have no theoretical guarantees for finding the best policy in multi-agent settings. However, in practice, prior works have reported good performance with independent algorithms in some domains and bad performance in others. Moreover, a comprehensive study of the strengths and weaknesses of independent algorithms is lacking in the literature. In this paper, we carry out an empirical comparison of the performance of independent algorithms on four PettingZoo environments that span the three main categories of multi-agent environments, i.e., cooperative, competitive, and mixed. We show that in fully-observable environments, independent algorithms can perform on par with multi-agent algorithms in cooperative and competitive settings. For the mixed environments, we show that agents trained via independent algorithms learn to perform well individually, but fail to learn to cooperate with allies and compete with enemies. We also show that adding recurrence improves the learning of independent algorithms in cooperative partially observable environments.

2. NLP-DigiPath

   Analysis of Language Embeddings for Classification of Unstructured Pathology Reports

   Aishwarya Krishna Allada, Yuanxin Wang, Veni Jindal, Morteza Babaie, H.R. Tizhoosh, and Mark Crowley

   In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, Nov, 2021. [10 citations]

   Abs Notes

   Note: This paper makes a comparative analysis of various natural language embedding models on the novel domain of digital pathology reports and attempts to determine the strengths and weaknesses of different wyas of dealing with these large and variable language datasets.

   Abstract: A pathology report is one of the most significant medical documents providing interpretive insights into the visual appearance of the patient’s biopsy sample. In digital pathology, high-resolution images of tissue samples are stored along with pathology reports. Despite the valuable information that pathology reports hold, they are not used in any systematic manner to promote computational pathology. In this work, we focus on analyzing the reports, which are generally unstructured documents written in English with sophisticated and highly specialized medical terminology. We provide a comparative analysis of various embedding models like BioBERT, Clinical BioBERT, BioMed-RoBERTa and Term Frequency-Inverse Document Frequency (TF-IDF), a traditional NLP technique, as well as the combination of embeddings from pre-trained models with TF-IDF. Our results demonstrate the effectiveness of various word embedding techniques for pathology reports.

3. PO-MFRL

   Partially Observable Mean Field Reinforcement Learning

   Sriram Ganapathi Subramanian, Matthew Taylor, Mark Crowley, and Pascal Poupart.

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems, London, United Kingdom. May, 2021. [23 citations]

   Abs arXiv PDF

   Abstract: Traditional multi-agent reinforcement learning algorithms are not scalable to environments with more than a few agents, since these algorithms are exponential in the number of agents. Recent research has introduced successful methods to scale multi-agent reinforcement learning algorithms to many agent scenarios using mean field theory. Previous work in this field assumes that an agent has access to exact cumulative metrics regarding the mean field behaviour of the system, which it can then use to take its actions. In this paper, we relax this assumption and maintain a distribution to model the uncertainty regarding the mean field of the system. We consider two different settings for this problem. In the first setting, only agents in a fixed neighbourhood are visible, while in the second setting, the visibility of agents is determined at random based on distances. For each of these settings, we introduce a Q-learning based algorithm that can learn effectively. We prove that this Q-learning estimate stays very close to the Nash Q-value (under a common set of assumptions) for the first setting. We also empirically show our algorithms outperform multiple baselines in three different games in the MAgents framework, which supports large environments with many agents learning simultaneously to achieve possibly distinct goals.

4. Amrl

   Active Measure Reinforcement Learning for Observation Cost Minimization: A framework for minimizing measurement costs in reinforcement learning

   Colin Bellinger, Rory Coles, Mark Crowley, and Isaac Tamblyn.

   In Canadian Conference on Artificial Intelligence. Springer, 2021. [28 citations]

   Abs arXiv PDF Notes

   Note: Neurips 2022 paper doing similar idea cited us https://hyp.is/go?url=https%3A%2F%2Fproceedings.neurips.cc%2Fpaper%2F2021%2Ffile%2F83e8fe6279ad25f15b23c6298c6a3584-Paper.pdf&group=__world__

   Abstract: Markov Decision Processes (MDP) with explicit measurement cost are a class of en- vironments in which the agent learns to maximize the costed return. Here, we define the costed return as the discounted sum of rewards minus the sum of the explicit cost of measuring the next state. The RL agent can freely explore the relationship between actions and rewards but is charged each time it measures the next state. Thus, an op- timal agent must learn a policy without making a large number of measurements. We propose the active measure RL framework (Amrl) as a solution to this novel class of problem, and contrast it with standard reinforcement learning under full observability and planning under partially observability. We demonstrate that Amrl-Q agents learn to shift from a reliance on costly measurements to exploiting a learned transition model in order to reduce the number of real-world measurements and achieve a higher costed return. Our results demonstrate the superiority of Amrl-Q over standard RL methods, Q-learning and Dyna-Q, and POMCP for planning under a POMDP in environments with explicit measurement costs.

2020

1. iMondrian

   Isolation Mondrian Forest for Batch and Online Anomaly Detection

   Haoran Ma, Benyamin Ghojogh, Maria N Samad, Dongyu Zheng, and Mark Crowley

   In IEEE International Conference on Systems, Man, and Cybernetics (IEEE-SMC-2020). IEEE SMC, Toronto, Canada (virtual). Oct, 2020. [38 citations]

   Abs arXiv PDF Code Hypoth Notes

   Note: The algorithm fuses two ideas, "isolation" from ensemble trees methods for anomaly detection and "Mondrian forests" which can learn flexible regression models from streaming data.

   Abstract: We propose a new method, named isolation Mondrian forest (iMondrian forest), for batch and online anomaly detection. The proposed method is a novel hybrid of isolation forest and Mondrian forest which are existing methods for batch anomaly detection and online random forest, respectively. iMondrian forest takes the idea of isolation, using the depth of a node in a tree, and implements it in the Mondrian forest structure. The result is a new data structure which can accept streaming data in an online manner while being used for anomaly detection. Our experiments show that iMondrian forest mostly performs better than isolation forest in batch settings and has better or comparable performance against other batch and online anomaly detection methods.

2. WildfireMLRev

   A review of machine learning applications in wildfire science and management

   Piyush Jain, Sean CP Coogan, Sriram Ganapathi Subramanian, Mark Crowley, Steve Taylor, and Mike D Flannigan.

   Environmental Reviews. 28, (3). Canadian Science Publishing, Jul, 2020. [631 citations]

   Abs arXiv PDF Slides URL

   Abstract: Artificial intelligence has been applied in wildfire science and management since the 1990s, with early applications including neural networks and expert systems. Since then the field has rapidly progressed congruently with the wide adoption of machine learning (ML) methods in the environmental sciences. Here, we present a scoping review of ML applications in wildfire science and management. Our overall objective is to improve awareness of ML methods among wildfire researchers and managers, as well as illustrate the diverse and challenging range of problems in wildfire science available to ML data scientists. To that end, we first present an overview of popular ML approaches used in wildfire science to date, and then review the use of ML in wildfire science as broadly categorized into six problem domains, including: 1) fuels characterization, fire detection, and mapping; 2) fire weather and climate change; 3) fire occurrence, susceptibility, and risk; 4) fire behavior prediction; 5) fire effects; and 6) fire management. Furthermore, we discuss the advantages and limitations of various ML approaches relating to data size, computational requirements, generalizability, and interpretability, as well as identify opportunities for future advances in the science and management of wildfires within a data science context. In total, we identified 298 relevant publications, where the most frequently used ML methods across problem domains included random forests, MaxEnt, artificial neural networks, decision trees, support vector machines, and genetic algorithms. As such, there exists opportunities to apply more current ML methods — including deep learning and agent based learning — in the wildfire sciences, especially in instances involving very large multivariate datasets. We must recognize, however, that despite the ability of ML models to learn on their own, expertise in wildfire science is necessary to ensure realistic modelling of fire processes across multiple scales, while the complexity of some ML methods, such as deep learning, requires a dedicated and sophisticated knowledge of their application. Finally, we stress that the wildfire research and management communities play an active role in providing relevant, high quality, and freely available wildfire data for use by practitioners of ML methods.

3. Deep Multi Agent Reinforcement Learning for Autonomous Driving

   Sushrut Bhalla, Sriram Ganapathi Subramanian, and Mark Crowley

   In Canadian Conference on Artificial Intelligence. May, 2020. [68 citations]

   PDF URL
4. Distributed Nonlinear Model Predictive Control and Metric Learning for Heterogeneous Vehicle Platooning with Cut-in/Cut-out Maneuvers

   Mohammad Hossein Basiri, Benyamin Ghojogh, Nasser L Azad, Sebastian Fischmeister, Fakhri Karray, and Mark Crowley

   In Proceeding of the 59th IEEE Conference on Decision and Control (CDC-2020). Jeju Island, Korea (virtual). Dec, 2020. [37 citations]

2019

1. TAC-ITS

   Comparison of Deep Learning models for Determining Road Surface Condition from Roadside Camera Images and Weather Data

   J. Carrillo, M. Crowley, G. Pan, and L. Fu.

   In TAC-ITS Canada Joint Conference. Halifax, Canada. 2019. [29 citations]

   Abs URL

   Abstract: Road maintenance during the Winter season is a safety critical and resource demanding operation. One of its key activities is determining road surface condition (RSC) in order to prioritize roads and allocate cleaning efforts such as plowing or salting. Two conventional approaches for determining RSC are: visual examination of roadside camera images by trained personnel and patrolling the roads to perform on-site inspections. However, with more than 500 cameras collecting images across Ontario, visual examination becomes a resource-intensive activity, difficult to scale especially during periods of snow storms. This paper presents the preliminary results of an ongoing study focused on improving the efficiency of road maintenance operations. We use multiple Deep Learning models to automatically determine RSC from roadside camera images and weather variables, extending previous research where similar methods have been used to deal with the problem. The dataset we use was collected during the 2017-2018 Winter season from 40 stations connected to the Ontario Road Weather Information System (RWIS), it includes 14.000 labelled images and 70.000 weather measurements. In particular, we train and evaluate the performance of seven state-of-the-art models from the Computer Vision literature, including the recent DenseNet, NASNet, and MobileNet. Also, by integrating observations from weather variables, the models are able to better ascertain RSC under poor visibility conditions.

2. ECML

   Compact Representation of a Multi-dimensional Combustion Manifold Using Deep Neural Networks

   Sushrut Bhalla, Matthew Yao, Jean-Pierre Hickey, and Mark Crowley

   In European Conference on Machine Learning (ECML-19). Wurzburg, Germany. 2019. [22 citations]

   Abs PDF Poster Slides

   Abstract: The computational challenges in turbulent combustion simulations stem from the physical complexities and multi-scale nature of the problem which make it intractable to compute scaleesolving simulations. For most engineering applications, the large scale separation between the flame (typically submillimeter scale) and the characteristic turbulent flow (typically centimeter or meter scale) allows us to evoke simplifying assumptions–such as done for the flamelet model to precompute all the chemical reactions and map them to a low order manifold. The resulting manifold is then tabulated and looked up at runtime. As the physical complexity of combustion simulations increases (including radiation, soot formation, pressure variations etc.) the dimensionality of the resulting manifold grows which impedes an efficient tabulation and look up. In this paper we present a novel approach to model the multidimensional combustion manifold. We approximate the combustion manifold using a neural network function approximator and use it to predict the temperature and composition of the reaction. We present a novel training procedure which is developed to generate a smooth output curve for temperature over the course of a reaction. We then evaluate our work against the current approach of tabulation with linear interpolation in combustion simulations. We also provide an ablation study of our training procedure in the context of overfitting in our model. The combustion dataset used for the modeling of combustion of H2 and O2 in this work is released alongside this paper.

2018

1. MCTS+A3C

   Combining MCTS and A3C for prediction of spatially spreading processes in forest wildfire settings

   Sriram Ganapathi Subramanian, and Mark Crowley

   In Canadian Conference on Artificial Intelligence. Toronto, Ontario, Canada. 2018. [15 citations]

   Abs PDF URL

   Abstract: In recent years, Deep Reinforcement Learning (RL) algorithms have shown super-human performance in a variety Atari and classic board games like chess and GO. Research into applications of RL in other domains with spatial considerations like environmental planning are still in their nascent stages. In this paper, we introduce a novel combination of Monte-Carlo Tree Search (MCTS) and A3C algorithms on an online simulator of a wildfire, on a pair of forest fires in Northern Alberta (Fort McMurray and Richardson fires) and on historical Saskatchewan fires previously compared by others to a physics-based simulator. We conduct several experiments to predict fire spread for several days before and after the given spatial information of fire spread and ignition points. Our results show that the advancements in Deep RL applications in the gaming world have advantages in spatially spreading real-world problems like forest fires. \textcopyright Springer International Publishing AG, part of Springer Nature 2018.

2. Blue Sky Ideas in Artificial Intelligence Education from the EAAI 2017 New and Future AI Educator Program

   Eric Eaton, Sven Koenig, Claudia Schulz, Francesco Maurelli, John Lee, Joshua Eckroth, Mark Crowley, Richard G Freedman, Rogelio E Cardona-Rivera, Tiago Machado, and Tom Williams.

   ACM, New York, NY, USA. Feb, 2018. [63 citations]

   URL
3. Using Spatial Reinforcement Learning to Build Forest Wildfire Dynamics Models From Satellite Images

   Sriram Ganapathi Subramanian, and Mark Crowley

   Frontiers in ICT. 5, (6). Frontiers, Apr, 2018. [63 citations]

   Abs PDF URL

   Abstract: Machine learning algorithms have increased tremendously in power in recent years but have yet to be fully utilized in many ecology and sustainable resource management domains such as wildlife reserve design, forest fire management and invasive species spread. One thing these domains have in common is that they contain dynamics that can be characterized as a Spatially Spreading Process (SSP) which requires many parameters to be set precisely to model the dynamics, spread rates and directional biases of the elements which are spreading. We present related work in Artificial Intelligence and Machine Learning for SSP sustainability domains including forest wildfire prediction. We then introduce a novel approach for learning in SSP domains using Reinforcement Learning (RL) where fire is the agent at any cell in the landscape and the set of actions the fire can take from a location at any point in time includes spreading North, South, East, West or not spreading. This approach inverts the usual RL setup since the dynamics of the corresponding Markov Decision Process (MDP) is a known function for immediate wildfire spread. Meanwhile, we learn an agent policy for a predictive model of the dynamics of a complex spatially-spreading process. Rewards are provided for correctly classifying which cells are on fire or not compared to satellite and other related data. We examine the behaviour of five RL algorithms on this problem: Value Iteration, Policy Iteration, Q-Learning, Monte Carlo Tree Search and Asynchronous Advantage Actor-Critic (A3C). We compare to a Gaussian process based supervised learning approach and discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We also discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We validate our approach with satellite image data of two massive wildfire events in Northern Alberta, Canada, the Fort McMurray fire of 2016 and the Richardson fire of 2011. The results show that we can learn predictive, agent-based policies as models of spatial dynamics using RL on readily available satellite images that other methods and have many additional advantages in terms of generalizability and interpretability.

2017

1. Learning Forest Wildfire Dynamics from Satellite Images Using Reinforcement Learning

   Sriram Ganapathi Subramanian, and Mark Crowley

   In Conference on Reinforcement Learning and Decision Making. Ann Arbor, MI, USA.. 2017. [24 citations]

2016

1. Anomaly Detection Using Inter-Arrival Curves for Real-time Systems

   Mahmoud Salem, Mark Crowley, and Sebastian Fischmeister.

   In 2016 28th Euromicro Conference on Real-Time Systems. Toulouse, France. Jul, 2016. [38 citations]

   Abs URL

   Abstract: —Real-time embedded systems are a significant class of applications, poised to grow even further as automated vehicles and the Internet of Things become a reality. An important problem for these systems is to detect anomalies during operation. Anomaly detection is a form of classification, which can be driven by data collected from the system at execution time. We propose inter-arrival curves as a novel analytic modelling technique for discrete event traces. Our approach relates to the existing technique of arrival curves and expands the technique to anomaly detection. Inter-arrival curves analyze the behaviour of events within a trace by providing upper and lower bounds to their inter-arrival occurrence. We exploit inter-arrival curves in a classification framework that detects deviations within these bounds for anomaly detection. Also, we show how inter-arrival curves act as good features to extract recurrent behaviour that these systems often exhibit. We demonstrate the feasibility and viability of the fully implemented approach with an industrial automotive case study (CAN traces) as well as a deployed aerospace case study (RTOS kernel traces).

2015

1. PAC Optimal MDP Planning with Application to Invasive Species Management

   Majid Alkaee Taleghan, Thomas G. Dietterich, Mark Crowley, Kim Hall, and H. Jo Albers.

   Journal of Machine Learning Research. 16, 2015. [29 citations]

   Abs URL

   Abstract: In a simulator-defined MDP, the Markovian dynamics and rewards are provided in the form of a simulator from which samples can be drawn. This paper studies MDP planning algorithms that attempt to minimize the number of simulator calls before terminating and outputting a policy that is approximately optimal with high probability. The paper introduces two heuristics for efficient exploration and an improved confidence interval that enables earlier termination with probabilis- tic guarantees. We prove that the heuristics and the confidence interval are sound and produce with high probability an approximately optimal policy in polynomial time. Experiments on two benchmark problems and two instances of an invasive species management problem show that the improved confidence intervals and the new search heuristics yield reductions of between 8% and 47% in the number of simulator calls required to reach near-optimal policies.

2014

2013

1. Allowing a wildfire to burn: Estimating the effect on future fire suppression costs

   Rachel M. Houtman, Claire A. Montgomery, Aaron R. Gagnon, David E. Calkin, Thomas G. Dietterich, Sean McGregor, and Mark Crowley

   International Journal of Wildland Fire. 22, (7). 2013. [116 citations]

   Abs PDF URL

   Abstract: Where a legacy of aggressive wildland fire suppression has left forests in need of fuel reduction, allowing wildland fire to burn may provide fuel treatment benefits, thereby reducing suppression costs from subsequent fires. The least-cost-plus-net-value-change model of wildland fire economics includes benefits of wildfire in a framework for evaluating suppression options. In this study, we estimated one component of that benefit – the expected present value of the reduction in suppression costs for subsequent fires arising from the fuel treatment effect of a current fire. To that end, we employed Monte Carlo methods to generate a set of scenarios for subsequent fire ignition and weather events, which are referred to as sample paths, for a study area in central Oregon. We simulated fire on the landscape over a 100-year time horizon using existing models of fire behaviour, vegetation and fuels development, and suppression effectiveness, and we estimated suppression costs using an existing suppression cost model. Our estimates suggest that the potential cost savings may be substantial. Further research is needed to estimate the full least-cost-plus-net-value-change model. This line of research will extend the set of tools available for developing wildfire management plans for forested landscapes.

2. PAC Optimal Planning for Invasive Species Management: Improved Exploration for Reinforcement Learning from Simulator-Defined MDPs

   Thomas G Dietterich, Majid Alkaee Taleghan, and Mark Crowley

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2013). Bellevue, WA, USA. 2013. [22 citations]

   Abs URL

   Abstract: Often the most practical way to define a Markov Decision Process (MDP) is as a simulator that, given a state and an action, produces a resulting state and immediate reward sampled from the corresponding distributions. Simulators in natural resource management can be very expensive to execute, so that the time required to solve such MDPs is dominated by the number of calls to the simulator. This paper presents an algorithm, DDV, that combines improved confidence intervals on the Q values (as in interval estimation) with a novel upper bound on the discounted state occupancy probabilities to intelligently choose state-action pairs to explore. We prove that this algorithm terminates with a policy whose value is within εof the optimal policy (with probability 1−δ) after making only polynomially-many calls to the simulator. Experiments on one benchmark MDP and on an MDP for invasive species management show very large reductions in the number of simulator calls required.

3. Cyclic causal models with discrete variables: Markov chain equilibrium semantics and sample ordering

   David Poole, and Mark Crowley

   In International Joint Conference on Artificial Intelligence (IJCAI). Beijing, China. 2013. [18 citations]

   Abs URL

   Abstract: We analyze the foundations of cyclic causal models for discrete variables, and compare structural equation models (SEMs) to an alternative semantics as the equilibrium (stationary) distribution of a Markov chain. We show under general conditions, discrete cyclic SEMs cannot have independent noise, even in the simplest case, cyclic structural equation models imply constraints on the noise. We give a formalization of an alternative Markov chain equilibrium semantics which requires not only the causal graph, but also a sample order. We show how the resulting equilibrium is a function of the sample ordering, both theoretically and empirically.

2012

2011

1. Policy gradient planning for environmental decision making with existing simulators

   Mark Crowley, and David Poole.

   In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence (AAAI). San Francisco. 2011. [14 citations]

   Abs URL

   Abstract: In environmental and natural resource planning domains actions are taken at a large number of locations over multiple time periods. These problems have enormous state and action spaces, spatial correlation between actions, uncertainty and complex utility models. We present an approach for modeling these planning problems as factored Markov decision processes. The reward model can contain local and global components as well as spatial constraints between locations. The transition dynamics can be provided by existing simulators developed by domain experts. We propose a landscape policy defined as the equilibrium distribution of a Markov chain built from many locally-parameterized policies. This policy is optimized using a policy gradient algorithm. Experiments using a forestry simulator demonstrate the algorithm’s ability to devise policies for sustainable harvest planning of a forest. Copyright \textcopyright 2011, Association for the Advancement of Artificial Intelligence. All rights reserved.

2010

2009

2007

2005