Mark Crowley | publications

Also see:

2021

Analysis of Language Embeddings for Classification of Unstructured Pathology Reports

Allada, Aishwarya Krishna, Wang, Yuanxin, Jindal, Veni, Babee, Morteza, Tizhoosh, H.R., and Crowley, Mark

In International Conference of the IEEE Engineering in Medicine and Biology Society 2021

Abs

A pathology report is one of the most significant medical documents providing interpretive insights into the visual appearance of the patient’s biopsy sample. In digital pathology, high-resolution images of tissue samples are stored along with pathology reports. Despite the valuable information that pathology reports hold, they are not used in any systematic manner to promote computational pathology. In this work, we focus on analyzing the reports, which are generally unstructured documents written in English with sophisticated and highly specialized medical terminology. We provide a comparative analysis of various embedding models like BioBERT, Clinical BioBERT, BioMed-RoBERTa and Term Frequency-Inverse Document Frequency (TF-IDF), a traditional NLP technique, as well as the combination of embeddings from pre-trained models with TF-IDF. Our results demonstrate the effectiveness of various word embedding techniques for pathology reports.
Magnification Generalization for Histopathology Image Embedding

Sikaroudi, Milad, Ghojogh, Benyamin, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

In IEEE International Symposium on Biomedical Imaging (ISBI) 2021

arXiv
Partially Observable Mean Field Reinforcement Learning

Ganapathi Subramanian, Sriram, Taylor, Matthew, Crowley, Mark, and Poupart, Pascal

In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems 2021

Abs arXiv

Traditional multi-agent reinforcement learning algorithms are not scalable to environments with more than a few agents, since these algorithms are exponential in the number of agents. Recent research has introduced successful methods to scale multi-agent reinforcement learning algorithms to many agent scenarios using mean field theory. Previous work in this field assumes that an agent has access to exact cumulative metrics regarding the mean field behaviour of the system, which it can then use to take its actions. In this paper, we relax this assumption and maintain a distribution to model the uncertainty regarding the mean field of the system. We consider two different settings for this problem. In the first setting, only agents in a fixed neighbourhood are visible, while in the second setting, the visibility of agents is determined at random based on distances. For each of these settings, we introduce a Q-learning based algorithm that can learn effectively. We prove that this Q-learning estimate stays very close to the Nash Q-value (under a common set of assumptions) for the first setting. We also empirically show our algorithms outperform multiple baselines in three different games in the MAgents framework, which supports large environments with many agents learning simultaneously to achieve possibly distinct goals.
Emotion Modelling for Search and Rescue (SAR) Robot to Human Communications

Akgun, Sami Alperen, Ghafurian, Moojan, Crowley, Mark, and Dautenhahn, Kerstin

TBD 2021
Batch-Incremental Triplet Sampling for Training Triplet Networks Using Bayesian Updating Theorem

Sikaroudi, Milad, Ghojogh, Benyamin, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

In 25th International Conference on Pattern Recognition (ICPR) 2021

PDF
Active Measure Reinforcement Learning for Observation Cost Minimization: A framework for minimizing measurement costs in reinforcement learning

Bellinger, Colin, Coles, Rory, Crowley, Mark, and Tamblyn, Isaac

In Canadian Conference on Artificial Intelligence 2021

Abs PDF

Markov Decision Processes (MDP) with explicit measurement cost are a class of en- vironments in which the agent learns to maximize the costed return. Here, we define the costed return as the discounted sum of rewards minus the sum of the explicit cost of measuring the next state. The RL agent can freely explore the relationship between actions and rewards but is charged each time it measures the next state. Thus, an op- timal agent must learn a policy without making a large number of measurements. We propose the active measure RL framework (Amrl) as a solution to this novel class of problem, and contrast it with standard reinforcement learning under full observability and planning under partially observability. We demonstrate that Amrl-Q agents learn to shift from a reliance on costly measurements to exploiting a learned transition model in order to reduce the number of real-world measurements and achieve a higher costed return. Our results demonstrate the superiority of Amrl-Q over standard RL methods, Q-learning and Dyna-Q, and POMCP for planning under a POMDP in environments with explicit measurement costs.

2020

Using Emotions to Complement Multi-Modal Human-Robot Interaction in Urban Search and Rescue Scenarios

Akgun, Sami Alperen, Ghafurian, Moojan, Crowley, Mark, and Dautenhahn, Kerstin

In Proceedings of the 2020 International Conference on Multimodal Interaction 2020

Abs

An experiment is presented to investigate whether there is consensus in mapping emotions to messages/situations in urban search and rescue scenarios, where efficiency and effectiveness of interactions are key to success. We studied mappings between 10 specific messages, presented in two different communication styles, reflecting common situations that might happen during search and rescue missions, and the emotions exhibited by robots in those situations. The data was obtained through a Mechanical Turk study with 78 participants. Our findings support the feasibility of using emotions as an additional communication channel to improve multi-modal human-robot interaction for urban search and rescue robots, and suggests that these mappings are robust, i.e. are not affected by the robot’s communication style.
Recognition of a Robot’s Affective Expressions under Conditions with Limited Visibility

Ghafurian, Moojan, Akgun, Sami Alperen, Crowley, Mark, and Dautenhahn, Kerstin

In International Conference on Human-Robot Interaction 2020

Abs

The capability of showing affective expressions is important for the design of social robots in many contexts, especially where the robot is designed to communicate with humans. It is reasonable to expect that, similar to all other interaction modalities, communicating with affective expressions is not without limitations. In this paper, we present two online video studies (N=72 and N=50) and investigate if/how much the recognition of affective displays of a zoomorphic robot is affected under situations with different levels of visibility. Five affective expressions combined with five visibility effects were studied. The intensity of the effects was more pronounced in the second experiment. While visual constraints affected recognition of expressions, our results showed that affective displays of the robot conveyed through its head and body gestures can be robust and recognition rates can still be high even under severe constraints. This study supported the effectiveness of using affective displays as a complementary communication modality in human-robot interaction in situations with low visibility.
Offline versus Online Triplet Mining based on Extreme Distances of Histopathology Patches

Sikaroudi, Milad, Ghojogh, Benyamin, Safarpoor, Amir, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

In 15th International Symposium on Visual Computing (ISCV 2020) 2020

Abs arXiv PDF

We analyze the effect of offline and online triplet mining for colorectal cancer (CRC) histopathology dataset containing 100,000 patches. We consider the extreme, i.e., farthest and nearest patches with respect to a given anchor, both in online and offline mining. While many works focus solely on how to select the triplets online (batch-wise), we also study the effect of extreme distances and neighbor patches before training in an offline fashion. We analyze the impacts of extreme cases for offline versus online mining, including easy positive, batch semi-hard, and batch hard triplet mining as well as the neighborhood component analysis loss, its proxy version, and distance weighted sampling. We also investigate online approaches based on extreme distance and comprehensively compare the performance of offline and online mining based on the data patterns and explain offline mining as a tractable generalization of the online mining with large mini-batch size. As well, we discuss the relations of different colorectal tissue types in terms of extreme distances. We found that offline mining can generate a better statistical representation of the population by working on the whole dataset.
Isolation Mondrian Forest for Batch and Online Anomaly Detection

Ma, Haoran, Ghojogh, Benyamin, Samad, Maria N, Zheng, Dongyu, and Crowley, Mark

In IEEE International Conference on Systems, Man, and Cybernetics (IEEE-SMC-2020) 2020
Multi-Level Collaborative Control System With Dual Neural Network Planning For Autonomous Vehicle Control In A Noisy Environment

Du, Zhiyuan, Lull, Joseph, Malhan, Rajesh, Ganapathi Subramanian, Sriram, Bhalla, Sushrut, Sambee, Jaspreet, and Crowley, Mark

2020
Quantile-Quantile Embedding for Distribution Transformation, Manifold Embedding, and Image Embedding with Choice of Embedding Distribution

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

Journal of the Optical Society of America A 2020
A review of machine learning applications in wildfire science and management

Jain, Piyush, Coogan, Sean CP, Ganapathi Subramanian, Sriram, Crowley, Mark, Taylor, Steve, and Flannigan, Mike D

Environmental Reviews 2020

arXiv PDF
Deep Multi Agent Reinforcement Learning for Autonomous Driving

Bhalla, Sushrut, Ganapathi Subramanian, Sriram, and Crowley, Mark

In Canadian Conference on Artificial Intelligence 2020
Supervision and Source Domain Impact on Representation Learning: A Histopathology Case Study

Sikaroudi, Milad, Safarpoor, Amir, Ghojogh, Benyamin, Shafiei, Sobhan, Crowley, Mark, and Tizhoosh, HR

In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’20) 2020

Abs

As many algorithms depend on a suitable representation of data, learning unique features is considered a crucial task. Although supervised techniques using deep neural networks have boosted the performance of representation learning, the need for a large set of labeled data limits the application of such methods. As an example, high-quality delineations of regions of interest in the field of pathology is a tedious and time-consuming task due to the large image dimensions. In this work, we explored the performance of a deep neural network and triplet loss in the area of representation learning. We investigated the notion of similarity and dissimilarity in pathology whole-slide images and compared different setups from unsupervised and semi-supervised to supervised learning in our experiments. Additionally, different approaches were tested, applying few-shot learning on two publicly available pathology image datasets. We achieved high accuracy and generalization when the learned representations were applied to two different pathology datasets
Distributed Nonlinear Model Predictive Control and Metric Learning for Heterogeneous Vehicle Platooning with Cut-in/Cut-out Maneuvers

Basiri, Mohammad Hossein, Ghojogh, Benyamin, Azad, Nasser L, Fischmeister, Sebastian, Karray, Fakhri, and Crowley, Mark

In Proceeding of the 59th IEEE Conference on Decision and Control (CDC-2020) 2020
Fisher Discriminant Triplet and Contrastive Losses for Training Siamese Networks

Ghojogh, Benyamin, Sikaroudi, Milad, Shafiei, Sobhan, Tizhoosh, H.R., Karray, Fakhri, and Crowley, Mark

In IEEE International Joint Conference on Neural Networks (IJCNN) 2020
Weighted Fisher Discriminant Analysis in the Input and Feature Spaces

Ghojogh, Benyamin, Sikaroudi, Milad, Tizhoosh, H.R., Karray, Fakhri, and Crowley, Mark

In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020
Backprojection for Training Feedforward Neural Networks in the Input and Feature Spaces

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020
Theoretical Insights into the Use of Structural Similarity Index In Generative Models and Inferential Autoencoders

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020
Generalized Subspace Learning by Roweis Discriminant Analysis

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020

Abs

We present a new method which generalizes subspace learning based on eigenvalue and generalized eigenvalue problems. This method, Roweis Discriminant Analysis (RDA), is named after Sam Roweis to whom the field of subspace learning owes significantly. RDA is a family of infinite number of algorithms where Principal Component Analysis (PCA), Supervised PCA (SPCA), and Fisher Discriminant Analysis (FDA) are special cases. One of the extreme special cases, which we name Double Supervised Discriminant Analysis (DSDA), uses the labels twice, it is novel and has not appeared elsewhere. We propose a dual for RDA for some special cases. We also propose kernel RDA, generalizing kernel PCA, kernel SPCA, and kernel FDA, using both dual RDA and representation theory. Our theoretical analysis explains previously known facts such as why SPCA can use regression but FDA cannot, why PCA and SPCA have duals but FDA does not, why kernel PCA and kernel SPCA use kernel trick but kernel FDA does not, and why PCA is the best linear method for reconstruction. Roweisfaces and kernel Roweisfaces are also proposed generalizing eigenfaces, Fisherfaces, supervised eigenfaces, and their kernel variants. We also report experiments showing the effectiveness of RDA and kernel RDA on some benchmark datasets.
Anomaly Detection and Prototype Selection Using Polyhedron Curvature

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

In Canadian Conference on Artificial Intelligence 2020
Reinforcement Learning in a Physics-Inspired Semi-Markov Environment

Bellinger, Colin, Coles, Rory, Crowley, Mark, and Tamblyn, Isaac

In Canadian Conference on Artificial Intelligence 2020

2019

Semantic Workflows and Machine Learning for the Assessment of Carbon Storage by Urban Trees

Carrillo, Juan, Garijo, Daniel, Crowley, Mark, Gil, Yolanda, and Borda, Katherine

In Third International Workshop on Capturing Scientific Knowledge (Sciknow 2019), Collocated with the tenth International Conference on Knowledge Capture (K-CAP) 2019
Proceedings of the Third International Workshop on Capturing Scientific Knowledge (SciKnow19)

2019
Comparison of Deep Learning models for Determining Road Surface Condition from Roadside Camera Images and Weather Data

Carrillo, J., Crowley, M., Pan, G., and Fu, L.

In TAC-ITS Canada Joint Conference 2019

Abs

Road maintenance during the Winter season is a safety critical and resource demanding operation. One of its key activities is determining road surface condition (RSC) in order to prioritize roads and allocate cleaning efforts such as plowing or salting. Two conventional approaches for determining RSC are: visual examination of roadside camera images by trained personnel and patrolling the roads to perform on-site inspections. However, with more than 500 cameras collecting images across Ontario, visual examination becomes a resource-intensive activity, difficult to scale especially during periods of snow storms. This paper presents the preliminary results of an ongoing study focused on improving the efficiency of road maintenance operations. We use multiple Deep Learning models to automatically determine RSC from roadside camera images and weather variables, extending previous research where similar methods have been used to deal with the problem. The dataset we use was collected during the 2017-2018 Winter season from 40 stations connected to the Ontario Road Weather Information System (RWIS), it includes 14.000 labelled images and 70.000 weather measurements. In particular, we train and evaluate the performance of seven state-of-the-art models from the Computer Vision literature, including the recent DenseNet, NASNet, and MobileNet. Also, by integrating observations from weather variables, the models are able to better ascertain RSC under poor visibility conditions.
Instance Ranking and Numerosity Reduction Using Matrix Decompositionand Subspace Learning

Ghojogh, Benyamin, and Crowley, Mark

In Canadian Conference on Artificial Intelligence 2019

Abs

One way to deal with the ever increasing amount of available data for processing is to rank data instances by usefulness and reduce the dataset size. In this work, we introduce a framework to achieve this using matrix decomposition and subspace learning. Our central contribution is a novel similarity measure for data instances that uses the basis obtained from matrix decomposition of the dataset. Using this similarity measure, we propose several related algorithms for ranking data instances and performing numerosity reduction. We then validate the effectiveness of these algorithms for data reduction on several datasets for classification, regression, and clustering tasks.
Learning Multi-Agent Communication with Reinforcement Learning

Bhalla, Sushrut, Ganapathi Subramanian, Sriram, and Crowley, Mark

In Conference on Reinforcement Learning and Decision Making (RLDM-19) 2019

Abs

Deep Learning and back-propagation have been successfully used to perform centralized training with communication protocols among multiple agents in a cooperative environment. In this paper, we present techniques for centralized training of Multi-Agent (Deep) Reinforcement Learning (MARL) using the model-free Deep Q-Network (DQN) as the baseline model and communication between agents. We present two novel, scalable and centralized MARL training techniques (MA-MeSN, MA-BoN), which separate the message learning module from the policy module. The separation of these modules helps in faster convergence in complex domains like autonomous driving simulators. A second contribution uses a memory module to achieve a decentralized cooperative policy for execution and thus addresses the challenges of noise and communication bottlenecks in real-time communication channels. This paper theoretically and empirically compares our centralized and decentralized training algorithms to current research in the field of MARL. We also present and release a new OpenAI-Gym environment which can be used for multi-agent research as it simulates multiple autonomous cars driving cooperatively on a highway. We compare the performance of our centralized algorithms to DIAL and IMS based on cumulative reward achieved per episode. MA-MeSN and MA-BoN achieve a cumulative reward of at-least }263\backslash%} of the reward achieved by the DIAL and IMS. We also present an ablation study of the scalability of MA-BoN and see a linear increase in inference time and number of trainable parameters compared to quadratic increase for DIAL.
Locally Linear Image Structural Embedding for Image Structure Manifold Learning

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

In International Conference on Image Analysis and Recognition (ICIAR-19) 2019
Image Structure Subspace Learning Using Structural Similarity Index

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

In International Conference on Image Analysis and Recognition (ICIAR-19) 2019
Principal Component Analysis Using Structural Similarity Index for Images

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

In International Conference on Image Analysis and Recognition (ICIAR-19) 2019
Artificial Counselor System For Stock Investment

Nekoei Qachkanloo, Hadi, Ghojogh, Benyamin, Pasand, Ali Saheb, and Crowley, Mark

2019
Compact Representation of a Multi-dimensional Combustion Manifold Using Deep Neural Networks

Bhalla, Sushrut, Yao, Matthew, Hickey, Jean-Pierre, and Crowley, Mark

2019

Abs PDF Poster Slides

The computational challenges in turbulent combustion simulations stem from the physical complexities and multi-scale nature of the problem which make it intractable to compute scaleesolving simulations. For most engineering applications, the large scale separation between the flame (typically submillimeter scale) and the characteristic turbulent flow (typically centimeter or meter scale) allows us to evoke simplifying assumptions–such as done for the flamelet model to precompute all the chemical reactions and map them to a low order manifold. The resulting manifold is then tabulated and looked up at runtime. As the physical complexity of combustion simulations increases (including radiation, soot formation, pressure variations etc.) the dimensionality of the resulting manifold grows which impedes an efficient tabulation and look up. In this paper we present a novel approach to model the multidimensional combustion manifold. We approximate the combustion manifold using a neural network function approximator and use it to predict the temperature and composition of the reaction. We present a novel training procedure which is developed to generate a smooth output curve for temperature over the course of a reaction. We then evaluate our work against the current approach of tabulation with linear interpolation in combustion simulations. We also provide an ablation study of our training procedure in the context of overfitting in our model. The combustion dataset used for the modeling of combustion of H2 and O2 in this work is released alongside this paper.
Training Cooperative Agents for Multi-Agent Reinforcement Learning

Bhalla, Sushrut, Ganapathi Subramanian, Sriram, and Crowley, Mark

In Proc. of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019) 2019

Abs

Deep Learning and back-propagation has been successfully used to perform centralized training with communication protocols among multiple agents in a cooperative environment. In this paper we present techniques for centralized training of Multi-Agent (Deep) Reinforcement Learning (MARL) using the model-free Deep Q-Network as the baseline model and message sharing between agents. We present a novel, scalable, centralized MARL training technique, which separates the message learning module from the policy module. The separation of these modules helps in faster convergence in complex domains like autonomous driving simulators. A second contribution uses the centrally trained model to bootstrap training of distributed, independent, cooperative agent policies for execution and thus addresses the challenges of noise and communication bottlenecks in real-time communication channels. This paper theoretically and empirically compares our centralized training algorithms to current research in the field of MARL. We also present and release a new OpenAI-Gym environment which can be used for multi-agent research as it simulates multiple autonomous cars driving cooperatively on a highway.
Integration of Roadside Camera Images and Weather Data for monitoring Winter Road Surface Conditions

Carrillo, Juan, and Crowley, Mark

2019

Abs

Background/Context: During the Winter season, real-time monitoring of road surface conditions is critical for the safety of drivers and road maintenance operations. Previous research has evaluated the potential of image classification methods for detecting road snow coverage by processing images from roadside cameras installed in RWIS (Road Weather Information System) stations. However, it is a challenging task due to limitations such as image resolution, camera angle, and illumination. Two common approaches to improve the accuracy of image classification methods are: adding more input features to the model and increasing the number of samples in the training dataset. Additional input features can be weather variables and more sample images can be added by including other roadside cameras. Although RWIS stations are equipped with both cameras and weather measurement instruments, they are only a subset of the total number of roadside cameras installed across transportation networks, most of which do not have weather measurement instruments. Thus, improvements in use of image data could benefit from additional data sources. Aims/Objectives: The first objective of this study is to complete an exploratory data analysis over three data sources in Ontario: RWIS stations, all the other MTO (Ministry of Transportation of Ontario) roadside cameras, and Environment Canada weather stations. The second objective is to determine the feasibility of integrating these three datasets into a more extensive and richer dataset with weather variables as additional features and other MTO roadside cameras as additional sources of images. Methods/Targets: First, we quantify the advantage of adding other MTO roadside cameras using spatial statistics, the number of monitored roads, and the coverage of ecoregions with different climate regimes. We then analyze experimental variograms from the literature and determine the feasibility of using Environment Canada stations and RWIS stations to interpolate weather variables for all the other MTO roadside cameras without weather instruments. Results/Activities: By adding all other MTO cameras as image data sources, the total number of cameras in the dataset increases from 139 to 578 across Ontario. The average distance to the nearest camera decreases from 38.4km to 9.4km, and the number of monitored roads increases approximately four times. Additionally, six times more cameras are available in the four most populated ecoregions in Ontario. The experimental variograms show that it is feasible to interpolate weather variables with reasonable accuracy. Moreover, observations in the three datasets are collected with similar frequency, which facilitates our data integration approach. Discussion/Deliverables: Integrating these three datasets is feasible and can benefit the design and development of automated image classification methods for monitoring road snow coverage. We do not consider data from pavement-embedded sensors, an additional line of research may explore the integration of this data. Our approach can provide actionable insights which can be used to more selectively perform manual patrolling to better identify road surface conditions. Conclusions: Our initial results are promising and demonstrate that additional, image only datasets can be added to road monitoring data by using existing multimodal sensors as ground truth, which will lead to greater performance on the future image classification tasks.

2018

A Complementary Approach to Improve WildFire Prediction Systems.

Subramanian, Sriram Ganapathi, and Crowley, Mark

In Neural Information Processing Systems (AI for social good workshop) 2018
Combining MCTS and A3C for prediction of spatially spreading processes in forest wildfire settings

Ganapathi Subramanian, Sriram, and Crowley, Mark

In Canadian Conference on Artificial Intelligence 2018

Abs

In recent years, Deep Reinforcement Learning (RL) algorithms have shown super-human performance in a variety Atari and classic board games like chess and GO. Research into applications of RL in other domains with spatial considerations like environmental planning are still in their nascent stages. In this paper, we introduce a novel combination of Monte-Carlo Tree Search (MCTS) and A3C algorithms on an online simulator of a wildfire, on a pair of forest fires in Northern Alberta (Fort McMurray and Richardson fires) and on historical Saskatchewan fires previously compared by others to a physics-based simulator. We conduct several experiments to predict fire spread for several days before and after the given spatial information of fire spread and ignition points. Our results show that the advancements in Deep RL applications in the gaming world have advantages in spatially spreading real-world problems like forest fires. \textcopyright Springer International Publishing AG, part of Springer Nature 2018.
Principal Sample Analysis for Data Reduction

Ghojogh, Benyamin, and Crowley, Mark

In 2018 IEEE International Conference on Big Knowledge (ICBK) 2018
Blue Sky Ideas in Artificial Intelligence Education from the EAAI 2017 New and Future AI Educator Program

Eaton, Eric, Koenig, Sven, Schulz, Claudia, Maurelli, Francesco, Lee, John, Eckroth, Joshua, Crowley, Mark, Freedman, Richard G, Cardona-Rivera, Rogelio E, Machado, Tiago, and Williams, Tom

2018
Decision Assist for Self-Driving Cars

Ganapathi Subramanian, Sriram, Sambee, Jaspreet Singh, Ghojogh, Benyamin, and Crowley, Mark

In Canadian Conference on Artificial Intelligence 2018

Abs

Research into self-driving cars has grown enormously in the last decade primarily due to the advances in the fields of machine intelligence and image processing. An under-appreciated aspect of self-driving cars is actively avoiding high traffic zones, low visibility zones, and routes with rough weather conditions by learning different conditions and making decisions based on trained experiences. This paper addresses this challenge by introducing a novel hierarchical structure for dynamic path planning and experiential learning for vehicles. A multistage system is proposed for detecting and compensating for weather, lighting, and traffic conditions as well as a novel adaptive path planning algorithm named Checked State A3C. This algorithm improves upon the existing A3C Reinforcement Learning (RL) algorithm by adding state memory which provides the ability to learn an adaptive model of the best decisions to take from experience. \textcopyright Springer International Publishing AG, part of Springer Nature 2018.
Using Spatial Reinforcement Learning to Build Forest Wildfire Dynamics Models From Satellite Images

Ganapathi Subramanian, Sriram, and Crowley, Mark

Frontiers in ICT 2018

Abs

Machine learning algorithms have increased tremendously in power in recent years but have yet to be fully utilized in many ecology and sustainable resource management domains such as wildlife reserve design, forest fire management and invasive species spread. One thing these domains have in common is that they contain dynamics that can be characterized as a Spatially Spreading Process (SSP) which requires many parameters to be set precisely to model the dynamics, spread rates and directional biases of the elements which are spreading. We present related work in Artificial Intelligence and Machine Learning for SSP sustainability domains including forest wildfire prediction. We then introduce a novel approach for learning in SSP domains using Reinforcement Learning (RL) where fire is the agent at any cell in the landscape and the set of actions the fire can take from a location at any point in time includes spreading North, South, East, West or not spreading. This approach inverts the usual RL setup since the dynamics of the corresponding Markov Decision Process (MDP) is a known function for immediate wildfire spread. Meanwhile, we learn an agent policy for a predictive model of the dynamics of a complex spatially-spreading process. Rewards are provided for correctly classifying which cells are on fire or not compared to satellite and other related data. We examine the behaviour of five RL algorithms on this problem: Value Iteration, Policy Iteration, Q-Learning, Monte Carlo Tree Search and Asynchronous Advantage Actor-Critic (A3C). We compare to a Gaussian process based supervised learning approach and discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We also discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We validate our approach with satellite image data of two massive wildfire events in Northern Alberta, Canada, the Fort McMurray fire of 2016 and the Richardson fire of 2011. The results show that we can learn predictive, agent-based policies as models of spatial dynamics using RL on readily available satellite images that other methods and have many additional advantages in terms of generalizability and interpretability.

2017

Learning Forest Wildfire Dynamics from Satellite Images Using Reinforcement Learning

Subramanian, Sriram Ganapathi, and Crowley, Mark

In Conference on Reinforcement Learning and Decision Making 2017
Big Metadata : Machine Learning on Encrypted Communications

Fernick, Jennifer, and Crowley, Mark

In RSA Conference 2017
Application of probabilistically-weighted graphs to image-based diagnosis of Alzheimer’s disease using diffusion MRI

Maryam, Syeda, McCrackin, Laura, Crowley, Mark, Rathi, Yogesh, and Michailovich, Oleg

In Proceedings of SPIE 101324, Medical Imaging 2017 : Computer-Aided Diagnosis 2017
AI Education Through Real World Problems

Crowley, Mark

In The Seventh Symposium on Educational Advances in Artificial Intellgience. 2017

2016

Inter-Arrival Curves for Multi-Mode and Online Anomaly Detection

Salem, Mahmoud, Crowley, Mark, and Fischmeister, Sebastian

In Euromicro Conference on Real-Time Systems 2016 - Work-in-Progress Proceedings 2016
Anomaly Detection Using Inter-Arrival Curves for Real-time Systems

Salem, Mahmoud, Crowley, Mark, and Fischmeister, Sebastian

In 2016 28th Euromicro Conference on Real-Time Systems 2016

Abs

—Real-time embedded systems are a significant class of applications, poised to grow even further as automated vehicles and the Internet of Things become a reality. An important problem for these systems is to detect anomalies during operation. Anomaly detection is a form of classification, which can be driven by data collected from the system at execution time. We propose inter-arrival curves as a novel analytic modelling technique for discrete event traces. Our approach relates to the existing technique of arrival curves and expands the technique to anomaly detection. Inter-arrival curves analyze the behaviour of events within a trace by providing upper and lower bounds to their inter-arrival occurrence. We exploit inter-arrival curves in a classification framework that detects deviations within these bounds for anomaly detection. Also, we show how inter-arrival curves act as good features to extract recurrent behaviour that these systems often exhibit. We demonstrate the feasibility and viability of the fully implemented approach with an industrial automotive case study (CAN traces) as well as a deployed aerospace case study (RTOS kernel traces).

2015

PAC Optimal MDP Planning with Application to Invasive Species Management

Taleghan, Majid Alkaee, Dietterich, Thomas G., Crowley, Mark, Hall, Kim, and Albers, H. Jo

Journal of Machine Learning Research 2015

Abs

In a simulator-defined MDP, the Markovian dynamics and rewards are provided in the form of a simulator from which samples can be drawn. This paper studies MDP planning algorithms that attempt to minimize the number of simulator calls before terminating and outputting a policy that is approximately optimal with high probability. The paper introduces two heuristics for efficient exploration and an improved confidence interval that enables earlier termination with probabilis- tic guarantees. We prove that the heuristics and the confidence interval are sound and produce with high probability an approximately optimal policy in polynomial time. Experiments on two benchmark problems and two instances of an invasive species management problem show that the improved confidence intervals and the new search heuristics yield reductions of between 8% and 47% in the number of simulator calls required to reach near-optimal policies.
Answering Simple Questions About Spatially Spreading Systems

Crowley, Mark

In 2015 Summer Solstice: 7th International Conference on Discrete Models of Complex Systems 2015

2014

Using equilibrium policy gradients for spatiotemporal planning in forest ecosystem management

Crowley, Mark

IEEE Transactions on Computers 2014

Abs

Spatiotemporal planning involves making choices at multiple locations in space over some planning horizon to maximize utility and satisfy various constraints. In Forest Ecosystem Management, the problem is to choose actions for thousands of locations each year including harvesting, treating trees for fire or pests, or doing nothing. The utility models could place value on sale of lumber, ecosystem sustainability or employment levels and incorporate legal and logistical constraints on actions such as avoiding large contiguous areas of clearcutting. Simulators developed by forestry researchers provide detailed dynamics but are generally inaccesible black boxes. We model spatiotemporal planning as a factored Markov decision process and present a policy gradient planning algorithm to optimize a stochastic spatial policy using simulated dynamics. It is common in environmental and resource planning to have actions at different locations be spatially interelated, this makes representation and planning challenging. We define a global spatial policy in terms of interacting local policies defining distributions over actions at each location conditioned on actions at nearby locations. Markov chain Monte Carlo simulation is used to sample landscape policies and estimate their gradients. Evaluation is carried out on a forestry planning problem with 1,880 locations using a variety of value models and constraints. Index

2014

Using equilibrium policy gradients for spatiotemporal planning in forest ecosystem management

Crowley, Mark

IEEE Transactions on Computers 2014

Abs

Spatiotemporal planning involves making choices at multiple locations in space over some planning horizon to maximize utility and satisfy various constraints. In Forest Ecosystem Management, the problem is to choose actions for thousands of locations each year including harvesting, treating trees for fire or pests, or doing nothing. The utility models could place value on sale of lumber, ecosystem sustainability or employment levels and incorporate legal and logistical constraints on actions such as avoiding large contiguous areas of clearcutting. Simulators developed by forestry researchers provide detailed dynamics but are generally inaccesible black boxes. We model spatiotemporal planning as a factored Markov decision process and present a policy gradient planning algorithm to optimize a stochastic spatial policy using simulated dynamics. It is common in environmental and resource planning to have actions at different locations be spatially interelated, this makes representation and planning challenging. We define a global spatial policy in terms of interacting local policies defining distributions over actions at each location conditioned on actions at nearby locations. Markov chain Monte Carlo simulation is used to sample landscape policies and estimate their gradients. Evaluation is carried out on a forestry planning problem with 1,880 locations using a variety of value models and constraints. Index

2013

Allowing a wildfire to burn: Estimating the effect on future fire suppression costs

Houtman, Rachel M., Montgomery, Claire A., Gagnon, Aaron R., Calkin, David E., Dietterich, Thomas G., McGregor, Sean, and Crowley, Mark

International Journal of Wildland Fire 2013

Abs

Where a legacy of aggressive wildland fire suppression has left forests in need of fuel reduction, allowing wildland fire to burn may provide fuel treatment benefits, thereby reducing suppression costs from subsequent fires. The least-cost-plus-net-value-change model of wildland fire economics includes benefits of wildfire in a framework for evaluating suppression options. In this study, we estimated one component of that benefit – the expected present value of the reduction in suppression costs for subsequent fires arising from the fuel treatment effect of a current fire. To that end, we employed Monte Carlo methods to generate a set of scenarios for subsequent fire ignition and weather events, which are referred to as sample paths, for a study area in central Oregon. We simulated fire on the landscape over a 100-year time horizon using existing models of fire behaviour, vegetation and fuels development, and suppression effectiveness, and we estimated suppression costs using an existing suppression cost model. Our estimates suggest that the potential cost savings may be substantial. Further research is needed to estimate the full least-cost-plus-net-value-change model. This line of research will extend the set of tools available for developing wildfire management plans for forested landscapes.
PAC Optimal Planning for Invasive Species Management: Improved Exploration for Reinforcement Learning from Simulator-Defined MDPs

Dietterich, Thomas G, Alkaee Taleghan, Majid, and Crowley, Mark

In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2013) 2013

Abs

Often the most practical way to define a Markov Decision Process (MDP) is as a simulator that, given a state and an action, produces a resulting state and immediate reward sam- pled from the corresponding distributions. Simulators in nat- ural resource management can be very expensive to execute, so that the time required to solve such MDPs is dominated by the number of calls to the simulator. This paper presents an algorithm, DDV, that combines improved confidence inter- vals on the Q values (as in interval estimation) with a novel upper bound on the discounted state occupancy probabilities to intelligently choose state-action pairs to explore. We prove that this algorithm terminates with a policy whose value is within εof the optimal policy (with probability 1−δ) after making only polynomially-many calls to the simulator. Ex- periments on one benchmark MDP and on an MDP for inva- sive species management show very large reductions in the number of simulator calls required.
Cyclic causal models with discrete variables: Markov chain equilibrium semantics and sample ordering

Poole, David, and Crowley, Mark

In IJCAI International Joint Conference on Artificial Intelligence 2013

Abs

We analyze the foundations of cyclic causal models for discrete variables, and compare structural equation models (SEMs) to an alternative semantics as the equilibrium (stationary) distribution of a Markov chain. We show under general conditions, discrete cyclic SEMs cannot have independent noise, even in the simplest case, cyclic structural equation models imply constraints on the noise. We give a formalization of an alternative Markov chain equilibrium semantics which requires not only the causal graph, but also a sample order. We show how the resulting equilibrium is a function of the sample ordering, both theoretically and empirically.
Policy Gradient Optimization Using Equilibrium Policies for Spatial Planning Domains

Crowley, Mark

In 13th INFORMS Computing Society Conference 2013

2012

Managing Invasive Species in a River Network

Hall, Kim, Alkaee Taleghan, Majid, Albers, Heidi J., Dietterich, Thomas, and Crowley, Mark

In Third International Conference on Computational Sustainability 2012

Abs

There are many pest species that invade valuable ecosystems and damage ecosystem services. An important management goal is to limit the spread of these species or even eradicate them. From the computational perspective, these problems can be formulated as Markov Decision Processes. However, solving them is difficult, because of their spatial nature. Viewed abstractly, the current state of the ecosystem can be formalized as a graph. Each node of the graph may be invaded or not, and at each time step, a “local” action must be selected at each node. Consequently, at the level of the whole MDP, each global action is a vector of local actions of length N, the number of nodes in the graph. The exponentially large action space raises major computational problems. During the state transitions, the invasive species can spread along the edges of the graph. We study a specific instance of this general problem based on the spread of Tamarisk in river networks. In this formulation, we work with the dual of the standard river network graph. Each node corresponds to a stretch of the river between two confluences (also called a “reach”). We divide each reach into H “slots”, where each slot can either be occupied by a single native tree, occupied by a single Tamarisk tree, or empty. In each time step, the plants (both native and invader) first must survive through winter (with some probability). Then, the surviving trees produce seeds that disperse through the river network according to a dispersal kernel that prefers downstream movement with high probability. Each dispersing seed lands at some slot in some node of the network. All seeds arriving at a slot compete to establish themselves in that slot (if the slot is empty). In our model, there are three local actions: simple eradication of invasive plants, eradication followed by planting of native plants, or doing nothing at all. Doing nothing costs nothing, while eradication is more expensive, and eradication plus planting is the most expensive. One local action must be chosen for each node (reach) at each time step, and it applies to all H slots at that node. The actions only succeed stochastically. The planning goal is to minimize the expected discounted sum of the cost of invasive damage and the cost of management actions. To solve this planning problem, we apply the standard value iteration algorithm. However, standard value iteration requires a fully-specified state transition matrix. In our problem, it is easy to write a simulator for drawing samples of the dispersal and competition processes, but it is computationally intractable to compute the exact transition probabilities. Hence, we estimate those probabilities by drawing a large number of samples from the simulator. Once we have an optimal policy on the approximated MDP, we explore its behaviour both directly and via simulated rollouts. The optimal policy can be visualized with one picture per state annotated by the types of plants in each reach and the optimal action that should be taken in each reach. We also run the optimal policy forward in time on many simulated trajectories in order to gather statistics about its behaviour such as: how long it takes to reach a steady state, average cost of a policy over time, or the frequency with which completely invaded or uninvaded states are reached. Comparisons of these statistics show that in many situations our optimal policies greatly outperform established rules of thumb from the literature. An example of a rule of thumb is to always prioritize treatment of upstream over downstream reaches. Another rule targets the most invaded reaches (those with the most slots occupied by Tamarisk) first. We have found that the optimality of these rules of thumb depends strongly on the dispersal and competition parameters. Some contributions this work makes to bioeconomics and ecology are showing evidence that the stochastic spread of the invasion process needs to be modelled and that the spatial characteristics of the system under invasion are relevant to optimal management. The computational challenges raised by this problem have led us into research on methods for planning that can provide bounds on the optimality of the policy while minimizing the number of calls to expensive simulators. We are also studying algorithms that can scale to large graphs with thousands of nodes and large out-degrees. Finally, we see potential in these problems for intelligently reducing the size of the state space and the complexity of the policy description by utilizing domain knowledge.

2011

Policy gradient planning for environmental decision making with existing simulators

Crowley, Mark, and Poole, David

In Proceedings of the National Conference on Artificial Intelligence 2011

Abs

In environmental and natural resource planning domains actions are taken at a large number of locations over multiple time periods. These problems have enormous state and action spaces, spatial correlation between actions, uncertainty and complex utility models. We present an approach for modeling these planning problems as factored Markov decision processes. The reward model can contain local and global components as well as spatial constraints between locations. The transition dynamics can be provided by existing simulators developed by domain experts. We propose a landscape policy defined as the equilibrium distribution of a Markov chain built from many locally-parameterized policies. This policy is optimized using a policy gradient algorithm. Experiments using a forestry simulator demonstrate the algorithm’s ability to devise policies for sustainable harvest planning of a forest. Copyright \textcopyright 2011, Association for the Advancement of Artificial Intelligence. All rights reserved.
Equilibrium Policy Gradients for Spatiotemporal Planning

Crowley, Mark

2011

Abs

In spatiotemporal planning, agents choose actions at multiple locations in space over some planning horizon to maximize their utility and satisfy various constraints. In forestry planning, for example, the problem is to choose actions for thousands of locations in the forest each year. The actions at each location could include harvesting trees, treating trees against disease and pests, or doing nothing. A utility model could place value on sale of forest products, ecosystem sustainability or employment levels, and could incorporate legal and logistical constraints such as avoiding large contiguous areas of clearcutting and managing road access. Planning requires a model of the dynamics. Existing simulators developed by forestry researchers can provide detailed models of the dynamics of a forest over time, but these simulators are often not designed for use in automated planning. This thesis presents spatiotemoral planning in terms of factored Markov decision processes. A policy gradient planning algorithm optimizes a stochastic spatial policy using existing simulators for dynamics. When a planning problem includes spatial interaction between locations, deciding on an action to carry out at one location requires considering the actions performed at other locations. This spatial interdependence is common in forestry and other environmental planning problems and makes policy representation and planning challenging. We define a spatial policy in terms of local policies defined as distributions over actions at one location conditioned upon actions at other locations. A policy gradient planning algorithm using this spatial policy is presented which uses Markov Chain Monte Carlo simulation to sample the landscape policy, estimate its gradient and use this gradient to guide policy improvement. Evaluation is carried out on a forestry planning problem with 1880 locations using a variety of value models and constraints. The distribution over joint actions at all locations can be seen as the equilibrium of a cyclic causal model. This equilibrium semantics is compared to Structural Equation Models. We also define an algorithm for approximating the equilibrium distribution for cyclic causal networks which exploits graphical structure and analyse when the algorithm is exact.

2010

Circuits and logic in the lab : Toward a coherent picture of computation

Patitsas, Elizabeth, Voll, Kimberly, Crowley, Mark, and Wolfman, Steven

In 15th Western Canadian Conference on Computing Education 2010

Abs

We describe extensive modifications made over time to a first year computer science course at the University of British Columbia covering logic and digital circuits (among other topics). Smoothly integrating the hardware-based labs with the more theory-based lectures into a cohesive picture of computation has always been a challenge in this course. The seeming disconnect between implementation and abstraction has historically led to frustration and dissatisfaction among students. We describe changes to the lab curriculum, equipment logistics, the style of in-lab activities and evaluation. We have also made logistical changes to the management and ongoing training of teaching assistants, allowing us to better anchor our larger course story into the lab curriculum. These changes have greatly improved student and TA opinions of the lab experience, as well as the overall course.

2009

Seeing the Forest Despite the Trees : Large Scale Spatial-Temporal Decision Making

Crowley, Mark, Nelson, John, and Poole, David

In Conference on Uncertainty in Artificial Intelligence (UAI09) 2009

Abs

We introduce a challenging real world planning problem where actions must be taken at each location in a spatial area at each point in time. We use forestry planning as the motivating application. In Large Scale Spatial-Temporal (LSST) planning problems, the state and action spaces are defined as the cross-products of many local state and action spaces spread over a large spatial area such as a city or forest. These problems possess state uncertainty, have complex utility functions involving spatial constraints and we generally must rely on simulations rather than an explicit transition model. We define LSST problems as reinforcement learning prob- lems and present a solution using policy gradients. We compare two different policy formulations: an explicit policy that identifies each location in space and the action to take there, and an abstract policy that defines the proportion of actions to take across all locations in space. We show that the abstract policy is more robust and achieves higher rewards with far fewer parameters than the elementary policy. This abstract policy is also a better fit to the properties that practitioners in LSST problem domains require for such methods to be widely useful