publications

2021

1. Generative locally linear embedding: A module for manifold unfolding and visualization

   Ghojogh, Benyamin, Ghodsi, Ali, Karray, Fakhri, and Crowley, Mark

   Software Impacts 2021

   Abs PDF

   Data often have nonlinear patterns in machine learning. One can unfold the nonlinear manifold of a dataset for low-dimensional visualization and feature extraction. Locally Linear Embedding (LLE) is a nonlinear spectral method for dimensionality reduction and manifold unfolding. It embeds data using the same linear reconstruction weights as in the input space. In this paper, we propose an open source module which not only implements LLE, but also includes implementations of two generative LLE algorithms whose linear reconstruction phases are stochastic. Using this module, one can generate as many manifold unfoldings as desired for data visualization or feature extraction.

2. Acceleration of Large Margin Metric Learning for Nearest Neighbor Classification Using Triplet Mining and Stratified Sampling

   Poorheravi*, Parisa, Ghojogh*, Benyamin, Gaudet, Vincent, Karray, Fakhri, and Crowley, Mark

   Journal of Computational Vision and Imaging Systems 2021

   Abs

   Metric learning is a technique in manifold learning to find a projection subspace for increasing and decreasing the inter- and intra-class variances, respectively. Some metric learning methods are based on triplet learning with anchor-positive-negative triplets. Large margin metric learning for nearest neighbor classification is one of the fundamental methods to do this. Recently, Siamese networks have been introduced with the triplet loss. Many triplet mining methods have been developed for Siamese nets; however, these techniques have not been applied on the triplets of large margin metric learning. In this work, inspired by the mining methods for Siamese nets, we propose several triplet mining techniques for large margin metric learning. Moreover, a hierarchical approach is proposed, for acceleration and scalability of optimization, where triplets are selected by stratified sampling in hierarchical hyper-spheres. We analyze the proposed methods on three publicly available datasets.

3. Integrating Affective Expressions into the Search and Rescue Context in order to Improve Non-Verbal Human-Robot Interaction

   Akgun, Sami Alperen, Ghafurian, Moojan, Crowley, Mark, and Dautenhahn, Kerstin

   In Workshop on Exploring Applications for Autonomous Non-Verbal Human-Robot Interactions (HRI) 2021
4. Magnification Generalization for Histopathology Image Embedding

   Sikaroudi, Milad, Ghojogh, Benyamin, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

   In IEEE International Symposium on Biomedical Imaging (ISBI) 2021

   Abs arXiv

   Histopathology image embedding is an active research area in computer vision. Most of the embedding models exclusively concentrate on a specific magnification level. However, a useful task in histopathology embedding is to train an embedding space regardless of the magnification level. Two main approaches for tackling this goal are domain adaptation and domain generalization, where the target magnification levels may or may not be introduced to the model in training, respectively. Although magnification adaptation is a well-studied topic in the literature, this paper, to the best of our knowledge, is the first work on magnification generalization for histopathology image embedding. We use an episodic trainable domain generalization technique for magnification generalization, namely Model Agnostic Learning of Semantic Features (MASF), which works based on the Model Agnostic Meta-Learning (MAML) concept. Our experimental results on a breast cancer histopathology dataset with four different magnification levels show the proposed method’s effectiveness for magnification generalization.

5. Partially Observable Mean Field Reinforcement Learning

   Ganapathi Subramanian, Sriram, Taylor, Matthew, Crowley, Mark, and Poupart, Pascal

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2021

   Abs arXiv PDF

   Traditional multi-agent reinforcement learning algorithms are not scalable to environments with more than a few agents, since these algorithms are exponential in the number of agents. Recent research has introduced successful methods to scale multi-agent reinforcement learning algorithms to many agent scenarios using mean field theory. Previous work in this field assumes that an agent has access to exact cumulative metrics regarding the mean field behaviour of the system, which it can then use to take its actions. In this paper, we relax this assumption and maintain a distribution to model the uncertainty regarding the mean field of the system. We consider two different settings for this problem. In the first setting, only agents in a fixed neighbourhood are visible, while in the second setting, the visibility of agents is determined at random based on distances. For each of these settings, we introduce a Q-learning based algorithm that can learn effectively. We prove that this Q-learning estimate stays very close to the Nash Q-value (under a common set of assumptions) for the first setting. We also empirically show our algorithms outperform multiple baselines in three different games in the MAgents framework, which supports large environments with many agents learning simultaneously to achieve possibly distinct goals.

6. Batch-Incremental Triplet Sampling for Training Triplet Networks Using Bayesian Updating Theorem

   Sikaroudi, Milad, Ghojogh, Benyamin, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

   In 25th International Conference on Pattern Recognition (ICPR) 2021

   Abs arXiv PDF

   Variants of Triplet networks are robust entities for learning a discriminative embedding subspace. There exist different triplet mining approaches for selecting the most suitable training triplets. Some of these mining methods rely on the extreme distances between instances, and some others make use of sampling. However, sampling from stochastic distributions of data rather than sampling merely from the existing embedding instances can provide more discriminative information. In this work, we sample triplets from distributions of data rather than from existing instances. We consider a multivariate normal distribution for the embedding of each class. Using Bayesian updating and conjugate priors, we update the distributions of classes dynamically by receiving the new mini-batches of training data. The proposed triplet mining with Bayesian updating can be used with any triplet-based loss function, e.g., triplet-loss or Neighborhood Component Analysis (NCA) loss. Accordingly, Our triplet mining approaches are called Bayesian Updating Triplet (BUT) and Bayesian Updating NCA (BUNCA), depending on which loss function is being used. Experimental results on two public datasets, namely MNIST and histopathology colorectal cancer (CRC), substantiate the effectiveness of the proposed triplet mining method.

7. Active Measure Reinforcement Learning for Observation Cost Minimization: A framework for minimizing measurement costs in reinforcement learning

   Bellinger, Colin, Coles, Rory, Crowley, Mark, and Tamblyn, Isaac

   In Canadian Conference on Artificial Intelligence 2021

   Abs arXiv PDF

   Markov Decision Processes (MDP) with explicit measurement cost are a class of en- vironments in which the agent learns to maximize the costed return. Here, we define the costed return as the discounted sum of rewards minus the sum of the explicit cost of measuring the next state. The RL agent can freely explore the relationship between actions and rewards but is charged each time it measures the next state. Thus, an op- timal agent must learn a policy without making a large number of measurements. We propose the active measure RL framework (Amrl) as a solution to this novel class of problem, and contrast it with standard reinforcement learning under full observability and planning under partially observability. We demonstrate that Amrl-Q agents learn to shift from a reliance on costly measurements to exploiting a learned transition model in order to reduce the number of real-world measurements and achieve a higher costed return. Our results demonstrate the superiority of Amrl-Q over standard RL methods, Q-learning and Dyna-Q, and POMCP for planning under a POMDP in environments with explicit measurement costs.

8. Recognition of a Robot’s Affective Expressions under Conditions with Limited Visibility

   Ghafurian, Moojan, Akgun, Sami Alperen, Crowley, Mark, and Dautenhahn, Kerstin

   In 18th International Conference promoted by the IFIP Technical Committee 13 on Human–Computer Interaction (INTERACT 2021) 2021

   Abs PDF

   The capability of showing affective expressions is important for the design of social robots in many contexts, especially where the robot is designed to communicate with humans. It is reasonable to expect that, similar to all other interaction modalities, communicating with affective expressions is not without limitations. In this paper, we present two online video studies (N=72 and N=50) and investigate if/how much the recognition of affective displays of a zoomorphic robot is affected under situations with different levels of visibility. Five affective expressions combined with five visibility effects were studied. The intensity of the effects was more pronounced in the second experiment. While visual constraints affected recognition of expressions, our results showed that affective displays of the robot conveyed through its head and body gestures can be robust and recognition rates can still be high even under severe constraints. This study supported the effectiveness of using affective displays as a complementary communication modality in human-robot interaction in situations with low visibility.

9. Quantile–Quantile Embedding for distribution transformation and manifold embedding with ability to choose the embedding distribution

   Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

   Machine Learning with Applications (MLWA) 2021

   Abs arXiv PDF

   We propose a new embedding method, named Quantile-Quantile Embedding (QQE), for distribution transformation and manifold embedding with the ability to choose the embedding distribution. QQE, which uses the concept of quantile-quantile plot from visual statistical tests, can transform the distribution of data to any theoretical desired distribution or empirical reference sample. Moreover, QQE gives the user a choice of embedding distribution in embedding the manifold of data into the low dimensional embedding space. It can also be used for modifying the embedding distribution of other dimensionality reduction methods, such as PCA, t-SNE, and deep metric learning, for better representation or visualization of data. We propose QQE in both unsupervised and supervised forms. QQE can also transform a distribution to either an exact reference distribution or its shape. We show that QQE allows for better discrimination of classes in some cases. Our experiments on different synthetic and image datasets show the effectiveness of the proposed embedding method.

2020

1. Using Emotions to Complement Multi-Modal Human-Robot Interaction in Urban Search and Rescue Scenarios

   Akgun, Sami Alperen, Ghafurian, Moojan, Crowley, Mark, and Dautenhahn, Kerstin

   In Proceedings of the 2020 International Conference on Multimodal Interaction (ICMI) 2020

   Abs

   An experiment is presented to investigate whether there is consensus in mapping emotions to messages/situations in urban search and rescue scenarios, where efficiency and effectiveness of interactions are key to success. We studied mappings between 10 specific messages, presented in two different communication styles, reflecting common situations that might happen during search and rescue missions, and the emotions exhibited by robots in those situations. The data was obtained through a Mechanical Turk study with 78 participants. Our findings support the feasibility of using emotions as an additional communication channel to improve multi-modal human-robot interaction for urban search and rescue robots, and suggests that these mappings are robust, i.e. are not affected by the robot’s communication style.

2. Offline versus Online Triplet Mining based on Extreme Distances of Histopathology Patches

   Sikaroudi, Milad, Ghojogh, Benyamin, Safarpoor, Amir, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

   In 15th International Symposium on Visual Computing (ISCV 2020) 2020

   Abs arXiv PDF

   We analyze the effect of offline and online triplet mining for colorectal cancer (CRC) histopathology dataset containing 100,000 patches. We consider the extreme, i.e., farthest and nearest patches with respect to a given anchor, both in online and offline mining. While many works focus solely on how to select the triplets online (batch-wise), we also study the effect of extreme distances and neighbor patches before training in an offline fashion. We analyze the impacts of extreme cases for offline versus online mining, including easy positive, batch semi-hard, and batch hard triplet mining as well as the neighborhood component analysis loss, its proxy version, and distance weighted sampling. We also investigate online approaches based on extreme distance and comprehensively compare the performance of offline and online mining based on the data patterns and explain offline mining as a tractable generalization of the online mining with large mini-batch size. As well, we discuss the relations of different colorectal tissue types in terms of extreme distances. We found that offline mining can generate a better statistical representation of the population by working on the whole dataset.

3. Isolation Mondrian Forest for Batch and Online Anomaly Detection

   Ma, Haoran, Ghojogh, Benyamin, Samad, Maria N, Zheng, Dongyu, and Crowley, Mark

   In IEEE International Conference on Systems, Man, and Cybernetics (IEEE-SMC-2020) 2020

   Abs PDF

   We propose a new method, named isolation Mon- drian forest (iMondrian forest), for batch and online anomaly detection. The proposed method is a novel hybrid of isolation forest and Mondrian forest which are existing methods for batch anomaly detection and online random forest, respectively. iMondrian forest takes the idea of isolation, using the depth of a node in a tree, and implements it in the Mondrian forest structure. The result is a new data structure which can accept streaming data in an online manner while being used for anomaly detection. Our experiments show that iMondrian forest mostly performs better than isolation forest in batch settings and has better or comparable performance against other batch and online anomaly detection methods.

4. Multi-Level Collaborative Control System With Dual Neural Network Planning For Autonomous Vehicle Control In A Noisy Environment

   Du, Zhiyuan, Lull, Joseph, Malhan, Rajesh, Ganapathi Subramanian, Sriram, Bhalla, Sushrut, Sambee, Jaspreet, and Crowley, Mark

   2020
5. A review of machine learning applications in wildfire science and management

   Jain, Piyush, Coogan, Sean CP, Ganapathi Subramanian, Sriram, Crowley, Mark, Taylor, Steve, and Flannigan, Mike D

   Environmental Reviews 2020

   Abs arXiv PDF Slides

   Artificial intelligence has been applied in wildfire science and management since the 1990s, with early applications including neural networks and expert systems. Since then the field has rapidly progressed congruently with the wide adoption of machine learning (ML) methods in the environmental sciences. Here, we present a scoping review of ML applications in wildfire science and management. Our overall objective is to improve awareness of ML methods among wildfire researchers and managers, as well as illustrate the diverse and challenging range of problems in wildfire science available to ML data scientists. To that end, we first present an overview of popular ML approaches used in wildfire science to date, and then review the use of ML in wildfire science as broadly categorized into six problem domains, including: 1) fuels characterization, fire detection, and mapping; 2) fire weather and climate change; 3) fire occurrence, susceptibility, and risk; 4) fire behavior prediction; 5) fire effects; and 6) fire management. Furthermore, we discuss the advantages and limitations of various ML approaches relating to data size, computational requirements, generalizability, and interpretability, as well as identify opportunities for future advances in the science and management of wildfires within a data science context. In total, we identified 298 relevant publications, where the most frequently used ML methods across problem domains included random forests, MaxEnt, artificial neural networks, decision trees, support vector machines, and genetic algorithms. As such, there exists opportunities to apply more current ML methods — including deep learning and agent based learning — in the wildfire sciences, especially in instances involving very large multivariate datasets. We must recognize, however, that despite the ability of ML models to learn on their own, expertise in wildfire science is necessary to ensure realistic modelling of fire processes across multiple scales, while the complexity of some ML methods, such as deep learning, requires a dedicated and sophisticated knowledge of their application. Finally, we stress that the wildfire research and management communities play an active role in providing relevant, high quality, and freely available wildfire data for use by practitioners of ML methods.

6. Deep Multi Agent Reinforcement Learning for Autonomous Driving

   Bhalla, Sushrut, Ganapathi Subramanian, Sriram, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2020
7. Supervision and Source Domain Impact on Representation Learning: A Histopathology Case Study

   Sikaroudi, Milad, Safarpoor, Amir, Ghojogh, Benyamin, Shafiei, Sobhan, Crowley, Mark, and Tizhoosh, HR

   In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’20) 2020

   Abs arXiv PDF

   As many algorithms depend on a suitable representation of data, learning unique features is considered a crucial task. Although supervised techniques using deep neural networks have boosted the performance of representation learning, the need for a large set of labeled data limits the application of such methods. As an example, high-quality delineations of regions of interest in the field of pathology is a tedious and time-consuming task due to the large image dimensions. In this work, we explored the performance of a deep neural network and triplet loss in the area of representation learning. We investigated the notion of similarity and dissimilarity in pathology whole-slide images and compared different setups from unsupervised and semi-supervised to supervised learning in our experiments. Additionally, different approaches were tested, applying few-shot learning on two publicly available pathology image datasets. We achieved high accuracy and generalization when the learned representations were applied to two different pathology datasets

8. Distributed Nonlinear Model Predictive Control and Metric Learning for Heterogeneous Vehicle Platooning with Cut-in/Cut-out Maneuvers

   Basiri, Mohammad Hossein, Ghojogh, Benyamin, Azad, Nasser L, Fischmeister, Sebastian, Karray, Fakhri, and Crowley, Mark

   In Proceeding of the 59th IEEE Conference on Decision and Control (CDC-2020) 2020
9. Fisher Discriminant Triplet and Contrastive Losses for Training Siamese Networks

   Ghojogh, Benyamin, Sikaroudi, Milad, Shafiei, Sobhan, Tizhoosh, H.R., Karray, Fakhri, and Crowley, Mark

   In IEEE International Joint Conference on Neural Networks (IJCNN) 2020
10. Weighted Fisher Discriminant Analysis in the Input and Feature Spaces

    Ghojogh, Benyamin, Sikaroudi, Milad, Tizhoosh, H.R., Karray, Fakhri, and Crowley, Mark

    In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020
11. Backprojection for Training Feedforward Neural Networks in the Input and Feature Spaces

    Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

    In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020
12. Theoretical Insights into the Use of Structural Similarity Index In Generative Models and Inferential Autoencoders

    Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

    In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020
13. Generalized Subspace Learning by Roweis Discriminant Analysis

    Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

    In International Conference on Image Analysis and Recognition (ICIAR-2020) 2020

    Abs

    We present a new method which generalizes subspace learning based on eigenvalue and generalized eigenvalue problems. This method, Roweis Discriminant Analysis (RDA), is named after Sam Roweis to whom the field of subspace learning owes significantly. RDA is a family of infinite number of algorithms where Principal Component Analysis (PCA), Supervised PCA (SPCA), and Fisher Discriminant Analysis (FDA) are special cases. One of the extreme special cases, which we name Double Supervised Discriminant Analysis (DSDA), uses the labels twice, it is novel and has not appeared elsewhere. We propose a dual for RDA for some special cases. We also propose kernel RDA, generalizing kernel PCA, kernel SPCA, and kernel FDA, using both dual RDA and representation theory. Our theoretical analysis explains previously known facts such as why SPCA can use regression but FDA cannot, why PCA and SPCA have duals but FDA does not, why kernel PCA and kernel SPCA use kernel trick but kernel FDA does not, and why PCA is the best linear method for reconstruction. Roweisfaces and kernel Roweisfaces are also proposed generalizing eigenfaces, Fisherfaces, supervised eigenfaces, and their kernel variants. We also report experiments showing the effectiveness of RDA and kernel RDA on some benchmark datasets.

14. Anomaly Detection and Prototype Selection Using Polyhedron Curvature

    Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

    In Canadian Conference on Artificial Intelligence 2020
15. Reinforcement Learning in a Physics-Inspired Semi-Markov Environment

    Bellinger, Colin, Coles, Rory, Crowley, Mark, and Tamblyn, Isaac

    In Canadian Conference on Artificial Intelligence 2020

2019

1. Semantic Workflows and Machine Learning for the Assessment of Carbon Storage by Urban Trees

   Carrillo, Juan, Garijo, Daniel, Crowley, Mark, Gil, Yolanda, and Borda, Katherine

   In Third International Workshop on Capturing Scientific Knowledge (Sciknow 2019), Collocated with the tenth International Conference on Knowledge Capture (K-CAP) 2019
2. Comparison of Deep Learning models for Determining Road Surface Condition from Roadside Camera Images and Weather Data

   Carrillo, J., Crowley, M., Pan, G., and Fu, L.

   In TAC-ITS Canada Joint Conference 2019

   Abs

   Road maintenance during the Winter season is a safety critical and resource demanding operation. One of its key activities is determining road surface condition (RSC) in order to prioritize roads and allocate cleaning efforts such as plowing or salting. Two conventional approaches for determining RSC are: visual examination of roadside camera images by trained personnel and patrolling the roads to perform on-site inspections. However, with more than 500 cameras collecting images across Ontario, visual examination becomes a resource-intensive activity, difficult to scale especially during periods of snow storms. This paper presents the preliminary results of an ongoing study focused on improving the efficiency of road maintenance operations. We use multiple Deep Learning models to automatically determine RSC from roadside camera images and weather variables, extending previous research where similar methods have been used to deal with the problem. The dataset we use was collected during the 2017-2018 Winter season from 40 stations connected to the Ontario Road Weather Information System (RWIS), it includes 14.000 labelled images and 70.000 weather measurements. In particular, we train and evaluate the performance of seven state-of-the-art models from the Computer Vision literature, including the recent DenseNet, NASNet, and MobileNet. Also, by integrating observations from weather variables, the models are able to better ascertain RSC under poor visibility conditions.

3. Instance Ranking and Numerosity Reduction Using Matrix Decompositionand Subspace Learning

   Ghojogh, Benyamin, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2019

   Abs

   One way to deal with the ever increasing amount of available data for processing is to rank data instances by usefulness and reduce the dataset size. In this work, we introduce a framework to achieve this using matrix decomposition and subspace learning. Our central contribution is a novel similarity measure for data instances that uses the basis obtained from matrix decomposition of the dataset. Using this similarity measure, we propose several related algorithms for ranking data instances and performing numerosity reduction. We then validate the effectiveness of these algorithms for data reduction on several datasets for classification, regression, and clustering tasks.

4. Learning Multi-Agent Communication with Reinforcement Learning

   Bhalla, Sushrut, Ganapathi Subramanian, Sriram, and Crowley, Mark

   In Conference on Reinforcement Learning and Decision Making (RLDM-19) 2019

   Abs

   Deep Learning and back-propagation have been successfully used to perform centralized training with communication protocols among multiple agents in a cooperative environment. In this paper, we present techniques for centralized training of Multi-Agent (Deep) Reinforcement Learning (MARL) using the model-free Deep Q-Network (DQN) as the baseline model and communication between agents. We present two novel, scalable and centralized MARL training techniques (MA-MeSN, MA-BoN), which separate the message learning module from the policy module. The separation of these modules helps in faster convergence in complex domains like autonomous driving simulators. A second contribution uses a memory module to achieve a decentralized cooperative policy for execution and thus addresses the challenges of noise and communication bottlenecks in real-time communication channels. This paper theoretically and empirically compares our centralized and decentralized training algorithms to current research in the field of MARL. We also present and release a new OpenAI-Gym environment which can be used for multi-agent research as it simulates multiple autonomous cars driving cooperatively on a highway. We compare the performance of our centralized algorithms to DIAL and IMS based on cumulative reward achieved per episode. MA-MeSN and MA-BoN achieve a cumulative reward of at-least }263\backslash%} of the reward achieved by the DIAL and IMS. We also present an ablation study of the scalability of MA-BoN and see a linear increase in inference time and number of trainable parameters compared to quadratic increase for DIAL.

5. Locally Linear Image Structural Embedding for Image Structure Manifold Learning

   Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

   In International Conference on Image Analysis and Recognition (ICIAR-19) 2019
6. Image Structure Subspace Learning Using Structural Similarity Index

   Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

   In International Conference on Image Analysis and Recognition (ICIAR-19) 2019
7. Principal Component Analysis Using Structural Similarity Index for Images

   Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

   In International Conference on Image Analysis and Recognition (ICIAR-19) 2019
8. Artificial Counselor System For Stock Investment

   Nekoei Qachkanloo, Hadi, Ghojogh, Benyamin, Pasand, Ali Saheb, and Crowley, Mark

   In Innovative Applications of Artificial Intelligence (IAAI-19) 2019
9. Compact Representation of a Multi-dimensional Combustion Manifold Using Deep Neural Networks

   Bhalla, Sushrut, Yao, Matthew, Hickey, Jean-Pierre, and Crowley, Mark

   2019

   Abs PDF Poster Slides

   The computational challenges in turbulent combustion simulations stem from the physical complexities and multi-scale nature of the problem which make it intractable to compute scaleesolving simulations. For most engineering applications, the large scale separation between the flame (typically submillimeter scale) and the characteristic turbulent flow (typically centimeter or meter scale) allows us to evoke simplifying assumptions–such as done for the flamelet model to precompute all the chemical reactions and map them to a low order manifold. The resulting manifold is then tabulated and looked up at runtime. As the physical complexity of combustion simulations increases (including radiation, soot formation, pressure variations etc.) the dimensionality of the resulting manifold grows which impedes an efficient tabulation and look up. In this paper we present a novel approach to model the multidimensional combustion manifold. We approximate the combustion manifold using a neural network function approximator and use it to predict the temperature and composition of the reaction. We present a novel training procedure which is developed to generate a smooth output curve for temperature over the course of a reaction. We then evaluate our work against the current approach of tabulation with linear interpolation in combustion simulations. We also provide an ablation study of our training procedure in the context of overfitting in our model. The combustion dataset used for the modeling of combustion of H2 and O2 in this work is released alongside this paper.

10. Training Cooperative Agents for Multi-Agent Reinforcement Learning

    Bhalla, Sushrut, Ganapathi Subramanian, Sriram, and Crowley, Mark

    In Proc. of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019) 2019

    Abs

    Deep Learning and back-propagation has been successfully used to perform centralized training with communication protocols among multiple agents in a cooperative environment. In this paper we present techniques for centralized training of Multi-Agent (Deep) Reinforcement Learning (MARL) using the model-free Deep Q-Network as the baseline model and message sharing between agents. We present a novel, scalable, centralized MARL training technique, which separates the message learning module from the policy module. The separation of these modules helps in faster convergence in complex domains like autonomous driving simulators. A second contribution uses the centrally trained model to bootstrap training of distributed, independent, cooperative agent policies for execution and thus addresses the challenges of noise and communication bottlenecks in real-time communication channels. This paper theoretically and empirically compares our centralized training algorithms to current research in the field of MARL. We also present and release a new OpenAI-Gym environment which can be used for multi-agent research as it simulates multiple autonomous cars driving cooperatively on a highway.

11. Integration of Roadside Camera Images and Weather Data for monitoring Winter Road Surface Conditions

    Carrillo, Juan, and Crowley, Mark

    In Canadian Association of Road Safety Professionals (CARSP) Conference 2019

    Abs

    Background/Context: During the Winter season, real-time monitoring of road surface conditions is critical for the safety of drivers and road maintenance operations. Previous research has evaluated the potential of image classification methods for detecting road snow coverage by processing images from roadside cameras installed in RWIS (Road Weather Information System) stations. However, it is a challenging task due to limitations such as image resolution, camera angle, and illumination. Two common approaches to improve the accuracy of image classification methods are: adding more input features to the model and increasing the number of samples in the training dataset. Additional input features can be weather variables and more sample images can be added by including other roadside cameras. Although RWIS stations are equipped with both cameras and weather measurement instruments, they are only a subset of the total number of roadside cameras installed across transportation networks, most of which do not have weather measurement instruments. Thus, improvements in use of image data could benefit from additional data sources. Aims/Objectives: The first objective of this study is to complete an exploratory data analysis over three data sources in Ontario: RWIS stations, all the other MTO (Ministry of Transportation of Ontario) roadside cameras, and Environment Canada weather stations. The second objective is to determine the feasibility of integrating these three datasets into a more extensive and richer dataset with weather variables as additional features and other MTO roadside cameras as additional sources of images. Methods/Targets: First, we quantify the advantage of adding other MTO roadside cameras using spatial statistics, the number of monitored roads, and the coverage of ecoregions with different climate regimes. We then analyze experimental variograms from the literature and determine the feasibility of using Environment Canada stations and RWIS stations to interpolate weather variables for all the other MTO roadside cameras without weather instruments. Results/Activities: By adding all other MTO cameras as image data sources, the total number of cameras in the dataset increases from 139 to 578 across Ontario. The average distance to the nearest camera decreases from 38.4km to 9.4km, and the number of monitored roads increases approximately four times. Additionally, six times more cameras are available in the four most populated ecoregions in Ontario. The experimental variograms show that it is feasible to interpolate weather variables with reasonable accuracy. Moreover, observations in the three datasets are collected with similar frequency, which facilitates our data integration approach. Discussion/Deliverables: Integrating these three datasets is feasible and can benefit the design and development of automated image classification methods for monitoring road snow coverage. We do not consider data from pavement-embedded sensors, an additional line of research may explore the integration of this data. Our approach can provide actionable insights which can be used to more selectively perform manual patrolling to better identify road surface conditions. Conclusions: Our initial results are promising and demonstrate that additional, image only datasets can be added to road monitoring data by using existing multimodal sensors as ground truth, which will lead to greater performance on the future image classification tasks.

2018

1. A Complementary Approach to Improve WildFire Prediction Systems.

   Subramanian, Sriram Ganapathi, and Crowley, Mark

   In Neural Information Processing Systems (AI for social good workshop) 2018
2. Combining MCTS and A3C for prediction of spatially spreading processes in forest wildfire settings

   Ganapathi Subramanian, Sriram, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2018

   Abs

   In recent years, Deep Reinforcement Learning (RL) algorithms have shown super-human performance in a variety Atari and classic board games like chess and GO. Research into applications of RL in other domains with spatial considerations like environmental planning are still in their nascent stages. In this paper, we introduce a novel combination of Monte-Carlo Tree Search (MCTS) and A3C algorithms on an online simulator of a wildfire, on a pair of forest fires in Northern Alberta (Fort McMurray and Richardson fires) and on historical Saskatchewan fires previously compared by others to a physics-based simulator. We conduct several experiments to predict fire spread for several days before and after the given spatial information of fire spread and ignition points. Our results show that the advancements in Deep RL applications in the gaming world have advantages in spatially spreading real-world problems like forest fires. \textcopyright Springer International Publishing AG, part of Springer Nature 2018.

3. Principal Sample Analysis for Data Reduction

   Ghojogh, Benyamin, and Crowley, Mark

   In 2018 IEEE International Conference on Big Knowledge (ICBK) 2018
4. Blue Sky Ideas in Artificial Intelligence Education from the EAAI 2017 New and Future AI Educator Program

   Eaton, Eric, Koenig, Sven, Schulz, Claudia, Maurelli, Francesco, Lee, John, Eckroth, Joshua, Crowley, Mark, Freedman, Richard G, Cardona-Rivera, Rogelio E, Machado, Tiago, and Williams, Tom

   2018
5. Decision Assist for Self-Driving Cars

   Ganapathi Subramanian, Sriram, Sambee, Jaspreet Singh, Ghojogh, Benyamin, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2018

   Abs

   Research into self-driving cars has grown enormously in the last decade primarily due to the advances in the fields of machine intelligence and image processing. An under-appreciated aspect of self-driving cars is actively avoiding high traffic zones, low visibility zones, and routes with rough weather conditions by learning different conditions and making decisions based on trained experiences. This paper addresses this challenge by introducing a novel hierarchical structure for dynamic path planning and experiential learning for vehicles. A multistage system is proposed for detecting and compensating for weather, lighting, and traffic conditions as well as a novel adaptive path planning algorithm named Checked State A3C. This algorithm improves upon the existing A3C Reinforcement Learning (RL) algorithm by adding state memory which provides the ability to learn an adaptive model of the best decisions to take from experience. \textcopyright Springer International Publishing AG, part of Springer Nature 2018.

6. Using Spatial Reinforcement Learning to Build Forest Wildfire Dynamics Models From Satellite Images

   Ganapathi Subramanian, Sriram, and Crowley, Mark

   Frontiers in ICT 2018

   Abs

   Machine learning algorithms have increased tremendously in power in recent years but have yet to be fully utilized in many ecology and sustainable resource management domains such as wildlife reserve design, forest fire management and invasive species spread. One thing these domains have in common is that they contain dynamics that can be characterized as a Spatially Spreading Process (SSP) which requires many parameters to be set precisely to model the dynamics, spread rates and directional biases of the elements which are spreading. We present related work in Artificial Intelligence and Machine Learning for SSP sustainability domains including forest wildfire prediction. We then introduce a novel approach for learning in SSP domains using Reinforcement Learning (RL) where fire is the agent at any cell in the landscape and the set of actions the fire can take from a location at any point in time includes spreading North, South, East, West or not spreading. This approach inverts the usual RL setup since the dynamics of the corresponding Markov Decision Process (MDP) is a known function for immediate wildfire spread. Meanwhile, we learn an agent policy for a predictive model of the dynamics of a complex spatially-spreading process. Rewards are provided for correctly classifying which cells are on fire or not compared to satellite and other related data. We examine the behaviour of five RL algorithms on this problem: Value Iteration, Policy Iteration, Q-Learning, Monte Carlo Tree Search and Asynchronous Advantage Actor-Critic (A3C). We compare to a Gaussian process based supervised learning approach and discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We also discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We validate our approach with satellite image data of two massive wildfire events in Northern Alberta, Canada, the Fort McMurray fire of 2016 and the Richardson fire of 2011. The results show that we can learn predictive, agent-based policies as models of spatial dynamics using RL on readily available satellite images that other methods and have many additional advantages in terms of generalizability and interpretability.

2017

1. Learning Forest Wildfire Dynamics from Satellite Images Using Reinforcement Learning

   Subramanian, Sriram Ganapathi, and Crowley, Mark

   In Conference on Reinforcement Learning and Decision Making 2017
2. Big Metadata : Machine Learning on Encrypted Communications

   Fernick, Jennifer, and Crowley, Mark

   In RSA Conference 2017
3. Application of probabilistically-weighted graphs to image-based diagnosis of Alzheimer’s disease using diffusion MRI

   Maryam, Syeda, McCrackin, Laura, Crowley, Mark, Rathi, Yogesh, and Michailovich, Oleg

   In Proceedings of SPIE 101324, Medical Imaging 2017 : Computer-Aided Diagnosis 2017
4. AI Education Through Real World Problems

   Crowley, Mark

   In The Seventh Symposium on Educational Advances in Artificial Intellgience. 2017

2016

1. Inter-Arrival Curves for Multi-Mode and Online Anomaly Detection

   Salem, Mahmoud, Crowley, Mark, and Fischmeister, Sebastian

   In Euromicro Conference on Real-Time Systems 2016 - Work-in-Progress Proceedings 2016
2. Anomaly Detection Using Inter-Arrival Curves for Real-time Systems

   Salem, Mahmoud, Crowley, Mark, and Fischmeister, Sebastian

   In 2016 28th Euromicro Conference on Real-Time Systems 2016

   Abs

   —Real-time embedded systems are a significant class of applications, poised to grow even further as automated vehicles and the Internet of Things become a reality. An important problem for these systems is to detect anomalies during operation. Anomaly detection is a form of classification, which can be driven by data collected from the system at execution time. We propose inter-arrival curves as a novel analytic modelling technique for discrete event traces. Our approach relates to the existing technique of arrival curves and expands the technique to anomaly detection. Inter-arrival curves analyze the behaviour of events within a trace by providing upper and lower bounds to their inter-arrival occurrence. We exploit inter-arrival curves in a classification framework that detects deviations within these bounds for anomaly detection. Also, we show how inter-arrival curves act as good features to extract recurrent behaviour that these systems often exhibit. We demonstrate the feasibility and viability of the fully implemented approach with an industrial automotive case study (CAN traces) as well as a deployed aerospace case study (RTOS kernel traces).

2015

1. PAC Optimal MDP Planning with Application to Invasive Species Management

   Taleghan, Majid Alkaee, Dietterich, Thomas G., Crowley, Mark, Hall, Kim, and Albers, H. Jo

   Journal of Machine Learning Research 2015

   Abs

   In a simulator-defined MDP, the Markovian dynamics and rewards are provided in the form of a simulator from which samples can be drawn. This paper studies MDP planning algorithms that attempt to minimize the number of simulator calls before terminating and outputting a policy that is approximately optimal with high probability. The paper introduces two heuristics for efficient exploration and an improved confidence interval that enables earlier termination with probabilis- tic guarantees. We prove that the heuristics and the confidence interval are sound and produce with high probability an approximately optimal policy in polynomial time. Experiments on two benchmark problems and two instances of an invasive species management problem show that the improved confidence intervals and the new search heuristics yield reductions of between 8% and 47% in the number of simulator calls required to reach near-optimal policies.

2. Answering Simple Questions About Spatially Spreading Systems

   Crowley, Mark

   In 2015 Summer Solstice: 7th International Conference on Discrete Models of Complex Systems 2015

2014

1. Using equilibrium policy gradients for spatiotemporal planning in forest ecosystem management

   Crowley, Mark

   IEEE Transactions on Computers 2014

   Abs

   Spatiotemporal planning involves making choices at multiple locations in space over some planning horizon to maximize utility and satisfy various constraints. In Forest Ecosystem Management, the problem is to choose actions for thousands of locations each year including harvesting, treating trees for fire or pests, or doing nothing. The utility models could place value on sale of lumber, ecosystem sustainability or employment levels and incorporate legal and logistical constraints on actions such as avoiding large contiguous areas of clearcutting. Simulators developed by forestry researchers provide detailed dynamics but are generally inaccesible black boxes. We model spatiotemporal planning as a factored Markov decision process and present a policy gradient planning algorithm to optimize a stochastic spatial policy using simulated dynamics. It is common in environmental and resource planning to have actions at different locations be spatially interelated, this makes representation and planning challenging. We define a global spatial policy in terms of interacting local policies defining distributions over actions at each location conditioned on actions at nearby locations. Markov chain Monte Carlo simulation is used to sample landscape policies and estimate their gradients. Evaluation is carried out on a forestry planning problem with 1,880 locations using a variety of value models and constraints. Index

2014

1. Using equilibrium policy gradients for spatiotemporal planning in forest ecosystem management

   Crowley, Mark

   IEEE Transactions on Computers 2014

   Abs

   Spatiotemporal planning involves making choices at multiple locations in space over some planning horizon to maximize utility and satisfy various constraints. In Forest Ecosystem Management, the problem is to choose actions for thousands of locations each year including harvesting, treating trees for fire or pests, or doing nothing. The utility models could place value on sale of lumber, ecosystem sustainability or employment levels and incorporate legal and logistical constraints on actions such as avoiding large contiguous areas of clearcutting. Simulators developed by forestry researchers provide detailed dynamics but are generally inaccesible black boxes. We model spatiotemporal planning as a factored Markov decision process and present a policy gradient planning algorithm to optimize a stochastic spatial policy using simulated dynamics. It is common in environmental and resource planning to have actions at different locations be spatially interelated, this makes representation and planning challenging. We define a global spatial policy in terms of interacting local policies defining distributions over actions at each location conditioned on actions at nearby locations. Markov chain Monte Carlo simulation is used to sample landscape policies and estimate their gradients. Evaluation is carried out on a forestry planning problem with 1,880 locations using a variety of value models and constraints. Index

2013

1. Allowing a wildfire to burn: Estimating the effect on future fire suppression costs

   Houtman, Rachel M., Montgomery, Claire A., Gagnon, Aaron R., Calkin, David E., Dietterich, Thomas G., McGregor, Sean, and Crowley, Mark

   International Journal of Wildland Fire 2013

   Abs

   Where a legacy of aggressive wildland fire suppression has left forests in need of fuel reduction, allowing wildland fire to burn may provide fuel treatment benefits, thereby reducing suppression costs from subsequent fires. The least-cost-plus-net-value-change model of wildland fire economics includes benefits of wildfire in a framework for evaluating suppression options. In this study, we estimated one component of that benefit – the expected present value of the reduction in suppression costs for subsequent fires arising from the fuel treatment effect of a current fire. To that end, we employed Monte Carlo methods to generate a set of scenarios for subsequent fire ignition and weather events, which are referred to as sample paths, for a study area in central Oregon. We simulated fire on the landscape over a 100-year time horizon using existing models of fire behaviour, vegetation and fuels development, and suppression effectiveness, and we estimated suppression costs using an existing suppression cost model. Our estimates suggest that the potential cost savings may be substantial. Further research is needed to estimate the full least-cost-plus-net-value-change model. This line of research will extend the set of tools available for developing wildfire management plans for forested landscapes.

2. PAC Optimal Planning for Invasive Species Management: Improved Exploration for Reinforcement Learning from Simulator-Defined MDPs

   Dietterich, Thomas G, Alkaee Taleghan, Majid, and Crowley, Mark

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2013) 2013

   Abs

   Often the most practical way to define a Markov Decision Process (MDP) is as a simulator that, given a state and an action, produces a resulting state and immediate reward sam- pled from the corresponding distributions. Simulators in nat- ural resource management can be very expensive to execute, so that the time required to solve such MDPs is dominated by the number of calls to the simulator. This paper presents an algorithm, DDV, that combines improved confidence inter- vals on the Q values (as in interval estimation) with a novel upper bound on the discounted state occupancy probabilities to intelligently choose state-action pairs to explore. We prove that this algorithm terminates with a policy whose value is within εof the optimal policy (with probability 1−δ) after making only polynomially-many calls to the simulator. Ex- periments on one benchmark MDP and on an MDP for inva- sive species management show very large reductions in the number of simulator calls required.

3. Cyclic causal models with discrete variables: Markov chain equilibrium semantics and sample ordering

   Poole, David, and Crowley, Mark

   In IJCAI International Joint Conference on Artificial Intelligence 2013

   Abs

   We analyze the foundations of cyclic causal models for discrete variables, and compare structural equation models (SEMs) to an alternative semantics as the equilibrium (stationary) distribution of a Markov chain. We show under general conditions, discrete cyclic SEMs cannot have independent noise, even in the simplest case, cyclic structural equation models imply constraints on the noise. We give a formalization of an alternative Markov chain equilibrium semantics which requires not only the causal graph, but also a sample order. We show how the resulting equilibrium is a function of the sample ordering, both theoretically and empirically.

4. Policy Gradient Optimization Using Equilibrium Policies for Spatial Planning Domains

   Crowley, Mark

   In 13th INFORMS Computing Society Conference 2013

2012

1. Managing Invasive Species in a River Network

   Hall, Kim, Alkaee Taleghan, Majid, Albers, Heidi J., Dietterich, Thomas, and Crowley, Mark

   In Third International Conference on Computational Sustainability 2012

   Abs

   There are many pest species that invade valuable ecosystems and damage ecosystem services. An important management goal is to limit the spread of these species or even eradicate them. From the computational perspective, these problems can be formulated as Markov Decision Processes. However, solving them is difficult, because of their spatial nature. Viewed abstractly, the current state of the ecosystem can be formalized as a graph. Each node of the graph may be invaded or not, and at each time step, a “local” action must be selected at each node. Consequently, at the level of the whole MDP, each global action is a vector of local actions of length N, the number of nodes in the graph. The exponentially large action space raises major computational problems. During the state transitions, the invasive species can spread along the edges of the graph. We study a specific instance of this general problem based on the spread of Tamarisk in river networks. In this formulation, we work with the dual of the standard river network graph. Each node corresponds to a stretch of the river between two confluences (also called a “reach”). We divide each reach into H “slots”, where each slot can either be occupied by a single native tree, occupied by a single Tamarisk tree, or empty. In each time step, the plants (both native and invader) first must survive through winter (with some probability). Then, the surviving trees produce seeds that disperse through the river network according to a dispersal kernel that prefers downstream movement with high probability. Each dispersing seed lands at some slot in some node of the network. All seeds arriving at a slot compete to establish themselves in that slot (if the slot is empty). In our model, there are three local actions: simple eradication of invasive plants, eradication followed by planting of native plants, or doing nothing at all. Doing nothing costs nothing, while eradication is more expensive, and eradication plus planting is the most expensive. One local action must be chosen for each node (reach) at each time step, and it applies to all H slots at that node. The actions only succeed stochastically. The planning goal is to minimize the expected discounted sum of the cost of invasive damage and the cost of management actions. To solve this planning problem, we apply the standard value iteration algorithm. However, standard value iteration requires a fully-specified state transition matrix. In our problem, it is easy to write a simulator for drawing samples of the dispersal and competition processes, but it is computationally intractable to compute the exact transition probabilities. Hence, we estimate those probabilities by drawing a large number of samples from the simulator. Once we have an optimal policy on the approximated MDP, we explore its behaviour both directly and via simulated rollouts. The optimal policy can be visualized with one picture per state annotated by the types of plants in each reach and the optimal action that should be taken in each reach. We also run the optimal policy forward in time on many simulated trajectories in order to gather statistics about its behaviour such as: how long it takes to reach a steady state, average cost of a policy over time, or the frequency with which completely invaded or uninvaded states are reached. Comparisons of these statistics show that in many situations our optimal policies greatly outperform established rules of thumb from the literature. An example of a rule of thumb is to always prioritize treatment of upstream over downstream reaches. Another rule targets the most invaded reaches (those with the most slots occupied by Tamarisk) first. We have found that the optimality of these rules of thumb depends strongly on the dispersal and competition parameters. Some contributions this work makes to bioeconomics and ecology are showing evidence that the stochastic spread of the invasion process needs to be modelled and that the spatial characteristics of the system under invasion are relevant to optimal management. The computational challenges raised by this problem have led us into research on methods for planning that can provide bounds on the optimality of the policy while minimizing the number of calls to expensive simulators. We are also studying algorithms that can scale to large graphs with thousands of nodes and large out-degrees. Finally, we see potential in these problems for intelligently reducing the size of the state space and the complexity of the policy description by utilizing domain knowledge.

2011

1. Policy gradient planning for environmental decision making with existing simulators

   Crowley, Mark, and Poole, David

   In Proceedings of the National Conference on Artificial Intelligence 2011

   Abs

   In environmental and natural resource planning domains actions are taken at a large number of locations over multiple time periods. These problems have enormous state and action spaces, spatial correlation between actions, uncertainty and complex utility models. We present an approach for modeling these planning problems as factored Markov decision processes. The reward model can contain local and global components as well as spatial constraints between locations. The transition dynamics can be provided by existing simulators developed by domain experts. We propose a landscape policy defined as the equilibrium distribution of a Markov chain built from many locally-parameterized policies. This policy is optimized using a policy gradient algorithm. Experiments using a forestry simulator demonstrate the algorithm’s ability to devise policies for sustainable harvest planning of a forest. Copyright \textcopyright 2011, Association for the Advancement of Artificial Intelligence. All rights reserved.

2. Equilibrium Policy Gradients for Spatiotemporal Planning

   Crowley, Mark

   2011

   Abs

   In spatiotemporal planning, agents choose actions at multiple locations in space over some planning horizon to maximize their utility and satisfy various constraints. In forestry planning, for example, the problem is to choose actions for thousands of locations in the forest each year. The actions at each location could include harvesting trees, treating trees against disease and pests, or doing nothing. A utility model could place value on sale of forest products, ecosystem sustainability or employment levels, and could incorporate legal and logistical constraints such as avoiding large contiguous areas of clearcutting and managing road access. Planning requires a model of the dynamics. Existing simulators developed by forestry researchers can provide detailed models of the dynamics of a forest over time, but these simulators are often not designed for use in automated planning. This thesis presents spatiotemoral planning in terms of factored Markov decision processes. A policy gradient planning algorithm optimizes a stochastic spatial policy using existing simulators for dynamics. When a planning problem includes spatial interaction between locations, deciding on an action to carry out at one location requires considering the actions performed at other locations. This spatial interdependence is common in forestry and other environmental planning problems and makes policy representation and planning challenging. We define a spatial policy in terms of local policies defined as distributions over actions at one location conditioned upon actions at other locations. A policy gradient planning algorithm using this spatial policy is presented which uses Markov Chain Monte Carlo simulation to sample the landscape policy, estimate its gradient and use this gradient to guide policy improvement. Evaluation is carried out on a forestry planning problem with 1880 locations using a variety of value models and constraints. The distribution over joint actions at all locations can be seen as the equilibrium of a cyclic causal model. This equilibrium semantics is compared to Structural Equation Models. We also define an algorithm for approximating the equilibrium distribution for cyclic causal networks which exploits graphical structure and analyse when the algorithm is exact.

2010

1. Circuits and logic in the lab : Toward a coherent picture of computation

   Patitsas, Elizabeth, Voll, Kimberly, Crowley, Mark, and Wolfman, Steven

   In 15th Western Canadian Conference on Computing Education 2010

   Abs

   We describe extensive modifications made over time to a first year computer science course at the University of British Columbia covering logic and digital circuits (among other topics). Smoothly integrating the hardware-based labs with the more theory-based lectures into a cohesive picture of computation has always been a challenge in this course. The seeming disconnect between implementation and abstraction has historically led to frustration and dissatisfaction among students. We describe changes to the lab curriculum, equipment logistics, the style of in-lab activities and evaluation. We have also made logistical changes to the management and ongoing training of teaching assistants, allowing us to better anchor our larger course story into the lab curriculum. These changes have greatly improved student and TA opinions of the lab experience, as well as the overall course.

2009

1. Seeing the Forest Despite the Trees : Large Scale Spatial-Temporal Decision Making

   Crowley, Mark, Nelson, John, and Poole, David

   In Conference on Uncertainty in Artificial Intelligence (UAI09) 2009

   Abs

   We introduce a challenging real world planning problem where actions must be taken at each location in a spatial area at each point in time. We use forestry planning as the motivating application. In Large Scale Spatial-Temporal (LSST) planning problems, the state and action spaces are defined as the cross-products of many local state and action spaces spread over a large spatial area such as a city or forest. These problems possess state uncertainty, have complex utility functions involving spatial constraints and we generally must rely on simulations rather than an explicit transition model. We define LSST problems as reinforcement learning prob- lems and present a solution using policy gradients. We compare two different policy formulations: an explicit policy that identifies each location in space and the action to take there, and an abstract policy that defines the proportion of actions to take across all locations in space. We show that the abstract policy is more robust and achieves higher rewards with far fewer parameters than the elementary policy. This abstract policy is also a better fit to the properties that practitioners in LSST problem domains require for such methods to be widely useful