showcase

2022

1. Mean Field MARL

   Decentralized Mean Field Games

   Ganapathi Subramanian, Sriram, Taylor, Mathew, Crowley, Mark, and Poupart, Pascal

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2022) 2022

   Abs

   Multiagent reinforcement learning algorithms have not been widely adopted in large scale environments with many agents as they often scale poorly with the number of agents. Using mean field theory to aggregate agents has been proposed as a solution to this problem. However, almost all previous methods in this area make a strong assumption of a centralized system where all the agents in the environment learn the same policy and are effectively indistinguishable from each other. In this paper, we relax this assumption about indistinguishable agents and propose a new mean field system known as Decentralized Mean Field Games, where each agent can be quite different from others. All agents learn independent policies in a decentralized fashion, based on their local observations. We define a theoretical solution concept for this system and provide a fixed point guarantee for a Q-learning based algorithm in this system. A practical consequence of our approach is that we can address a ‘chicken-and-egg’ problem in empirical mean field reinforcement learning algorithms. Further, we provide Q-learning and actor-critic algorithms that use the decentralized mean field learning approach and give stronger performances compared to common baselines in this area. In our setting, agents do not need to be clones of each other and learn in a fully decentralized fashion. Hence, for the first time, we show the application of mean field learning methods in fully competitive environments, large-scale continuous action space environments, and other environments with heterogeneous agents. Importantly, we also apply the mean field method in a ride-sharing problem using a real-world dataset. We propose a decentralized solution to this problem, which is more practical than existing centralized training methods.

2. Multi-Advisor-QL

   Multi-Agent Advisor Q-Learning

   Ganapathi Subramanian, Sriram, Larson, Kate, Taylor, Mathew, and Crowley, Mark

   Journal of Artificial Intelligence Research (JAIR) 2022

   URL

2021

1. MARLEmpircal

   Investigation of Independent Reinforcement Learning Algorithms in Multi-Agent Environments

   Lee, Ken Ming, Ganapathi Subramanian, Sriram, and Crowley, Mark

   Frontiers in Artificial Intelligence (Machine Learning and Artificial Intelligence) 2021
2. MARLEmpircal

   Investigation of Independent Reinforcement Learning Algorithms in Multi-Agent Environments

   Lee, Ken Ming, Ganapathi Subramanian, Sriram, and Crowley, Mark

   In NeurIPS 2021 Deep Reinforcement Learning Workshop 2021

   Abs arXiv

   Independent reinforcement learning algorithms have no theoretical guarantees for finding the best policy in multi-agent settings. However, in practice, prior works have reported good performance with independent algorithms in some domains and bad performance in others. Moreover, a comprehensive study of the strengths and weaknesses of independent algorithms is lacking in the literature. In this paper, we carry out an empirical comparison of the performance of independent algorithms on four PettingZoo environments that span the three main categories of multi-agent environments, i.e., cooperative, competitive, and mixed. We show that in fully-observable environments, independent algorithms can perform on par with multi-agent algorithms in cooperative and competitive settings. For the mixed environments, we show that agents trained via independent algorithms learn to perform well individually, but fail to learn to cooperate with allies and compete with enemies. We also show that adding recurrence improves the learning of independent algorithms in cooperative partially observable environments.

3. Acceleration of Large Margin Metric Learning for Nearest Neighbor Classification Using Triplet Mining and Stratified Sampling

   Poorheravi, Parisa, Ghojogh, Benyamin, Gaudet, Vincent, Karray, Fakhri, and Crowley, Mark

   Journal of Computational Vision and Imaging Systems 2021

   Abs PDF URL

   Metric learning is a technique in manifold learning to find a projection subspace for increasing and decreasing the inter- and intra-class variances, respectively. Some metric learning methods are based on triplet learning with anchor-positive-negative triplets. Large margin metric learning for nearest neighbor classification is one of the fundamental methods to do this. Recently, Siamese networks have been introduced with the triplet loss. Many triplet mining methods have been developed for Siamese nets; however, these techniques have not been applied on the triplets of large margin metric learning. In this work, inspired by the mining methods for Siamese nets, we propose several triplet mining techniques for large margin metric learning. Moreover, a hierarchical approach is proposed, for acceleration and scalability of optimization, where triplets are selected by stratified sampling in hierarchical hyper-spheres. We analyze the proposed methods on three publicly available datasets.

4. NLP-DigiPath

   Analysis of Language Embeddings for Classification of Unstructured Pathology Reports

   Allada, Aishwarya Krishna, Wang, Yuanxin, Jindal, Veni, Babee, Morteza, Tizhoosh, H.R., and Crowley, Mark

   In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2021

   Abs

   A pathology report is one of the most significant medical documents providing interpretive insights into the visual appearance of the patient’s biopsy sample. In digital pathology, high-resolution images of tissue samples are stored along with pathology reports. Despite the valuable information that pathology reports hold, they are not used in any systematic manner to promote computational pathology. In this work, we focus on analyzing the reports, which are generally unstructured documents written in English with sophisticated and highly specialized medical terminology. We provide a comparative analysis of various embedding models like BioBERT, Clinical BioBERT, BioMed-RoBERTa and Term Frequency-Inverse Document Frequency (TF-IDF), a traditional NLP technique, as well as the combination of embeddings from pre-trained models with TF-IDF. Our results demonstrate the effectiveness of various word embedding techniques for pathology reports.

5. VTFR-LBFGS-21

   Vector Transport Free Riemannian LBFGS for Optimization on Symmetric Positive Definite Matrix Manifolds

   Godaz, Reza, Ghojogh, Benyamin, Hosseini, Reshad, Monsefi, Reza, Karray, Fakhri, and Crowley, Mark

   In Asian Conference on Machine Learning (ACML) 2021

   Abs arXiv PDF Supp Poster URL

   This work concentrates on optimization on Riemannian manifolds. The Limited-memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) algorithm is a commonly used quasi-Newton method for numerical optimization in Euclidean spaces. Riemannian LBFGS (RLBFGS) is an extension of this method to Riemannian manifolds. RLBFGS involves computationally expensive vector transports as well as unfolding recursions using adjoint vector transports. In this article, we propose two mappings in the tangent space using the inverse second root and Cholesky decomposition. These mappings make both vector transport and adjoint vector transport identity and therefore isometric. Identity vector transport makes RLBFGS less computationally expensive and its isometry is also very useful in convergence analysis of RLBFGS. Moreover, under the proposed mappings, the Riemannian metric reduces to Euclidean inner product, which is much less computationally expensive. We focus on the Symmetric Positive Definite (SPD) manifolds which are beneficial in various fields such as data science and statistics. This work opens a research opportunity for extension of the proposed mappings to other well-known manifolds.

6. Partially Observable Mean Field Reinforcement Learning

   Ganapathi Subramanian, Sriram, Taylor, Matthew, Crowley, Mark, and Poupart, Pascal

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2021

   Abs arXiv PDF

   Traditional multi-agent reinforcement learning algorithms are not scalable to environments with more than a few agents, since these algorithms are exponential in the number of agents. Recent research has introduced successful methods to scale multi-agent reinforcement learning algorithms to many agent scenarios using mean field theory. Previous work in this field assumes that an agent has access to exact cumulative metrics regarding the mean field behaviour of the system, which it can then use to take its actions. In this paper, we relax this assumption and maintain a distribution to model the uncertainty regarding the mean field of the system. We consider two different settings for this problem. In the first setting, only agents in a fixed neighbourhood are visible, while in the second setting, the visibility of agents is determined at random based on distances. For each of these settings, we introduce a Q-learning based algorithm that can learn effectively. We prove that this Q-learning estimate stays very close to the Nash Q-value (under a common set of assumptions) for the first setting. We also empirically show our algorithms outperform multiple baselines in three different games in the MAgents framework, which supports large environments with many agents learning simultaneously to achieve possibly distinct goals.

7. Batch-Incremental Triplet Sampling for Training Triplet Networks Using Bayesian Updating Theorem

   Sikaroudi, Milad, Ghojogh, Benyamin, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

   In 25th International Conference on Pattern Recognition (ICPR) 2021

   Abs arXiv PDF URL

   Variants of Triplet networks are robust entities for learning a discriminative embedding subspace. There exist different triplet mining approaches for selecting the most suitable training triplets. Some of these mining methods rely on the extreme distances between instances, and some others make use of sampling. However, sampling from stochastic distributions of data rather than sampling merely from the existing embedding instances can provide more discriminative information. In this work, we sample triplets from distributions of data rather than from existing instances. We consider a multivariate normal distribution for the embedding of each class. Using Bayesian updating and conjugate priors, we update the distributions of classes dynamically by receiving the new mini-batches of training data. The proposed triplet mining with Bayesian updating can be used with any triplet-based loss function, e.g., triplet-loss or Neighborhood Component Analysis (NCA) loss. Accordingly, Our triplet mining approaches are called Bayesian Updating Triplet (BUT) and Bayesian Updating NCA (BUNCA), depending on which loss function is being used. Experimental results on two public datasets, namely MNIST and histopathology colorectal cancer (CRC), substantiate the effectiveness of the proposed triplet mining method.

8. Amrl

   Active Measure Reinforcement Learning for Observation Cost Minimization: A framework for minimizing measurement costs in reinforcement learning

   Bellinger, Colin, Coles, Rory, Crowley, Mark, and Tamblyn, Isaac

   In Canadian Conference on Artificial Intelligence 2021

   Abs arXiv PDF

   Markov Decision Processes (MDP) with explicit measurement cost are a class of en- vironments in which the agent learns to maximize the costed return. Here, we define the costed return as the discounted sum of rewards minus the sum of the explicit cost of measuring the next state. The RL agent can freely explore the relationship between actions and rewards but is charged each time it measures the next state. Thus, an op- timal agent must learn a policy without making a large number of measurements. We propose the active measure RL framework (Amrl) as a solution to this novel class of problem, and contrast it with standard reinforcement learning under full observability and planning under partially observability. We demonstrate that Amrl-Q agents learn to shift from a reliance on costly measurements to exploiting a learned transition model in order to reduce the number of real-world measurements and achieve a higher costed return. Our results demonstrate the superiority of Amrl-Q over standard RL methods, Q-learning and Dyna-Q, and POMCP for planning under a POMDP in environments with explicit measurement costs.

9. Recognition of a Robot’s Affective Expressions under Conditions with Limited Visibility

   Ghafurian, Moojan, Akgun, Sami Alperen, Crowley, Mark, and Dautenhahn, Kerstin

   In 18th International Conference promoted by the IFIP Technical Committee 13 on Human–Computer Interaction (INTERACT 2021) 2021

   Abs PDF

   The capability of showing affective expressions is important for the design of social robots in many contexts, especially where the robot is designed to communicate with humans. It is reasonable to expect that, similar to all other interaction modalities, communicating with affective expressions is not without limitations. In this paper, we present two online video studies (N=72 and N=50) and investigate if/how much the recognition of affective displays of a zoomorphic robot is affected under situations with different levels of visibility. Five affective expressions combined with five visibility effects were studied. The intensity of the effects was more pronounced in the second experiment. While visual constraints affected recognition of expressions, our results showed that affective displays of the robot conveyed through its head and body gestures can be robust and recognition rates can still be high even under severe constraints. This study supported the effectiveness of using affective displays as a complementary communication modality in human-robot interaction in situations with low visibility.

10. QQE

    Quantile–Quantile Embedding for distribution transformation and manifold embedding with ability to choose the embedding distribution

    Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

    Machine Learning with Applications (MLWA) 2021

    Abs arXiv PDF URL

    We propose a new embedding method, named Quantile-Quantile Embedding (QQE), for distribution transformation and manifold embedding with the ability to choose the embedding distribution. QQE, which uses the concept of quantile-quantile plot from visual statistical tests, can transform the distribution of data to any theoretical desired distribution or empirical reference sample. Moreover, QQE gives the user a choice of embedding distribution in embedding the manifold of data into the low dimensional embedding space. It can also be used for modifying the embedding distribution of other dimensionality reduction methods, such as PCA, t-SNE, and deep metric learning, for better representation or visualization of data. We propose QQE in both unsupervised and supervised forms. QQE can also transform a distribution to either an exact reference distribution or its shape. We show that QQE allows for better discrimination of classes in some cases. Our experiments on different synthetic and image datasets show the effectiveness of the proposed embedding method.

2020

1. iMondrian

   Isolation Mondrian Forest for Batch and Online Anomaly Detection

   Ma, Haoran, Ghojogh, Benyamin, Samad, Maria N, Zheng, Dongyu, and Crowley, Mark

   In IEEE International Conference on Systems, Man, and Cybernetics (IEEE-SMC-2020) 2020

   Abs PDF

   We propose a new method, named isolation Mon- drian forest (iMondrian forest), for batch and online anomaly detection. The proposed method is a novel hybrid of isolation forest and Mondrian forest which are existing methods for batch anomaly detection and online random forest, respectively. iMondrian forest takes the idea of isolation, using the depth of a node in a tree, and implements it in the Mondrian forest structure. The result is a new data structure which can accept streaming data in an online manner while being used for anomaly detection. Our experiments show that iMondrian forest mostly performs better than isolation forest in batch settings and has better or comparable performance against other batch and online anomaly detection methods.

2. patent

   Multi-Level Collaborative Control System With Dual Neural Network Planning For Autonomous Vehicle Control In A Noisy Environment

   Du, Zhiyuan, Lull, Joseph, Malhan, Rajesh, Ganapathi Subramanian, Sriram, Bhalla, Sushrut, Sambee, Jaspreet, Crowley, Mark, Fischmeister, Sebastian, Shin, Donghyun, Melek, William, Fidan, Baris, Woo, Ami, and Sahoo, Bismaya

   2020

   Abs PDF URL

   A RLP system for a host vehicle includes a memory and levels. The memory stores a RLP algorithm, which is a multi-agent collaborative DQN with PER algorithm. A first level includes a data processing module that provides sensor data, object location data, and state information of the host vehicle and other vehicles. A second level includes a coordinate location module that, based on the sensor data, the object location data, the state information, and a refined policy provided by the third level, generates an updated policy and a set of future coordinate locations implemented via the first level. A third level includes evaluation and target neural networks and a processor that executes instructions of the RLP algorithm for collaborative action planning between the host and other vehicles based on outputs of the evaluation and target networks and to generate the refined policy based on reward values associated with events.

3. A review of machine learning applications in wildfire science and management

   Jain, Piyush, Coogan, Sean CP, Ganapathi Subramanian, Sriram, Crowley, Mark, Taylor, Steve, and Flannigan, Mike D

   Environmental Reviews 2020

   Abs arXiv PDF Slides URL

   Artificial intelligence has been applied in wildfire science and management since the 1990s, with early applications including neural networks and expert systems. Since then the field has rapidly progressed congruently with the wide adoption of machine learning (ML) methods in the environmental sciences. Here, we present a scoping review of ML applications in wildfire science and management. Our overall objective is to improve awareness of ML methods among wildfire researchers and managers, as well as illustrate the diverse and challenging range of problems in wildfire science available to ML data scientists. To that end, we first present an overview of popular ML approaches used in wildfire science to date, and then review the use of ML in wildfire science as broadly categorized into six problem domains, including: 1) fuels characterization, fire detection, and mapping; 2) fire weather and climate change; 3) fire occurrence, susceptibility, and risk; 4) fire behavior prediction; 5) fire effects; and 6) fire management. Furthermore, we discuss the advantages and limitations of various ML approaches relating to data size, computational requirements, generalizability, and interpretability, as well as identify opportunities for future advances in the science and management of wildfires within a data science context. In total, we identified 298 relevant publications, where the most frequently used ML methods across problem domains included random forests, MaxEnt, artificial neural networks, decision trees, support vector machines, and genetic algorithms. As such, there exists opportunities to apply more current ML methods — including deep learning and agent based learning — in the wildfire sciences, especially in instances involving very large multivariate datasets. We must recognize, however, that despite the ability of ML models to learn on their own, expertise in wildfire science is necessary to ensure realistic modelling of fire processes across multiple scales, while the complexity of some ML methods, such as deep learning, requires a dedicated and sophisticated knowledge of their application. Finally, we stress that the wildfire research and management communities play an active role in providing relevant, high quality, and freely available wildfire data for use by practitioners of ML methods.

4. Supervision and Source Domain Impact on Representation Learning: A Histopathology Case Study

   Sikaroudi, Milad, Safarpoor, Amir, Ghojogh, Benyamin, Shafiei, Sobhan, Crowley, Mark, and Tizhoosh, HR

   In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’20) 2020

   Abs arXiv PDF

   As many algorithms depend on a suitable representation of data, learning unique features is considered a crucial task. Although supervised techniques using deep neural networks have boosted the performance of representation learning, the need for a large set of labeled data limits the application of such methods. As an example, high-quality delineations of regions of interest in the field of pathology is a tedious and time-consuming task due to the large image dimensions. In this work, we explored the performance of a deep neural network and triplet loss in the area of representation learning. We investigated the notion of similarity and dissimilarity in pathology whole-slide images and compared different setups from unsupervised and semi-supervised to supervised learning in our experiments. Additionally, different approaches were tested, applying few-shot learning on two publicly available pathology image datasets. We achieved high accuracy and generalization when the learned representations were applied to two different pathology datasets

5. Distributed Nonlinear Model Predictive Control and Metric Learning for Heterogeneous Vehicle Platooning with Cut-in/Cut-out Maneuvers

   Basiri, Mohammad Hossein, Ghojogh, Benyamin, Azad, Nasser L, Fischmeister, Sebastian, Karray, Fakhri, and Crowley, Mark

   In Proceeding of the 59th IEEE Conference on Decision and Control (CDC-2020) 2020

2019

1. Semantic Workflows and Machine Learning for the Assessment of Carbon Storage by Urban Trees

   Carrillo, Juan, Garijo, Daniel, Crowley, Mark, Gil, Yolanda, and Borda, Katherine

   In Third International Workshop on Capturing Scientific Knowledge (Sciknow 2019), Collocated with the tenth International Conference on Knowledge Capture (K-CAP) 2019
2. Instance Ranking and Numerosity Reduction Using Matrix Decompositionand Subspace Learning

   Ghojogh, Benyamin, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2019

   Abs

   One way to deal with the ever increasing amount of available data for processing is to rank data instances by usefulness and reduce the dataset size. In this work, we introduce a framework to achieve this using matrix decomposition and subspace learning. Our central contribution is a novel similarity measure for data instances that uses the basis obtained from matrix decomposition of the dataset. Using this similarity measure, we propose several related algorithms for ranking data instances and performing numerosity reduction. We then validate the effectiveness of these algorithms for data reduction on several datasets for classification, regression, and clustering tasks.

3. Principal Component Analysis Using Structural Similarity Index for Images

   Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

   In International Conference on Image Analysis and Recognition (ICIAR-19) 2019

2018

1. MCTS+A3C

   Combining MCTS and A3C for prediction of spatially spreading processes in forest wildfire settings

   Ganapathi Subramanian, Sriram, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2018

   Abs PDF URL

   In recent years, Deep Reinforcement Learning (RL) algorithms have shown super-human performance in a variety Atari and classic board games like chess and GO. Research into applications of RL in other domains with spatial considerations like environmental planning are still in their nascent stages. In this paper, we introduce a novel combination of Monte-Carlo Tree Search (MCTS) and A3C algorithms on an online simulator of a wildfire, on a pair of forest fires in Northern Alberta (Fort McMurray and Richardson fires) and on historical Saskatchewan fires previously compared by others to a physics-based simulator. We conduct several experiments to predict fire spread for several days before and after the given spatial information of fire spread and ignition points. Our results show that the advancements in Deep RL applications in the gaming world have advantages in spatially spreading real-world problems like forest fires. \textcopyright Springer International Publishing AG, part of Springer Nature 2018.

2. Using Spatial Reinforcement Learning to Build Forest Wildfire Dynamics Models From Satellite Images

   Ganapathi Subramanian, Sriram, and Crowley, Mark

   Frontiers in ICT 2018

   Abs PDF URL

   Machine learning algorithms have increased tremendously in power in recent years but have yet to be fully utilized in many ecology and sustainable resource management domains such as wildlife reserve design, forest fire management and invasive species spread. One thing these domains have in common is that they contain dynamics that can be characterized as a Spatially Spreading Process (SSP) which requires many parameters to be set precisely to model the dynamics, spread rates and directional biases of the elements which are spreading. We present related work in Artificial Intelligence and Machine Learning for SSP sustainability domains including forest wildfire prediction. We then introduce a novel approach for learning in SSP domains using Reinforcement Learning (RL) where fire is the agent at any cell in the landscape and the set of actions the fire can take from a location at any point in time includes spreading North, South, East, West or not spreading. This approach inverts the usual RL setup since the dynamics of the corresponding Markov Decision Process (MDP) is a known function for immediate wildfire spread. Meanwhile, we learn an agent policy for a predictive model of the dynamics of a complex spatially-spreading process. Rewards are provided for correctly classifying which cells are on fire or not compared to satellite and other related data. We examine the behaviour of five RL algorithms on this problem: Value Iteration, Policy Iteration, Q-Learning, Monte Carlo Tree Search and Asynchronous Advantage Actor-Critic (A3C). We compare to a Gaussian process based supervised learning approach and discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We also discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We validate our approach with satellite image data of two massive wildfire events in Northern Alberta, Canada, the Fort McMurray fire of 2016 and the Richardson fire of 2011. The results show that we can learn predictive, agent-based policies as models of spatial dynamics using RL on readily available satellite images that other methods and have many additional advantages in terms of generalizability and interpretability.

2017

1. Learning Forest Wildfire Dynamics from Satellite Images Using Reinforcement Learning

   Subramanian, Sriram Ganapathi, and Crowley, Mark

   In Conference on Reinforcement Learning and Decision Making 2017
2. AI Education Through Real World Problems

   Crowley, Mark

   In The Seventh Symposium on Educational Advances in Artificial Intellgience. 2017

2016

1. Anomaly Detection Using Inter-Arrival Curves for Real-time Systems

   Salem, Mahmoud, Crowley, Mark, and Fischmeister, Sebastian

   In 2016 28th Euromicro Conference on Real-Time Systems 2016

   Abs URL

   —Real-time embedded systems are a significant class of applications, poised to grow even further as automated vehicles and the Internet of Things become a reality. An important problem for these systems is to detect anomalies during operation. Anomaly detection is a form of classification, which can be driven by data collected from the system at execution time. We propose inter-arrival curves as a novel analytic modelling technique for discrete event traces. Our approach relates to the existing technique of arrival curves and expands the technique to anomaly detection. Inter-arrival curves analyze the behaviour of events within a trace by providing upper and lower bounds to their inter-arrival occurrence. We exploit inter-arrival curves in a classification framework that detects deviations within these bounds for anomaly detection. Also, we show how inter-arrival curves act as good features to extract recurrent behaviour that these systems often exhibit. We demonstrate the feasibility and viability of the fully implemented approach with an industrial automotive case study (CAN traces) as well as a deployed aerospace case study (RTOS kernel traces).

2015

1. PAC Optimal MDP Planning with Application to Invasive Species Management

   Taleghan, Majid Alkaee, Dietterich, Thomas G., Crowley, Mark, Hall, Kim, and Albers, H. Jo

   Journal of Machine Learning Research 2015

   Abs URL

   In a simulator-defined MDP, the Markovian dynamics and rewards are provided in the form of a simulator from which samples can be drawn. This paper studies MDP planning algorithms that attempt to minimize the number of simulator calls before terminating and outputting a policy that is approximately optimal with high probability. The paper introduces two heuristics for efficient exploration and an improved confidence interval that enables earlier termination with probabilis- tic guarantees. We prove that the heuristics and the confidence interval are sound and produce with high probability an approximately optimal policy in polynomial time. Experiments on two benchmark problems and two instances of an invasive species management problem show that the improved confidence intervals and the new search heuristics yield reductions of between 8% and 47% in the number of simulator calls required to reach near-optimal policies.

2014

1. Using equilibrium policy gradients for spatiotemporal planning in forest ecosystem management

   Crowley, Mark

   IEEE Transactions on Computers 2014

   Abs URL

   Spatiotemporal planning involves making choices at multiple locations in space over some planning horizon to maximize utility and satisfy various constraints. In Forest Ecosystem Management, the problem is to choose actions for thousands of locations each year including harvesting, treating trees for fire or pests, or doing nothing. The utility models could place value on sale of lumber, ecosystem sustainability or employment levels and incorporate legal and logistical constraints on actions such as avoiding large contiguous areas of clearcutting. Simulators developed by forestry researchers provide detailed dynamics but are generally inaccesible black boxes. We model spatiotemporal planning as a factored Markov decision process and present a policy gradient planning algorithm to optimize a stochastic spatial policy using simulated dynamics. It is common in environmental and resource planning to have actions at different locations be spatially interelated, this makes representation and planning challenging. We define a global spatial policy in terms of interacting local policies defining distributions over actions at each location conditioned on actions at nearby locations. Markov chain Monte Carlo simulation is used to sample landscape policies and estimate their gradients. Evaluation is carried out on a forestry planning problem with 1,880 locations using a variety of value models and constraints. Index

2013

1. Allowing a wildfire to burn: Estimating the effect on future fire suppression costs

   Houtman, Rachel M., Montgomery, Claire A., Gagnon, Aaron R., Calkin, David E., Dietterich, Thomas G., McGregor, Sean, and Crowley, Mark

   International Journal of Wildland Fire 2013

   Abs PDF URL

   Where a legacy of aggressive wildland fire suppression has left forests in need of fuel reduction, allowing wildland fire to burn may provide fuel treatment benefits, thereby reducing suppression costs from subsequent fires. The least-cost-plus-net-value-change model of wildland fire economics includes benefits of wildfire in a framework for evaluating suppression options. In this study, we estimated one component of that benefit – the expected present value of the reduction in suppression costs for subsequent fires arising from the fuel treatment effect of a current fire. To that end, we employed Monte Carlo methods to generate a set of scenarios for subsequent fire ignition and weather events, which are referred to as sample paths, for a study area in central Oregon. We simulated fire on the landscape over a 100-year time horizon using existing models of fire behaviour, vegetation and fuels development, and suppression effectiveness, and we estimated suppression costs using an existing suppression cost model. Our estimates suggest that the potential cost savings may be substantial. Further research is needed to estimate the full least-cost-plus-net-value-change model. This line of research will extend the set of tools available for developing wildfire management plans for forested landscapes.

2. PAC Optimal Planning for Invasive Species Management: Improved Exploration for Reinforcement Learning from Simulator-Defined MDPs

   Dietterich, Thomas G, Alkaee Taleghan, Majid, and Crowley, Mark

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2013) 2013

   Abs URL

   Often the most practical way to define a Markov Decision Process (MDP) is as a simulator that, given a state and an action, produces a resulting state and immediate reward sampled from the corresponding distributions. Simulators in natural resource management can be very expensive to execute, so that the time required to solve such MDPs is dominated by the number of calls to the simulator. This paper presents an algorithm, DDV, that combines improved confidence intervals on the Q values (as in interval estimation) with a novel upper bound on the discounted state occupancy probabilities to intelligently choose state-action pairs to explore. We prove that this algorithm terminates with a policy whose value is within εof the optimal policy (with probability 1−δ) after making only polynomially-many calls to the simulator. Experiments on one benchmark MDP and on an MDP for invasive species management show very large reductions in the number of simulator calls required.

3. Cyclic causal models with discrete variables: Markov chain equilibrium semantics and sample ordering

   Poole, David, and Crowley, Mark

   In IJCAI International Joint Conference on Artificial Intelligence 2013

   Abs URL

   We analyze the foundations of cyclic causal models for discrete variables, and compare structural equation models (SEMs) to an alternative semantics as the equilibrium (stationary) distribution of a Markov chain. We show under general conditions, discrete cyclic SEMs cannot have independent noise, even in the simplest case, cyclic structural equation models imply constraints on the noise. We give a formalization of an alternative Markov chain equilibrium semantics which requires not only the causal graph, but also a sample order. We show how the resulting equilibrium is a function of the sample ordering, both theoretically and empirically.

2012

2011