showcase

2021

1. Partially Observable Mean Field Reinforcement Learning

   Ganapathi Subramanian, Sriram, Taylor, Matthew, Crowley, Mark, and Poupart, Pascal

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems 2021

   Abs arXiv

   Traditional multi-agent reinforcement learning algorithms are not scalable to environments with more than a few agents, since these algorithms are exponential in the number of agents. Recent research has introduced successful methods to scale multi-agent reinforcement learning algorithms to many agent scenarios using mean field theory. Previous work in this field assumes that an agent has access to exact cumulative metrics regarding the mean field behaviour of the system, which it can then use to take its actions. In this paper, we relax this assumption and maintain a distribution to model the uncertainty regarding the mean field of the system. We consider two different settings for this problem. In the first setting, only agents in a fixed neighbourhood are visible, while in the second setting, the visibility of agents is determined at random based on distances. For each of these settings, we introduce a Q-learning based algorithm that can learn effectively. We prove that this Q-learning estimate stays very close to the Nash Q-value (under a common set of assumptions) for the first setting. We also empirically show our algorithms outperform multiple baselines in three different games in the MAgents framework, which supports large environments with many agents learning simultaneously to achieve possibly distinct goals.

2. Batch-Incremental Triplet Sampling for Training Triplet Networks Using Bayesian Updating Theorem

   Sikaroudi, Milad, Ghojogh, Benyamin, Karray, Fakhri, Crowley, Mark, and Tizhoosh, H. R.

   In 25th International Conference on Pattern Recognition (ICPR) 2021

   PDF
3. Active Measure Reinforcement Learning for Observation Cost Minimization: A framework for minimizing measurement costs in reinforcement learning

   Bellinger, Colin, Coles, Rory, Crowley, Mark, and Tamblyn, Isaac

   In Canadian Conference on Artificial Intelligence 2021

   Abs PDF

   Markov Decision Processes (MDP) with explicit measurement cost are a class of en- vironments in which the agent learns to maximize the costed return. Here, we define the costed return as the discounted sum of rewards minus the sum of the explicit cost of measuring the next state. The RL agent can freely explore the relationship between actions and rewards but is charged each time it measures the next state. Thus, an op- timal agent must learn a policy without making a large number of measurements. We propose the active measure RL framework (Amrl) as a solution to this novel class of problem, and contrast it with standard reinforcement learning under full observability and planning under partially observability. We demonstrate that Amrl-Q agents learn to shift from a reliance on costly measurements to exploiting a learned transition model in order to reduce the number of real-world measurements and achieve a higher costed return. Our results demonstrate the superiority of Amrl-Q over standard RL methods, Q-learning and Dyna-Q, and POMCP for planning under a POMDP in environments with explicit measurement costs.

2020

1. Recognition of a Robot’s Affective Expressions under Conditions with Limited Visibility

   Ghafurian, Moojan, Akgun, Sami Alperen, Crowley, Mark, and Dautenhahn, Kerstin

   In International Conference on Human-Robot Interaction 2020

   Abs

   The capability of showing affective expressions is important for the design of social robots in many contexts, especially where the robot is designed to communicate with humans. It is reasonable to expect that, similar to all other interaction modalities, communicating with affective expressions is not without limitations. In this paper, we present two online video studies (N=72 and N=50) and investigate if/how much the recognition of affective displays of a zoomorphic robot is affected under situations with different levels of visibility. Five affective expressions combined with five visibility effects were studied. The intensity of the effects was more pronounced in the second experiment. While visual constraints affected recognition of expressions, our results showed that affective displays of the robot conveyed through its head and body gestures can be robust and recognition rates can still be high even under severe constraints. This study supported the effectiveness of using affective displays as a complementary communication modality in human-robot interaction in situations with low visibility.

2. Isolation Mondrian Forest for Batch and Online Anomaly Detection

   Ma, Haoran, Ghojogh, Benyamin, Samad, Maria N, Zheng, Dongyu, and Crowley, Mark

   In IEEE International Conference on Systems, Man, and Cybernetics (IEEE-SMC-2020) 2020
3. A review of machine learning applications in wildfire science and management

   Jain, Piyush, Coogan, Sean CP, Ganapathi Subramanian, Sriram, Crowley, Mark, Taylor, Steve, and Flannigan, Mike D

   Environmental Reviews 2020

   arXiv PDF
4. Deep Multi Agent Reinforcement Learning for Autonomous Driving

   Bhalla, Sushrut, Ganapathi Subramanian, Sriram, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2020
5. Supervision and Source Domain Impact on Representation Learning: A Histopathology Case Study

   Sikaroudi, Milad, Safarpoor, Amir, Ghojogh, Benyamin, Shafiei, Sobhan, Crowley, Mark, and Tizhoosh, HR

   In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’20) 2020

   Abs

   As many algorithms depend on a suitable representation of data, learning unique features is considered a crucial task. Although supervised techniques using deep neural networks have boosted the performance of representation learning, the need for a large set of labeled data limits the application of such methods. As an example, high-quality delineations of regions of interest in the field of pathology is a tedious and time-consuming task due to the large image dimensions. In this work, we explored the performance of a deep neural network and triplet loss in the area of representation learning. We investigated the notion of similarity and dissimilarity in pathology whole-slide images and compared different setups from unsupervised and semi-supervised to supervised learning in our experiments. Additionally, different approaches were tested, applying few-shot learning on two publicly available pathology image datasets. We achieved high accuracy and generalization when the learned representations were applied to two different pathology datasets

6. Distributed Nonlinear Model Predictive Control and Metric Learning for Heterogeneous Vehicle Platooning with Cut-in/Cut-out Maneuvers

   Basiri, Mohammad Hossein, Ghojogh, Benyamin, Azad, Nasser L, Fischmeister, Sebastian, Karray, Fakhri, and Crowley, Mark

   In Proceeding of the 59th IEEE Conference on Decision and Control (CDC-2020) 2020
7. Reinforcement Learning in a Physics-Inspired Semi-Markov Environment

   Bellinger, Colin, Coles, Rory, Crowley, Mark, and Tamblyn, Isaac

   In Canadian Conference on Artificial Intelligence 2020

2019

1. Semantic Workflows and Machine Learning for the Assessment of Carbon Storage by Urban Trees

   Carrillo, Juan, Garijo, Daniel, Crowley, Mark, Gil, Yolanda, and Borda, Katherine

   In Third International Workshop on Capturing Scientific Knowledge (Sciknow 2019), Collocated with the tenth International Conference on Knowledge Capture (K-CAP) 2019
2. Instance Ranking and Numerosity Reduction Using Matrix Decompositionand Subspace Learning

   Ghojogh, Benyamin, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2019

   Abs

   One way to deal with the ever increasing amount of available data for processing is to rank data instances by usefulness and reduce the dataset size. In this work, we introduce a framework to achieve this using matrix decomposition and subspace learning. Our central contribution is a novel similarity measure for data instances that uses the basis obtained from matrix decomposition of the dataset. Using this similarity measure, we propose several related algorithms for ranking data instances and performing numerosity reduction. We then validate the effectiveness of these algorithms for data reduction on several datasets for classification, regression, and clustering tasks.

3. Principal Component Analysis Using Structural Similarity Index for Images

   Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark

   In International Conference on Image Analysis and Recognition (ICIAR-19) 2019

2018

1. Combining MCTS and A3C for prediction of spatially spreading processes in forest wildfire settings

   Ganapathi Subramanian, Sriram, and Crowley, Mark

   In Canadian Conference on Artificial Intelligence 2018

   Abs

   In recent years, Deep Reinforcement Learning (RL) algorithms have shown super-human performance in a variety Atari and classic board games like chess and GO. Research into applications of RL in other domains with spatial considerations like environmental planning are still in their nascent stages. In this paper, we introduce a novel combination of Monte-Carlo Tree Search (MCTS) and A3C algorithms on an online simulator of a wildfire, on a pair of forest fires in Northern Alberta (Fort McMurray and Richardson fires) and on historical Saskatchewan fires previously compared by others to a physics-based simulator. We conduct several experiments to predict fire spread for several days before and after the given spatial information of fire spread and ignition points. Our results show that the advancements in Deep RL applications in the gaming world have advantages in spatially spreading real-world problems like forest fires. \textcopyright Springer International Publishing AG, part of Springer Nature 2018.

2. Using Spatial Reinforcement Learning to Build Forest Wildfire Dynamics Models From Satellite Images

   Ganapathi Subramanian, Sriram, and Crowley, Mark

   Frontiers in ICT 2018

   Abs

   Machine learning algorithms have increased tremendously in power in recent years but have yet to be fully utilized in many ecology and sustainable resource management domains such as wildlife reserve design, forest fire management and invasive species spread. One thing these domains have in common is that they contain dynamics that can be characterized as a Spatially Spreading Process (SSP) which requires many parameters to be set precisely to model the dynamics, spread rates and directional biases of the elements which are spreading. We present related work in Artificial Intelligence and Machine Learning for SSP sustainability domains including forest wildfire prediction. We then introduce a novel approach for learning in SSP domains using Reinforcement Learning (RL) where fire is the agent at any cell in the landscape and the set of actions the fire can take from a location at any point in time includes spreading North, South, East, West or not spreading. This approach inverts the usual RL setup since the dynamics of the corresponding Markov Decision Process (MDP) is a known function for immediate wildfire spread. Meanwhile, we learn an agent policy for a predictive model of the dynamics of a complex spatially-spreading process. Rewards are provided for correctly classifying which cells are on fire or not compared to satellite and other related data. We examine the behaviour of five RL algorithms on this problem: Value Iteration, Policy Iteration, Q-Learning, Monte Carlo Tree Search and Asynchronous Advantage Actor-Critic (A3C). We compare to a Gaussian process based supervised learning approach and discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We also discuss the relation of our approach to manually constructed, state-of-the-art methods from forest wildfire modelling. We validate our approach with satellite image data of two massive wildfire events in Northern Alberta, Canada, the Fort McMurray fire of 2016 and the Richardson fire of 2011. The results show that we can learn predictive, agent-based policies as models of spatial dynamics using RL on readily available satellite images that other methods and have many additional advantages in terms of generalizability and interpretability.

2017

1. Learning Forest Wildfire Dynamics from Satellite Images Using Reinforcement Learning

   Subramanian, Sriram Ganapathi, and Crowley, Mark

   In Conference on Reinforcement Learning and Decision Making 2017
2. AI Education Through Real World Problems

   Crowley, Mark

   In The Seventh Symposium on Educational Advances in Artificial Intellgience. 2017

2016

1. Anomaly Detection Using Inter-Arrival Curves for Real-time Systems

   Salem, Mahmoud, Crowley, Mark, and Fischmeister, Sebastian

   In 2016 28th Euromicro Conference on Real-Time Systems 2016

   Abs

   —Real-time embedded systems are a significant class of applications, poised to grow even further as automated vehicles and the Internet of Things become a reality. An important problem for these systems is to detect anomalies during operation. Anomaly detection is a form of classification, which can be driven by data collected from the system at execution time. We propose inter-arrival curves as a novel analytic modelling technique for discrete event traces. Our approach relates to the existing technique of arrival curves and expands the technique to anomaly detection. Inter-arrival curves analyze the behaviour of events within a trace by providing upper and lower bounds to their inter-arrival occurrence. We exploit inter-arrival curves in a classification framework that detects deviations within these bounds for anomaly detection. Also, we show how inter-arrival curves act as good features to extract recurrent behaviour that these systems often exhibit. We demonstrate the feasibility and viability of the fully implemented approach with an industrial automotive case study (CAN traces) as well as a deployed aerospace case study (RTOS kernel traces).

2015

1. PAC Optimal MDP Planning with Application to Invasive Species Management

   Taleghan, Majid Alkaee, Dietterich, Thomas G., Crowley, Mark, Hall, Kim, and Albers, H. Jo

   Journal of Machine Learning Research 2015

   Abs

   In a simulator-defined MDP, the Markovian dynamics and rewards are provided in the form of a simulator from which samples can be drawn. This paper studies MDP planning algorithms that attempt to minimize the number of simulator calls before terminating and outputting a policy that is approximately optimal with high probability. The paper introduces two heuristics for efficient exploration and an improved confidence interval that enables earlier termination with probabilis- tic guarantees. We prove that the heuristics and the confidence interval are sound and produce with high probability an approximately optimal policy in polynomial time. Experiments on two benchmark problems and two instances of an invasive species management problem show that the improved confidence intervals and the new search heuristics yield reductions of between 8% and 47% in the number of simulator calls required to reach near-optimal policies.

2014

1. Using equilibrium policy gradients for spatiotemporal planning in forest ecosystem management

   Crowley, Mark

   IEEE Transactions on Computers 2014

   Abs

   Spatiotemporal planning involves making choices at multiple locations in space over some planning horizon to maximize utility and satisfy various constraints. In Forest Ecosystem Management, the problem is to choose actions for thousands of locations each year including harvesting, treating trees for fire or pests, or doing nothing. The utility models could place value on sale of lumber, ecosystem sustainability or employment levels and incorporate legal and logistical constraints on actions such as avoiding large contiguous areas of clearcutting. Simulators developed by forestry researchers provide detailed dynamics but are generally inaccesible black boxes. We model spatiotemporal planning as a factored Markov decision process and present a policy gradient planning algorithm to optimize a stochastic spatial policy using simulated dynamics. It is common in environmental and resource planning to have actions at different locations be spatially interelated, this makes representation and planning challenging. We define a global spatial policy in terms of interacting local policies defining distributions over actions at each location conditioned on actions at nearby locations. Markov chain Monte Carlo simulation is used to sample landscape policies and estimate their gradients. Evaluation is carried out on a forestry planning problem with 1,880 locations using a variety of value models and constraints. Index

2013

1. Allowing a wildfire to burn: Estimating the effect on future fire suppression costs

   Houtman, Rachel M., Montgomery, Claire A., Gagnon, Aaron R., Calkin, David E., Dietterich, Thomas G., McGregor, Sean, and Crowley, Mark

   International Journal of Wildland Fire 2013

   Abs

   Where a legacy of aggressive wildland fire suppression has left forests in need of fuel reduction, allowing wildland fire to burn may provide fuel treatment benefits, thereby reducing suppression costs from subsequent fires. The least-cost-plus-net-value-change model of wildland fire economics includes benefits of wildfire in a framework for evaluating suppression options. In this study, we estimated one component of that benefit – the expected present value of the reduction in suppression costs for subsequent fires arising from the fuel treatment effect of a current fire. To that end, we employed Monte Carlo methods to generate a set of scenarios for subsequent fire ignition and weather events, which are referred to as sample paths, for a study area in central Oregon. We simulated fire on the landscape over a 100-year time horizon using existing models of fire behaviour, vegetation and fuels development, and suppression effectiveness, and we estimated suppression costs using an existing suppression cost model. Our estimates suggest that the potential cost savings may be substantial. Further research is needed to estimate the full least-cost-plus-net-value-change model. This line of research will extend the set of tools available for developing wildfire management plans for forested landscapes.

2. PAC Optimal Planning for Invasive Species Management: Improved Exploration for Reinforcement Learning from Simulator-Defined MDPs

   Dietterich, Thomas G, Alkaee Taleghan, Majid, and Crowley, Mark

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2013) 2013

   Abs

   Often the most practical way to define a Markov Decision Process (MDP) is as a simulator that, given a state and an action, produces a resulting state and immediate reward sam- pled from the corresponding distributions. Simulators in nat- ural resource management can be very expensive to execute, so that the time required to solve such MDPs is dominated by the number of calls to the simulator. This paper presents an algorithm, DDV, that combines improved confidence inter- vals on the Q values (as in interval estimation) with a novel upper bound on the discounted state occupancy probabilities to intelligently choose state-action pairs to explore. We prove that this algorithm terminates with a policy whose value is within εof the optimal policy (with probability 1−δ) after making only polynomially-many calls to the simulator. Ex- periments on one benchmark MDP and on an MDP for inva- sive species management show very large reductions in the number of simulator calls required.

3. Cyclic causal models with discrete variables: Markov chain equilibrium semantics and sample ordering

   Poole, David, and Crowley, Mark

   In IJCAI International Joint Conference on Artificial Intelligence 2013

   Abs

   We analyze the foundations of cyclic causal models for discrete variables, and compare structural equation models (SEMs) to an alternative semantics as the equilibrium (stationary) distribution of a Markov chain. We show under general conditions, discrete cyclic SEMs cannot have independent noise, even in the simplest case, cyclic structural equation models imply constraints on the noise. We give a formalization of an alternative Markov chain equilibrium semantics which requires not only the causal graph, but also a sample order. We show how the resulting equilibrium is a function of the sample ordering, both theoretically and empirically.

2012

2011