student theses

2024

1. MASc Thesis

   Adversarial Black-Box Testing of Self-Driving Vehicle Algorithms with Reinforcement Learning

   Oleksandra Nahorna.

   UWSpace Thesis Repository, University of Waterloo, Dec, 2024.

   Abs URL

   Abstract: When we hear autonomous or ego vehicles, we imagine a safe, self-driving device. However, this designation is unsuitable for describing a self-driving car because the car’s automation is still being refined, and safety is being studied and constantly improved. Right now, we can observe mostly the vehicles that are not fully autonomous to be present on the road in the physical world. The autonomy introduced requires the driver to be able to switch and take control of the vehicle in the case of an emergency occurrence. On the contrary, for the car to be able to apply full autonomy and be recognized and approved by the government to be placed on the road, the vehicle algorithm should be able to resolve and predict cases that may occur on the road and be able to respond to the situation utilizing the decision-making approach. The physical world setting may be constrained in terms of stress-testing the ego-vehicle before its availability to the regular user-driver. Therefore, to broaden the ability to stress-test autonomous vehicles in various scenarios, the manufacturer of the autonomous vehicle and its algorithm may benefit from utilizing the ability to stress-test vehicles in the virtual vehicle setting. For the work presented instead of enhancing the ability of safety enhancement of the autonomous vehicle by training it as a white box using the algorithm exposed by the manufacturer - we are introducing the adversarial virtual environment setting with generated accidents by the algorithm for the provider/manufacturer of the autonomous vehicle and its algorithm to be stress-tested in a black-box approach without the exposure of the technology (ego-vehicle algorithm) by the manufacturer. Using the Reinforcement Learning technique, a single-agent or multi-agent attacker vehicle is trained to reproduce collisions in a virtual environment. The cases generated can be different, both simple and difficult to mimic a real-world setting. When we describe autonomous vehicle adversarial blackbox testing, we identify the target vehicle as a representation of the autonomous vehicle by determining the action space, kinematics, and behaviour of the vehicle mentioned as autonomous throughout the work. This vehicle representation mimics the autonomous vehicle approach/action/decision-making within the 2D setting of the chosen virtual environment, with constraints driven by the virtual environment. This work challenges realism, namely, it compares the likelihood of emergencies in generated car accidents to real-life cases. To do this, as in any other area, we study historical data on road accidents. These situations are compared with selectively generated instances, and a successful comparison is recorded and described by creating a narrative for the generated situation. Thereby smoothing the boundary between virtual and real accidents on the road. It is important to note that reproducing and preparing ego-vehicle for any situations that may arise on the road in a virtual environment can be an economical approach. After all, to train the model, only time is needed depending on the number of generated episodes, thereby reducing the cost of testing or pre-testing. And it will also help to recreate scenarios that are problematic for recreation in real, physical space. Testing cars using different algorithms can help to identify a pattern or algorithm suitable for a specific environment, type of road, and number of actors on the road. This work seeks to publicize and recommend using a virtual environment to improve the automotive industry.

2023

1. MASc Thesis

   Aggressiveness-Regulated Multi-agent Stress Testing of Autonomous Vehicles

   Xiaoliang Zhou.

   UWSpace Thesis Repository, University of Waterloo, Sep, 2023.

   Abs URL

   Abstract: The emerging era of autonomous vehicles (AVs) presents unprecedented potential for transforming global transportation. As these vehicles begin to permeate our streets, the challenge of ensuring their safety, especially in unprecedented scenarios, looms large, due to the infrequent occurrence of high-risk scenarios within an essentially infinite number of test cases. This Master’s thesis explores the intricate challenge of stress testing autonomous vehicles in simulated environments. The study delves into the application of multi-agent reinforcement learning (MARL) as a tool for stress testing AVs. Although MARL demands higher computational resources, it demonstrates strong ability in uncovering complex accident scenarios. This marks a shift from the state-of-the-art which deploys single-agent reinforcement algorithms that encounter limitations both in the quality of the generated accident scenarios and in their ability to generate complex accident scenarios as the number of traffic participants increases. Central to our approach is the integration of constraints that regulate the level of aggressiveness of traffic participants to induce more realistic and insightful accident scenarios. The thesis also presents the highway-attack-env, an environment for black-box AV testing that allows the assessment of both single and multi-agent reinforcement learning algorithms. The contributions of this research include the introduction of the aforementioned environment and a comprehensive benchmark, as well as a comparative analysis of single-agent and MARL algorithms, underscoring the superiority of the proposed multi-agent, aggressiveness-regulated methodology for AV validation.

2. PhD Thesis

   Deep Learning-Based Driver Behavior Detection on Simulated and Real Data

   Zehra Camlica.

   UWSpace Thesis Repository, University of Waterloo, Sep, 2023.

   Abs URL

   Abstract: Driver behaviour has a significant influence on vehicle accidents. Measuring and providing feedback on driver behaviour can provide significant benefits for understanding and improving road safety. Mobile phones can be leveraged for the detection of driver actions and characteristics from the broadest population of drivers. Mobile phones also offer easy accessibility for cost-effective and reliable information with the built-in sensors available on them, such as the Global Positional System (GPS) and Inertial Measurement Unit (IMU). However, when it comes to a larger scale, obtaining labelled data from these mobile devices is still far from optimal for low-cost and reliable applications due to noise and missing data. In this study, data obtained from mobile phone sensors is simulated as a time series dataset using a traffic simulator and a robotic simulator. Then, the dataset is used with deep learning methods to classify both manoeuvres and driver behaviours, focusing specifically on aggressive driver behaviour. We propose a novel method using two Convolutional Neural Networks Convolutional Neural Networks (CNN) working in parallel to classify driver behaviours while classifying manoeuvres (i.e., aggressive right lane-change). We claim that the Parallel Convolutional Neural Network (PCNN) not only speeds up training time but also increases performance since having information about the manoeuvre helps improve behaviour classification performance. To validate this, first, a single task CNN for manoeuvre classification and a single task CNN aggressive/non-aggressive behaviour classifiers were built separately. The utility of the classifiers was demonstrated on a large simulated dataset created using the Sumo and Webots simulators. Subsequently, the PCNN classifier has been trained and validated on the big simulated dataset and a small driven dataset. We have also collected a dataset driven on real road by using GPS and IMU sensors and the PCNN model has been tested on this real dataset to investigate whether a classifier trained on simulated data can generalize to real data. In addition to this method, we propose a method of Spatial CNN with Attention (SCNN-A) layer to apply to our time series data with extracting more high-level features from spatial data for classification purposes.

3. MASc Thesis

   Understanding Driving Behaviours: Classification and Regression Approaches

   Josh Qixuan Sun.

   UWSpace Thesis Repository, University of Waterloo, Sep, 2023.

   Abs URL

   Abstract: The emergence of self-driving technology is poised to revolutionize transportation by empowering vehicles to operate autonomously. These advancements are classified into distinct levels of automation, delineating the progression from rudimentary driver assistance to complete autonomy. However, these advancements are accompanied by a spectrum of challenges. For example, the intricate interplay between autonomous and human-driven vehicles necessitates intuitive human interaction capabilities. This thesis focuses on driver behaviour learning, which is a way towards a safe and personal self-driving system. The eventual goal of driver behaviour learning is to equip self-driving vehicles with a deep understanding of human driving patterns. Leveraging learned human behaviours, self-driving vehicles can mirror familiar and intuitive driving actions, thereby fostering harmonious and predictable engagements on the road. This thesis focuses on investigating using different machine learning methods to learn an accurate and robust predictor of driver behaviours from multiple drivers and also personalize outputs for each individual driver. We define three driving behaviours. 1. Driver Identification: Given an observed driving execution history, identify the driver in the vehicle. 2. Lane Positioning: Given an observed driving execution history, predict the subject vehicle in the lane in the foreseeable future. 3. Lane Keeping: Given an observed driving execution history, predict the speed and steering angle of the subject vehicle in the foreseeable future. In the experiment, we mainly test with LSTM and Transformer, which are commonly used in time-series data. Specifically, we conduct experiments with LSTM and Transformer encoders to examine their capacity to model driving data in three behaviours. Furthermore, we employ LSTM and Transformer decoders to forecast lane positions, speeds, and angles. Variations in hyperparameter settings, such as width and depth, are investigated for all three behaviors in search of potential optimal configurations.

2022

2021

1. PhD Thesis

   Data Reduction Algorithms in Machine Learning and Data Science

   Benyamin Ghojogh.

   UWSpace Thesis Repository, University of Waterloo, Feb, 2021.

   Abs URL

   Abstract: Raw data are usually required to be pre-processed for better representation or discrimination of classes. This pre-processing can be done by data reduction, i.e., either reduction in dimensionality or numerosity (cardinality). Dimensionality reduction can be used for feature extraction or data visualization. Numerosity reduction is useful for ranking data points or finding the most and least important data points. This thesis proposes several algorithms for data reduction, known as dimensionality and numerosity reduction, in machine learning and data science. Dimensionality reduction tackles feature extraction and feature selection methods while numerosity reduction includes prototype selection and prototype generation approaches. This thesis focuses on feature extraction and prototype selection for data reduction. Dimensionality reduction methods can be divided into three categories, i.e., spectral, probabilistic, and neural network-based methods.

2. MASc Thesis

   Histopathology Image Analysis and NLP for Digital Pathology.

   Aishwarya Krishna Allada.

   UWSpace Thesis Repository, University of Waterloo, Waterloo, Canada. Aug, 2021.

   PDF URL

2020

1. MASc Thesis

   Stackelberg Multi-Agent Reinforcement Learning for Hierarchical Environments

   Sahil Pereira.

   UWSpace Thesis Repository, University of Waterloo, May, 2020.

   Abs URL

   Abstract: This thesis explores the application of multi-agent reinforcement learning in domains containing asymmetries between agents, caused by differences in information and position, resulting in a hierarchy of leaders and followers. Leaders are agents that have access to follower agent policies and the ability to commit to an action before the followers. These followers can observe actions taken by leaders and respond to maximize their own payoffs. Since leaders know the follower policies, they can manipulate the followers to elicit a better payoff for themselves. In this work, we focus on the problem of training agents in a multi-agent setting with continuous actions at different levels of hierarchy to obtain the best payoffs at their given positions. To address this problem we propose a new algorithm, Stackelberg Multi-Agent Reinforcement Learning (SMARL) that incorporates the Stackelberg equilibrium concept into the multi-agent deep deterministic policy gradient (MADDPG) algorithm. This enables us to efficiently train agents at all levels in the hierarchy. Since maximization over a continuous action space is intractable, we propose a method to solve our Stackelberg formulation for continuous actions using conditional actions and gradient descent. We evaluate our algorithm on multiple mixed cooperative and competitive multi-agent domains, consisting of our custom built highway driving environment and a subset of the multi-agent particle environments. We show that agents trained using our proposed algorithm outperform those trained with existing methods in most hierarchical domains, and are comparable in the rest.

2. MASc Thesis

   Deep Multi Agent Reinforcement Learning for Autonomous Driving

   Sushrut Bhalla.

   UWSpace Thesis Repository, University of Waterloo, Apr, 2020.

   Abs URL

   Abstract: Deep Learning and back-propagation have been successfully used to perform centralized training with communication protocols among multiple agents in a cooperative Multi-Agent Deep Reinforcement Learning (MARL) environment. In this work, I present techniques for centralized training of MARL agents in large scale environments and compare my work against current state of the art techniques. This work uses model-free Deep Q-Network (DQN) as the baseline model and allows inter agent communication for cooperative policy learning. I present two novel, scalable and centralized MARL training techniques (MA-MeSN, MA-BoN), which are developed under the principle that the behavior policy and message/communication policies have different optimization criteria. Thus, this work presents models which separate the message learning module from the behavior policy learning module. As shown in the experiments, the separation of these modules helps in faster convergence in complex domains like autonomous driving simulators and achieves better results than the current techniques in literature. Subsequently, this work presents two novel techniques for achieving decentralized execution for the communication based cooperative policy. The first technique uses behavior cloning as a method of cloning an expert cooperative policy to a decentralized agent without message sharing. In the second method, the behavior policy is coupled with a memory module which is local to each model. This memory model is used by the independent agents to mimic the communication policies of other agents and thus generate an independent behavior policy. This decentralized approach has minimal effect on degradation of the overall cumulative reward achieved by the centralized policy. Using a fully decentralized approach allows us to address the challenges of noise and communication bottlenecks in real-time communication channels. In this work, I theoretically and empirically compare the centralized and decentralized training algorithms to current research in the field of MARL. As part of this thesis, I also developed a large scale multi-agent testing environment. It is a new OpenAI-Gym environment which can be used for large scale multi-agent research as it simulates multiple autonomous cars driving cooperatively on a highway in the presence of a bad actor. I compare the performance of the centralized algorithms to existing state-of-the-art algorithms, for ex, DIAL and IMS which are based on cumulative reward achieved per episode and other metrics. MA-MeSN and MA-BoN achieve a cumulative reward of at-least 263% higher than the reward achieved by the DIAL and IMS. I also present an ablation study of the scalability of MA-BoN and show that MA-MeSN and MA-BoN algorithms only exhibit a linear increase in inference time and number of trainable parameters compared to quadratic increase for DIAL.

2019

1. MASc Thesis

   Deep Learning and Spatial Statistics for Determining Road Surface Condition

   Carrillo Garcia, and Juan Manuel.

   UWSpace Thesis Repository, University of Waterloo, Aug, 2019.

   Abs URL

   Abstract: Machine Learning (ML), and especially Deep Learning (DL) methods, have evolved rapidly over the last years and showed remarkable advances in research areas such as computer vision and natural language processing; however, there are still engineering applications in industries such as transportation where DL methods have not been applied yet or that can be benefited from an integrated approach using DL in addition to other methods. For countries in Northern latitudes, one of such applications is Monitoring Road Surface Condition (RSC) during the Winter season for improving road safety and road maintenance operations. In this study, we introduce a novel approach for monitoring of RSC that integrates DL methods and Spatial Statistics (SS) to simultaneously process data from roadside cameras and weather stations to determine automatically the category of snow coverage at sample locations across a region of interest. Our approach integrates the advantages of SS for interpolating spatial variables and the strengths of DL for Computer Vision tasks, particularly for image classification. On one hand, SS models serve to understand the spatial autocorrelation of random variables and to determine their expected values in unsampled locations based on a number of near observations. On the other hand, DL models extract relevant patterns from a large number of training images and learn a mapping from input images to a set of predefined labels. We implement and evaluate our approach using data collected in the province of Ontario during the 2017-2018 Winter season. Specifically, we included data from three separate sources, Environment Canada (EC) Weather stations, Road Weather Information System (RWIS) stations, and roadside cameras from the Ministry of Transportation of Ontario (MTO). To the best of our knowledge, this is the first study that integrates both DL and SS techniques for processing the three data sources with the goal of monitoring RSC. The DL models we implement and compared are Inception, Inception-Resnet, Xception, DenseNet, MobileNetv2, and NASNet. All of these models have achieved remarkable results for image classification in well-known benchmarks. The SS models we evaluate are Ordinary Kriging (OK), Radial Basis Functions (RBF), and Inverse Distance Weighted (IDW). The first provides a comprehensive understanding of the spatial autocorrelation for each particular variable, while the second and third allow a faster implementation. Our integrated approach works by combining the output feature vector from the DL model with the interpolated values from the SS model to output a more robust prediction of RSC for the locations of interest.

2. MASc Thesis

   Deep Representation Learning and Prediction for Forest Wildfires

   Pardis Zohouri Haghian.

   UWSpace Thesis Repository, University of Waterloo, May, 2019.

   Abs URL

   Abstract: An average of 8000 forest wildfires occurs each year in Canada burning an average of 2.5M ha/year as reported by the Government of Canada. Given the current rate of climate change, this number is expected to increase each year. Being able to predict how the fires spread would play a critical role in fire risk management. However, given the complexity of the natural processes that influence a fire system, most of the models used for simulating wildfires are computationally expensive and need a high variety of information about the environmental parameters to be able to give good performances. Deep learning algorithms allow computers to learn from experience and understand the world in terms of a hierarchy of concepts, with each concept defined in terms of its relation to simpler concepts. We propose a deep learning predictor that uses a Deep Convolutional Auto-Encoder to learn the key structures of a forest wildfire spread from images and a Long Short Term Memory to predict the next phase of the fire. We divided the predictor problem in three phases: find a dataset of wildfires, learning the essential structure of forest fire, and predict the next image. We first present the simulated wildfires dataset and the algorithm we applied on it to make it more suitable to the model. Then we present the Deep Forest Wildfire Auto-Encoder and its implementation using the Caffe framework. Particular attention is given to the design considerations and to the best practice used to implement the model. We also present the design of the Deep Forest Wildfire Predictor, and some possible future variations of it.

2018

1. MASc Thesis

   Population-Level Indicators of Physical Activity, Sedentary Behaviour and Sleep in Canada Based on Twitter

   Olivier Nguyen.

   UWSpace Thesis Repository, University of Waterloo, Aug, 2018.

   Abs URL

   Abstract: Social media platforms contain large amounts of freely and publicly available data that could be used to measure population characteristics across different geographical regions. Analyzing public data sources such as social media data has shown promising results for public health measures and monitoring. This thesis addresses challenges in building sys- tems that collect high-volumes of data from social media platforms. More specifically, we look at Twitter data processing, filtering, and aggregation to provide population-level in- dicators of physical activity, sedentary behavior, and sleep (PASS). In the first part of the thesis, we go over the whole machine learning pipeline built: (i) Twitter data collection from November 2017 to May 2018; (ii) data preparation through manual annotation, key- word filtering, and an active learning technique for the labelling of 10,283 tweets; and (iii) training a classifier to identify PASS related tweets. Training the model involves building an initial classifier to efficiently find relevant tweets in subsequent annotation iterations. Our classifiers include an ensemble model consisting of several shallow machine learning algorithms, along with deep learning algorithms. In the second part of the thesis, we look at the performance of different solutions. We provide benchmark results for the task of classifying PASS related tweets for the various algorithms considered. We also derive health indicators by aggregating and computing the proportion of classified tweets by province and compare our metrics with the prevalence of obesity, diabetes and mood disorders from the Canadian Community Health Survey. Our work shows how machine learning can be used to complement public health data and better inform health policy makers to improve the lives of Canadians.