Mark Crowley | Lab News April 2023

Manifold Learning Book finally released

Our long-awaited, much discussed (by us at least), textbook on Manifold Learning and Dimensionality Reduction (Ghojogh et al., 2023) has now been published officially via Springer Nature and the text is now available at Amazon and other fine booksellers {;)}.

The reference and teaching textbook is on the broad topic area of my former PhD student Benyamin Ghojogh’s thesis. Benyamin led the massive effort involved in the completion of this textbook. We’re confident it will be a fantastic resource for students and researchers wanting a wide view of this field but also one built from the ground up from the fundamentals at each step.

These are fundamental tools and concepts for all data analysis and machine learning professionals. So if you’re interested in a deep dive into this important area, and take a look at our book.

ICML Paper Accepted

Bagi, S. S. G., Gharaee, Z., Schulte, O., & Crowley, M. (2023). Generative Causal Representation Learning for Out-of-Distribution Motion Forecasting. International Conference on Machine Learning (ICML), 17.

With all the excitement recently in advances in Artificial Intelligence and Machine Learning (AI/ML), especially Generative AI models such as OpenAI’s GPT, there is a huge need for addressing a broader notion of AI that will be needed for use in the real world. Most of these methods can interpolate, predict, generate new patterns from data but they do this by learning correlations between features in the data, and datapoints over time. But there is a huge literature with methods and theory that deal with other kinds of relationships we can use to describe data, such as causality, uncertainty, confidence, and truth.

In this paper (Bagi et al., 2023) we propose a new framework for leveraging causal information to improve robustness of learning in the presence of distribution shift. Empirical results on the pedestrian motion forecasting domain support our theoretical findings. Prof. Crowley will be presenting this world at ICML 2023 in Hawai’i in July 2023. You can read the whole paper on Arxiv right now.

References:

Textbook

Elements of Dimensionality Reduction and Manifold Learning

Benyamin Ghojogh, Mark Crowley, Fakhri Karray, and Ali Ghodsi.

Springer Nature, Feb, 2023.

Abs URL

Dimensionality reduction, also known as manifold learning, is an area of machine learning used for extracting informative features from data, for better representation of data or separation between classes. This book presents a cohesive review of linear and nonlinear dimensionality reduction and manifold learning. Three main aspects of dimensionality reduction are covered – spectral dimensionality reduction, probabilistic dimensionality reduction, and neural network-based dimensionality reduction, which have geometric, probabilistic, and information-theoretic points of view to dimensionality reduction, respectively. This book delves into basic concepts and recent developments in the field of dimensionality reduction and manifold learning, providing the reader with a comprehensive understanding. The necessary background and preliminaries, on linear algebra, optimization, and kernels, are also explained in the book to ensure a comprehensive understanding of the algorithms. The tools this book introduces are applied to various applications involving feature extraction, image processing, computer vision, and signal processing. This book is applicable to a wide audience who would like to acquire a deep understanding of the various ways to extract, transform, and understand the structure of data. The intended audiences are academics, students, and industry professionals. Academic researchers and students can use this book as a textbook for machine learning and dimensionality reduction. Data scientists, machine learning scientists, computer vision scientists, computer scientists, statisticians, and mathematicians in the fields of applied mathematics, statistical learning, Riemannian manifolds, subspace analysis, linear algebra, and optimization can use this book as a reference book for both technical and applied concepts. Industry professionals, including applied engineers, data engineers, and engineers in various fields of science dealing with machine learning, can use this book as a guidebook for feature extraction from their data as the raw data in industry often requires pre-processing. Data feature extraction is useful in various fields of science including engineering, physics, chemistry, biometrics, biomedical signals and images, etc. This book is structured as a reference textbook, so that it can be used for advanced courses, as an in-depth supplementary resource or for researchers or practitioners who want to learn about dimensionality reduction and manifold learning. The book is grounded in theory, but provides thorough explanations and diverse examples to improve the readers comprehension of the advanced topics. Advanced methods are explained in a step-by-step manner so that readers of all levels can follow the reasoning and come to a deep understanding of the concepts. This book does not assume the reader has an advanced theoretical background in machine learning and provides necessary background, though an undergraduate-level background in linear algebra and calculus is recommended.

GCRL

Generative Causal Representation Learning for Out-of-Distribution Motion Forecasting

Shayan Shirahmad Gale Bagi, Zahra Gharaee, Oliver Schulte, and Mark Crowley

In International Conference on Machine Learning (ICML). Honolulu, Hawaii, USA. Jul, 2023.

Abs arXiv PDF Code Poster Hypoth

Conventional supervised learning methods typically assume i.i.d samples and are found to be sensitive to out-of-distribution (OOD) data. We propose Generative Causal Representation Learning (GCRL) which leverages causality to facilitate knowledge transfer under distribution shifts. While we evaluate the effectiveness of our proposed method in human trajectory prediction models, GCRL can be applied to other domains as well. First, we propose a novel causal model that explains the generative factors in motion forecasting datasets using features that are common across all environments and with features that are specific to each environment. Selection variables are used to determine which parts of the model can be directly transferred to a new environment without fine-tuning. Second, we propose an end-to-end variational learning paradigm to learn the causal mechanisms that generate observations from features. GCRL is supported by strong theoretical results that imply identifiability of the causal model under certain assumptions. Experimental results on synthetic and real-world motion forecasting datasets show the robustness and effectiveness of our proposed method for knowledge transfer under zero-shot and low-shot settings by substantially outperforming the prior motion forecasting models on out-of-distribution prediction.