Vision Language Navigation

Our Papers on Vision Language Navigation

1. Do Navigation VLA Models Ground What They Are Told? An Empirical Audit of Landmark-Vision Alignment in VLN

   Josh Qixuan Sun, Xiaoying Xing, Yurun Chen, Huaiyuan Weng, Wentao Zhu, and Mark Crowley

   In Winter Conference on Applications of Computer Vision (WACV). Jan, 2027.

   Abs PDF Supp

   Vision-and-language navigation (VLN) requires models to connect route instructions with visual observations when selecting actions, but standard navigation metrics do not reveal which visual evidence supports those actions. We introduce LandMarkVLN, a 7,228-sample evaluation dataset constructed from VLN benchmarks. Each sample contains a navigation instruction, a singular landmark phrase, current and historical observations, same-instance landmark masks, and local reference actions. Using LandMarkVLN, we audit eight released navigation VLA models with two complementary analyses. First, attribution-based vision-action (VxA) analysis measures whether visual attribution localizes the named landmark and whether such localization is more frequent for acceptable local actions. Only two of 60 VxA evaluations support a positive vision-action association. Second, matched visual intervention compares the output change caused by editing the landmark region with that caused by applying the same edit to a comparable non-landmark region. To distinguish landmark-specific effects from general visual responsiveness, we also apply a full-frame edit that removes image structure from all observations in the evaluated temporal scope. All 23 configurations show a positive mean response to the full-frame edit, but only six support a larger response to the landmark than to the matched control. These results show that acceptable local actions and broad visual sensitivity are not, by themselves, reliable evidence that a navigation VLA model relies on the instructed landmark.

2. V^2-VLNCE

   View Invariant Learning for Vision-Language Navigation in Continuous Environments

   Josh Qixuan Sun, Huaiyuan Weng, Xiaoying Xing, Chul Min Yeum, and Mark Crowley

   IEEE Robotics and Automation Letters (IEEE RA-L). 11, (5). 2026.

   Abs PDF Code URL

   Vision-Language Navigation in Continuous Environments (VLNCE), where an agent follows instructions and moves freely to reach a destination, is a key research problem in embodied AI. However, most existing approaches are sensitive to viewpoint changes, i.e. variations in camera height and viewing angle. Here we introduce a more general scenario, V2-VLNCE (VLNCE with Varied Viewpoints) and propose a view-invariant post-training framework, called VIL (View Invariant Learning), that makes existing navigation policies more robust to changes in camera viewpoint. VIL employs a contrastive learning framework to learn sparse and view-invariant features. We also introduce a teacher-student framework for the Waypoint Predictor Module, a standard part of VLNCE baselines, where a view-dependent teacher model distills knowledge into a view-invariant student model. We employ an end-to-end training paradigm to jointly optimize these components. Empirical results show that our method outperforms state-of-the-art approaches on V2-VLNCE by 8-15% measured on Success Rate for two standard benchmark datasets R2R-CE and RxR-CE. Evaluation of VIL in standard VLNCE settings shows that despite being trained for varied viewpoints, VIL often still improves performance. On the harder RxR-CE dataset, our method also achieved state-of-the-art performance across all metrics. This suggests that adding VIL does not diminish the standard viewpoint performance and can serve as a plug-and-play posttraining method. We further evaluate VIL for simulated camera placements derived from real robot configurations (e.g. Stretch RE1, LoCoBot), showing consistent improvements of performance. Finally, we present a proof-of-concept real-robot evaluation in two physical environments using a panoramic RGB sensor combined with LiDAR. These results show that VIL improves robustness not only in simulation but also in real-world navigation scenarios, making it a practical strategy for embodied agents. The code is available at https://github.com/realjoshqsun/V2-VLNCE.