Mark Crowley | Paper Accepted at WACV 2027 on Empirical Auditing of Landmark Grounding in VLN

We are excited to share that our paper (Sun et al., 2027) has been accepted to the Winter Conference on Applications of Computer Vision (WACV 2027)!

Led by Ph.D. student Josh Sun with help from all his coauthors, this work performs a kind of audit into whether existing Vision-Language Action (VLA) navigation models actually utilize the landmark instructions they are given rather than solving the task independent of those instructions. If this seems like a strange question, think groudning as a kind of allignment question. Measuring “performance” of the latest large-scale, language foundation models, whether they include vision inputs or not, is tricky for many reasons. Just one of those reasons is that these are very general pre-trained foundation models that are extremely effective at coming up with adequate solutions to general requests given their vast pre-training process.

The allignment question here is whether the model’s behaviour is grounded in the request we gave it. In other words, does it succeed at the task because of anything we told it or if it could have solved it with even less information? We have found that commong validation experiments don’t always test whether the base model being used could solve things on its own.

In this work we look at various ways to isolate what parts of the prompt and data given to the model were necessary and which were not. The result is somewhat surprising.

Sun, J. Q., Xing, X., Chen, Y., Weng, H., Zhu, W., & Crowley, M. (2027). Do Navigation VLA Models Ground What They Are Told? An Empirical Audit of Landmark-Vision Alignment in VLN. Winter Conference on Applications of Computer Vision (WACV), 10.

Josh did a stellar job spearheading this project, building LandMarkVLN, a 7,228-sample diagnostic evaluation benchmark, and conducting rigorous visual interventions across eight released VLA models. Josh demonstrated that local navigation actions and visual sensitivity do not necessarily imply that VLA models rely on instructed landmarks, giving us crucial insights for building truly reliable embodied AI agents.

Josh will present the work at the WACV conference in Florida in January 2027.

References:

Do Navigation VLA Models Ground What They Are Told? An Empirical Audit of Landmark-Vision Alignment in VLN

Josh Qixuan Sun, Xiaoying Xing, Yurun Chen, Huaiyuan Weng, Wentao Zhu, and Mark Crowley

In Winter Conference on Applications of Computer Vision (WACV). Jan, 2027.

Abs PDF Supp

Vision-and-language navigation (VLN) requires models to connect route instructions with visual observations when selecting actions, but standard navigation metrics do not reveal which visual evidence supports those actions. We introduce LandMarkVLN, a 7,228-sample evaluation dataset constructed from VLN benchmarks. Each sample contains a navigation instruction, a singular landmark phrase, current and historical observations, same-instance landmark masks, and local reference actions. Using LandMarkVLN, we audit eight released navigation VLA models with two complementary analyses. First, attribution-based vision-action (VxA) analysis measures whether visual attribution localizes the named landmark and whether such localization is more frequent for acceptable local actions. Only two of 60 VxA evaluations support a positive vision-action association. Second, matched visual intervention compares the output change caused by editing the landmark region with that caused by applying the same edit to a comparable non-landmark region. To distinguish landmark-specific effects from general visual responsiveness, we also apply a full-frame edit that removes image structure from all observations in the evaluated temporal scope. All 23 configurations show a positive mean response to the full-frame edit, but only six support a larger response to the landmark than to the matched control. These results show that acceptable local actions and broad visual sensitivity are not, by themselves, reliable evidence that a navigation VLA model relies on the instructed landmark.