ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models

Dingming Li^1,2*, Hongxing Li^1*, Zixuan Wang¹, Yuchen Yan¹, Hang Zhang¹, Siqi Chen¹, Guiyang Hou¹, Shengpei Jiang³, Wenqi Zhang¹, Yongliang Shen^1†, Weiming Lu¹, Yueting Zhuang¹,

¹Zhejiang University, ²University of Electronic Science and Technology of China, ³The Chinese University of Hong Kong
Preprint. Under review.
^*Equal Contribution, ^†Corresponding Author

Paper Code arXiv 🤗 Dataset

Learning from model weights

This work presents a range of spatial localization tasks requiring reasoning from both camera-centric and human-centric perspectives, revealing the challenges visual-language models (VLMs) face in multi-viewpoint spatial understanding. Current VLMs are predominantly trained on image-text pairs from the web that lack explicit 3D spatial annotations, limiting their cross-perspective spatial reasoning capabilities. To address this gap, we introduce ViewSpatial-Bench, the first comprehensive benchmark for evaluating spatial localization abilities across camera and human viewpoints spanning five distinct task types. Our benchmark leverages an automated 3D orientation annotation pipeline that efficiently generates diverse, scalable image datasets with precise directional labels. Additionally, this pipeline enables the creation of spatially annotated training data, substantially enhancing VLMs' multi-view spatial reasoning and cross-perspective generalization abilities.

Abstract

Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content, but significant challenges persist in tasks requiring cross-viewpoint understanding and spatial reasoning. We identify a critical limitation: current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints when required to adopt another entity's spatial frame of reference. We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation across five distinct task types, supported by an automated 3D annotation pipeline that generates precise directional labels. Comprehensive evaluation of diverse VLMs on ViewSpatial-Bench reveals a significant performance disparity: models demonstrate reasonable performance on camera-perspective tasks but exhibit reduced accuracy when reasoning from a human viewpoint. By fine-tuning VLMs on our multi-perspective spatial dataset, we achieve an overall performance improvement of 46.24% across tasks, highlighting the efficacy of our approach. Our work establishes a crucial benchmark for spatial intelligence in embodied AI systems and provides empirical evidence that modeling 3D spatial relationships enhances VLMs' corresponding spatial comprehension capabilities.

ViewSpatial-Bench construction pipeline

Directional Weight Score

ViewSpatial-Bench comprises five localization recognition tasks across two complementary perspective frameworks. From the camera perspective: (1) Object Relative Direction recognition(Cam-Rel. Dir.), which determines spatial relationships between objects directly from images; (2) Object View Orientation recognition(Cam-Obj. Oir.), which identifies the gaze direction of individuals relative to the camera from an egocentric viewpoint. These tasks evaluate VLMs' intuitive, egocentric spatial understanding abilities. From the human perspective: (3) Object Relative Direction recognition(Per-Rel. Dir.), which involves adopting the viewpoint of a character in the image to determine the spatial relationships of other objects from their perspective; (4) Object View Orientation recognition(Per-Obj. Oir.), which requires assuming the position of a character in the image to determine the direction of their gaze; (5) Scene Simulation Relative Direction recognition(Per-Sce. Sim.), which requires modeling oneself within a spatial scene across sequential frames to determine relative positions of other objects. These latter three tasks assess VLMs' abstract, perception-dependent spatial awareness while accommodating complex human pose variations and spatial information in embodied scenarios.

Question: Standing at desk, gazing at pillow, where should shelves be?

Choices:A. back-left B. front-right C. front-left D. back-right
Answer: B. front-right

Question type: Per-Sce. Sim.

Question: How is the chair positioned with respect to the pillow?

Choices:A. right B. back C. left D. above
Answer: A. right

Question type: Cam-Rel. Dir.

Question: Taking the camera lens as the front, what direction is the man looking toward?

Choices:A. back B. back-right C. left D. front
Answer: C. left

Question type: Cam-Obj. Dir.

Question: From the perspective of the man in white clothes, where is the man in red clothes?

Choices:A. front B. right C. left D. back-left
Answer: B. right

Question type: Per-Rel. Dir.

Question: As the man in black in the photo, in which direction are you facing?

Choices:A. back B. left C. front-right D. back-right
Answer: B. left

Question type: Per-Obj. Dir.

Multi-View Spatial Model

We present Multi-View Spatial Model (MVSM), developed to address limitations in perspective-dependent spatial reasoning in vision-language models. Following the ViewSpatial-Bench pipeline, we constructed a training dataset of ~43K diverse spatial relationship samples across five task categories, utilizing automated spatial annotations from ScanNet and MS-COCO data, supplemented with Spatial-MM for person-perspective tasks. Using consistent language templates and standardized directional classifications, we implemented a Multi-Perspective Fine-Tuning strategy on Qwen2.5-VL (3B) to enhance reasoning across different observational viewpoints. This approach enables MVSM to develop unified 3D spatial relationship representations that robustly support both camera and human perspective reasoning.

Zero-Shot Evaluation Performance

Directional Weight Score

Accuracy comparison across multiple VLMs on camera and human perspective spatial tasks. Our Multi-View Spatial Model (MVSM) significantly outperforms all baseline models across all task categories, demonstrating the effectiveness of our multi-perspective spatial fine-tuning approach. These results reveal fundamental limitations in perspective-based spatial reasoning capabilities among current VLMs. Even powerful proprietary models like GPT-4o (34.98%) and Gemini-2.0-Flash (32.56%) perform only marginally above random chance (26.33%), confirming our hypothesis that standard VLMs struggle with perspective-dependent spatial reasoning despite their strong performance on other vision-language tasks.

ViewSpatial-Bench Examples (Part 1). Performance comparison of three models (Qwen2.5-VL (3B), GPT-4o, and MVSM) on five spatial reasoning tasks from human perspective and camera perspective.

ViewSpatial-Bench Examples (Part 2).

ViewSpatial-Bench Examples (Part 3).

BibTeX

@misc{li2025viewspatialbenchevaluatingmultiperspectivespatial,
      title={ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models},
      author={Dingming Li and Hongxing Li and Zixuan Wang and Yuchen Yan and Hang Zhang and Siqi Chen and Guiyang Hou and Shengpei Jiang and Wenqi Zhang and Yongliang Shen and Weiming Lu and Yueting Zhuang},
      year={2025},
      eprint={2505.21500},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2505.21500},
}