SpaceQwen2.5-VL-3B-Instruct/README.md

base_model, datasets, language, library_name, license_name, license_link, pipeline_tag, tags, new_version, model-index

base_model	datasets	language	library_name	license_name	license_link	pipeline_tag	tags	new_version	model-index
Qwen/Qwen2.5-VL-3B-Instruct	remyxai/OpenSpaces	en	transformers	qwen-research	https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE	image-text-to-text	remyx vqasynth spatial-reasoning multimodal vlm vision-language robotics distance-estimation embodied-ai quantitative-spatial-reasoning	remyxai/SpaceThinker-Qwen2.5VL-3B	name results SpaceQwen2.5-VL-3B-Instruct task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type 3DSRBench benchmark type name value success_rate Overall Success Rate 0.515 name success_rate 3D Positional Relation / Orientation 0.4706 name success_rate Object Localization / 3D Localization 0.5629 name success_rate Object Properties / Size 0.5116 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type BLINK benchmark type name value success_rate Overall Success Rate 0.5 name success_rate 3D Positional Relation / Orientation 0.6503 name success_rate Counting / Object Counting 0.6083 name success_rate Depth and Distance / Relative 0.5161 name success_rate Object Localization / 2D Localization 0.4426 name success_rate Point and Object Tracking / Point Correspondence 0.2849 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type MMIU benchmark type name value success_rate Overall Success Rate 0.3045 name success_rate Camera and Image Transformation / 2D Transformation 0.245 name success_rate Camera and Image Transformation / 3D Camera Pose 0.215 name success_rate Camera and Image Transformation / Camera Motion 0.4436 name success_rate Depth and Distance / Absolute 0.265 name success_rate Object Localization / 3D Localization 0.48 name success_rate Point and Object Tracking / 3D Tracking 0.24 name success_rate Point and Object Tracking / Point Correspondence 0.28 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type MMVP benchmark type name value success_rate Overall Success Rate 0.5767 name success_rate Others / Miscellaneous 0.5767 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type QSpatialBench-Plus benchmark type name value success_rate Overall Success Rate 0.3663 name success_rate Depth and Distance / Absolute 0.3663 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type QSpatialBench-ScanNet benchmark type name value success_rate Overall Success Rate 0.33 name success_rate Depth and Distance / Absolute 0.216 name success_rate Object Properties / Size 0.4444 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type RealWorldQA benchmark type name value success_rate Overall Success Rate 0.4392 name success_rate Others / Miscellaneous 0.4392 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type SpatialSense benchmark type name value success_rate Overall Success Rate 0.6554 name success_rate 3D Positional Relation / Orientation 0.6554 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type VGBench benchmark type name value success_rate Overall Success Rate 0.2615 name success_rate Camera and Image Transformation / 2D Transformation 0.2277 name success_rate Camera and Image Transformation / 3D Camera Pose 0.2438 name success_rate Depth and Distance / Absolute 0.2696 name success_rate Depth and Distance / Relative 0.1945 name success_rate Object Localization / 3D Localization 0.3733 name success_rate Point and Object Tracking / 3D Tracking 0.2655 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type VSI-Bench_8 benchmark type name value success_rate Overall Success Rate 0.2322 name success_rate 3D Positional Relation / Orientation 0.3843 name success_rate Counting / Object Counting 0.1715 name success_rate Depth and Distance / Absolute 0.0299 name success_rate Depth and Distance / Relative 0.3521 name success_rate Object Properties / Size 0.2323 name success_rate Others / Miscellaneous 0.2525 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type VSR-ZeroShot benchmark type name value success_rate Overall Success Rate 0.7373 name success_rate 3D Positional Relation / Orientation 0.7373 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type cvbench benchmark type name value success_rate Overall Success Rate 0.5179 name success_rate Counting / Object Counting 0.6168 name success_rate Depth and Distance / Relative 0.4925 name success_rate Object Localization / 3D Localization 0.4446 task dataset metrics results_by_subcategory type name visual-question-answering Spatial Reasoning name type spatialbench benchmark type name value success_rate Overall Success Rate 0.4879 name success_rate 3D Positional Relation / Orientation 0.5294 name success_rate Counting / Object Counting 0.7 name success_rate Object Properties / Existence 0.45 name success_rate Object Properties / Reachability 0.5 name success_rate Object Properties / Size 0.25

SpaceQwen2.5-VL-3B-Instruct

The model was presented in the paper OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models. More information can be found at the project page.

• Model Type: Multimodal, Vision-Language Model
• Architecture: Qwen2.5-VL-3B-Instruct
• Model Size: 3.75B parameters (FP16)
• Finetuned from: Qwen/Qwen2.5-VL-3B-Instruct
• Finetune Strategy: LoRA (Low-Rank Adaptation)
• License: Apache-2.0

Model Overview

This model uses data synthesis techniques and publicly available models to reproduce the work described in SpatialVLM to enhance the spatial reasoning of multimodal models. With a pipeline of expert models, we can infer spatial relationships between objects in a scene to create a VQA dataset for spatial reasoning.

Running SpaceQwen2.5-VL-3B-Instruct

Ollama

To launch with ollama, run:

ollama run hf.co/remyxai/SpaceQwen2.5-VL-3B-Instruct:latest

Transformers

Install qwen dependencies:

pip install qwen-vl-utils[decord]==0.0.8

To run inference on a sample image:

from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "remyxai/SpaceQwen2.5-VL-3B-Instruct", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("remyxai/SpaceQwen2.5-VL-3B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://raw.githubusercontent.com/remyxai/VQASynth/refs/heads/main/assets/warehouse_sample_2.jpeg",
            },
            {"type": "text", "text": "What is the height of the man in the red hat in feet?"},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

GGUF

Or run SpaceQwen2.5-VL-3B-Instruct using llama.cpp:

./llama-qwen2vl-cli -m /path/to/SpaceQwen2.5-VL-3B-Instruct/SpaceQwen2.5-VL-3B-Instruct-F16.gguf \
                    --mmproj /path/to/SpaceQwen2.5-VL-3B-Instruct/spaceqwen2.5-vl-3b-instruct-vision.gguf \
                    -p "What's the height of the man in the red hat?" \
                    --image /path/to/warehouse_sample_2.jpeg --threads 24 -ngl 99

Dataset & Training

SpaceQwen2.5-VL-3B-Instruct uses LoRA to fine-tune Qwen2.5-VL-3B-Instruct on the OpenSpaces dataset.

Dataset Summary:

• ~10k synthetic spatial reasoning traces

• Question types: spatial relations (distances (units), above, left-of, contains, closest to)

• Format: image (RGB) + question + answer

• Dataset: OpenSpaces

• Code: VQASynth

• Reference: SpatialVLM

Scripts for LoRA SFT available at trl

Model Evaluation

SpatialScore

SpaceQwen shines in the 3D positional relations categories of the SpatialScore-Hard comparison featured in the table below:

Read more about the comprehensive spatial reasoning benchmark: SpatialScore.

The following chart compares performance between SpaceQwen and SpaceThinker on the SpatialScore benchmarks sources.

OmniSpatial

OmniSpatial is another comprehensive spatial reasoning benchmark that assesses dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking capabilities.

Learn more about OmniSpatial.

SpaCE-10

Model	Overall	EQ	SQ	SA	OO	OS	EP	FR	SP	Source
InternVL2.5-4B	36.01	34.30	34.40	43.60	44.40	16.50	31.10	50.10	33.70	Table
SpaceThinker	32.72	32.73	24.81	47.26	50.33	33.63	9.25	37.54	26.25	GPT Eval
SpaceOm	32.32	32.47	24.81	47.63	50.00	32.52	9.12	37.04	25.00	GPT Eval
SpaceQwen	31.98	31.19	25.89	41.61	51.98	35.18	10.97	36.54	22.50	GPT Eval
Qwen2.5-VL-3B-Instruct	30.00	31.70	45.50	39.00	43.00	25.30	11.50	22.80	21.20	Table

Legend:

• EQ: Entity Quantification
• SQ: Scene Quantification
• SA: Size Assessment
• OO: Object-Object spatial relations
• OS: Object-Scene spatial relations
• EP: Entity Presence
• FR: Functional Reasoning
• SP: Spatial Planning

ℹ️ Note: Scores for SpaceQwen, SpaceThinker, SpaceOm are generated via gpt_eval_score on single-choice (*-single) versions of the SpaCE-10 benchmark tasks. Other entries reflect leaderboard accuracy scores from the official SpaCE-10 evaluation table.

Read more about the SpaCE-10 benchmark or see results here

SIRI-Bench

SIRI-Bench is a video-based benchmark designed to evaluate complex spatial reasoning capabilities

MindCube

MindCube is a benchmark for assessing Spatial Mental Modeling from Limited Views

⚠️ Limitations & Ethical Considerations

• Performance may degrade in cluttered environments or camera perspective.
• This model was fine-tuned using synthetic reasoning over an internet image dataset.
• Multimodal biases inherent to the base model (Qwen2.5-VL) may persist.
• Not intended for use in safety-critical or legal decision-making.

Users are encouraged to evaluate outputs critically and consider fine-tuning for domain-specific safety and performance.

Citation

@article{chen2024spatialvlm,
  title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
  author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
  journal = {arXiv preprint arXiv:2401.12168},
  year = {2024},
  url = {https://arxiv.org/abs/2401.12168},
}

@misc{qwen2.5-VL,
    title = {Qwen2.5-VL},
    url = {https://qwenlm.github.io/blog/qwen2.5-vl/},
    author = {Qwen Team},
    month = {January},
    year = {2025}
}

@article{wu2025spatialscore,
    author    = {Wu, Haoning and Huang, Xiao and Chen, Yaohui and Zhang, Ya and Wang, Yanfeng and Xie, Weidi},
    title     = {SpatialScore: Towards Unified Evaluation for Multimodal Spatial Understanding},
    journal   = {arXiv preprint arXiv:2505.17012},
    year      = {2025},
}

@article{omnispatial25,
  title   = {OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models},
  author  = {Mengdi Jia and Zekun Qi and Shaochen Zhang and Wenyao Zhang and Xinqiang Yu and Jiawei He and He Wang and Li Yi},
  journal = {arXiv preprint arXiv:2506.03135},
  year = {2025}
}

@article{song2025siribench,
  title   = {{SIRI-Bench}: Challenging VLMs’ Spatial Intelligence through Complex Reasoning Tasks},
  author  = {Song, Zijian and Lin, Xiaoxin and Huang, Qiuming and Wang, Guangrun and Lin, Liang},
  journal = {arXiv preprint arXiv:2506.14512},
  year    = {2025},
  url     = {https://arxiv.org/abs/2506.14512}
}

@misc{yin2025spatial,
  title        = {Spatial Mental Modeling from Limited Views},
  author       = {Baiqiao Yin and Qineng Wang and Pingyue Zhang and Jianshu Zhang
                  and Kangrui Wang and Zihan Wang and Jieyu Zhang
                  and Keshigeyan Chandrasegaran and Han Liu and Ranjay Krishna
                  and Saining Xie and Manling Li and Jiajun Wu and Li Fei-Fei},
  year         = {2025},
  archivePrefix= {arXiv},
  eprint       = {2506.21458},
  primaryClass = {cs.AI},
  url          = {https://arxiv.org/abs/2506.21458}
}