31 lines
1.2 KiB
Markdown
31 lines
1.2 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model: Qwen/Qwen3-VL-2B-Instruct
|
||
|
|
library_name: transformers
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen3-VL-2B-Instruct-LLM
|
||
|
|
|
||
|
|
The language model of [Qwen/Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct),
|
||
|
|
extracted as a plain `Qwen3ForCausalLM` checkpoint (vision tower dropped).
|
||
|
|
|
||
|
|
The Qwen3-VL-2B text tower is architecturally identical to Qwen3-1.7B dense
|
||
|
|
(hidden 2048 / 28 layers / 16 Q + 8 KV heads / head_dim 128 / FFN 6144 /
|
||
|
|
vocab 151936 / tied embeddings). Its interleaved MRoPE assigns text tokens the
|
||
|
|
same index on all three axes, which reduces exactly to standard 1D RoPE, so
|
||
|
|
this export keeps the VL `rope_theta` (5e6) and drops `rope_scaling`.
|
||
|
|
|
||
|
|
**Equivalence**: text-only logits match the original VL model bit-for-bit in
|
||
|
|
fp32 (`max |logit diff| = 0.0` on a held-out prompt).
|
||
|
|
|
||
|
|
Load with:
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("shushengyang/Qwen3-VL-2B-Instruct-LLM")
|
||
|
|
```
|
||
|
|
|
||
|
|
Produced by `personal/shusheng/mllm/misc/convert_qwen3vl_llm_to_qwen3.py`
|
||
|
|
(amilabs monorepo); intended as an LLM initialization for ScaleWM MLLM
|
||
|
|
experiments (vision-language-pretrained LLM vs text-only Qwen3-1.7B).
|