vla-1.7b-qwen3-v2/README.md

---
license: apache-2.0
language:
  - en
tags:
  - robotics
  - vla
  - vision-language-action
  - 3d-pose
  - qwen3
  - megatron
  - multimodal
pipeline_tag: text-generation
library_name: transformers
---

# VLA 1.7B — Qwen3 v2

A 1.7B parameter Vision-Language-Action model, migrated to a **Qwen3** backbone and
trained on a **5-source, ~32B-token multimodal mix** (video, 3D pose, audio, image+caption).
This is the project's first Qwen3-based VLA model, and the first trained after fixing
the "stuck in one modality" failure mode found in the previous model.

## Key facts

| | |
|---|---|
| **Architecture** | Qwen3 (28 layers, hidden 2048, intermediate 6144, 16 attn heads / 8 KV heads (GQA), qk-layernorm, RoPE θ=1e6, tied embeddings) |
| **Parameters** | 1.94B (including embeddings for 257,920 vocab) |
| **Vocab size** | 257,920 (Qwen3 base ~151,669 + 106,232 VLA tokens, padded) |
| **Tokenizer** | [EmpathicRobotics/tokenizer-vla-qwen3](https://huggingface.co/EmpathicRobotics/tokenizer-vla-qwen3) |
| **Training data** | ~32.01B tokens across 5 sources: FineVideo-VLA v6, MixtureVitae-Omni, OmniVideo-100K, synth-llava, emotional-roleplay |
| **Training** | 7,632 iters (1 epoch), 64 nodes × 4 GH200 GPUs, global batch 1024, seq len 4096 |
| **Final loss** | Train: 1.694, Val: 1.7526 (PPL 5.77), Test: 1.7722 (PPL 5.88) |
| **Precision** | bf16 |
| **Context length** | 4,096 tokens |

## What this model does

Given a text prompt (activity description, image seed2 block, or partial modality
sequence), the model generates an interleaved multimodal token sequence spanning
6 categories it was trained on:

```
<seed2_N> ...                          # 1 FPS semantic image/video keyframes (vocab 8192)
<cosmos_N> ... </cosmos>               # 8-frame spatial video tokens (vocab 64000)
<snac_N> ... </snac>                   # SNAC audio codec tokens (12,288)
<speech> ... </speech>                 # inline spoken-dialogue text
<caption> ... </caption>               # inline visual caption text
<agent> <fps_30> <pelvis> ... </agent> # 3D human pose, 17 H36M joints
```

## Progress vs. the previous model

The first model ([vla-1.7b-pab-spline-adaptive](https://huggingface.co/EmpathicRobotics/vla-1.7b-pab-spline-adaptive))
passed agent-completion but **failed modality transitions**: it stayed in `seed2` mode and
never transitioned to `cosmos`/`avclm`/`agent` from text alone. This model no longer has
that failure mode — it transitions freely across all 6 trained categories, in both
greedy and sampled decoding.

Strongest evidence: given **only** 32 real `<seed2_N>` tokens from a held-out image record
(no other text hint), the model generated a topically-correct caption closely matching the
real ground truth, then closed `</caption><|im_end|>` cleanly — genuine image↔text
cross-modal binding, not template noise.

It can also produce full agent (3D pose) blocks that decode to valid, non-degenerate
coordinates, and — verified for the first time on this model's own generation, not just
training data — `cosmos` video tokens that decode to a real, playable video via
[`Cosmos-Tokenizer-DV8x16x16`](https://github.com/NVIDIA/Cosmos-Tokenizer), and `snac`
audio tokens that decode to a real, non-silent waveform via
[SNAC](https://github.com/hubertsiuzdak/snac) (`hubertsiuzdak/snac_24khz`).

All 3 non-text modalities the model actually produces in volume (`cosmos`, `snac`, `seed2`)
now have a working decoder in the project repo (`tools/decode/`) and have each been
round-tripped on real ground-truth tokens. `seed2` is generative rather than a
deterministic codec round-trip — see Known limitations.

## Known limitations

- **Greedy decoding can degenerate into repeated-token loops** inside long `cosmos`
  runs (e.g. the same token repeating 6-8 times), which can burn the generation budget
  before reaching `<fps_N>`/`agent`. Sampling with `repetition_penalty>1` mitigates this.
- **Sampling trades accuracy for diversity**: in the image-captioning test, sampled
  generation occasionally hallucinated details (e.g. an invented name) not present in
  the source image; greedy decoding did not.
- **`cosmos` tokens dominate generation**: aggregated across all test prompts, `cosmos`
  is 61-77% of all non-text VLA tokens produced (vs. a minority share for
  agent/seed2/snac combined). This is largely structural (one cosmos chunk costs a fixed
  200 tokens vs. ~1-4 tokens for the others), but it does mean cosmos runs can consume
  most of a generation's token budget before reaching `<fps_N>`/`agent`.
- **`avc_lm` tokens are essentially unused** — discarded at the data-flatten stage before
  training (to control token count), so the model rarely if ever produces them.
- **`seed2`→image reconstruction is generative, not a deterministic round-trip.**
  Seed2Tokenizer has no pixel decoder of its own; reconstruction conditions a diffusion
  img2img pipeline (`StableUnCLIPImg2ImgPipeline`) on the token embeddings to *generate*
  a plausible image, unlike `cosmos`/`snac`'s lossy-but-deterministic codec decoders — two
  runs of the same tokens can come out visually different. Verified end-to-end on 32 real
  ground-truth `<seed2_N>` tokens (`tools/decode/decode_seed2.py`) — the diffusion weights
  now come from a community mirror (`sd2-community/stable-diffusion-2-1-unclip`), since
  the original `stabilityai/stable-diffusion-2-1-unclip` was removed from HuggingFace.
- **Evaluation so far is qualitative** (manual inspection of generated tokens/decoded
  media) — no MPJPE, BLEU/CIDEr, or closed-loop task-success metric has been run yet.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "EmpathicRobotics/vla-1.7b-qwen3-v2",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("EmpathicRobotics/vla-1.7b-qwen3-v2")

prompt = (
    "### Context: Person raises both arms above head.\n"
    "<seed2_3758> <seed2_2157> <cosmos_58567> "
    "<fps_30> <pelvis> <pelvis_t_0> <pelvis_x_128> <pelvis_y_128> <pelvis_z_128>"
)
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    input_ids, max_new_tokens=500,
    do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.3,
)
print(tokenizer.decode(output[0]))
```

### Encoding real media into tokens (so you can actually prompt the model)

The `## Usage` prompt above uses pre-picked token ids as a demo. To send the
model *real* media -- e.g. "here's a photo, continue the scene" or "here's
a real motion clip, keep going" -- encode it first with the 4 encoders below
(**verified working 2026-07-23**, each tested end-to-end: real media ->
tokens -> decoded/compared back against the original). Bundled in this repo
the same way as the decoders (`tools/encode/`), no separate `git clone`
needed.

```bash
# Image -> <seed2_N> tokens (32 ids, auto-downloads the Q-Former checkpoint
# from ontocord/seed2 if not cached locally)
python tools/encode/encode_seed2.py --image photo.jpg

# 8 video frames -> <cosmos_N> tokens (200 ids -- this model's OLD
# window=8/square-crop convention, NOT the newer 2026-07-23 aspect-preserving
# one; auto-downloads encoder.jit from nvidia/Cosmos-Tokenizer-DV8x16x16)
python tools/encode/encode_cosmos.py --frames f0.png f1.png f2.png f3.png f4.png f5.png f6.png f7.png

# Audio/video file -> <snac_N> tokens (listen-format, <snac> wrapper --
# this model never saw the newer <listen>/<speak> convention or speak-format L2)
python tools/encode/encode_snac.py --input clip.wav

# Real 3D pose (8 frames x 17 joints x xyz, metres, root-centred) -> <agent>
# tokens -- for "give the model a real motion capture / pose-pipeline output,
# have it continue" (same behavior already verified: agent completion PASS)
python tools/encode/encode_agent.py --input pose.npy   # shape (8, 17, 3)
```

Splice the printed token block into your prompt (e.g. after `### Context:
...`) the same way the `## Usage` example does, then call `model.generate()`
as shown there.

### Decoding generated tokens back to media

The decoder scripts + their vendored dependencies are bundled directly in
**this repo** (`tools/`) -- one `snapshot_download` gets everything, no
separate `git clone` needed. (Also mirrored at
[github.com/TieuDaoChanNhan/finevideo-vla](https://github.com/TieuDaoChanNhan/finevideo-vla)
if you'd rather browse/clone the code on its own.) **Verified working
2026-07-23** with no cluster/internal access required, each tested end-to-end
on real tokens this model actually generated.

```bash
python -c "
from huggingface_hub import snapshot_download
snapshot_download('EmpathicRobotics/vla-1.7b-qwen3-v2', allow_patterns=['tools/*', 'tools/**/*'])
"
pip install scipy numpy torch torchvision imageio-ffmpeg soundfile snac huggingface_hub
cd <snapshot-download-cache-dir-printed-above>
```

**Agent tokens -> 3D pose** (pure Python, no extra downloads):
```bash
python tools/eval/decode_agent_tokens.py --input generated_tokens.txt --output poses.json
```

**Cosmos tokens -> video** (auto-downloads the ~350MB decoder checkpoint from
[nvidia/Cosmos-Tokenizer-DV8x16x16](https://huggingface.co/nvidia/Cosmos-Tokenizer-DV8x16x16)
on first run):
```bash
python tools/decode/decode_cosmos.py --tokens 58345,57843,... --output out.mp4
# this model's cosmos chunks are exactly 200 raw ids each (8 frames, 160x160,
# square-cropped -- the DV8x16x16 checkpoint's own token grid for that input
# size). A later dataset pivot (2026-07-23, aspect-preserving/896 tokens)
# does NOT apply to this model -- it was trained entirely on the 200-token/
# square-crop convention.
```

**SNAC tokens -> audio** (auto-downloads `hubertsiuzdak/snac_24khz` from HF):
```bash
python tools/decode/decode_snac.py --tokens 130911,134940,... --format listen --output out.wav
# this model only ever saw "listen" format (3 tokens/base-frame, <snac>
# wrapper) -- do NOT use --format speak, that's a newer (2026-07-23)
# convention this model was never trained on.
```

**Seed2 tokens -> image** (auto-downloads the ~2.6GB Q-Former checkpoint from
the tokenizer's own public repo,
[ontocord/seed2](https://huggingface.co/ontocord/seed2), plus a ~5GB
diffusion img2img pipeline on first run -- this one is a generative
*reconstruction*, not a deterministic decode, so expect run-to-run and
prompt-to-prompt variation in the exact pixels even for the same tokens):
```bash
python tools/decode/decode_seed2.py --tokens 6750,680,2472,... --output out.png
# exactly 32 raw ids per image (Seed2Tokenizer's fixed Q-former query length)
```

## Training details

### Loss curve

| Iter | Loss |
|---|---|
| 50 | 6.472 |
| 500 | 2.840 |
| 1000 | 2.154 |
| 2000 | 1.953 |
| 4000 | 1.826 |
| 6000 | 1.767 |
| 7600 | 1.694 |
| 7632 (val) | 1.7526 (PPL 5.77) |
| 7632 (test) | 1.7722 (PPL 5.88) |

### Config

- **Batch**: GBS 1024, seq_len 4096 → 32.01B tokens trained (exactly 1 epoch)
- **Infrastructure**: 64 nodes × 4 GH200 GPUs (256 total), ~284 TFLOP/s/GPU, ~21,800 tok/s/GPU
- **Framework**: Megatron-LM via oellm-autoexp

### Data mix

| Source | Tokens |
|---|---|
| MixtureVitae-Omni | 20.39B |
| FineVideo-VLA v6 | 10.93B |
| OmniVideo-100K (video) | 0.54B |
| synth-llava | 0.10B |
| emotional-roleplay (SNAC TTS) | 0.05B |
| **Total** | **~32.01B** |

## Citation

```bibtex
@misc{empathicrobotics2026vlaqwen3,
  title={VLA 1.7B Qwen3 v2: Multi-Source Multimodal Vision-Language-Action Pretraining},
  author={EmpathicRobotics},
  year={2026},
  url={https://huggingface.co/EmpathicRobotics/vla-1.7b-qwen3-v2}
}
```