85 lines
4.3 KiB
Markdown
85 lines
4.3 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
- zh
|
||
|
|
base_model: Qwen/Qwen3-VL-8B-Thinking
|
||
|
|
tags:
|
||
|
|
- gui-agent
|
||
|
|
- desktop
|
||
|
|
- multimodal
|
||
|
|
- sft
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: image-text-to-text
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen3-VL-8B-Thinking-Desktop-SFT
|
||
|
|
|
||
|
|
This is the **Desktop SFT** model from the [UI-MOPD](https://elisspectre.github.io/UI-MOPD/) project — an 8B model supervised-fine-tuned on desktop GUI interaction data. This checkpoint is released as an **auxiliary artifact** to demonstrate the quality of the Uni-GUI SFT data on the 8B tier and to enable community work such as Model-Merge (TIES / DARE) with other 8B GUI agents.
|
||
|
|
|
||
|
|
> **Note:** This model is **not** used in the UI-MOPD student training path. In the UI-MOPD pipeline, the 8B student is cold-started directly from the `Qwen3-VL-8B-Thinking` base and trained via on-policy distillation from the 32B teachers.
|
||
|
|
|
||
|
|
## Model Description
|
||
|
|
|
||
|
|
Qwen3-VL-8B-Thinking-Desktop-SFT is fine-tuned from [Qwen3-VL-8B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking) on desktop interaction trajectories from the **Uni-GUI** dataset. It is trained independently on the same desktop data used by the 32B Desktop Teacher, but at the 8B scale.
|
||
|
|
|
||
|
|
### Key Highlights
|
||
|
|
|
||
|
|
- **Base Model**: Qwen3-VL-8B-Thinking
|
||
|
|
- **Training Data**: Desktop subset of Uni-GUI (interaction trajectories from OSWorld-style desktop environments)
|
||
|
|
- **Training**: Supervised fine-tuning (SFT) on desktop GUI interaction trajectories
|
||
|
|
- **Role**: Auxiliary artifact — demonstrates Uni-GUI data quality on the 8B tier and enables Model-Merge experiments. **Not used in the UI-MOPD student training path.**
|
||
|
|
|
||
|
|
## Relationship to the UI-MOPD Pipeline
|
||
|
|
|
||
|
|
The actual UI-MOPD training pipeline is:
|
||
|
|
|
||
|
|
1. **Stage 1**: Supervised fine-tuning of **Qwen3-VL-32B-Thinking** on platform-specific data to produce a Desktop Teacher (46.3% on OSWorld) and a Mobile Teacher (16.2% on MobileWorld).
|
||
|
|
2. **Stage 2**: The 8B student is **cold-started from `Qwen3-VL-8B-Thinking` base** (not from any SFT checkpoint) and trained via DAPO (reinforcement learning) with multi-teacher on-policy distillation using platform-conditioned routing from the 32B teachers.
|
||
|
|
|
||
|
|
This 8B SFT model sits **outside** the distillation pipeline. It is trained independently on the same Uni-GUI desktop subset and released separately for the community.
|
||
|
|
|
||
|
|
## Intended Use
|
||
|
|
|
||
|
|
This model is designed to:
|
||
|
|
- Be used as a standalone **desktop GUI agent** for executing desktop tasks (e.g., file management, web browsing, application control)
|
||
|
|
- Serve as a **reference** for evaluating the quality of the Uni-GUI SFT data on the 8B scale
|
||
|
|
- Provide a **base for Model-Merge / TIES / DARE** experiments with other 8B GUI agents
|
||
|
|
- Offer a **baseline** for comparing SFT-only vs. distillation-enhanced performance
|
||
|
|
|
||
|
|
## How to Use
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
|
||
|
|
|
||
|
|
model = Qwen3VLForConditionalGeneration.from_pretrained(
|
||
|
|
"UI-MOPD/Qwen3-VL-8B-Thinking-Desktop-SFT",
|
||
|
|
torch_dtype="auto",
|
||
|
|
device_map="auto",
|
||
|
|
)
|
||
|
|
processor = AutoProcessor.from_pretrained("UI-MOPD/Qwen3-VL-8B-Thinking-Desktop-SFT")
|
||
|
|
```
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
```bibtex
|
||
|
|
@misc{lian2026uimopdmultiplatformonpolicydistillation,
|
||
|
|
title={UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning},
|
||
|
|
author={Niu Lian and Alan Chen and Zhehao Yu and Chengzhen Duan and Fazhan Liu and Hui Liu and Pei Fu and Jian Luan and Yaowei Wang and Shu-Tao Xia and Jinpeng Wang},
|
||
|
|
year={2026},
|
||
|
|
eprint={2607.04425},
|
||
|
|
archivePrefix={arXiv},
|
||
|
|
primaryClass={cs.CL},
|
||
|
|
url={https://arxiv.org/abs/2607.04425},
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Related Resources
|
||
|
|
|
||
|
|
- **Project Page**: [https://elisspectre.github.io/UI-MOPD/](https://elisspectre.github.io/UI-MOPD/)
|
||
|
|
- **Code**: [https://github.com/EliSpectre/UI-MOPD](https://github.com/EliSpectre/UI-MOPD)
|
||
|
|
- **Paper**: [UI-MOPD (arXiv)](https://arxiv.org/abs/2607.04425)
|
||
|
|
- **Desktop Teacher**: [UI-MOPD/Qwen3-VL-32B-Thinking-Desktop-Teacher](https://huggingface.co/UI-MOPD/Qwen3-VL-32B-Thinking-Desktop-Teacher)
|
||
|
|
- **Mobile Teacher**: [UI-MOPD/Qwen3-VL-32B-Thinking-Mobile-Teacher](https://huggingface.co/UI-MOPD/Qwen3-VL-32B-Thinking-Mobile-Teacher)
|
||
|
|
- **Student Model** (cold-started from base, not from this SFT): [UI-MOPD/Qwen3-VL-8B-Thinking-UI-MOPD-Student](https://huggingface.co/UI-MOPD/Qwen3-VL-8B-Thinking-UI-MOPD-Student)
|