42 lines
1.2 KiB
Markdown
42 lines
1.2 KiB
Markdown
---
|
|
license: apache-2.0
|
|
tags:
|
|
- prime-rl
|
|
- moe
|
|
- test-model
|
|
library_name: transformers
|
|
---
|
|
|
|
<div align="center">
|
|
<img src="https://cdn-avatars.huggingface.co/v1/production/uploads/61e020e4a343274bb132e138/H2mcdPRWtl4iKLd-OYYBc.jpeg" width="200"/>
|
|
</div>
|
|
|
|
# qwen3-moe-tiny
|
|
|
|
A small (~670M parameter) Qwen3 MoE model for testing only. It is generally compatible with vLLM and HuggingFace Transformers but is meant to be used with [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl).
|
|
|
|
Fine-tuned on [PrimeIntellect/Reverse-Text-SFT](https://huggingface.co/datasets/PrimeIntellect/Reverse-Text-SFT) to provide a non-trivial distribution for KL divergence during RL.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
uv run rl @ configs/ci/integration/rl_moe/qwen3_moe.toml
|
|
```
|
|
|
|
See the [Testing MoE at Small Scale](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/docs/testing-moe-at-small-scale.md) guide for full instructions.
|
|
|
|
## Model Details
|
|
|
|
| Parameter | Value |
|
|
|-----------|-------|
|
|
| Hidden size | 1024 |
|
|
| Layers | 24 |
|
|
| Experts | 16 |
|
|
| Active experts | 4 |
|
|
| Parameters | ~670M |
|
|
|
|
## Links
|
|
|
|
- [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) - RL training framework
|
|
- [PrimeIntellect](https://www.primeintellect.ai/) - Building infrastructure for decentralized AI
|