39 lines
2.0 KiB
Markdown
39 lines
2.0 KiB
Markdown
---
|
|
library_name: transformers
|
|
tags:
|
|
- compression
|
|
- expert-merging
|
|
- moe
|
|
license: apache-2.0
|
|
base_model:
|
|
- Qwen/Qwen3-30B-A3B-Instruct-2507
|
|
---
|
|
|
|
# Qwen3-30B-A3B-Instruct-2507-REAP
|
|
|
|
This model is a compressed version of [Qwen/Qwen3-30B-A3B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507).
|
|
It is obtained by reducing the number of experts in each MoE layer from 128 to 96 using the REAP baseline method as described in https://bknyaz.github.io/blog/2026/moe/.
|
|
The compressed model has 23B params (44GB) instead of 31B (57GB) of the original model,
|
|
reducing storage and GPU memory requirements by roughly 25%. At the same time,
|
|
the model retains >=90% of the original model's performance on a variety of benchmarks (see Results section below).
|
|
Additional efficiency optimization (e.g., quantization) can be added similarly to the original model.
|
|
|
|
See additional details at [Qwen3-30B-A3B-Instruct-2507-REAM](https://huggingface.co/SamsungSAILMontreal/Qwen3-30B-A3B-Instruct-2507-REAM).
|
|
|
|
### Results
|
|
|
|
| Model | Winogrande | ARC-C | ARC-E | BoolQ | HellaSwag | MMLU | OpenBookQA | RTE | AVG |
|
|
|----------------------------------|------------|-------|-------|-------|-----------|------|------------|------|------|
|
|
| Qwen3-30B-A3B-Instruct-2507 | 73.2 | 60.7 | 85.1 | 88.7 | 61.2 | 80.1 | 32.4 | 76.5 | 69.7 |
|
|
| Qwen3-30B-A3B-Instruct-2507-REAP | 71.7 | 49.3 | 77.4 | 88.1 | 56.5 | 69.3 | 29.6 | 78.3 | 65.0 |
|
|
|
|
|
|
| Model | IFeval | AIME25 | GSM8K | GPQA-D | HumanEval | LiveCodeBench | AVG |
|
|
|----------------------------------|--------|--------|-------|--------|-----------|---------------|-------|
|
|
| Qwen3-30B-A3B-Instruct-2507 | 90.4 | 56.7 | 89.3 | 47.0 | 93.3 | 48.6 | 70.9 |
|
|
| Qwen3-30B-A3B-Instruct-2507-REAP | 88.0 | 56.7 | 87.9 | 37.9 | 81.7 | 33.0 | 64.2 |
|
|
|
|
## License
|
|
|
|
Please refer to the license of the original model [Qwen/Qwen3-30B-A3B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507).
|