97 lines
4.3 KiB
Markdown
97 lines
4.3 KiB
Markdown
|
---
|
|||
|
|
library_name: transformers
|
|||
|
|
base_model: Qwen/Qwen3-8B-Base
|
|||
|
|
tags:
|
|||
|
|
- multilingual
|
|||
|
|
- reasoning
|
|||
|
|
- LLM
|
|||
|
|
- qwen3
|
|||
|
|
license: apache-2.0
|
|||
|
|
datasets:
|
|||
|
|
- lightonai/Dolci-Think-SFT-32B-Multilingual
|
|||
|
|
language:
|
|||
|
|
- sw
|
|||
|
|
- en
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-8B-SW-Pivot-EN
|
|||
|
|
|
|||
|
|
`Qwen3-8B-SW-Pivot-EN` is an **English-pivoted reasoning model** fine-tuned from [`Qwen/Qwen3-8B-Base`](https://huggingface.co/Qwen/Qwen3-8B-Base) on Swahili questions and answers. This model receives questions in Swahili, produces its **entire reasoning trace in English**, then delivers the final answer in Swahili.
|
|||
|
|
|
|||
|
|
It is released alongside the paper [**Rethinking the Multilingual Reasoning Gap with Layer Swap**](https://arxiv.org/abs/2605.26735).
|
|||
|
|
|
|||
|
|
## Model details
|
|||
|
|
|
|||
|
|
- **Base model:** `Qwen/Qwen3-8B-Base`
|
|||
|
|
- **Language:** Swahili (Q&A) with English CoT
|
|||
|
|
- **Training:** Full SFT, ~10B tokens, 2 epochs
|
|||
|
|
- **Context length:** 32,768 tokens
|
|||
|
|
- **Dataset:** [`lightonai/Dolci-Think-SFT-32B-Multilingual`](https://huggingface.co/datasets/lightonai/Dolci-Think-SFT-32B-Multilingual) (Swahili Q&A with English CoT).
|
|||
|
|
|
|||
|
|
> [!NOTE]
|
|||
|
|
> The model was trained on data derived from `allenai/Dolci-Think-SFT-32B`, released under the ODC-BY-1.0 license.
|
|||
|
|
|
|||
|
|
## Related models
|
|||
|
|
|
|||
|
|
This model is part of a Swahili specialist trio designed to study the native reasoning gap:
|
|||
|
|
|
|||
|
|
| Model | CoT language | Description |
|
|||
|
|
|---|---|---|
|
|||
|
|
| [`lightonai/Qwen3-8B-SW`](https://huggingface.co/lightonai/Qwen3-8B-SW) | Swahili | Native reasoning specialist |
|
|||
|
|
| [`lightonai/Qwen3-8B-SW-Swap`](https://huggingface.co/lightonai/Qwen3-8B-SW-Swap) | Swahili | Layer Swap: middle layers (L13–L22) of `Qwen3-8B-EN` transplanted into `Qwen3-8B-SW` |
|
|||
|
|
| [`lightonai/Qwen3-8B-SW-Pivot-EN`](https://huggingface.co/lightonai/Qwen3-8B-SW-Pivot-EN) | English | Same Swahili Q&A pairs, but CoT in English |
|
|||
|
|
| [`lightonai/Qwen3-8B-EN`](https://huggingface.co/lightonai/Qwen3-8B-EN) | English | English specialist |
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
All scores are mean accuracy (%) on the **Swahili** version of each benchmark, with sample standard deviation across runs. AIME 24/25 is averaged over 30 runs; the others over 10 runs, using the recommended generation parameters.
|
|||
|
|
|
|||
|
|
| Model | MGSM-Rev2 | Global-MMLU-Lite | GPQA-Diamond | AIME 24/25 | HumanEvalPlus | Average |
|
|||
|
|
|---|:---:|:---:|:---:|:---:|:---:|:---:|
|
|||
|
|
| `Qwen3-8B-SW` | 93.16 | 61.98 | 49.39 | 47.67 | 82.69 | 66.98 |
|
|||
|
|
| `Qwen3-8B-SW-Swap` | <u>96.12</u> | 64.10 | 49.29 | 50.33 | <u>85.62</u> | 69.09 |
|
|||
|
|
| `Qwen3-8B-SW-Pivot-EN` | 89.68 | <u>66.00</u> | <u>52.73</u> | <u>59.67</u> | 84.50 | <u>70.52</u> |
|
|||
|
|
| `Qwen3-8B-EN` | 35.88 | 33.88 | 36.82 | 24.78 | 58.44 | 37.96 |
|
|||
|
|
|
|||
|
|
**Benchmarks used:**
|
|||
|
|
|
|||
|
|
- [`lightonai/gpqa_diamond_multilingual`](https://huggingface.co/datasets/lightonai/gpqa_diamond_multilingual)
|
|||
|
|
- [`lightonai/aime24_multilingual`](https://huggingface.co/datasets/lightonai/aime24_multilingual)
|
|||
|
|
- [`lightonai/aime25_multilingual`](https://huggingface.co/datasets/lightonai/aime25_multilingual)
|
|||
|
|
- [`lightonai/HumanEvalPlus_multilingual`](https://huggingface.co/datasets/lightonai/HumanEvalPlus_multilingual)
|
|||
|
|
- [`lightonai/mgsm-rev2`](https://huggingface.co/datasets/lightonai/mgsm-rev2)
|
|||
|
|
- [`CohereLabs/Global-MMLU-Lite`](https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite)
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model_name = "lightonai/Qwen3-8B-SW-Pivot-EN"
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
|
|||
|
|
|
|||
|
|
messages = [{"role": "user", "content": "Suluhisha: 24 × 17 = ?"}]
|
|||
|
|
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
|
|||
|
|
|
|||
|
|
outputs = model.generate(inputs, max_new_tokens=32768, temperature=1.0, top_p=0.95, top_k=20)
|
|||
|
|
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Recommended sampling: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0`.
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
|
|||
|
|
If you find our work helpful, feel free to give us a cite.
|
|||
|
|
|
|||
|
|
```bibtex
|
|||
|
|
@misc{lasbordes2026rethinking,
|
|||
|
|
title = {Rethinking the Multilingual Reasoning Gap with Layer Swap},
|
|||
|
|
author = {Lasbordes, Maxence and Chatelain, Amélie and Seddah, Djamé},
|
|||
|
|
year = {2026},
|
|||
|
|
eprint = {2605.26735},
|
|||
|
|
archivePrefix= {arXiv},
|
|||
|
|
primaryClass = {cs.CL}
|
|||
|
|
}
|
|||
|
|
```
|