115 lines
5.3 KiB
Markdown
115 lines
5.3 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen1.5-0.5B
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
language:
|
||
- en
|
||
tags:
|
||
- reasoning
|
||
- chain-of-thought
|
||
- math
|
||
- rlvr
|
||
- qwen2
|
||
---
|
||
|
||
# Argonne-Qwen1.5-0.5B-think
|
||
|
||
A **0.46B-parameter chain-of-thought reasoner**: the base model **Qwen1.5-0.5B** fine-tuned with a fully
|
||
open five-stage reasoning recipe — general SFT → DPO → CoT-SFT → **STaR** self-improvement → **GRPO/RLVR**
|
||
(verifiable reward). It produces an explicit `<think> … </think>` reasoning trace followed by a final
|
||
`\boxed{}` answer on math word problems.
|
||
|
||
The point of this model is to show how much math-reasoning ability a **well-pretrained sub-1B base** can be
|
||
given by a light, fully open post-training recipe. It is a compact research reasoner — useful for studying
|
||
test-time compute on small models — **not** a production-grade or competition-math solver.
|
||
|
||
## Benchmarks (honest, held-out)
|
||
|
||
Evaluated on **contamination-free** elementary math-word-problem benchmarks — SVAMP, ASDiv, MAWPS (clean)
|
||
and GSM-Plus (semi-clean, adversarial) — that appear in **none** of the training stages. `n=500/set`,
|
||
with-think decoding. Metrics: **greedy / +budget-forcing / self-consistency@32 / pass@32** (%).
|
||
|
||
| set | greedy | +budget | self-cons@32 | pass@32 |
|
||
|---|:---:|:---:|:---:|:---:|
|
||
| SVAMP | 33.4 | 33.4 | 44.4 | 83.6 |
|
||
| ASDiv | 49.2 | 48.6 | 58.4 | 83.8 |
|
||
| MAWPS | 39.8 | 39.6 | 47.4 | 78.4 |
|
||
| GSM-Plus (adversarial) | 16.8 | 16.8 | 21.2 | 60.2 |
|
||
| **mean (SVAMP/ASDiv/MAWPS)** | **40.8** | 40.5 | **50.1** | **81.9** |
|
||
|
||
Read plainly: single-shot greedy is ~41% on the clean sets, self-consistency lifts it to ~50%, and the
|
||
model **can** solve far more than it reliably answers in one pass — **pass@32 ≈ 82%**. These are
|
||
absolute-modest numbers for a 0.46B model; it is a small reasoner, and the greedy→pass@K headroom is the
|
||
interesting part.
|
||
|
||
> **Note on GSM8K:** the CoT/STaR/GRPO training overlaps GSM8K, so GSM8K is **not** a valid held-out
|
||
> benchmark for this model. Use the clean SVAMP / ASDiv / MAWPS numbers above as the honest signal.
|
||
|
||
## How it was trained
|
||
|
||
Base **Qwen1.5-0.5B** → five stages (base-agnostic harness `reasoning/reason_control/` + RLVR in the
|
||
[training repo](https://github.com/PursuitOfDataScience/ArgonneAI)):
|
||
|
||
1. **General SFT** on [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (LR 2e-5).
|
||
2. **DPO** preference alignment on [argilla/dpo-mix-7k](https://huggingface.co/datasets/argilla/dpo-mix-7k) (LR 5e-6, β 0.1).
|
||
3. **CoT-SFT** on a `<think>…</think>` + `\boxed{}` math-reasoning mix (LR 1e-5, 1 epoch).
|
||
4. **STaR** — self-distillation on the model's own verified-correct, concise (terminating) traces on
|
||
GSM8K-train + MATH-L1-3 (two rounds; fixes the non-termination that caps greedy).
|
||
5. **GRPO / RLVR** — group-relative policy optimization with a verifiable reward (correct `\boxed` on
|
||
GSM8K-train), early-stopped to avoid over-optimization.
|
||
|
||
Cumulatively, greedy accuracy on the clean sets rose from ~24% (after stage 3) to ~41% (after stage 5) —
|
||
STaR and RLVR mostly **convert** the large latent pass@K ceiling into single-shot answers rather than
|
||
raising the ceiling itself.
|
||
|
||
## Inference
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
mid = "PursuitOfDataScience/Argonne-Qwen1.5-0.5B-think"
|
||
tok = AutoTokenizer.from_pretrained(mid)
|
||
model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.bfloat16).to("cuda").eval()
|
||
|
||
msgs = [{"role": "user", "content": "Ana has 3 boxes with 12 pencils each. She gives away 8. How many are left?"}]
|
||
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to("cuda")
|
||
out = model.generate(ids, max_new_tokens=512, do_sample=False)
|
||
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
|
||
# -> <think> 3 * 12 = 36 ... 36 - 8 = 28 ... </think> The answer is $\boxed{28}$.
|
||
```
|
||
|
||
Uses the standard ChatML format (`<|im_start|>` / `<|im_end|>`). For the best accuracy, sample `K≈16–32`
|
||
traces (`temperature 0.8, top_p 0.95`) and majority-vote the `\boxed{}` answers (self-consistency ≈ 50%).
|
||
|
||
## Limitations
|
||
|
||
- **0.46B parameters** — a small reasoner. Absolute accuracy on clean grade-school math is ~41% greedy /
|
||
~50% self-consistency; expect errors, especially on multi-step or adversarial problems (GSM-Plus greedy
|
||
~17%).
|
||
- **Scope:** trained and evaluated on English elementary math word problems + general chat. Not a
|
||
general-purpose or competition-math model.
|
||
- **GSM8K is not a valid benchmark here** (train-mix overlap) — use the clean numbers above.
|
||
|
||
## Base model & license
|
||
|
||
Fine-tuned from [Qwen/Qwen1.5-0.5B](https://huggingface.co/Qwen/Qwen1.5-0.5B) (Apache-2.0) and released under
|
||
Apache-2.0; please also observe the base model's terms.
|
||
|
||
## Provenance
|
||
|
||
Training + evaluation code: [PursuitOfDataScience/ArgonneAI](https://github.com/PursuitOfDataScience/ArgonneAI)
|
||
(`reasoning/reason_control/` recipe, `reasoning/grpo.py` RLVR, `reasoning/clean_eval.py` honest evaluator).
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{argonne_qwen05b_think_2026,
|
||
title = {Argonne-Qwen1.5-0.5B-think: a compact chain-of-thought reasoner (SFT->DPO->CoT->STaR->RLVR)},
|
||
author = {PursuitOfDataScience},
|
||
year = {2026},
|
||
url = {https://huggingface.co/PursuitOfDataScience/Argonne-Qwen1.5-0.5B-think}
|
||
}
|
||
```
|