59 lines
1.9 KiB
Markdown
59 lines
1.9 KiB
Markdown
|
---
|
|||
|
|
base_model: allenai/OLMo-3-1025-7B
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- grpo
|
|||
|
|
- rlhf
|
|||
|
|
- instruction-following
|
|||
|
|
- olmo
|
|||
|
|
license: apache-2.0
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# OLMo-3 7B Instruct-Only (GRPO)
|
|||
|
|
|
|||
|
|
Fine-tuned from [allenai/OLMo-3-1025-7B](https://huggingface.co/allenai/OLMo-3-1025-7B) using GRPO (Group Relative Policy Optimization) on instruction-following tasks.
|
|||
|
|
|
|||
|
|
## Training Details
|
|||
|
|
|
|||
|
|
- **Base model**: allenai/OLMo-3-1025-7B
|
|||
|
|
- **Training method**: GRPO with RL-Zero (no supervised warmup)
|
|||
|
|
- **Dataset**: [allenai/Dolci-RLZero-IF-7B](https://huggingface.co/datasets/allenai/Dolci-RLZero-IF-7B) (IFEval instruction-following)
|
|||
|
|
- **Config**: `if_valley_thinker` — valley length penalty (512–4096 token sweet spot) + think token reward shaping
|
|||
|
|
- **Chat template**: OLMo thinker (prefills `<think>` tag for chain-of-thought reasoning)
|
|||
|
|
- **Precision**: bfloat16
|
|||
|
|
- **Hardware**: 2 nodes × 4 NVIDIA GH200 120GB GPUs (Isambard-AI)
|
|||
|
|
|
|||
|
|
## Reward Components
|
|||
|
|
|
|||
|
|
| Component | Description |
|
|||
|
|
|-----------|-------------|
|
|||
|
|
| IFEval verifiable reward | Binary per-constraint score for instruction-following |
|
|||
|
|
| Valley length penalty | Penalizes responses <512 or >4096 tokens (coeff: -0.001) |
|
|||
|
|
| Think tag reward | +0.125 for correct `</think>` closure |
|
|||
|
|
| Think length penalty | -0.1 if thinking block <10 words |
|
|||
|
|
|
|||
|
|
## Performance (late-stage averages)
|
|||
|
|
|
|||
|
|
| Metric | Value |
|
|||
|
|
|--------|-------|
|
|||
|
|
| IFEval correct rate | 0.88 |
|
|||
|
|
| Training reward | 6.36 |
|
|||
|
|
| Think word count | ~886 words |
|
|||
|
|
| Sequence length | ~1353 tokens |
|
|||
|
|
|
|||
|
|
## Checkpoints
|
|||
|
|
|
|||
|
|
Each training checkpoint is available as a separate branch/revision:
|
|||
|
|
- `main` — step 3800 (latest)
|
|||
|
|
- `step_600` through `step_3600` — intermediate checkpoints (every 200 steps)
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
# Load latest
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained("camgeodesic/olmo3-7b-instruct-only")
|
|||
|
|
|
|||
|
|
# Load specific checkpoint
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained("camgeodesic/olmo3-7b-instruct-only", revision="step_2000")
|
|||
|
|
```
|