103 lines
5.1 KiB
Markdown
103 lines
5.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen3-1.7B-Base
|
||
datasets:
|
||
- POLARIS-Project/Polaris-Dataset-53K
|
||
tags:
|
||
- reinforcement-learning
|
||
- sgd
|
||
- grpo
|
||
- math
|
||
- reasoning
|
||
- low-rank-analysis
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Qwen3-1.7B-Base + Polaris RL (PLAIN SGD, step 300, peak val)
|
||
|
||
This is the **peak-validation checkpoint (step 300/1550)** of a plain-SGD RL fine-tune of `Qwen/Qwen3-1.7B-Base` on `POLARIS-Project/Polaris-Dataset-53K`. Released as part of an ICML-2026 study on the **low-rank structure of SGD vs Adam RL updates** for batched-LoRA inference.
|
||
|
||
## Why this checkpoint exists
|
||
|
||
The motivation is to compare the SVD-compressibility of `ΔW = W_ft − W_base` between SGD-trained and Adam-trained RL fine-tunes. Existing open RL FTs (POLARIS, Skywork-OR1, DeepCoder, AceReason, ORZ, DAPO) are all Adam-trained; we needed a same-base same-recipe SGD counterpart. **POLARIS-1.7B-Preview** is the upstream Adam-trained reference for this exact base + dataset combination — diff this model's ΔW against POLARIS-1.7B-Preview's ΔW for the head-to-head SGD-vs-Adam compressibility comparison.
|
||
|
||
## Training recipe
|
||
|
||
| field | value |
|
||
|---|---|
|
||
| base model | `Qwen/Qwen3-1.7B-Base` |
|
||
| dataset | `POLARIS-Project/Polaris-Dataset-53K` (52,779 train / 512 val) |
|
||
| algorithm | GRPO (`adv_estimator=grpo`, `use_kl_loss=False`, `entropy_coeff=0`, `use_kl_in_reward=False`) |
|
||
| optimizer | **PLAIN SGD** — `momentum=0.0`, `nesterov=false`, `dampening=0.0`, `weight_decay=0.0` |
|
||
| learning rate | `1e-1` (constant) |
|
||
| train batch size | 128 (1 grad step per rollout batch) |
|
||
| ppo_micro_batch_size_per_gpu | 4 |
|
||
| rollout.n | 4 |
|
||
| rollout.temperature | 1.0 |
|
||
| max_prompt_length | 1024 |
|
||
| max_response_length | 8192 |
|
||
| epochs at this checkpoint | ~0.73 (step 300 / 412 per epoch) |
|
||
| hardware | 4× B200 (179 GB) |
|
||
| step time | ~65 s/step |
|
||
| trainer | [verl](https://github.com/volcengine/verl) (FSDP + vLLM rollout) |
|
||
|
||
The "PLAIN SGD" choice is scientifically load-bearing — every claim about SGD update compressibility relies on the update being the **pure first-order gradient**.
|
||
|
||
## LR exploration (relevant context)
|
||
|
||
Plain SGD on Qwen3-Base + math RL has a narrow stable LR window:
|
||
- `lr=1e-2`: gradient signal too weak — stalled.
|
||
- `lr=2e-1`: catastrophic policy collapse (response_length → 8192 cap, rewards pinned at -1).
|
||
- `lr=1`: instant collapse from step 1.
|
||
- **`lr=1e-1`** ← this run: stable-but-slow regime where val acc actually rose from 0.
|
||
|
||
## Results
|
||
|
||
| metric | value |
|
||
|---|---|
|
||
| baseline val acc (step 0) | 0.0% |
|
||
| **val acc at this ckpt (step 300)** | **14.29%** ← peak across the run |
|
||
| val acc at step 550 | 13.7% |
|
||
| val acc at step 700 | 2.7% |
|
||
| val acc at step 1500+ | 0% (collapsed) |
|
||
|
||
The val acc trajectory peaked at step 300, slowly drifted, then collapsed back to 0% in the second half of training — a textbook SGD-without-momentum drift on noisy advantages. **This checkpoint is therefore the one you want for downstream use or analysis** — for the final-epoch checkpoint (collapsed) see [`Sinestro38/qwen3-1p7b-sgd-polaris-step1550-final`](https://huggingface.co/Sinestro38/qwen3-1p7b-sgd-polaris-step1550-final).
|
||
|
||
## Use
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
m = AutoModelForCausalLM.from_pretrained(
|
||
"Sinestro38/qwen3-1p7b-sgd-polaris-step300-best-val",
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="cuda",
|
||
)
|
||
tok = AutoTokenizer.from_pretrained("Sinestro38/qwen3-1p7b-sgd-polaris-step300-best-val")
|
||
|
||
# Math problems work best with the boxed-answer suffix
|
||
prompt = "Find all integer solutions to x^2 + y^2 = 25. Let's think step by step and output the final answer within \\boxed{}."
|
||
msgs = [{"role": "user", "content": prompt}]
|
||
ids = tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
|
||
out = m.generate(ids, max_new_tokens=2048, do_sample=True, temperature=0.6)
|
||
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
## Caveats
|
||
|
||
- This is a **base model** (Qwen3-1.7B-Base) fine-tuned with RL — there's no SFT step before. It only learned to box answers in the prose-style "Answer: X" format that matches verl's default Minerva regex (the Polaris template asks for `\boxed{}` but the base learned to emit "Answer:" patterns that score correctly).
|
||
- Trained on math only; no code, no general instruction-following data.
|
||
- ΔW magnitude is small relative to base weights (feature for compressibility study, not a bug).
|
||
|
||
## Citation context
|
||
|
||
Work in progress — being submitted to ICML 2026 with a paper on plain-SGD RL update compressibility for batched-LoRA serving.
|
||
|
||
Related models in this study:
|
||
- [`Sinestro38/qwen3-1p7b-sgd-polaris-step1550-final`](https://huggingface.co/Sinestro38/qwen3-1p7b-sgd-polaris-step1550-final) — same run, final (collapsed) checkpoint
|
||
- [`Sinestro38/dsr1-qwen7b-sgd-polaris-step100-best-val`](https://huggingface.co/Sinestro38/dsr1-qwen7b-sgd-polaris-step100-best-val) — same recipe, scaled to 7B with DS-R1-Distill base
|
||
- [`Sinestro38/dsr1-qwen7b-sgd-polaris-step412-final`](https://huggingface.co/Sinestro38/dsr1-qwen7b-sgd-polaris-step412-final) — 7B final
|