108 lines
3.6 KiB
Markdown
108 lines
3.6 KiB
Markdown
---
|
||
library_name: transformers
|
||
base_model: google/gemma-3-270m
|
||
license: gemma
|
||
datasets:
|
||
- Skywork/Skywork-Reward-Preference-80K-v0.2
|
||
language:
|
||
- en
|
||
tags:
|
||
- gemma3
|
||
- sft
|
||
- research
|
||
- skywork
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# gemma-3-270m SFT'd on Skywork-Reward-Preference-80K (chosen-only)
|
||
|
||
A response-only SFT of [`google/gemma-3-270m`](https://huggingface.co/google/gemma-3-270m)
|
||
on the **chosen** responses of
|
||
[`Skywork/Skywork-Reward-Preference-80K-v0.2`](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.2).
|
||
|
||
This is the anchor checkpoint used as the SFT reference policy for a research
|
||
project on **data-space DPO** — applying preference signal by reweighting SFT
|
||
samples through a susceptibility matrix, instead of via DPO's weight-space
|
||
gradient descent. It is **not** intended as a useful chat model — at 270M
|
||
parameters and a single epoch of SFT, generations are at best "coherent text
|
||
on the topic of the prompt". The role of this checkpoint is to be a
|
||
controlled, reproducible anchor for sampling-based experiments.
|
||
|
||
## Training recipe
|
||
|
||
| | |
|
||
|---|---|
|
||
| Base | `google/gemma-3-270m` (pretrained, no chat template) |
|
||
| Data | `Skywork/Skywork-Reward-Preference-80K-v0.2`, chosen responses only (~77K examples) |
|
||
| Epochs | 1 |
|
||
| Learning rate | 2e-5, cosine schedule, 3% warmup |
|
||
| Batch size | 16 per device × 4 GPUs = 64 effective |
|
||
| Precision | bf16 + gradient checkpointing |
|
||
| Loss | response-only cross-entropy (prompt tokens masked to -100) |
|
||
| Total steps | 602 |
|
||
| Final train loss | 1.41 |
|
||
| Wall time | ~4 min on 8× H100 |
|
||
|
||
## Prompt format
|
||
|
||
The base model has no `chat_template`, so we use a fixed plain-text format
|
||
matching the SFT distribution:
|
||
|
||
```text
|
||
Human: <user message>
|
||
Assistant: ASSISTANT_RESPONSE
|
||
```
|
||
|
||
Internally the prompt prefix is `"Human: <user>\n\nAssistant:"` and the loss is
|
||
computed only on the tokens following `Assistant:`.
|
||
|
||
## Quick start
|
||
|
||
```python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
mid = "bfpill/gemma-3-270m-sft-skywork-chosen"
|
||
tok = AutoTokenizer.from_pretrained(mid)
|
||
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16).cuda().eval()
|
||
|
||
prompt = f"Human: What is the capital of France?\n\nAssistant:"
|
||
ids = tok(prompt, return_tensors="pt").to(model.device)
|
||
out = model.generate(
|
||
**ids, max_new_tokens=80, do_sample=False,
|
||
repetition_penalty=1.15, pad_token_id=tok.eos_token_id,
|
||
)
|
||
print(tok.decode(out[0, ids.input_ids.shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- 270M params, 1 epoch of SFT on ~77K examples. Generations are short and
|
||
often factually incorrect.
|
||
- Trained exclusively on the Skywork prompt distribution; performance on
|
||
other distributions (instruction-following, dialogue, code, math)
|
||
reflects the base model plus a thin Skywork-flavoured veneer.
|
||
- Not preference-tuned. The whole point of this checkpoint is to be the
|
||
anchor for a follow-up data-space patterning experiment that is the
|
||
*replacement* for the usual DPO step.
|
||
- Inherits the base model's biases and license restrictions.
|
||
|
||
## License
|
||
|
||
Gemma Terms of Use — see [Google's Gemma license](https://ai.google.dev/gemma/terms).
|
||
This SFT preserves the base model's weights structure; usage is subject to
|
||
the same terms.
|
||
|
||
## Source code
|
||
|
||
The trainer that produced this checkpoint:
|
||
`rm-patterning/policy/sft_anchor.py` in
|
||
`timaeus-research/timaeus@max/rm-scaling` (commit `3a3e0755e` and later).
|
||
|
||
```bash
|
||
torchrun --nproc_per_node=8 rm-patterning/policy/sft_anchor.py \
|
||
--base-model google/gemma-3-270m \
|
||
--output-dir ./gemma-3-270m-sft-skywork-chosen \
|
||
--epochs 1 --learning-rate 2e-5 --batch-size 16 --max-length 1024
|
||
```
|