119 lines
4.6 KiB
Markdown
119 lines
4.6 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
base_model:
|
||
- Qwen/Qwen3-8B
|
||
datasets:
|
||
- SlowGuess/abforge-data
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- ablation-study
|
||
- scientific-reasoning
|
||
- post-training
|
||
- qwen3
|
||
---
|
||
|
||
# ABForge-Qwen3-8B
|
||
|
||
<p align="center">
|
||
📄 <a href="https://arxiv.org/abs/XXXX.XXXXX">ArXiv</a>
|
||
 ｜  💻 <a href="https://github.com/SlowGuess/Abforge_1">Code</a>
|
||
 ｜  🤗 <a href="https://huggingface.co/collections/SlowGuess/abforge-6a2ac561d0e97f11e409dd75">Collection</a>
|
||
</p>
|
||
|
||
## About
|
||
|
||
This repository contains **ABForge-Qwen3-8B**, the model presented in *ABForge:
|
||
Post-Training for Paper-Grounded Ablation Design*. Given a paper's methodology with its
|
||
ablation content removed, it proposes the ablation objectives the paper should investigate
|
||
and designs a rigorous experiment plan for each — both from a single checkpoint.
|
||
|
||
The model is obtained through an **SFT → GRPO** pipeline on
|
||
[Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B), trained throughout on a 1:1 mixture of the
|
||
two tasks:
|
||
|
||
1. Supervised fine-tuning on the ABForge SFT corpus, one full epoch →
|
||
[ABForge-Qwen3-8B-SFT](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-SFT)
|
||
2. Rubric-guided GRPO, 200 updates, each rollout routed to its task's reward by
|
||
`data_source` → **this model**
|
||
|
||
## Links
|
||
|
||
| Resource | Link |
|
||
|---|---|
|
||
| Code | [SlowGuess/Abforge_1](https://github.com/SlowGuess/Abforge_1) |
|
||
| Training & evaluation data | [SlowGuess/abforge-data](https://huggingface.co/datasets/SlowGuess/abforge-data) |
|
||
| SFT checkpoint | [SlowGuess/ABForge-Qwen3-8B-SFT](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-SFT) |
|
||
| RL-only checkpoint | [SlowGuess/ABForge-Qwen3-8B-RL](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-RL) |
|
||
| Per-paper outputs & judge rationales | `outputs/task{1,2}/*/abforge.jsonl` in the data repo |
|
||
|
||
## Training data
|
||
|
||
[SlowGuess/abforge-data](https://huggingface.co/datasets/SlowGuess/abforge-data) ships a
|
||
single table under `train/`, one row per paper, built by a semi-automated audit-in-the-loop
|
||
pipeline over research papers from major ML, NLP and CV venues. Both stages select their rows
|
||
from that table by its split columns and mix the two tasks 1:1:
|
||
|
||
- **SFT** — `in_sft_task1` (45,961 papers) + `in_sft_task2` (36,955)
|
||
- **GRPO** — `in_rl_task1` + `in_rl_task2` (30,000 papers each, disjoint from the SFT pool)
|
||
|
||
The benchmark papers carry no training flag, so they cannot leak into either stage. The
|
||
preprocessing scripts in the code release turn the table into the training parquets directly.
|
||
|
||
## Performance
|
||
|
||
AblationBench, automated rubric-based LLM-as-a-Judge evaluation (`eval/ablationbench_200.jsonl`,
|
||
200 papers, judge `claude-sonnet-4-6`). Task 1 is ablation objective identification
|
||
(`paper_score`); Task 2 is ablation plan synthesis (`design_score`, ×100).
|
||
|
||
| Model | Task 1 | Task 2 |
|
||
|---|---|---|
|
||
| [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) (base) | 44.4 | 43.4 |
|
||
| [ABForge-Qwen3-8B-SFT](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-SFT) (SFT only) | 30.7 | 52.2 |
|
||
| [ABForge-Qwen3-8B-RL](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-RL) (RL only) | 52.2 | 54.9 |
|
||
| **ABForge-Qwen3-8B** (SFT → GRPO) | **55.9** | **62.4** |
|
||
|
||
> Each stage on its own falls short: SFT trades Task 1 for Task 2, and GRPO from the base model
|
||
> plateaus below the full pipeline. Running GRPO from the SFT checkpoint is worth +3.7 on Task 1
|
||
> and +7.5 on Task 2 over RL alone.
|
||
|
||
## Evaluation
|
||
|
||
Reproduce the numbers above with the [code release](https://github.com/SlowGuess/Abforge_1):
|
||
|
||
```bash
|
||
git clone https://github.com/SlowGuess/Abforge_1 && cd Abforge_1
|
||
huggingface-cli download SlowGuess/abforge-data --repo-type dataset \
|
||
--include "eval/*" --local-dir data
|
||
|
||
python run_inference_local.py --task 1 \
|
||
--input data/eval/ablationbench_200.jsonl \
|
||
--output outputs/task1_infer.jsonl \
|
||
--model-path SlowGuess/ABForge-Qwen3-8B \
|
||
--dtype bf16 --device-map auto \
|
||
--max-new-tokens 5120 --temperature 0.0 --stop-on '</Result>'
|
||
|
||
export JUDGE_API_BASE=https://api.openai.com/v1
|
||
export JUDGE_API_KEY=...
|
||
export JUDGE_MODEL=...
|
||
scripts/evaluate_task1.sh outputs/task1_infer.jsonl
|
||
```
|
||
|
||
Swap `--task 2`, `--stop-on '</Proposed_Plan>'` and `scripts/evaluate_task2.sh` for Task 2. The
|
||
model is trained on the prompt templates in the code release and the rubric evaluator expects
|
||
the matching output structure, so use those templates and greedy decoding.
|
||
|
||
## Citation
|
||
|
||
<!-- TODO: replace with the final paper citation once available. -->
|
||
|
||
```bibtex
|
||
@misc{abforge2026,
|
||
title={ABForge: Post-Training for Paper-Grounded Ablation Design},
|
||
author={TODO},
|
||
year={2026},
|
||
}
|
||
```
|