110 lines
4.3 KiB
Markdown
110 lines
4.3 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
base_model:
|
||
- Qwen/Qwen3-8B
|
||
datasets:
|
||
- SlowGuess/abforge-data
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- ablation-study
|
||
- scientific-reasoning
|
||
- post-training
|
||
- qwen3
|
||
---
|
||
|
||
# ABForge-Qwen3-8B-SFT
|
||
|
||
<p align="center">
|
||
📄 <a href="https://arxiv.org/abs/XXXX.XXXXX">ArXiv</a>
|
||
 ｜  💻 <a href="https://github.com/SlowGuess/Abforge_1">Code</a>
|
||
 ｜  🤗 <a href="https://huggingface.co/collections/SlowGuess/abforge-6a2ac561d0e97f11e409dd75">Collection</a>
|
||
</p>
|
||
|
||
## About
|
||
|
||
This repository contains the **SFT stage** of **ABForge**, presented in *ABForge:
|
||
Post-Training for Paper-Grounded Ablation Design*. Given a paper's methodology with its
|
||
ablation content removed, ABForge proposes the ablation objectives the paper should investigate
|
||
and designs a rigorous experiment plan for each — both from a single checkpoint.
|
||
|
||
This model is [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) supervised fine-tuned for one
|
||
full epoch on a 1:1 mixture of the two tasks, with no RL. It initializes the GRPO stage of
|
||
[ABForge-Qwen3-8B](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B) and is the *SFT only* row
|
||
of the paper's post-training ablation.
|
||
|
||
## Links
|
||
|
||
| Resource | Link |
|
||
|---|---|
|
||
| Code | [SlowGuess/Abforge_1](https://github.com/SlowGuess/Abforge_1) |
|
||
| Training & evaluation data | [SlowGuess/abforge-data](https://huggingface.co/datasets/SlowGuess/abforge-data) |
|
||
| Released model (SFT → GRPO) | [SlowGuess/ABForge-Qwen3-8B](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B) |
|
||
| RL-only checkpoint | [SlowGuess/ABForge-Qwen3-8B-RL](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-RL) |
|
||
| Per-paper outputs & judge rationales | `outputs/task{1,2}/*/abforge-sft.jsonl` in the data repo |
|
||
|
||
## Training data
|
||
|
||
[SlowGuess/abforge-data](https://huggingface.co/datasets/SlowGuess/abforge-data) ships a
|
||
single table under `train/`, one row per paper, built by a semi-automated audit-in-the-loop
|
||
pipeline over research papers from major ML, NLP and CV venues. This model trains on the rows
|
||
flagged `in_sft_task1` (45,961 papers) and `in_sft_task2` (36,955), shuffled into a 1:1 task
|
||
mixture for one full epoch. The benchmark papers carry no training flag, so they cannot leak in.
|
||
|
||
## Performance
|
||
|
||
AblationBench, automated rubric-based LLM-as-a-Judge evaluation (`eval/ablationbench_200.jsonl`,
|
||
200 papers, judge `claude-sonnet-4-6`). Task 1 is ablation objective identification
|
||
(`paper_score`); Task 2 is ablation plan synthesis (`design_score`, ×100).
|
||
|
||
| Model | Task 1 | Task 2 |
|
||
|---|---|---|
|
||
| [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) (base) | 44.4 | 43.4 |
|
||
| **ABForge-Qwen3-8B-SFT** (this model, SFT only) | **30.7** | **52.2** |
|
||
| [ABForge-Qwen3-8B-RL](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-RL) (RL only) | 52.2 | 54.9 |
|
||
| [ABForge-Qwen3-8B](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B) (SFT → GRPO) | 55.9 | 62.4 |
|
||
|
||
> SFT alone lifts Task 2 but costs Task 1 relative to the base model. Its value is as an RL
|
||
> initialization: GRPO started from here beats GRPO started from the base model by +3.7 on
|
||
> Task 1 and +7.5 on Task 2.
|
||
|
||
## Evaluation
|
||
|
||
Reproduce the numbers above with the [code release](https://github.com/SlowGuess/Abforge_1):
|
||
|
||
```bash
|
||
git clone https://github.com/SlowGuess/Abforge_1 && cd Abforge_1
|
||
huggingface-cli download SlowGuess/abforge-data --repo-type dataset \
|
||
--include "eval/*" --local-dir data
|
||
|
||
python run_inference_local.py --task 1 \
|
||
--input data/eval/ablationbench_200.jsonl \
|
||
--output outputs/task1_infer.jsonl \
|
||
--model-path SlowGuess/ABForge-Qwen3-8B-SFT \
|
||
--dtype bf16 --device-map auto \
|
||
--max-new-tokens 5120 --temperature 0.0 --stop-on '</Result>'
|
||
|
||
export JUDGE_API_BASE=https://api.openai.com/v1
|
||
export JUDGE_API_KEY=...
|
||
export JUDGE_MODEL=...
|
||
scripts/evaluate_task1.sh outputs/task1_infer.jsonl
|
||
```
|
||
|
||
Swap `--task 2`, `--stop-on '</Proposed_Plan>'` and `scripts/evaluate_task2.sh` for Task 2. The
|
||
model is trained on the prompt templates in the code release and the rubric evaluator expects
|
||
the matching output structure, so use those templates and greedy decoding.
|
||
|
||
## Citation
|
||
|
||
<!-- TODO: replace with the final paper citation once available. -->
|
||
|
||
```bibtex
|
||
@misc{abforge2026,
|
||
title={ABForge: Post-Training for Paper-Grounded Ablation Design},
|
||
author={TODO},
|
||
year={2026},
|
||
}
|
||
```
|