初始化项目，由ModelHub XC社区提供模型
Model: SlowGuess/ABForge-Qwen3-8B-SFT Source: Original Platform
This commit is contained in:
ModelHub XC
109
README.md
Normal file
109
README.md
Normal file
@@ -0,0 +1,109 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
base_model:
|
||||
- Qwen/Qwen3-8B
|
||||
datasets:
|
||||
- SlowGuess/abforge-data
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- ablation-study
|
||||
- scientific-reasoning
|
||||
- post-training
|
||||
- qwen3
|
||||
---
|
||||
|
||||
# ABForge-Qwen3-8B-SFT
|
||||
|
||||
<p align="center">
|
||||
📄 <a href="https://arxiv.org/abs/XXXX.XXXXX">ArXiv</a>
|
||||
 ｜  💻 <a href="https://github.com/SlowGuess/Abforge_1">Code</a>
|
||||
 ｜  🤗 <a href="https://huggingface.co/collections/SlowGuess/abforge-6a2ac561d0e97f11e409dd75">Collection</a>
|
||||
</p>
|
||||
|
||||
## About
|
||||
|
||||
This repository contains the **SFT stage** of **ABForge**, presented in *ABForge:
|
||||
Post-Training for Paper-Grounded Ablation Design*. Given a paper's methodology with its
|
||||
ablation content removed, ABForge proposes the ablation objectives the paper should investigate
|
||||
and designs a rigorous experiment plan for each — both from a single checkpoint.
|
||||
|
||||
This model is [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) supervised fine-tuned for one
|
||||
full epoch on a 1:1 mixture of the two tasks, with no RL. It initializes the GRPO stage of
|
||||
[ABForge-Qwen3-8B](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B) and is the *SFT only* row
|
||||
of the paper's post-training ablation.
|
||||
|
||||
## Links
|
||||
|
||||
| Resource | Link |
|
||||
|---|---|
|
||||
| Code | [SlowGuess/Abforge_1](https://github.com/SlowGuess/Abforge_1) |
|
||||
| Training & evaluation data | [SlowGuess/abforge-data](https://huggingface.co/datasets/SlowGuess/abforge-data) |
|
||||
| Released model (SFT → GRPO) | [SlowGuess/ABForge-Qwen3-8B](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B) |
|
||||
| RL-only checkpoint | [SlowGuess/ABForge-Qwen3-8B-RL](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-RL) |
|
||||
| Per-paper outputs & judge rationales | `outputs/task{1,2}/*/abforge-sft.jsonl` in the data repo |
|
||||
|
||||
## Training data
|
||||
|
||||
[SlowGuess/abforge-data](https://huggingface.co/datasets/SlowGuess/abforge-data) ships a
|
||||
single table under `train/`, one row per paper, built by a semi-automated audit-in-the-loop
|
||||
pipeline over research papers from major ML, NLP and CV venues. This model trains on the rows
|
||||
flagged `in_sft_task1` (45,961 papers) and `in_sft_task2` (36,955), shuffled into a 1:1 task
|
||||
mixture for one full epoch. The benchmark papers carry no training flag, so they cannot leak in.
|
||||
|
||||
## Performance
|
||||
|
||||
AblationBench, automated rubric-based LLM-as-a-Judge evaluation (`eval/ablationbench_200.jsonl`,
|
||||
200 papers, judge `claude-sonnet-4-6`). Task 1 is ablation objective identification
|
||||
(`paper_score`); Task 2 is ablation plan synthesis (`design_score`, ×100).
|
||||
|
||||
| Model | Task 1 | Task 2 |
|
||||
|---|---|---|
|
||||
| [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) (base) | 44.4 | 43.4 |
|
||||
| **ABForge-Qwen3-8B-SFT** (this model, SFT only) | **30.7** | **52.2** |
|
||||
| [ABForge-Qwen3-8B-RL](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-RL) (RL only) | 52.2 | 54.9 |
|
||||
| [ABForge-Qwen3-8B](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B) (SFT → GRPO) | 55.9 | 62.4 |
|
||||
|
||||
> SFT alone lifts Task 2 but costs Task 1 relative to the base model. Its value is as an RL
|
||||
> initialization: GRPO started from here beats GRPO started from the base model by +3.7 on
|
||||
> Task 1 and +7.5 on Task 2.
|
||||
|
||||
## Evaluation
|
||||
|
||||
Reproduce the numbers above with the [code release](https://github.com/SlowGuess/Abforge_1):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/SlowGuess/Abforge_1 && cd Abforge_1
|
||||
huggingface-cli download SlowGuess/abforge-data --repo-type dataset \
|
||||
--include "eval/*" --local-dir data
|
||||
|
||||
python run_inference_local.py --task 1 \
|
||||
--input data/eval/ablationbench_200.jsonl \
|
||||
--output outputs/task1_infer.jsonl \
|
||||
--model-path SlowGuess/ABForge-Qwen3-8B-SFT \
|
||||
--dtype bf16 --device-map auto \
|
||||
--max-new-tokens 5120 --temperature 0.0 --stop-on '</Result>'
|
||||
|
||||
export JUDGE_API_BASE=https://api.openai.com/v1
|
||||
export JUDGE_API_KEY=...
|
||||
export JUDGE_MODEL=...
|
||||
scripts/evaluate_task1.sh outputs/task1_infer.jsonl
|
||||
```
|
||||
|
||||
Swap `--task 2`, `--stop-on '</Proposed_Plan>'` and `scripts/evaluate_task2.sh` for Task 2. The
|
||||
model is trained on the prompt templates in the code release and the rubric evaluator expects
|
||||
the matching output structure, so use those templates and greedy decoding.
|
||||
|
||||
## Citation
|
||||
|
||||
<!-- TODO: replace with the final paper citation once available. -->
|
||||
|
||||
```bibtex
|
||||
@misc{abforge2026,
|
||||
title={ABForge: Post-Training for Paper-Grounded Ablation Design},
|
||||
author={TODO},
|
||||
year={2026},
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user