74 lines
2.5 KiB
Markdown
74 lines
2.5 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model: Qwen/Qwen3-8B
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- ablation-study
|
||
|
|
- scientific-reasoning
|
||
|
|
- post-training
|
||
|
|
- qwen3
|
||
|
|
---
|
||
|
|
|
||
|
|
# ABForge-Qwen3-8B-Task1
|
||
|
|
|
||
|
|
An **ABForge** model for **Task 1: Ablation Objective Generation**.
|
||
|
|
|
||
|
|
ABForge is a post-training pipeline for paper-grounded ablation design. This checkpoint is
|
||
|
|
post-trained with the full ABForge pipeline: supervised fine-tuning from `Qwen/Qwen3-8B` followed by rubric-guided GRPO (**SFT → GRPO**).
|
||
|
|
|
||
|
|
## Task
|
||
|
|
|
||
|
|
Given the ablation-free context of a research paper, the model proposes candidate ablation objectives, each expressed as a **Target Module** (the component to ablate) paired with a **Research Question** it is meant to answer.
|
||
|
|
|
||
|
|
## Training data
|
||
|
|
|
||
|
|
SFT on `train/sft_task1_45961.jsonl`, then GRPO on `train/RL_task1_30K.jsonl`, from [`SlowGuess/abforge-data`](https://huggingface.co/datasets/SlowGuess/abforge-data)
|
||
|
|
(derived from CC-licensed research papers). Evaluation uses the held-out **AblationBench** split
|
||
|
|
(`eval/ablationbench_200.jsonl`) of the same dataset.
|
||
|
|
|
||
|
|
## Related models (Task 1)
|
||
|
|
|
||
|
|
- [`SlowGuess/ABForge-Qwen3-8B-Task1`](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-Task1) (this model)
|
||
|
|
- [`SlowGuess/ABForge-Qwen3-8B-Task1-SFT`](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-Task1-SFT)
|
||
|
|
- [`SlowGuess/ABForge-Qwen3-8B-Task1-RL`](https://huggingface.co/SlowGuess/ABForge-Qwen3-8B-Task1-RL)
|
||
|
|
|
||
|
|
## Evaluation
|
||
|
|
|
||
|
|
Reproduce AblationBench evaluation with the [`SlowGuess/Abforge_1`](https://github.com/SlowGuess/Abforge_1) code:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
git clone https://github.com/SlowGuess/Abforge_1 && cd Abforge_1
|
||
|
|
huggingface-cli download SlowGuess/abforge-data --repo-type dataset --local-dir data
|
||
|
|
|
||
|
|
export MODEL_PATH=SlowGuess/ABForge-Qwen3-8B-Task1
|
||
|
|
|
||
|
|
# 1. Generate predictions on AblationBench
|
||
|
|
python run_inference_local.py --task 1 \
|
||
|
|
--input data/eval/ablationbench_200.jsonl \
|
||
|
|
--output preds.jsonl \
|
||
|
|
--model-path "$MODEL_PATH" --dtype bf16 --max-new-tokens 4096
|
||
|
|
|
||
|
|
# 2. Score against the fixed AblationBench rubric (Claude judge)
|
||
|
|
export ANTHROPIC_API_KEY=<your-key>
|
||
|
|
python scripts/eval_task1_claude.py --input preds.jsonl --output scored.jsonl
|
||
|
|
```
|
||
|
|
|
||
|
|
## Links
|
||
|
|
|
||
|
|
- Dataset: [`SlowGuess/abforge-data`](https://huggingface.co/datasets/SlowGuess/abforge-data)
|
||
|
|
- Code: [`SlowGuess/Abforge_1`](https://github.com/SlowGuess/Abforge_1)
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
```bibtex
|
||
|
|
@misc{abforge,
|
||
|
|
title = {ABForge: A Post-Training Pipeline for Paper-Grounded Ablation Design},
|
||
|
|
author = {ABForge authors},
|
||
|
|
year = {2026},
|
||
|
|
howpublished = {\url{https://github.com/SlowGuess/Abforge_1}}
|
||
|
|
}
|
||
|
|
```
|