50 lines
2.1 KiB
Markdown
50 lines
2.1 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model:
|
||
|
|
- Qwen/Qwen3-1.7B-Base
|
||
|
|
tags:
|
||
|
|
- swe-bench
|
||
|
|
- swe-zero
|
||
|
|
- sft
|
||
|
|
- mini-swe-agent
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# SWE-ZERO-10K-Qwen3-1.7B-Base
|
||
|
|
|
||
|
|
Qwen3-1.7B-Base SFT on a 10K random sample of [SWE-ZERO trajectories](https://huggingface.co/datasets/AlienKevin/SWE-ZERO-12M-trajectories) (right-truncated to 8K tokens), evaluated on SWE-bench Verified.
|
||
|
|
|
||
|
|
## Eval result
|
||
|
|
|
||
|
|
**pass@1 = 7/100 = 7%** on the 100-task SWE-bench Verified slice (latest trial per task).
|
||
|
|
|
||
|
|
For full eval details + per-task trajectories, see the eval dataset: [AlienKevin/SWE-ZERO-10K-Qwen3-1.7B-Base-eval](https://huggingface.co/datasets/AlienKevin/SWE-ZERO-10K-Qwen3-1.7B-Base-eval).
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
- **Base:** `Qwen/Qwen3-1.7B-Base`
|
||
|
|
- **SFT data:** 10K random sample from [AlienKevin/SWE-ZERO-12M-trajectories @ 2f328e1d](https://huggingface.co/datasets/AlienKevin/SWE-ZERO-12M-trajectories/tree/2f328e1dcea8286aa8eb67ff5ec80c7fd4c99450), right-truncated to 8K tokens
|
||
|
|
- **Model arch:** `max_seq_len=32768` (Llama 3 RoPE scaling from 8192) — data truncated to 8K but RoPE precomputed to 32K so positions past 8K stay in distribution
|
||
|
|
- **TPU:** v5p-8 (~1.5h training)
|
||
|
|
- **Optimizer:** AdamW (β1=0.9, β2=0.95, ε=1e-8), lr=2e-5, weight_decay=0.1, max_grad_norm=30, cosine schedule, warmup=0.03, min_lr_ratio=0.1
|
||
|
|
- **Batch:** 8 global, 1249 steps
|
||
|
|
- **Tracking:** [marin#5611](https://github.com/marin-community/marin/issues/5611)
|
||
|
|
|
||
|
|
## Critical detail: `eos_token_id`
|
||
|
|
|
||
|
|
The HF config has `eos_token_id: [151643, 151645]` so vLLM stops at both `<|endoftext|>` AND `<|im_end|>` (Qwen3 chat-template turn boundary). The default base model only includes 151643; SFT-trained models that aren't patched will keep generating past the natural turn end until `max_tokens`.
|
||
|
|
|
||
|
|
## Inference
|
||
|
|
|
||
|
|
```python
|
||
|
|
from vllm import LLM, SamplingParams
|
||
|
|
|
||
|
|
llm = LLM(model="AlienKevin/SWE-ZERO-10K-Qwen3-1.7B-Base", max_model_len=32768)
|
||
|
|
params = SamplingParams(temperature=1.0, max_tokens=4096)
|
||
|
|
# Use the Qwen3 chat template; the eos_token_id list will be picked up automatically.
|
||
|
|
```
|
||
|
|
|
||
|
|
For the mini-swe-agent eval harness used in our results, see the eval dataset for the harbor adapter + agent config.
|