69 lines
2.4 KiB
Markdown
69 lines
2.4 KiB
Markdown
---
|
|
license: other
|
|
base_model: Qwen/Qwen3-8B-Base
|
|
tags:
|
|
- code
|
|
- reinforcement-learning
|
|
- pkpo
|
|
- livecodebench
|
|
---
|
|
|
|
# Qwen3-8B Code PKPO
|
|
|
|
This repo contains a Qwen/Qwen3-8B-Base derivative trained for a small agentic
|
|
coding experiment using the shared tool path in `agent_core.py` and
|
|
`shipped_tool.py`.
|
|
|
|
## Method
|
|
|
|
- Base: `Qwen/Qwen3-8B-Base`.
|
|
- Prompt/template: custom `<think>...</think><answer>...</answer>` template saved
|
|
in the tokenizer. The generation prompt ends with `Assistant: <think>`.
|
|
- Tool protocol: no system role; instructions are merged into the first user
|
|
message; strict user/assistant alternation; plain-text `Tool type` and
|
|
`Tool query` calls.
|
|
- Training data: `deepmind/code_contests` train split only, filtered to old
|
|
stdin/stdout problems. The LiveCodeBench eval subset is not used for training.
|
|
- Reward: binary hidden-test pass/fail.
|
|
- PKPO: `sloo_minus_one` from the paper for `k >= 2`; centered `k=1` rewards for
|
|
the first and final stages. No GRPO-style reward normalization is applied.
|
|
- Schedule actually run: `[1, 8, 1] (shipped: stage1_group3)`.
|
|
|
|
The run was intentionally small to fit the free-credit budget and deadline.
|
|
Results should be treated as a reproducible experiment, not a leaderboard model.
|
|
|
|
## Results
|
|
|
|
Evaluation uses `livecodebench/code_generation_lite` `v6`, a fixed subset saved
|
|
at `eval/eval_subset.json`, temperature 1.0, and the same one-turn tool path used
|
|
for training.
|
|
|
|
| model | pass@1 estimate |
|
|
|---|---:|
|
|
| base before training | 0.1111 |
|
|
| final merged model | 0.0556 |
|
|
|
|
Raw files:
|
|
|
|
- `eval/baseline_results.json`
|
|
- `eval/final_results.json`
|
|
- `eval/eval_subset.json`
|
|
|
|
## Usage
|
|
|
|
Serve with vLLM:
|
|
|
|
```bash
|
|
vllm serve bk1dr/qwen3-8b-code-pkpo --trust-remote-code --max-model-len 8192
|
|
```
|
|
|
|
Run the shipped tool:
|
|
|
|
```bash
|
|
python shipped_tool.py --base-url http://127.0.0.1:8000/v1 --model bk1dr/qwen3-8b-code-pkpo --max-turns 1 --cp < problem.txt
|
|
```
|
|
|
|
## Run Notes
|
|
|
|
Run pkpo_20260709T184830Z: full PKPO schedule k=1->8->1 with a LoRA checkpoint after every group. The shipped weights are checkpoint 'stage1_group3', selected by validation on the fixed eval subset (per-checkpoint pass@1: {"after_sft": 0.027777777777777776, "stage1_group3": 0.05555555555555555, "stage2_group1": 0.027777777777777776}). The full-schedule endpoint regressed on the subset (see eval/full_schedule_endpoint_results.json); intermediate checkpoint selection is part of the documented training procedure. Selection+merge elapsed 13.3 min on one H100.
|