bk1dr/qwen3-8b-code-pkpo
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: bk1dr/qwen3-8b-code-pkpo
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-30 09:33:20 +08:00
commit 8060a07a88
29 changed files with 158812 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

68
README.md Normal file
View File

@@ -0,0 +1,68 @@
---
license: other
base_model: Qwen/Qwen3-8B-Base
tags:
- code
- reinforcement-learning
- pkpo
- livecodebench
---
# Qwen3-8B Code PKPO
This repo contains a Qwen/Qwen3-8B-Base derivative trained for a small agentic
coding experiment using the shared tool path in `agent_core.py` and
`shipped_tool.py`.
## Method
- Base: `Qwen/Qwen3-8B-Base`.
- Prompt/template: custom `<think>...</think><answer>...</answer>` template saved
in the tokenizer. The generation prompt ends with `Assistant: <think>`.
- Tool protocol: no system role; instructions are merged into the first user
message; strict user/assistant alternation; plain-text `Tool type` and
`Tool query` calls.
- Training data: `deepmind/code_contests` train split only, filtered to old
stdin/stdout problems. The LiveCodeBench eval subset is not used for training.
- Reward: binary hidden-test pass/fail.
- PKPO: `sloo_minus_one` from the paper for `k >= 2`; centered `k=1` rewards for
the first and final stages. No GRPO-style reward normalization is applied.
- Schedule actually run: `[1, 8, 1] (shipped: stage1_group3)`.
The run was intentionally small to fit the free-credit budget and deadline.
Results should be treated as a reproducible experiment, not a leaderboard model.
## Results
Evaluation uses `livecodebench/code_generation_lite` `v6`, a fixed subset saved
at `eval/eval_subset.json`, temperature 1.0, and the same one-turn tool path used
for training.
| model | pass@1 estimate |
|---|---:|
| base before training | 0.1111 |
| final merged model | 0.0556 |
Raw files:
- `eval/baseline_results.json`
- `eval/final_results.json`
- `eval/eval_subset.json`
## Usage
Serve with vLLM:
```bash
vllm serve bk1dr/qwen3-8b-code-pkpo --trust-remote-code --max-model-len 8192
```
Run the shipped tool:
```bash
python shipped_tool.py --base-url http://127.0.0.1:8000/v1 --model bk1dr/qwen3-8b-code-pkpo --max-turns 1 --cp < problem.txt
```
## Run Notes
Run pkpo_20260709T184830Z: full PKPO schedule k=1->8->1 with a LoRA checkpoint after every group. The shipped weights are checkpoint 'stage1_group3', selected by validation on the fixed eval subset (per-checkpoint pass@1: {"after_sft": 0.027777777777777776, "stage1_group3": 0.05555555555555555, "stage2_group1": 0.027777777777777776}). The full-schedule endpoint regressed on the subset (see eval/full_schedule_endpoint_results.json); intermediate checkpoint selection is part of the documented training procedure. Selection+merge elapsed 13.3 min on one H100.

28
added_tokens.json Normal file
View File

@@ -0,0 +1,28 @@
{
"</think>": 151668,
"</tool_call>": 151658,
"</tool_response>": 151666,
"<think>": 151667,
"<tool_call>": 151657,
"<tool_response>": 151665,
"<|box_end|>": 151649,
"<|box_start|>": 151648,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|image_pad|>": 151655,
"<|object_ref_end|>": 151647,
"<|object_ref_start|>": 151646,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|video_pad|>": 151656,
"<|vision_end|>": 151653,
"<|vision_pad|>": 151654,
"<|vision_start|>": 151652
}

273
agent_core.py Normal file
View File

@@ -0,0 +1,273 @@
"""Shared agent core: chat template, instructions, tool-call parsing, episode logic.
This exact module drives (1) RL training rollouts, (2) every evaluation, and
(3) the shipped standalone tool — zero train/inference mismatch.
Conventions (inherited from the reference client):
- NO system role: instructions are merged into the first user message.
- Strict user/assistant alternation; tool output comes back as the next user
message, prefixed "Tool output: ".
- Plain-text tool calls, types "bash" and "final", lenient regex parsing, and
parse errors are fed back to the model as the next user message.
Format: reasoning goes in <think>...</think>, the tool call in <answer>...</answer>
(the reference's "Thoughts:" field is dropped — it is redundant with <think>).
Assistant turns end with "</answer>" followed by EOS. Past assistant turns are
rendered with the <think> block stripped (saves context; template handles it).
"""
import re
# ----------------------------------------------------------------------------
# Chat template
# ----------------------------------------------------------------------------
PREAMBLE = (
"A conversation between User and Assistant. The user asks a question, and the "
"Assistant solves it. The assistant first thinks about the reasoning process in "
"the mind and then provides the user with the answer. The reasoning process and "
"answer are enclosed within <think>...</think> and <answer>...</answer> tags, "
"respectively, i.e., <think> reasoning process here </think> "
"<answer>answer here </answer>."
)
# Past assistant messages are rendered with everything up to and including
# </think> removed. The generation prompt ends with "Assistant: <think>" + newline.
CHAT_TEMPLATE = (
"{{- '" + PREAMBLE.replace("'", "\\'") + "\\n\\n' -}}"
"{%- for message in messages -%}"
"{%- if message['role'] == 'user' -%}"
"{{- 'User: ' + message['content'] + '\\n\\n' -}}"
"{%- elif message['role'] == 'assistant' -%}"
"{%- set content = message['content'] -%}"
"{%- if '</think>' in content -%}"
"{%- set content = content.split('</think>')[-1] -%}"
"{%- endif -%}"
"{{- 'Assistant: ' + content.strip() + '\\n\\n' -}}"
"{%- endif -%}"
"{%- endfor -%}"
"{%- if add_generation_prompt -%}"
"{{- 'Assistant: <think>\\n' -}}"
"{%- endif -%}"
)
THINK_PREFIX = "<think>\n" # generation prompt ends with this; completions start after it
STOP_STRING = "</answer>"
# Sampling defaults — used for RL rollouts, every eval, and shipped generation_config.
GEN_TEMPERATURE = 1.0
GEN_TOP_P = 0.95
MAX_TURN_TOKENS = 1024 # shared training/eval/shipped-tool per-turn output budget
def render_messages(messages, add_generation_prompt: bool = True) -> str:
"""Render messages exactly like CHAT_TEMPLATE, without depending on Jinja.
Offline vLLM/Transformers calls use this function; server-mode vLLM uses the
tokenizer chat template below. Keep both byte-identical.
"""
parts = [PREAMBLE + "\n\n"]
for message in messages:
role = message["role"]
content = message["content"]
if role == "user":
parts.append("User: " + content + "\n\n")
elif role == "assistant":
if "</think>" in content:
content = content.split("</think>")[-1]
parts.append("Assistant: " + content.strip() + "\n\n")
if add_generation_prompt:
parts.append("Assistant: <think>\n")
return "".join(parts)
# ----------------------------------------------------------------------------
# Instructions (merged into the first user message)
# ----------------------------------------------------------------------------
TOOL_INSTRUCTIONS = """You are an AI agentic coding assistant. You complete the user's task by calling tools step by step, one tool call per turn.
First reason inside <think>...</think>, then put exactly one tool call inside <answer>...</answer> in exactly this format:
<answer>Tool type: [bash or final]
Tool query: [the shell command to run, or the final answer]</answer>
Available tools:
1. bash: runs a shell command in an isolated Linux sandbox and returns its output (e.g. write files with heredocs, run python, test code). The sandbox is stateless: the working directory and variables reset between calls, so chain commands with && where needed.
2. final: ends the task; put the complete final answer in Tool query.
The next user message will contain the tool output as "Tool output: ...". Keep outputs small (use head/tail). After your final tool call the episode ends."""
def coding_task_message(statement: str, starter_code: str = "", max_turns: int = 2) -> str:
"""First user message for a competitive-programming task (training AND eval)."""
if starter_code and starter_code.strip():
req = (
"Write a Python 3 solution that completes the following starter code, "
"keeping the exact same class name and method signature:\n"
"```python\n" + starter_code.strip() + "\n```"
)
else:
req = ("Write a complete Python 3 program that reads the input from stdin "
"and prints the required output to stdout.")
n_bash = max_turns - 1
one_turn_rule = (
"- This is a one-turn episode: NEVER call `bash`; submit the complete Python solution "
"with your single `final` tool call immediately.\n"
if max_turns == 1 else ""
)
return f"""{TOOL_INSTRUCTIONS}
Task: solve this competitive programming problem in Python 3.
{statement.strip()}
{req}
Rules:
- You have at most {max_turns} tool calls in total, so at most {n_bash} bash call{"s" if n_bash != 1 else ""}; you may use bash to test your solution on the example input before submitting.
{one_turn_rule}- Your last tool call must be "final" and its Tool query must contain ONLY your complete solution inside one ```python ... ``` code block.
- The solution is judged against hidden tests; handle all edge cases and stay within a few seconds of runtime."""
PARSE_ERROR_FEEDBACK = """Parsing Error: {err}
Please repeat, strictly matching this format:
<answer>Tool type: [bash or final]
Tool query: [the shell command, or the final answer]</answer>"""
# ----------------------------------------------------------------------------
# Parsing
# ----------------------------------------------------------------------------
def parse_action(text: str):
"""Parse an assistant completion into (tool_type, tool_query, error).
`text` is the completion text (reasoning + <answer> block, with or without
the closing tag). Lenient: falls back to scanning the whole text.
"""
m = re.search(r"<answer>(.*?)</answer>", text, re.DOTALL)
if m:
seg = m.group(1)
elif "<answer>" in text:
seg = text.split("<answer>", 1)[1]
else:
seg = text.split("</think>")[-1]
tt = re.search(r"Tool type:\s*(.*?)(?=\n\s*Tool query:|$)", seg, re.DOTALL | re.IGNORECASE)
tq = re.search(r"Tool query:\s*(.*)", seg, re.DOTALL | re.IGNORECASE)
tool_type = tt.group(1).strip().lower() if tt else None
# The reference protocol presents its choices as ``[bash or final]`` and
# base completions commonly echo the brackets (``Tool type: [final]``).
# Treat that as the same lenient plain-text call rather than wasting a turn.
if tool_type:
tool_type = tool_type.strip(" \t[](){}<>`*_\"'")
tool_query = tq.group(1).strip() if tq else None
if not tool_type or tool_query is None or tool_query == "":
return None, None, ("Could not parse 'Tool type' or 'Tool query' from the response. "
"Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.")
if tool_type not in ("bash", "final"):
return None, None, f"Unknown tool type '{tool_type}'; must be 'bash' or 'final'."
if tool_type == "bash":
# strip a markdown fence around the command, reference-style
tool_query = re.sub(r"^```[a-zA-Z]*\n", "", tool_query)
tool_query = re.sub(r"\n```$", "", tool_query)
tool_query = tool_query.strip()
return tool_type, tool_query, None
def extract_code(final_answer: str) -> str:
"""Extract the python solution from a final answer (last ```python block, else raw)."""
blocks = re.findall(r"```(?:python|py)?\n(.*?)```", final_answer, re.DOTALL)
if blocks:
return blocks[-1].strip()
return final_answer.strip()
def normalize_completion(content: str, finish_reason: str = "stop") -> str:
"""Make server-mode completions byte-identical to training rollouts: the stop
string is part of the turn. If the server stripped it (or stopped at EOS right
after it), re-append."""
if STOP_STRING not in content and finish_reason == "stop" and "<answer>" in content:
content = content.rstrip() + STOP_STRING
return content
# ----------------------------------------------------------------------------
# Episode state machine (transport-agnostic)
# ----------------------------------------------------------------------------
class Episode:
"""One agentic episode with strict user/assistant alternation.
Use: messages -> (render+generate outside) -> step(completion) which returns
("bash", cmd) | ("final", answer) | ("continue", None) after appending messages.
After executing a bash command, call add_tool_output(out).
"""
def __init__(self, first_user_msg: str, max_turns: int = 2, meta=None):
self.messages = [{"role": "user", "content": first_user_msg}]
self.max_turns = max_turns
self.turns = 0
self.done = False
self.final_answer = None
self.end_reason = None
self.meta = meta or {}
def step(self, completion: str):
"""completion: model output text (starts after 'Assistant: <think>\\n')."""
assert not self.done
content = THINK_PREFIX + completion
self.messages.append({"role": "assistant", "content": content})
self.turns += 1
tool_type, tool_query, err = parse_action(completion)
if err is not None:
if self.turns >= self.max_turns:
self.done, self.end_reason = True, "parse_error"
return ("continue", None)
self.add_tool_output(PARSE_ERROR_FEEDBACK.format(err=err))
return ("continue", None)
if tool_type == "final":
self.done, self.end_reason = True, "final"
self.final_answer = tool_query
return ("final", tool_query)
# bash
if self.turns >= self.max_turns:
self.done, self.end_reason = True, "out_of_turns"
return ("continue", None)
return ("bash", tool_query)
def add_tool_output(self, output: str):
self.messages.append({"role": "user", "content": f"Tool output: {output}"})
def run_episode(generate_fn, bash_fn, first_user_msg: str, max_turns: int = 2, meta=None):
"""Sequential episode driver (eval + shipped tool). generate_fn(messages)->completion."""
ep = Episode(first_user_msg, max_turns=max_turns, meta=meta)
while not ep.done:
completion = generate_fn(ep.messages)
kind, arg = ep.step(completion)
if kind == "bash":
ep.add_tool_output(bash_fn(arg))
return ep
def openai_generate_fn(client, model: str, temperature: float = GEN_TEMPERATURE,
top_p: float = GEN_TOP_P, max_tokens: int = MAX_TURN_TOKENS):
"""generate_fn over an OpenAI-compatible chat endpoint (vLLM server). The server
applies this repo's chat template (shipped in the tokenizer)."""
def fn(messages):
kwargs = dict(model=model, messages=messages, temperature=temperature,
top_p=top_p, max_tokens=max_tokens, stop=[STOP_STRING])
try:
resp = client.chat.completions.create(
**kwargs, extra_body={"include_stop_str_in_output": True})
except Exception:
resp = client.chat.completions.create(**kwargs)
choice = resp.choices[0]
return normalize_completion(choice.message.content or "", choice.finish_reason)
return fn

1
chat_template.jinja Normal file
View File

@@ -0,0 +1 @@
{{- 'A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think>...</think> and <answer>...</answer> tags, respectively, i.e., <think> reasoning process here </think> <answer>answer here </answer>.\n\n' -}}{%- for message in messages -%}{%- if message['role'] == 'user' -%}{{- 'User: ' + message['content'] + '\n\n' -}}{%- elif message['role'] == 'assistant' -%}{%- set content = message['content'] -%}{%- if '</think>' in content -%}{%- set content = content.split('</think>')[-1] -%}{%- endif -%}{{- 'Assistant: ' + content.strip() + '\n\n' -}}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{- 'Assistant: <think>\n' -}}{%- endif -%}

30
config.json Normal file
View File

@@ -0,0 +1,30 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 12288,
"max_position_embeddings": 32768,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.52.4",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

338
eval/baseline_results.json Normal file
View File

@@ -0,0 +1,338 @@
{
"details": [
{
"difficulty": "hard",
"problem_id": "arc196_a",
"samples": [
{
"code_chars": 380,
"completion_chars": 2880,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 327,
"completion_chars": 4333,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 5950,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"code_chars": 318,
"completion_chars": 5576,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 316,
"completion_chars": 6824,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 540,
"completion_chars": 3408,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_f",
"samples": [
{
"code_chars": 332,
"completion_chars": 5783,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 325,
"completion_chars": 4684,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 351,
"completion_chars": 2437,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 194,
"completion_chars": 4745,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 360,
"completion_chars": 3907,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 6143,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
},
{
"difficulty": "easy",
"problem_id": "abc400_a",
"samples": [
{
"code_chars": 180,
"completion_chars": 982,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 1062,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"code_chars": 285,
"completion_chars": 1636,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 264,
"completion_chars": 1539,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 165,
"completion_chars": 886,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 175,
"completion_chars": 940,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "arc196_c",
"samples": [
{
"code_chars": 0,
"completion_chars": 9181,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": true,
"tool_type": null
},
{
"code_chars": 601,
"completion_chars": 7205,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 371,
"completion_chars": 3946,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 1489,
"completion_chars": 8048,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 5614,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"code_chars": 125,
"completion_chars": 7651,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "medium",
"problem_id": "abc400_d",
"samples": [
{
"code_chars": 2468,
"completion_chars": 5103,
"end": "timeout",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 3814,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"code_chars": 1121,
"completion_chars": 9244,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 3572,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"code_chars": 1874,
"completion_chars": 4560,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 3678,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_g",
"samples": [
{
"code_chars": 820,
"completion_chars": 3243,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 1127,
"completion_chars": 5958,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 842,
"completion_chars": 6255,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 326,
"completion_chars": 6262,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 869,
"completion_chars": 4210,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 3812,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
}
],
"eval_limit": 6,
"max_new_tokens": 1024,
"pass_at_1": 0.1111111111111111,
"passed_samples": 4,
"samples_per_problem": 6,
"tag": "baseline",
"temperature": 1.0,
"tool_turn_limit": 1,
"top_p": 0.95,
"total_samples": 36
}

247
eval/eval_subset.json Normal file
View File

@@ -0,0 +1,247 @@
[
{
"contest_date": "2025-04-06T00:00:00",
"difficulty": "hard",
"hidden_tests": [
{
"input": "4564\n253140288 293305612 750791934 354530934 444237786 846682985 925663544 35056",
"output_len": 14
},
{
"input": "63\n55449 10966 48278 28854 34199 76708 21835 56522 25159 46992 15075 8370 92014 ",
"output_len": 8
},
{
"input": "52\n68827 19447 33671 78983 20435 49677 76409 38586 94127 92586 61661 8698 11095 ",
"output_len": 8
}
],
"problem_id": "arc196_a",
"public_tests": [
{
"input": "4\n1 2 5 3",
"output": "5",
"testtype": "stdin"
},
{
"input": "7\n3 1 4 1 5 9 2",
"output": "14",
"testtype": "stdin"
},
{
"input": "5\n1 1 1 1 1",
"output": "0",
"testtype": "stdin"
}
],
"source": "livecodebench/code_generation_lite:v6",
"starter_code": "",
"statement": "You are given a length-N sequence A = (A_1, A_2, \\ldots, A_N).\nYou will repeatedly perform the following operation until the sequence has length at most 1: choose two adjacent numbers and remove both from the sequence.\nThe score obtained in one operation is the absolute difference of the two chosen numbers.\nFind the maximum possible total score obtained.\n\nInput\n\nThe input is given from Standard Input in the following format:\nN\nA_1 A_2 \\ldots A_N\n\nOutput\n\nPrint the maximum possible total score ob",
"title": "Adjacent Delete"
},
{
"contest_date": "2025-03-22T00:00:00",
"difficulty": "hard",
"hidden_tests": [
{
"input": "ZGBZMOLYGOLZUCBBPIAQVSSGHCYUYQWQNQJDENSNCDNCDNYEXAZONVNAPKXICLCDLWALLFAHLCTEGAGV",
"output_len": 166
},
{
"input": "UOMWVSNBMWSNYVWBFOCIWFOQPRTYABPKJOBZZNGRUCXEAMVNKAGAWYAVQTDGDTUG",
"output_len": 128
},
{
"input": "XCU",
"output_len": 6
}
],
"problem_id": "abc398_f",
"public_tests": [
{
"input": "ABC",
"output": "ABCBA",
"testtype": "stdin"
},
{
"input": "Z",
"output": "Z",
"testtype": "stdin"
},
{
"input": "TREE",
"output": "TREERT",
"testtype": "stdin"
}
],
"source": "livecodebench/code_generation_lite:v6",
"starter_code": "",
"statement": "Find one shortest palindrome that has S as its prefix.\n\nInput\n\nThe input is given from Standard Input in the following format:\nS\n\nOutput\n\nPrint the answer.\r\nIf multiple solutions exist, any of them is accepted.\n\nConstraints\n\n\n- S is a string of length between 1 and 500000, inclusive, consisting of uppercase English letters.\n\nSample Input 1\n\nABC\n\nSample Output 1\n\nABCBA\r\n\nABCBA is a shortest palindrome that has S= ABC as its prefix.\n\nSample Input 2\n\nZ\n\nSample Output 2\n\nZ\r\n\nZ is a shortest palindro",
"title": "ABCBA"
},
{
"contest_date": "2025-04-05T00:00:00",
"difficulty": "easy",
"hidden_tests": [
{
"input": "286",
"output_len": 3
},
{
"input": "20",
"output_len": 3
},
{
"input": "26",
"output_len": 3
}
],
"problem_id": "abc400_a",
"public_tests": [
{
"input": "10",
"output": "40",
"testtype": "stdin"
},
{
"input": "11",
"output": "-1",
"testtype": "stdin"
},
{
"input": "400",
"output": "1",
"testtype": "stdin"
}
],
"source": "livecodebench/code_generation_lite:v6",
"starter_code": "",
"statement": "In the ceremony commemorating ABC400, we want to arrange 400 people in a rectangular formation of A rows and B columns without any gaps.\nYou are given a positive integer A. Print the value of a positive integer B for which such an arrangement is possible. If there is no such positive integer B, print -1.\n\nInput\n\nThe input is given from Standard Input in the following format:\nA\n\nOutput\n\nPrint the value of B or -1 as specified by the problem statement.\n\nConstraints\n\n\n- A is an integer between 1 an",
"title": "ABC400 Party"
},
{
"contest_date": "2025-04-06T00:00:00",
"difficulty": "hard",
"hidden_tests": [
{
"input": "118\nWWWWBWBWWBWBWWWWWWWBWBWWBBBBBBBWBBWBBWBWBWBWBWWWBBBWBWBWBWWBWWWWWWBBWWBBWBWB",
"output_len": 2
},
{
"input": "108\nBBBWWBWBWWBWBBWWBWWBWBWWBWWWWBWWWBWBWBWWWBBWWWBBWWWBWBBBWWWWBBWBWBBBWBWWWBWB",
"output_len": 2
},
{
"input": "8\nWWBWBWWBWBBBWWBB",
"output_len": 2
}
],
"problem_id": "arc196_c",
"public_tests": [
{
"input": "2\nBWBW",
"output": "1",
"testtype": "stdin"
},
{
"input": "4\nBWWBWBWB",
"output": "0",
"testtype": "stdin"
},
{
"input": "9\nBWWBWBBBWWBWBBWWBW",
"output": "240792",
"testtype": "stdin"
}
],
"source": "livecodebench/code_generation_lite:v6",
"starter_code": "",
"statement": "There is a directed graph with 2N vertices and 2N-1 edges.\r\nThe vertices are numbered 1, 2, \\ldots, 2N, and the i-th edge is a directed edge from vertex i to vertex i+1.\nYou are given a length-2N string S = S_1 S_2 \\ldots S_{2N} consisting of N Ws and N Bs.\r\nVertex i is colored white if S_i is W, and black if S_i is B.\nYou will perform the following series of operations:\n\n- Partition the 2N vertices into N pairs, each consisting of one white vertex and one black vertex.\n- For each pair, add a di",
"title": "Strongly Connected"
},
{
"contest_date": "2025-04-05T00:00:00",
"difficulty": "medium",
"hidden_tests": [
{
"input": "2 2\n..\n..\n2 2 2 1",
"output_len": 2
},
{
"input": "8 5\n....#\n...#.\n.##.#\n###.#\n##...\n###.#\n.#..#\n.##..\n4 4 2 5",
"output_len": 2
},
{
"input": "6 4\n..##\n...#\n##..\n.#..\n..#.\n...#\n4 1 1 1",
"output_len": 2
}
],
"problem_id": "abc400_d",
"public_tests": [
{
"input": "10 10\n..........\n#########.\n#.......#.\n#..####.#.\n##....#.#.\n#####.#.#.\n.##.#.#.#.\n###.#.#.#.\n###.#.#.#.\n#.....#...\n1 1 7 1",
"output": "1",
"testtype": "stdin"
},
{
"input": "2 2\n.#\n#.\n1 1 2 2",
"output": "1",
"testtype": "stdin"
},
{
"input": "1 3\n.#.\n1 1 1 3",
"output": "1",
"testtype": "stdin"
},
{
"input": "20 20\n####################\n##...##....###...###\n#.....#.....#.....##\n#..#..#..#..#..#..##\n#..#..#....##..#####\n#.....#.....#..#####\n#.....#..#..#..#..##\n#..#..#.....#.....##\n#..#..#....###...###\n####################\n####################\n##..#..##...###...##\n##..#..#.....#.....#\n##..#..#..#..#..#..#\n##..#..#..#..#..#..#\n##.....#..#..#..#..#\n###....#..#..#..#..#\n#####..#.....#.....#\n#####..##...###...##\n####################\n3 3 18 18",
"output": "3",
"testtype": "stdin"
}
],
"source": "livecodebench/code_generation_lite:v6",
"starter_code": "",
"statement": "Takahashi is about to go buy eel at a fish shop.\nThe town where he lives is divided into a grid of H rows and W columns. Each cell is either a road or a wall.\r\nLet us denote the cell at the i-th row from the top (1\\leq i \\leq H) and the j-th column from the left (1\\leq j \\leq W) as cell (i,j).\r\nInformation about each cell is given by H strings S_1,S_2,\\ldots,S_H, each of length W. Specifically, if the j-th character of S_i (1\\leq i \\leq H,1\\leq j\\leq W) is ., cell (i,j) is a road; if it is #, ce",
"title": "Takahashi the Wall Breaker"
},
{
"contest_date": "2025-03-22T00:00:00",
"difficulty": "hard",
"hidden_tests": [
{
"input": "5 4\n1 2\n1 3\n1 4\n1 5",
"output_len": 10
},
{
"input": "128 2871\n1 68\n24 62\n13 107\n21 41\n6 89\n10 106\n24 88\n25 53\n21 44\n2 72\n2 95\n29 79\n1",
"output_len": 10
},
{
"input": "6 5\n1 6\n3 6\n2 6\n4 6\n5 6",
"output_len": 10
}
],
"problem_id": "abc398_g",
"public_tests": [
{
"input": "4 3\n1 2\n2 3\n3 4",
"output": "Aoki",
"testtype": "stdin"
},
{
"input": "4 2\n1 2\n3 4",
"output": "Takahashi",
"testtype": "stdin"
},
{
"input": "9 5\n2 9\n2 3\n4 6\n5 7\n1 8",
"output": "Aoki",
"testtype": "stdin"
}
],
"source": "livecodebench/code_generation_lite:v6",
"starter_code": "",
"statement": "You are given a simple undirected graph with N vertices and M edges, with vertices labeled 1 to N and edges labeled 1 to M. The i-th edge connects vertices U_i and V_i. Initially, G does not contain an odd cycle.\nTakahashi and Aoki will play a game using this graph G. With Aoki going first, they take turns performing the following operation:\n\n- Choose a pair of integers (i,j) with 1 \\leq i < j \\leq N that satisfies both of the following conditions, then add an edge connecting vertices i and j to",
"title": "Not Only Tree Game"
}
]

302
eval/final_results.json Normal file
View File

@@ -0,0 +1,302 @@
{
"details": [
{
"difficulty": "hard",
"problem_id": "arc196_a",
"samples": [
{
"completion_chars": 11690,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12444,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12570,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12044,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 1151,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 10895,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_f",
"samples": [
{
"completion_chars": 2877,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2932,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": true,
"tool_type": null
},
{
"completion_chars": 715,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2867,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2538,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2577,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "easy",
"problem_id": "abc400_a",
"samples": [
{
"completion_chars": 328,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 917,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 600,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 742,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 808,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 820,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "arc196_c",
"samples": [
{
"completion_chars": 5294,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1689,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 5976,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"completion_chars": 4300,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 4040,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 5089,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "medium",
"problem_id": "abc400_d",
"samples": [
{
"completion_chars": 8429,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 7583,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 7201,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 7008,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2737,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"completion_chars": 7069,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_g",
"samples": [
{
"completion_chars": 11630,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12637,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 4093,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 3088,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 13122,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 2073,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
}
],
"eval_limit": 6,
"max_new_tokens": 1024,
"pass_at_1": 0.05555555555555555,
"passed_samples": 2,
"samples_per_problem": 6,
"tag": "stage1_group3",
"temperature": 1.0,
"tool_turn_limit": 1,
"top_p": 0.95,
"total_samples": 36
}

338
eval/full_schedule_endpoint_results.json Normal file
View File

@@ -0,0 +1,338 @@
{
"details": [
{
"difficulty": "hard",
"problem_id": "arc196_a",
"samples": [
{
"code_chars": 719,
"completion_chars": 828,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 115,
"completion_chars": 2707,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 389,
"completion_chars": 1655,
"end": "timeout",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 214,
"completion_chars": 2386,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 282,
"completion_chars": 2263,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 303,
"completion_chars": 2072,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_f",
"samples": [
{
"code_chars": 165,
"completion_chars": 3251,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 160,
"completion_chars": 3358,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 877,
"completion_chars": 985,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 499,
"completion_chars": 2528,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 396,
"completion_chars": 2252,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 327,
"completion_chars": 2724,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "easy",
"problem_id": "abc400_a",
"samples": [
{
"code_chars": 125,
"completion_chars": 12597,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 88,
"completion_chars": 12702,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 143,
"completion_chars": 12602,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 103,
"completion_chars": 12689,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 30,
"completion_chars": 12918,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 2353,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
},
{
"difficulty": "hard",
"problem_id": "arc196_c",
"samples": [
{
"code_chars": 111,
"completion_chars": 3537,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 231,
"completion_chars": 2460,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 233,
"completion_chars": 3048,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 132,
"completion_chars": 3325,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 870,
"completion_chars": 979,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 389,
"completion_chars": 2591,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "medium",
"problem_id": "abc400_d",
"samples": [
{
"code_chars": 733,
"completion_chars": 9798,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 9689,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"code_chars": 1866,
"completion_chars": 7054,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 1347,
"completion_chars": 8086,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 2217,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"code_chars": 1162,
"completion_chars": 8951,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_g",
"samples": [
{
"code_chars": 0,
"completion_chars": 4543,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"code_chars": 2840,
"completion_chars": 3882,
"end": "timeout",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 347,
"completion_chars": 11776,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 0,
"completion_chars": 5113,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"code_chars": 791,
"completion_chars": 11156,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"code_chars": 1219,
"completion_chars": 10183,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
}
],
"eval_limit": 6,
"max_new_tokens": 1024,
"pass_at_1": 0.0,
"passed_samples": 0,
"samples_per_problem": 6,
"tag": "final",
"temperature": 1.0,
"tool_turn_limit": 1,
"top_p": 0.95,
"total_samples": 36
}

916
eval/selection_results.json Normal file
View File

@@ -0,0 +1,916 @@
{
"best": "stage1_group3",
"results": {
"after_sft": {
"details": [
{
"difficulty": "hard",
"problem_id": "arc196_a",
"samples": [
{
"completion_chars": 1731,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1841,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 781,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2276,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1543,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2677,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_f",
"samples": [
{
"completion_chars": 3067,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 2838,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 599,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 2426,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1308,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1743,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "easy",
"problem_id": "abc400_a",
"samples": [
{
"completion_chars": 376,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 520,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 725,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 666,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 294,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 431,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "arc196_c",
"samples": [
{
"completion_chars": 10417,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 11700,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1704,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 3592,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 12478,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 11832,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "medium",
"problem_id": "abc400_d",
"samples": [
{
"completion_chars": 5287,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 10882,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1065,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 1647,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 13229,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 9263,
"end": "timeout",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_g",
"samples": [
{
"completion_chars": 12369,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 10642,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 11360,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2075,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 11910,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 604,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
}
],
"eval_limit": 6,
"max_new_tokens": 1024,
"pass_at_1": 0.027777777777777776,
"passed_samples": 1,
"samples_per_problem": 6,
"tag": "after_sft",
"temperature": 1.0,
"tool_turn_limit": 1,
"top_p": 0.95,
"total_samples": 36
},
"stage1_group3": {
"details": [
{
"difficulty": "hard",
"problem_id": "arc196_a",
"samples": [
{
"completion_chars": 11690,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12444,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12570,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12044,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 1151,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 10895,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_f",
"samples": [
{
"completion_chars": 2877,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2932,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": true,
"tool_type": null
},
{
"completion_chars": 715,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2867,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2538,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2577,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "easy",
"problem_id": "abc400_a",
"samples": [
{
"completion_chars": 328,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 917,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 600,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 742,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 808,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 820,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "arc196_c",
"samples": [
{
"completion_chars": 5294,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1689,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 5976,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"completion_chars": 4300,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 4040,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 5089,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "medium",
"problem_id": "abc400_d",
"samples": [
{
"completion_chars": 8429,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 7583,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 7201,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 7008,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2737,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"completion_chars": 7069,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_g",
"samples": [
{
"completion_chars": 11630,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12637,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 4093,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 3088,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 13122,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 2073,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
}
],
"eval_limit": 6,
"max_new_tokens": 1024,
"pass_at_1": 0.05555555555555555,
"passed_samples": 2,
"samples_per_problem": 6,
"tag": "stage1_group3",
"temperature": 1.0,
"tool_turn_limit": 1,
"top_p": 0.95,
"total_samples": 36
},
"stage2_group1": {
"details": [
{
"difficulty": "hard",
"problem_id": "arc196_a",
"samples": [
{
"completion_chars": 11873,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 4103,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 10118,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 11800,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2309,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 2870,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_f",
"samples": [
{
"completion_chars": 1513,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1075,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1550,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1186,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1775,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 399,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "easy",
"problem_id": "abc400_a",
"samples": [
{
"completion_chars": 478,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 466,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 949,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 556,
"end": "ok",
"passed": true,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 328,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 434,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "arc196_c",
"samples": [
{
"completion_chars": 5189,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 4221,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 3838,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 1365,
"end": "format_error: final",
"passed": false,
"stop_seen": false,
"tool_type": "final"
},
{
"completion_chars": 4021,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 4585,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "medium",
"problem_id": "abc400_d",
"samples": [
{
"completion_chars": 1449,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 13228,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 7164,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 10712,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 9464,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 5995,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
}
]
},
{
"difficulty": "hard",
"problem_id": "abc398_g",
"samples": [
{
"completion_chars": 10181,
"end": "runtime_error",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 11037,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2580,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
},
{
"completion_chars": 10968,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 12204,
"end": "wrong_answer",
"passed": false,
"stop_seen": true,
"tool_type": "final"
},
{
"completion_chars": 2073,
"end": "format_error: Could not parse 'Tool type' or 'Tool query' from the response. Ensure you output an <answer> block with 'Tool type:' and 'Tool query:'.",
"passed": false,
"stop_seen": false,
"tool_type": null
}
]
}
],
"eval_limit": 6,
"max_new_tokens": 1024,
"pass_at_1": 0.027777777777777776,
"passed_samples": 1,
"samples_per_problem": 6,
"tag": "stage2_group1",
"temperature": 1.0,
"tool_turn_limit": 1,
"top_p": 0.95,
"total_samples": 36
}
},
"summary": {
"after_sft": 0.027777777777777776,
"stage1_group3": 0.05555555555555555,
"stage2_group1": 0.027777777777777776
}
}

770
eval/sft_log.json Normal file
View File

@@ -0,0 +1,770 @@
[
{
"grad_norm": 1.040154218673706,
"index": 1,
"loss": 0.48266512155532837,
"nll": 0.48266512155532837,
"problem_id": "438_A. The Child and Toy",
"skipped": null
},
{
"grad_norm": 1.1882520914077759,
"index": 2,
"loss": 0.5828834176063538,
"nll": 0.5828834176063538,
"problem_id": "300_A. Array",
"skipped": null
},
{
"grad_norm": 1.9626847505569458,
"index": 3,
"loss": 1.1390048265457153,
"nll": 1.1390048265457153,
"problem_id": "478_A. Initial Bet",
"skipped": null
},
{
"grad_norm": 1.5171997547149658,
"index": 4,
"loss": 0.9680904150009155,
"nll": 0.9680904150009155,
"problem_id": "676_A. Nicholas and Permutation",
"skipped": null
},
{
"grad_norm": 0.3860478699207306,
"index": 5,
"loss": 0.39563682675361633,
"nll": 0.39563682675361633,
"problem_id": "1031_B. Curiosity Has No Limits",
"skipped": null
},
{
"grad_norm": 1.020121455192566,
"index": 6,
"loss": 0.8472175002098083,
"nll": 0.8472175002098083,
"problem_id": "940_A. Points on the line",
"skipped": null
},
{
"grad_norm": 0.4967013895511627,
"index": 7,
"loss": 0.5509471893310547,
"nll": 0.5509471893310547,
"problem_id": "1080_C. Masha and two friends",
"skipped": null
},
{
"grad_norm": 1.479300856590271,
"index": 8,
"loss": 0.8386653661727905,
"nll": 0.8386653661727905,
"problem_id": "38_C. Blinds",
"skipped": null
},
{
"grad_norm": 0.8893219828605652,
"index": 9,
"loss": 0.3628019392490387,
"nll": 0.3628019392490387,
"problem_id": "489_B. BerSU Ball",
"skipped": null
},
{
"grad_norm": 1.4400527477264404,
"index": 10,
"loss": 0.9306145310401917,
"nll": 0.9306145310401917,
"problem_id": "787_A. The Monster",
"skipped": null
},
{
"grad_norm": 1.0569334030151367,
"index": 11,
"loss": 0.6781665086746216,
"nll": 0.6781665086746216,
"problem_id": "1070_D. Garbage Disposal",
"skipped": null
},
{
"grad_norm": 1.9829460382461548,
"index": 12,
"loss": 1.084402322769165,
"nll": 1.084402322769165,
"problem_id": "805_A. Fake NP",
"skipped": null
},
{
"grad_norm": 0.910430908203125,
"index": 13,
"loss": 0.7472644448280334,
"nll": 0.7472644448280334,
"problem_id": "1305_B. Kuroni and Simple Strings",
"skipped": null
},
{
"grad_norm": 0.8546664714813232,
"index": 14,
"loss": 0.4409652650356293,
"nll": 0.4409652650356293,
"problem_id": "742_A. Arpa\u2019s hard exam and Mehrdad\u2019s naive cheat",
"skipped": null
},
{
"grad_norm": 1.0713262557983398,
"index": 15,
"loss": 0.9237834215164185,
"nll": 0.9237834215164185,
"problem_id": "358_B. Dima and Text Messages",
"skipped": null
},
{
"grad_norm": 1.3963273763656616,
"index": 16,
"loss": 0.7768023610115051,
"nll": 0.7768023610115051,
"problem_id": "1443_A. Kids Seating",
"skipped": null
},
{
"grad_norm": 1.0952928066253662,
"index": 17,
"loss": 0.8175995945930481,
"nll": 0.8175995945930481,
"problem_id": "604_B. More Cowbell",
"skipped": null
},
{
"grad_norm": 1.6387556791305542,
"index": 18,
"loss": 1.1116256713867188,
"nll": 1.1116256713867188,
"problem_id": "1043_A. Elections",
"skipped": null
},
{
"grad_norm": 0.9755963683128357,
"index": 19,
"loss": 0.7564535737037659,
"nll": 0.7564535737037659,
"problem_id": "1054_B. Appending Mex",
"skipped": null
},
{
"grad_norm": 0.8682615756988525,
"index": 20,
"loss": 0.785297691822052,
"nll": 0.785297691822052,
"problem_id": "248_B. Chilly Willy",
"skipped": null
},
{
"grad_norm": 0.7661585807800293,
"index": 21,
"loss": 0.6494926810264587,
"nll": 0.6494926810264587,
"problem_id": "985_A. Chess Placing",
"skipped": null
},
{
"grad_norm": 2.292353630065918,
"index": 22,
"loss": 0.8781017065048218,
"nll": 0.8781017065048218,
"problem_id": "110_B. Lucky String",
"skipped": null
},
{
"grad_norm": 0.833644449710846,
"index": 23,
"loss": 0.504396378993988,
"nll": 0.504396378993988,
"problem_id": "519_B. A and B and Compilation Errors",
"skipped": null
},
{
"grad_norm": 0.9993244409561157,
"index": 24,
"loss": 0.8234552145004272,
"nll": 0.8234552145004272,
"problem_id": "615_A. Bulbs",
"skipped": null
},
{
"grad_norm": 1.3748852014541626,
"index": 25,
"loss": 0.5862762928009033,
"nll": 0.5862762928009033,
"problem_id": "270_B. Multithreading",
"skipped": null
},
{
"grad_norm": 1.3495038747787476,
"index": 26,
"loss": 1.5716842412948608,
"nll": 1.5716842412948608,
"problem_id": "272_C. Dima and Staircase",
"skipped": null
},
{
"grad_norm": 0.9883641600608826,
"index": 27,
"loss": 0.5327346324920654,
"nll": 0.5327346324920654,
"problem_id": "5_A. Chat Server's Outgoing Traffic",
"skipped": null
},
{
"grad_norm": 0.7346091270446777,
"index": 28,
"loss": 0.721221923828125,
"nll": 0.721221923828125,
"problem_id": "1408_B. Arrays Sum",
"skipped": null
},
{
"grad_norm": 1.3178147077560425,
"index": 29,
"loss": 0.7296963334083557,
"nll": 0.7296963334083557,
"problem_id": "177_D1. Encrypting Messages",
"skipped": null
},
{
"grad_norm": 1.020062804222107,
"index": 30,
"loss": 0.5247489213943481,
"nll": 0.5247489213943481,
"problem_id": "282_B. Painting Eggs",
"skipped": null
},
{
"grad_norm": 0.8763545155525208,
"index": 31,
"loss": 0.5632577538490295,
"nll": 0.5632577538490295,
"problem_id": "1334_A. Level Statistics",
"skipped": null
},
{
"grad_norm": 0.9953303337097168,
"index": 32,
"loss": 0.3932126760482788,
"nll": 0.3932126760482788,
"problem_id": "630_A. Again Twenty Five!",
"skipped": null
},
{
"grad_norm": 1.3474981784820557,
"index": 33,
"loss": 0.514382004737854,
"nll": 0.514382004737854,
"problem_id": "1366_A. Shovels and Swords",
"skipped": null
},
{
"grad_norm": 0.7742969393730164,
"index": 34,
"loss": 0.5816912055015564,
"nll": 0.5816912055015564,
"problem_id": "448_B. Suffix Structures",
"skipped": null
},
{
"grad_norm": 1.562877893447876,
"index": 35,
"loss": 0.8283543586730957,
"nll": 0.8283543586730957,
"problem_id": "1353_C. Board Moves",
"skipped": null
},
{
"grad_norm": 0.783785343170166,
"index": 36,
"loss": 0.3817444145679474,
"nll": 0.3817444145679474,
"problem_id": "259_B. Little Elephant and Magic Square",
"skipped": null
},
{
"grad_norm": 0.7053418755531311,
"index": 37,
"loss": 0.4648408889770508,
"nll": 0.4648408889770508,
"problem_id": "687_A. NP-Hard Problem",
"skipped": null
},
{
"grad_norm": 0.8954440355300903,
"index": 38,
"loss": 0.5581481456756592,
"nll": 0.5581481456756592,
"problem_id": "1121_A. Technogoblet of Fire",
"skipped": null
},
{
"grad_norm": 1.3505918979644775,
"index": 39,
"loss": 0.5477667450904846,
"nll": 0.5477667450904846,
"problem_id": "918_A. Eleven",
"skipped": null
},
{
"grad_norm": 1.0709880590438843,
"index": 40,
"loss": 0.7576397657394409,
"nll": 0.7576397657394409,
"problem_id": "1084_C. The Fair Nut and String",
"skipped": null
},
{
"grad_norm": 1.2410813570022583,
"index": 41,
"loss": 0.981418788433075,
"nll": 0.981418788433075,
"problem_id": "545_C. Woodcutters",
"skipped": null
},
{
"grad_norm": 0.44853660464286804,
"index": 42,
"loss": 0.20538830757141113,
"nll": 0.20538830757141113,
"problem_id": "1152_A. Neko Finds Grapes",
"skipped": null
},
{
"grad_norm": 0.7902032732963562,
"index": 43,
"loss": 0.43099430203437805,
"nll": 0.43099430203437805,
"problem_id": "385_A. Bear and Raspberry",
"skipped": null
},
{
"grad_norm": 0.8370153307914734,
"index": 44,
"loss": 0.5004207491874695,
"nll": 0.5004207491874695,
"problem_id": "1392_B. Omkar and Infinity Clock",
"skipped": null
},
{
"grad_norm": 0.6053890585899353,
"index": 45,
"loss": 0.527194082736969,
"nll": 0.527194082736969,
"problem_id": "1301_B. Motarack's Birthday",
"skipped": null
},
{
"grad_norm": 1.0937732458114624,
"index": 46,
"loss": 0.703790545463562,
"nll": 0.703790545463562,
"problem_id": "494_A. Treasure",
"skipped": null
},
{
"grad_norm": 0.902428150177002,
"index": 47,
"loss": 0.42323175072669983,
"nll": 0.42323175072669983,
"problem_id": "1366_B. Shuffle",
"skipped": null
},
{
"grad_norm": 0.6949153542518616,
"index": 48,
"loss": 0.37455669045448303,
"nll": 0.37455669045448303,
"problem_id": "69_A. Young Physicist",
"skipped": null
},
{
"grad_norm": 1.3904484510421753,
"index": 49,
"loss": 0.7064117193222046,
"nll": 0.7064117193222046,
"problem_id": "125_A. Measuring Lengths in Baden",
"skipped": null
},
{
"grad_norm": 1.2604994773864746,
"index": 50,
"loss": 0.6908637285232544,
"nll": 0.6908637285232544,
"problem_id": "45_I. TCMCF+++",
"skipped": null
},
{
"grad_norm": 1.3847407102584839,
"index": 51,
"loss": 0.6546027660369873,
"nll": 0.6546027660369873,
"problem_id": "740_B. Alyona and flowers",
"skipped": null
},
{
"grad_norm": 1.031480073928833,
"index": 52,
"loss": 0.6020389795303345,
"nll": 0.6020389795303345,
"problem_id": "136_B. Ternary Logic",
"skipped": null
},
{
"grad_norm": 0.9329264760017395,
"index": 53,
"loss": 0.766955554485321,
"nll": 0.766955554485321,
"problem_id": "1139_C. Edgy Trees",
"skipped": null
},
{
"grad_norm": 0.34977203607559204,
"index": 54,
"loss": 0.44151249527931213,
"nll": 0.44151249527931213,
"problem_id": "1481_B. New Colony",
"skipped": null
},
{
"grad_norm": 0.8771538734436035,
"index": 55,
"loss": 0.7079184055328369,
"nll": 0.7079184055328369,
"problem_id": "1382_A. Common Subsequence",
"skipped": null
},
{
"grad_norm": 1.0364067554473877,
"index": 56,
"loss": 0.8011530637741089,
"nll": 0.8011530637741089,
"problem_id": "248_A. Cupboards",
"skipped": null
},
{
"grad_norm": 0.8864939212799072,
"index": 57,
"loss": 0.5355629324913025,
"nll": 0.5355629324913025,
"problem_id": "268_A. Games",
"skipped": null
},
{
"grad_norm": 0.47688010334968567,
"index": 58,
"loss": 0.509407639503479,
"nll": 0.509407639503479,
"problem_id": "618_B. Guess the Permutation",
"skipped": null
},
{
"grad_norm": 1.1146557331085205,
"index": 59,
"loss": 0.9011950492858887,
"nll": 0.9011950492858887,
"problem_id": "1181_A. Chunga-Changa",
"skipped": null
},
{
"grad_norm": 0.5178598761558533,
"index": 60,
"loss": 0.2951464056968689,
"nll": 0.2951464056968689,
"problem_id": "1421_B. Putting Bricks in the Wall",
"skipped": null
},
{
"grad_norm": 0.9971789121627808,
"index": 61,
"loss": 0.5864426493644714,
"nll": 0.5864426493644714,
"problem_id": "554_C. Kyoya and Colored Balls",
"skipped": null
},
{
"grad_norm": 0.699429452419281,
"index": 62,
"loss": 0.4205191731452942,
"nll": 0.4205191731452942,
"problem_id": "1203_A. Circle of Students",
"skipped": null
},
{
"grad_norm": 0.7046372294425964,
"index": 63,
"loss": 0.8773186206817627,
"nll": 0.8773186206817627,
"problem_id": "1427_B. Chess Cheater",
"skipped": null
},
{
"grad_norm": 1.2316917181015015,
"index": 64,
"loss": 0.7916744351387024,
"nll": 0.7916744351387024,
"problem_id": "670_B. Game of Robots",
"skipped": null
},
{
"grad_norm": 0.7533033490180969,
"index": 65,
"loss": 0.5084953308105469,
"nll": 0.5084953308105469,
"problem_id": "1000_A. Codehorses T-shirts",
"skipped": null
},
{
"grad_norm": 0.4042784869670868,
"index": 66,
"loss": 0.3695657253265381,
"nll": 0.3695657253265381,
"problem_id": "1264_A. Beautiful Regional Contest",
"skipped": null
},
{
"grad_norm": 1.0112096071243286,
"index": 67,
"loss": 0.8849809169769287,
"nll": 0.8849809169769287,
"problem_id": "538_B. Quasi Binary",
"skipped": null
},
{
"grad_norm": 0.7484896183013916,
"index": 68,
"loss": 0.5725398659706116,
"nll": 0.5725398659706116,
"problem_id": "868_A. Bark to Unlock",
"skipped": null
},
{
"grad_norm": 0.4507679045200348,
"index": 69,
"loss": 0.40581223368644714,
"nll": 0.40581223368644714,
"problem_id": "99_B. Help Chef Gerasim",
"skipped": null
},
{
"grad_norm": 0.6510177850723267,
"index": 70,
"loss": 0.5508701205253601,
"nll": 0.5508701205253601,
"problem_id": "931_A. Friends Meeting",
"skipped": null
},
{
"grad_norm": 0.9200932383537292,
"index": 71,
"loss": 0.6416708827018738,
"nll": 0.6416708827018738,
"problem_id": "870_A. Search for Pretty Integers",
"skipped": null
},
{
"grad_norm": 0.8513489365577698,
"index": 72,
"loss": 1.0005862712860107,
"nll": 1.0005862712860107,
"problem_id": "1191_C. Tokitsukaze and Discard Items",
"skipped": null
},
{
"grad_norm": 0.4327949285507202,
"index": 73,
"loss": 0.5023372173309326,
"nll": 0.5023372173309326,
"problem_id": "505_B. Mr. Kitayuta's Colorful Graph",
"skipped": null
},
{
"grad_norm": 0.6228141784667969,
"index": 74,
"loss": 0.482375830411911,
"nll": 0.482375830411911,
"problem_id": "365_A. Good Number",
"skipped": null
},
{
"grad_norm": 0.9539189338684082,
"index": 75,
"loss": 0.5643600225448608,
"nll": 0.5643600225448608,
"problem_id": "513_A. Game",
"skipped": null
},
{
"grad_norm": 0.8559263944625854,
"index": 76,
"loss": 1.1494770050048828,
"nll": 1.1494770050048828,
"problem_id": "1159_A. A pile of stones",
"skipped": null
},
{
"grad_norm": 0.9862070083618164,
"index": 77,
"loss": 1.1389682292938232,
"nll": 1.1389682292938232,
"problem_id": "1398_A. Bad Triangle",
"skipped": null
},
{
"grad_norm": 1.9144845008850098,
"index": 78,
"loss": 0.7307466268539429,
"nll": 0.7307466268539429,
"problem_id": "483_C. Diverse Permutation",
"skipped": null
},
{
"grad_norm": 1.0251142978668213,
"index": 79,
"loss": 0.7849968075752258,
"nll": 0.7849968075752258,
"problem_id": "673_A. Bear and Game",
"skipped": null
},
{
"grad_norm": 1.3851652145385742,
"index": 80,
"loss": 0.9004217386245728,
"nll": 0.9004217386245728,
"problem_id": "604_A. Uncowed Forces",
"skipped": null
},
{
"grad_norm": 0.8206305503845215,
"index": 81,
"loss": 0.513279914855957,
"nll": 0.513279914855957,
"problem_id": "1059_A. Cashier",
"skipped": null
},
{
"grad_norm": 0.4920770823955536,
"index": 82,
"loss": 0.2822416424751282,
"nll": 0.2822416424751282,
"problem_id": "835_A. Key races",
"skipped": null
},
{
"grad_norm": 0.789677083492279,
"index": 83,
"loss": 0.6259894967079163,
"nll": 0.6259894967079163,
"problem_id": "1490_D. Permutation Transformation",
"skipped": null
},
{
"grad_norm": 0.9807916879653931,
"index": 84,
"loss": 0.98895663022995,
"nll": 0.98895663022995,
"problem_id": "443_B. Kolya and Tandem Repeat",
"skipped": null
},
{
"grad_norm": 0.6448408365249634,
"index": 85,
"loss": 0.4732319116592407,
"nll": 0.4732319116592407,
"problem_id": "580_B. Kefa and Company",
"skipped": null
},
{
"grad_norm": 0.6379855871200562,
"index": 86,
"loss": 0.470840722322464,
"nll": 0.470840722322464,
"problem_id": "143_A. Help Vasilisa the Wise 2",
"skipped": null
},
{
"grad_norm": 0.816592276096344,
"index": 87,
"loss": 0.8404387831687927,
"nll": 0.8404387831687927,
"problem_id": "1011_A. Stages",
"skipped": null
},
{
"grad_norm": 0.6529431343078613,
"index": 88,
"loss": 0.6879411935806274,
"nll": 0.6879411935806274,
"problem_id": "1269_B. Modulo Equality",
"skipped": null
},
{
"grad_norm": 0.6517459154129028,
"index": 89,
"loss": 0.5100618004798889,
"nll": 0.5100618004798889,
"problem_id": "1543_B. Customising the Track",
"skipped": null
},
{
"grad_norm": 0.5063552260398865,
"index": 90,
"loss": 0.49379613995552063,
"nll": 0.49379613995552063,
"problem_id": "177_C1. Party",
"skipped": null
},
{
"grad_norm": 1.0049214363098145,
"index": 91,
"loss": 0.4494905471801758,
"nll": 0.4494905471801758,
"problem_id": "379_A. New Year Candles",
"skipped": null
},
{
"grad_norm": 0.5677995681762695,
"index": 92,
"loss": 0.5990645885467529,
"nll": 0.5990645885467529,
"problem_id": "1474_B. Different Divisors",
"skipped": null
},
{
"grad_norm": 0.9032794237136841,
"index": 93,
"loss": 0.5278613567352295,
"nll": 0.5278613567352295,
"problem_id": "155_A. I_love_%username%",
"skipped": null
},
{
"grad_norm": 0.44249868392944336,
"index": 94,
"loss": 0.372690349817276,
"nll": 0.372690349817276,
"problem_id": "673_C. Bear and Colors",
"skipped": null
},
{
"grad_norm": 0.7681690454483032,
"index": 95,
"loss": 0.7501130104064941,
"nll": 0.7501130104064941,
"problem_id": "381_A. Sereja and Dima",
"skipped": null
},
{
"grad_norm": 1.172059416770935,
"index": 96,
"loss": 0.8275896310806274,
"nll": 0.8275896310806274,
"problem_id": "39_H. Multiplication Table",
"skipped": null
}
]

2010
eval/training_stage_log.json Normal file
View File

File diff suppressed because it is too large Load Diff

413
eval_lib.py Normal file
View File

@@ -0,0 +1,413 @@
"""Dataset, judging, and sandbox helpers for the PKPO coding run.
All generated code is executed inside Modal containers. This module still applies
per-process time and memory limits and uses a guarded Python runner that blocks
network-oriented imports.
"""
from __future__ import annotations
import base64
import json
import os
import pickle
import random
import re
import resource
import subprocess
import sys
import tempfile
import textwrap
import time
import zlib
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Any
from agent_core import coding_task_message, extract_code
@dataclass
class Problem:
source: str
problem_id: str
title: str
statement: str
public_tests: list[dict[str, str]]
hidden_tests: list[dict[str, str]]
starter_code: str = ""
difficulty: str = ""
contest_date: str = ""
def first_user_message(self, max_turns: int = 1) -> str:
return coding_task_message(
f"{self.title}\n\n{self.statement}",
starter_code=self.starter_code,
max_turns=max_turns,
)
def to_public_dict(self) -> dict[str, Any]:
d = asdict(self)
d["hidden_tests"] = [{"input": t.get("input", "")[:80], "output_len": len(t.get("output", ""))}
for t in self.hidden_tests[:3]]
d["statement"] = self.statement[:500]
return d
def _json_or_encoded_tests(value: Any) -> list[dict[str, str]]:
if value is None or value == "":
return []
if isinstance(value, list):
return value
if not isinstance(value, str):
return []
s = value.strip()
if not s:
return []
try:
obj = json.loads(s)
return obj if isinstance(obj, list) else []
except Exception:
pass
raw = zlib.decompress(base64.b64decode(s))
try:
obj = pickle.loads(raw)
except Exception:
obj = raw.decode("utf-8")
if isinstance(obj, str):
obj = json.loads(obj)
return obj if isinstance(obj, list) else []
def _stdin_tests(tests: list[dict[str, Any]], limit: int | None = None) -> list[dict[str, str]]:
out = []
for t in tests:
if t.get("testtype", "stdin") != "stdin":
continue
if "input" not in t or "output" not in t:
continue
out.append({"input": str(t["input"]), "output": str(t["output"]), "testtype": "stdin"})
if limit and len(out) >= limit:
break
return out
def load_lcb_v6_subset(limit: int = 12, seed: int = 7341) -> list[Problem]:
"""Latest code_generation_lite release file is v6/test6.jsonl."""
from datasets import load_dataset
ds = load_dataset("livecodebench/code_generation_lite", "v6", split="test", trust_remote_code=True)
candidates: list[Problem] = []
for row in ds:
if str(row.get("starter_code") or "").strip():
continue
public = _stdin_tests(_json_or_encoded_tests(row.get("public_test_cases")))
hidden = _stdin_tests(_json_or_encoded_tests(row.get("private_test_cases")))
if not public or not hidden:
continue
candidates.append(Problem(
source="livecodebench/code_generation_lite:v6",
problem_id=str(row["question_id"]),
title=str(row.get("question_title") or row["question_id"]),
statement=str(row.get("question_content") or ""),
public_tests=public,
hidden_tests=hidden,
starter_code=str(row.get("starter_code") or ""),
difficulty=str(row.get("difficulty") or ""),
contest_date=str(row.get("contest_date") or ""),
))
candidates.sort(key=lambda p: (p.contest_date, p.problem_id), reverse=True)
rng = random.Random(seed)
latest_pool = candidates[: max(limit * 4, limit)]
rng.shuffle(latest_pool)
return latest_pool[:limit]
def _tests_from_io_dict(value: Any, limit: int | None = None) -> list[dict[str, str]]:
if not isinstance(value, dict):
return []
ins = value.get("input") or []
outs = value.get("output") or []
if len(ins) != len(outs):
return []
out = []
for i, o in zip(ins, outs):
out.append({"input": str(i), "output": str(o), "testtype": "stdin"})
if limit and len(out) >= limit:
break
return out
def load_codecontest_train(
limit: int = 12,
seed: int = 20260709,
exclude_problem_ids: set[str] | None = None,
max_rows: int | None = None,
min_cf_rating: int = 800,
max_cf_rating: int = 1300,
) -> list[Problem]:
"""Sample valid old CodeContests train problems without an early-prefix bias.
``streaming=True`` yields a stable dataset prefix; shuffling only after
prematurely stopping at ``limit`` is not a sample. Reservoir sampling lets a
bounded scan supply a deterministic, broader curriculum while keeping SFT and
RL problem ids disjoint.
A Qwen3-8B base policy earns ~0 reward on unrated/hard Codeforces problems,
which starves PKPO of nonzero groups, so the RL pool is restricted to
problems with a KNOWN cf_rating inside [min_cf_rating, max_cf_rating].
"""
from datasets import load_dataset
stream = load_dataset("deepmind/code_contests", split="train", streaming=True)
excluded = exclude_problem_ids or set()
rng = random.Random(seed)
rows: list[Problem] = []
seen = 0
max_rows = max_rows or max(3000, limit * 45)
for row_idx, row in enumerate(stream, start=1):
if row_idx > max_rows:
break
rating = int(row.get("cf_rating") or 0)
if rating < min_cf_rating or rating > max_cf_rating:
continue
desc = str(row.get("description") or "")
if not desc or len(desc) > 4500:
continue
low = desc.lower()
if "interactive" in low or "output-only" in low:
continue
if str(row.get("input_file") or "").strip() or str(row.get("output_file") or "").strip():
continue
public = _tests_from_io_dict(row.get("public_tests"), limit=3)
generated = _tests_from_io_dict(row.get("generated_tests"), limit=18)
private = _tests_from_io_dict(row.get("private_tests"), limit=18)
hidden = generated or private
problem_id = str(row.get("name") or f"codecontest-{row_idx}")
if problem_id in excluded or not public or len(hidden) < 6:
continue
problem = Problem(
source="deepmind/code_contests:train",
problem_id=problem_id,
title=str(row.get("name") or "CodeContests problem"),
statement=desc,
public_tests=public,
hidden_tests=hidden,
difficulty=str(row.get("difficulty") or ""),
)
seen += 1
if len(rows) < limit:
rows.append(problem)
else:
replace_idx = rng.randrange(seen)
if replace_idx < limit:
rows[replace_idx] = problem
rng.shuffle(rows)
return rows
def python3_verified_solutions(
limit: int = 8,
seed: int = 20260710,
candidate_multiplier: int = 2,
max_rows: int | None = None,
min_cf_rating: int = 800,
max_cf_rating: int = 1500,
) -> list[tuple[Problem, str]]:
"""Return runnable CodeContests Python-3 reference solutions only.
``solutions.language`` is a ClassLabel integer in CodeContests: 1 is Python
(Python 2) while 3 is Python 3. The earlier loader ignored that paired field,
so it could SFT on ``raw_input``/bare-``print`` programs that fail our Python 3
judge. Each candidate here is also compiled and run on held-out generated
tests before it is admitted to the warm-up set.
"""
from datasets import load_dataset
stream = load_dataset("deepmind/code_contests", split="train", streaming=True)
pairs: list[tuple[Problem, str]] = []
wanted = max(limit, limit * max(1, candidate_multiplier))
max_rows = max_rows or min(6000, max(500, wanted * 40))
for row_idx, row in enumerate(stream, start=1):
if row_idx > max_rows:
break
rating = int(row.get("cf_rating") or 0)
if rating < min_cf_rating or rating > max_cf_rating:
continue
desc = str(row.get("description") or "")
if not desc or len(desc) > 4500:
continue
low = desc.lower()
if "interactive" in low or "output-only" in low:
continue
if str(row.get("input_file") or "").strip() or str(row.get("output_file") or "").strip():
continue
sols = row.get("solutions") or {}
languages = sols.get("language") or []
solutions = sols.get("solution") or []
public = _tests_from_io_dict(row.get("public_tests"), limit=2)
generated = _tests_from_io_dict(row.get("generated_tests"), limit=3)
private = _tests_from_io_dict(row.get("private_tests"), limit=3)
held_out = generated or private
if not public or not held_out:
continue
for language, raw_solution in zip(languages, solutions):
# CodeContests' ClassLabel id 3 is PYTHON3. Do not accept PYTHON (id
# 1), even if it happens to compile under the local interpreter.
if language != 3:
continue
solution = textwrap.dedent(str(raw_solution)).strip()
if not 40 <= len(solution) <= 7000:
continue
try:
compile(solution, "<codecontests-python3>", "exec")
except (SyntaxError, ValueError, TypeError):
continue
verdict = judge_code(solution, held_out, timeout_s=3, memory_mb=768, max_tests=3)
if not verdict.get("passed"):
continue
prob = Problem(
source="deepmind/code_contests:train",
problem_id=str(row.get("name") or f"sft-{len(pairs)}"),
title=str(row.get("name") or "CodeContests problem"),
statement=desc,
public_tests=public,
hidden_tests=held_out,
difficulty=str(row.get("difficulty") or ""),
)
pairs.append((prob, solution))
break
if len(pairs) >= wanted:
break
rng = random.Random(seed)
rng.shuffle(pairs)
return pairs[:limit]
GUARD = r'''
import builtins
import sys
blocked = {
"_socket", "socket", "ssl", "urllib", "http", "ftplib", "requests",
"subprocess", "multiprocessing", "ctypes",
}
real_import = builtins.__import__
def guarded_import(name, globals=None, locals=None, fromlist=(), level=0):
root = name.split(".", 1)[0]
if root in blocked:
raise ImportError(f"blocked import: {name}")
return real_import(name, globals, locals, fromlist, level)
builtins.__import__ = guarded_import
path = sys.argv[1]
with open(path, "r", encoding="utf-8", errors="replace") as f:
src = f.read()
ns = {"__name__": "__main__", "__file__": path}
exec(compile(src, path, "exec"), ns, ns)
'''
def _limit_child(timeout_s: int, memory_mb: int, cwd: str):
def inner():
os.chdir(cwd)
cpu = max(1, int(timeout_s) + 1)
for res, lim in [
(resource.RLIMIT_CPU, (cpu, cpu + 1)),
(resource.RLIMIT_AS, (memory_mb * 1024 * 1024, memory_mb * 1024 * 1024)),
(resource.RLIMIT_FSIZE, (32 * 1024 * 1024, 32 * 1024 * 1024)),
(resource.RLIMIT_NOFILE, (64, 64)),
]:
try:
resource.setrlimit(res, lim)
except Exception:
pass
return inner
def normalize_output(text: str) -> str:
text = text.replace("\r\n", "\n").replace("\r", "\n")
return "\n".join(line.rstrip() for line in text.strip().split("\n")).strip()
def judge_code(code: str, tests: list[dict[str, str]], timeout_s: int = 3,
memory_mb: int = 768, max_tests: int | None = None) -> dict[str, Any]:
tests = _stdin_tests(tests, limit=max_tests)
started = time.time()
if not code.strip():
return {
"passed": False,
"passed_tests": 0,
"total_tests": len(tests),
"error": "empty code",
"seconds": time.time() - started,
}
with tempfile.TemporaryDirectory(prefix="pkpo_exec_") as td:
sol = Path(td) / "solution.py"
runner = Path(td) / "guarded_runner.py"
sol.write_text(code, encoding="utf-8")
runner.write_text(GUARD, encoding="utf-8")
for idx, test in enumerate(tests):
try:
cp = subprocess.run(
[sys.executable, "-I", str(runner), str(sol)],
input=test["input"],
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
timeout=timeout_s + 1,
preexec_fn=_limit_child(timeout_s, memory_mb, td),
)
except subprocess.TimeoutExpired:
return {
"passed": False,
"passed_tests": idx,
"total_tests": len(tests),
"error": "timeout",
"seconds": time.time() - started,
}
if cp.returncode != 0:
return {
"passed": False,
"passed_tests": idx,
"total_tests": len(tests),
"error": "runtime_error",
"exit_code": cp.returncode,
"stderr": cp.stderr[-500:],
"seconds": time.time() - started,
}
got = normalize_output(cp.stdout)
want = normalize_output(test["output"])
if got != want:
return {
"passed": False,
"passed_tests": idx,
"total_tests": len(tests),
"error": "wrong_answer",
"got": got[:300],
"want": want[:300],
"seconds": time.time() - started,
}
return {"passed": True, "passed_tests": len(tests), "total_tests": len(tests), "seconds": time.time() - started}
def judge_final_answer(final_answer: str, tests: list[dict[str, str]], **kwargs) -> dict[str, Any]:
return judge_code(extract_code(final_answer), tests, **kwargs)
def compact_json_dump(path: str | Path, obj: Any):
Path(path).parent.mkdir(parents=True, exist_ok=True)
Path(path).write_text(json.dumps(obj, indent=2, sort_keys=True) + "\n", encoding="utf-8")
def short_completion_for_sft(solution: str) -> str:
solution = textwrap.dedent(solution).strip()
return (
"I will provide a direct Python solution.\n</think>\n"
"<answer>Tool type: final\nTool query: ```python\n"
+ solution
+ "\n```</answer>"
)

13
generation_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"_from_model_config": true,
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": 151643,
"max_new_tokens": 1024,
"pad_token_id": 151643,
"stop_strings": [
"</answer>"
],
"top_p": 0.95,
"transformers_version": "4.52.4"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f501648102b630bbde3dbb82e798314285b1ab40c82ef56af8c8c93fb44c8820
size 3996250744

3
model-00002-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:64e7a9c5de9192eed35320d4366e886b801050e7afdd96546f13417236a0f0f9
size 3993160032

3
model-00003-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:41905ceaa6f8af415ce3e5aa5f1e30987ae76c464482f73645b077d642d12b4d
size 3959604768

3
model-00004-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:366cfc55ebfb47f604184d5236f679ba458cb81cf02e6fa342a33d6801a78360
size 3187841392

3
model-00005-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fbf24915d47ea030bb68ab0b9488f4515a907185baa6dc26837c9c3f2326a550
size 1244659840

406
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,406 @@
{
"metadata": {
"total_size": 16381470720
},
"weight_map": {
"lm_head.weight": "model-00005-of-00005.safetensors",
"model.embed_tokens.weight": "model-00001-of-00005.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.10.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.10.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.11.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.12.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.13.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.13.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.14.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.14.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.14.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.15.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.15.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.16.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.16.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.17.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.18.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.18.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.18.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.19.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.19.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.19.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.20.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.20.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.20.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.21.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.21.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.21.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.22.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.22.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.22.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.23.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.23.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.23.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.24.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.24.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.25.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.25.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.25.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.26.input_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
"model.layers.26.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.26.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.27.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
"model.layers.28.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.28.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.28.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.28.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.28.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.28.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.28.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.28.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.28.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.28.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.28.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.29.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.29.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.29.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.29.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.29.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.29.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.29.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.29.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.29.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.29.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.29.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.30.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.30.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.30.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.30.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.30.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.30.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.30.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.30.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.30.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.30.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.30.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.31.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.31.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.31.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.31.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.31.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.31.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.31.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.31.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.31.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.31.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.31.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.32.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.32.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.32.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.32.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.32.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.32.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.32.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.32.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.32.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.32.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.32.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.33.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.33.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.33.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.33.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.33.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.33.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.33.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.33.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.33.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.33.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.33.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.34.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.34.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.34.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.34.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.34.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.34.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.34.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.34.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.34.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.34.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.34.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.35.input_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.35.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.35.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.35.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.35.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
"model.layers.35.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.35.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.35.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.35.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
"model.layers.35.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.35.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.4.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.6.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.7.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.7.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.7.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
"model.layers.8.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.8.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.8.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.9.input_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
"model.layers.9.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.9.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
"model.norm.weight": "model-00004-of-00005.safetensors"
}
}

98
pkpo.py Normal file
View File

@@ -0,0 +1,98 @@
"""PKPO reward transformations — Listing 1 of the paper (arXiv:2505.15201), verbatim.
sloo_minus_one is the s^(loo-1) estimator of Eq. (33)/(34): transformed rewards whose
sum estimates pass@k / max_g@k, with a k-1-subset LOO baseline. Requires n >= k+1.
At k=1 the paper uses untransformed rewards with mean centering (its k-1 baseline is
undefined); transform_rewards() below handles that case.
"""
from typing import Callable
import numpy as np
def _m_normed(N: int, K: int, i: int, j: int) -> float:
if i == j and i >= K - 1:
return (
K / (N - K + 1)
* np.prod(np.arange(i - K + 2, i + 1) / np.arange(N - K + 2, N + 1))
)
elif j > i and j >= K - 1 and K >= 2:
return (
K / (N - K + 1) * (K - 1) / N
* np.prod(np.arange(j - K + 2, j) / np.arange(N - K + 2, N))
)
return 0
def _m_diagonal(N: int, K: int) -> np.ndarray:
return np.array([_m_normed(N, K, i, i) for i in range(N)])
def rho(g: np.ndarray, K: int) -> float:
"""See Equation (12)."""
return (np.sort(g) * _m_diagonal(len(g), K)).sum()
def _delta(N: int, K: int, i: int) -> float:
return _m_normed(N, K, i, i + 1) - _m_normed(N, K, i + 1, i + 1)
def _deltas(N: int, K: int) -> np.ndarray:
return np.array([_delta(N - 1, K, i) for i in range(N - 2)])
def _sorted_apply(func: Callable) -> Callable:
def inner(x: np.ndarray, *args, **kwargs) -> np.ndarray:
i_sort = np.argsort(x)
func_x = np.zeros_like(x)
func_x[i_sort] = func(x[i_sort], *args, **kwargs)
return func_x
return inner
@_sorted_apply
def s(g: np.ndarray, K: int):
"""See Equation (19)."""
N = len(g)
c = g * _m_diagonal(N, K)
c[:(N - 1)] += g[1:] * _deltas(N + 1, K)
return np.cumsum(c[::-1])[::-1]
@_sorted_apply
def _b(g: np.ndarray, K: int) -> np.ndarray:
N = len(g)
w = (_m_diagonal(N - 1, K) * np.arange(1, N)).astype(float)
w[1:] += _deltas(N, K) * np.arange(1, N - 1)
c1 = np.array([(w * g[1:]).sum()])
c2 = (g[:-1] - g[1:]) * w
return np.cumsum(np.concatenate((c1, c2)))
def sloo(g: np.ndarray, K: int) -> np.ndarray:
"""See Equation (29)."""
return s(g, K) - _b(g, K) / (len(g) - 1)
def sloo_minus_one(g: np.ndarray, K: int) -> np.ndarray:
"""See Equation (33)."""
return s(g, K) - _b(g, K - 1) * K / (K - 1) / len(g)
def transform_rewards(g: np.ndarray, K: int) -> np.ndarray:
"""PKPO advantages for one group of n rollouts of the same prompt.
k >= 2: sloo_minus_one exactly as in Listing 1.
k == 1: s(g, 1) (= g/n, no transformation) with group-mean centering, which is
what the paper uses for its k_opt=1 runs ("without which the training diverges").
Scaled by n so advantage magnitude is O(reward) at every k (constant across
stages; equivalent to a learning-rate rescale).
"""
g = np.asarray(g, dtype=np.float64)
n = len(g)
if K >= 2:
out = sloo_minus_one(g, K)
else:
out = s(g, 1)
out = out - out.mean()
return out * n

57
shipped_tool.py Normal file
View File

@@ -0,0 +1,57 @@
#!/usr/bin/env python3
"""Standalone agentic coding tool for OpenAI-compatible endpoints."""
from __future__ import annotations
import argparse
import os
import subprocess
import sys
from openai import OpenAI
from agent_core import coding_task_message, openai_generate_fn, run_episode
def run_bash(cmd: str, timeout_s: int = 30) -> str:
cp = subprocess.run(
cmd,
shell=True,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
timeout=timeout_s,
)
out = cp.stdout[-12000:]
return f"Exit code: {cp.returncode}\n{out}"
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL", "http://127.0.0.1:8000/v1"))
ap.add_argument("--model", default=os.environ.get("MODEL", "bk1dr/qwen3-8b-code-pkpo"))
ap.add_argument("--api-key", default=os.environ.get("OPENAI_API_KEY", "EMPTY"))
ap.add_argument("--max-turns", type=int, default=1,
help="One-turn deployment default used for training and evaluation.")
ap.add_argument("--cp", action="store_true", help="Wrap stdin as a competitive-programming task.")
ap.add_argument("--starter-code", default="")
args = ap.parse_args()
prompt = sys.stdin.read()
if args.cp:
first = coding_task_message(prompt, starter_code=args.starter_code, max_turns=args.max_turns)
else:
first = prompt
client = OpenAI(base_url=args.base_url, api_key=args.api_key)
generate = openai_generate_fn(client, args.model)
ep = run_episode(generate, run_bash, first, max_turns=args.max_turns)
if ep.final_answer is not None:
print(ep.final_answer)
else:
print(f"Episode ended without final answer: {ep.end_reason}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())

31
special_tokens_map.json Normal file
View File

@@ -0,0 +1,31 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

BIN
tokenizer.json (Stored with Git LFS) Normal file
View File

Binary file not shown.

239
tokenizer_config.json Normal file
View File

@@ -0,0 +1,239 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151665": {
"content": "<tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151666": {
"content": "</tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151667": {
"content": "<think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151668": {
"content": "</think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

791
train_pkpo_modal.py Normal file
View File

@@ -0,0 +1,791 @@
"""Modal PKPO pipeline: baseline eval, tiny warmup, PKPO LoRA, upload."""
from __future__ import annotations
import gc
import inspect
import json
import os
import shutil
import time
from pathlib import Path
import modal
from modal_common import BASE_MODEL, CACHE_DIR, HF_REPO, WORK_DIR, hf_cache_vol, image, work_vol
app = modal.App("pkpo-train")
def _copy_source_bundle(dst: Path):
import agent_core
import eval_lib
import pkpo
import shipped_tool
import train_pkpo_modal
for mod, name in [
(agent_core, "agent_core.py"),
(eval_lib, "eval_lib.py"),
(pkpo, "pkpo.py"),
(shipped_tool, "shipped_tool.py"),
(train_pkpo_modal, "train_pkpo_modal.py"),
]:
Path(dst / name).write_text(Path(inspect.getfile(mod)).read_text(encoding="utf-8"), encoding="utf-8")
def _load_tokenizer(model_ref: str):
from transformers import AutoTokenizer
from agent_core import CHAT_TEMPLATE
tok = AutoTokenizer.from_pretrained(model_ref, trust_remote_code=True)
tok.chat_template = CHAT_TEMPLATE
if tok.pad_token is None:
tok.pad_token = tok.eos_token
return tok
def _write_generation_config(model, tok, dst: Path):
from transformers import GenerationConfig
from agent_core import GEN_TEMPERATURE, GEN_TOP_P, MAX_TURN_TOKENS, STOP_STRING
gen = GenerationConfig.from_model_config(model.config)
gen.do_sample = True
gen.temperature = GEN_TEMPERATURE
gen.top_p = GEN_TOP_P
gen.max_new_tokens = MAX_TURN_TOKENS
gen.pad_token_id = tok.pad_token_id
gen.eos_token_id = tok.eos_token_id
gen.stop_strings = [STOP_STRING]
gen.save_pretrained(dst)
def _read_json(path: Path, default):
if path.exists():
return json.loads(path.read_text(encoding="utf-8"))
return default
def _model_card(baseline, final, schedule, notes: str) -> str:
b = baseline.get("pass_at_1", 0.0) if baseline else 0.0
f = final.get("pass_at_1", 0.0) if final else 0.0
return f"""---
license: other
base_model: Qwen/Qwen3-8B-Base
tags:
- code
- reinforcement-learning
- pkpo
- livecodebench
---
# Qwen3-8B Code PKPO
This repo contains a Qwen/Qwen3-8B-Base derivative trained for a small agentic
coding experiment using the shared tool path in `agent_core.py` and
`shipped_tool.py`.
## Method
- Base: `Qwen/Qwen3-8B-Base`.
- Prompt/template: custom `<think>...</think><answer>...</answer>` template saved
in the tokenizer. The generation prompt ends with `Assistant: <think>`.
- Tool protocol: no system role; instructions are merged into the first user
message; strict user/assistant alternation; plain-text `Tool type` and
`Tool query` calls.
- Training data: `deepmind/code_contests` train split only, filtered to old
stdin/stdout problems. The LiveCodeBench eval subset is not used for training.
- Reward: binary hidden-test pass/fail.
- PKPO: `sloo_minus_one` from the paper for `k >= 2`; centered `k=1` rewards for
the first and final stages. No GRPO-style reward normalization is applied.
- Schedule actually run: `{schedule}`.
The run was intentionally small to fit the free-credit budget and deadline.
Results should be treated as a reproducible experiment, not a leaderboard model.
## Results
Evaluation uses `livecodebench/code_generation_lite` `v6`, a fixed subset saved
at `eval/eval_subset.json`, temperature 1.0, and the same one-turn tool path used
for training.
| model | pass@1 estimate |
|---|---:|
| base before training | {b:.4f} |
| final merged model | {f:.4f} |
Raw files:
- `eval/baseline_results.json`
- `eval/final_results.json`
- `eval/eval_subset.json`
## Usage
Serve with vLLM:
```bash
vllm serve bk1dr/qwen3-8b-code-pkpo --trust-remote-code --max-model-len 8192
```
Run the shipped tool:
```bash
python shipped_tool.py --base-url http://127.0.0.1:8000/v1 --model bk1dr/qwen3-8b-code-pkpo --max-turns 1 --cp < problem.txt
```
## Run Notes
{notes}
"""
@app.function(
image=image,
volumes={CACHE_DIR: hf_cache_vol, WORK_DIR: work_vol},
secrets=[modal.Secret.from_name("hf-secret")],
timeout=1800,
cpu=4,
memory=16384,
)
def push_scaffold():
from huggingface_hub import HfApi, snapshot_download
from transformers import AutoConfig
from agent_core import CHAT_TEMPLATE
base_path = snapshot_download(BASE_MODEL, cache_dir=CACHE_DIR)
out = Path(WORK_DIR) / "scaffold_repo"
if out.exists():
shutil.rmtree(out)
out.mkdir(parents=True)
tok = _load_tokenizer(base_path)
tok.save_pretrained(out)
cfg = AutoConfig.from_pretrained(base_path, trust_remote_code=True)
cfg.save_pretrained(out)
(out / "chat_template.jinja").write_text(CHAT_TEMPLATE, encoding="utf-8")
_copy_source_bundle(out)
(out / "README.md").write_text(_model_card({}, {}, "pending", "Scaffold pushed before GPU training."), encoding="utf-8")
HfApi().upload_folder(repo_id=HF_REPO, repo_type="model", folder_path=str(out), commit_message="Initial scaffold")
work_vol.commit()
return {"uploaded": str(out)}
@app.function(
image=image,
gpu="H100",
volumes={CACHE_DIR: hf_cache_vol},
timeout=15 * 60,
cpu=4,
memory=98304,
)
def smoke_batched_generation(max_new_tokens: int = 96):
"""Low-cost preflight for the exact batched stop-string generation API."""
import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from agent_core import CHAT_TEMPLATE, STOP_STRING, coding_task_message, render_messages
base_path = snapshot_download(BASE_MODEL, cache_dir=CACHE_DIR)
tok = AutoTokenizer.from_pretrained(base_path, trust_remote_code=True)
tok.chat_template = CHAT_TEMPLATE
if tok.pad_token is None:
tok.pad_token = tok.eos_token
tok.padding_side = "left"
model = AutoModelForCausalLM.from_pretrained(
base_path,
torch_dtype=torch.bfloat16,
device_map={"": 0},
trust_remote_code=True,
low_cpu_mem_usage=True,
)
prompt = render_messages([{
"role": "user",
"content": coding_task_message(
"Read an integer N and print N plus one.", max_turns=1
),
}], add_generation_prompt=True)
inputs = tok([prompt, prompt], return_tensors="pt", padding=True).to(model.device)
prompt_width = inputs.input_ids.shape[1]
with torch.no_grad():
out = model.generate(
**inputs,
do_sample=True,
temperature=1.0,
top_p=0.95,
max_new_tokens=max_new_tokens,
pad_token_id=tok.pad_token_id,
eos_token_id=tok.eos_token_id,
stop_strings=[STOP_STRING],
tokenizer=tok,
)
completions = [tok.decode(row[prompt_width:], skip_special_tokens=False) for row in out]
result = {
"batch_size": len(completions),
"completion_lengths": [len(c) for c in completions],
"stopped": [STOP_STRING in c for c in completions],
"samples": [c[:500] for c in completions],
}
print(json.dumps(result), flush=True)
return result
@app.function(
image=image,
gpu="H100",
volumes={CACHE_DIR: hf_cache_vol, WORK_DIR: work_vol},
secrets=[modal.Secret.from_name("hf-secret")],
timeout=125 * 60,
cpu=10,
memory=98304,
)
def train_eval_upload(
eval_limit: int = 6,
eval_samples: int = 6,
train_groups_per_stage: int = 4,
rollouts: int = 12,
max_new_tokens: int = 1024,
sft_examples: int = 96,
finish_by_epoch: float = 0.0,
):
"""Train a checkpointed one-turn agentic coding LoRA with PKPO.
A one-turn episode is intentional here: it uses the same ``Episode`` parser
and final-action contract as the shipped tool while making the short budget
practical. Every generated completion is durably written to a private text
trace and JSONL before it is judged.
"""
import hashlib
import numpy as np
import torch
from huggingface_hub import HfApi, snapshot_download
from peft import LoraConfig, get_peft_model
from torch.nn.utils import clip_grad_norm_
from transformers import AutoModelForCausalLM, AutoTokenizer
from agent_core import (CHAT_TEMPLATE, STOP_STRING, THINK_PREFIX, Episode,
extract_code, parse_action, render_messages)
from eval_lib import (compact_json_dump, judge_final_answer, load_codecontest_train,
load_lcb_v6_subset, python3_verified_solutions,
short_completion_for_sft)
from pkpo import transform_rewards
torch.backends.cuda.matmul.allow_tf32 = True
started = time.time()
# Wall-clock guard: the final eval + merge + save + upload need a fixed
# reserve that SFT/RL must never eat into. finish_by_epoch is when the
# merged upload must be DONE (epoch seconds).
finish_by = finish_by_epoch if finish_by_epoch > started else started + 100 * 60
FINAL_RESERVE_S = 30 * 60
rl_stop_at = finish_by - FINAL_RESERVE_S
print("time_guard config", {"now": int(started), "finish_by": int(finish_by),
"rl_stop_at": int(rl_stop_at)}, flush=True)
run_id = time.strftime("pkpo_%Y%m%dT%H%M%SZ", time.gmtime())
run_dir = Path(WORK_DIR) / "results" / run_id
checkpoint_root = Path(WORK_DIR) / "checkpoints" / run_id
run_dir.mkdir(parents=True, exist_ok=False)
checkpoint_root.mkdir(parents=True, exist_ok=True)
def save_json(name: str, value):
compact_json_dump(run_dir / name, value)
def safe_verdict(verdict: dict) -> dict:
"""Keep useful diagnostics without saving hidden test values."""
out = {
"passed": bool(verdict.get("passed", False)),
"error": verdict.get("error", "ok"),
"passed_tests": int(verdict.get("passed_tests", 0)),
"total_tests": int(verdict.get("total_tests", 0)),
}
for key in ("seconds", "exit_code"):
if key in verdict:
out[key] = verdict[key]
if verdict.get("stderr"):
out["stderr_tail"] = str(verdict["stderr"])[-500:]
return out
class RolloutTrace:
"""Private structured and plain-text traces, flushed per sampled output."""
def __init__(self, root: Path):
self.jsonl = (root / "rollouts.jsonl").open("a", encoding="utf-8")
self.text = (root / "raw_rollouts.txt").open("a", encoding="utf-8")
def _write(self, record: dict):
self.jsonl.write(json.dumps(record, ensure_ascii=False, sort_keys=True) + "\n")
self.jsonl.flush()
def rollout(self, record: dict):
self._write(record)
header = {
key: record.get(key)
for key in ("run_id", "phase", "stage", "k", "group", "candidate_attempt",
"sample", "problem_id", "tool_type", "stop_seen")
}
self.text.write("\n===== ROLLOUT =====\n")
self.text.write(json.dumps(header, sort_keys=True) + "\n")
self.text.write("--- completion_raw ---\n")
self.text.write(record.get("completion_raw", "") + "\n")
self.text.write("--- final_answer ---\n")
self.text.write(record.get("final_answer", "") + "\n")
self.text.write("--- verdict ---\n")
self.text.write(json.dumps(record.get("verdict", {}), ensure_ascii=False, sort_keys=True) + "\n")
self.text.write("===== END ROLLOUT =====\n")
self.text.flush()
def summary(self, record: dict):
self._write(record)
self.text.write("\n===== GROUP SUMMARY =====\n")
self.text.write(json.dumps(record, ensure_ascii=False, sort_keys=True) + "\n")
self.text.write("===== END GROUP SUMMARY =====\n")
self.text.flush()
def close(self):
self.jsonl.close()
self.text.close()
trace = RolloutTrace(run_dir)
base_path = snapshot_download(BASE_MODEL, cache_dir=CACHE_DIR)
tok = AutoTokenizer.from_pretrained(base_path, trust_remote_code=True)
tok.chat_template = CHAT_TEMPLATE
if tok.pad_token is None:
tok.pad_token = tok.eos_token
tok.padding_side = "left"
def load_model():
model = AutoModelForCausalLM.from_pretrained(
base_path,
torch_dtype=torch.bfloat16,
device_map={"": 0},
trust_remote_code=True,
low_cpu_mem_usage=True,
)
model.config.use_cache = False
model.gradient_checkpointing_enable()
lora = LoraConfig(
r=32,
lora_alpha=64,
lora_dropout=0.02,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora)
model.enable_input_require_grads()
model.print_trainable_parameters()
return model
model = load_model()
def generate_many(first_user_msg: str, count: int) -> list[str]:
"""Sample a group together; each row has an independent stop string."""
if count <= 0:
return []
prompt = render_messages([{"role": "user", "content": first_user_msg}], add_generation_prompt=True)
inputs = tok([prompt] * count, return_tensors="pt", padding=True).to(model.device)
input_width = inputs.input_ids.shape[1]
was_training = model.training
old_cache = getattr(model.config, "use_cache", False)
model.eval()
model.config.use_cache = True
try:
with torch.no_grad():
out = model.generate(
**inputs,
do_sample=True,
temperature=1.0,
top_p=0.95,
max_new_tokens=max_new_tokens,
pad_token_id=tok.pad_token_id,
eos_token_id=tok.eos_token_id,
use_cache=True,
stop_strings=[STOP_STRING],
tokenizer=tok,
)
finally:
model.config.use_cache = old_cache
if was_training:
model.train()
return [tok.decode(row[input_width:], skip_special_tokens=False) for row in out]
eval_subset = load_lcb_v6_subset(limit=eval_limit, seed=7341)
save_json("eval_subset.json", [p.to_public_dict() for p in eval_subset])
def sample_episodes(problem, count: int, phase: str, stage=None, k=None, group=None,
candidate_attempt=None, max_tests: int = 18):
"""Drive the exact one-turn ``Episode`` state machine for every sample."""
first_user_msg = problem.first_user_message(max_turns=1)
prompt_hash = hashlib.sha256(
render_messages([{"role": "user", "content": first_user_msg}], add_generation_prompt=True).encode()
).hexdigest()
completions = generate_many(first_user_msg, count)
samples = []
for sample_idx, completion in enumerate(completions):
episode = Episode(first_user_msg, max_turns=1)
kind, _ = episode.step(completion)
tool_type, _, parse_error = parse_action(completion)
stop_seen = STOP_STRING in completion
valid_final = bool(stop_seen and kind == "final" and episode.final_answer is not None)
if valid_final:
final_answer = episode.final_answer
extracted = extract_code(final_answer)
verdict = judge_final_answer(
final_answer, problem.hidden_tests, timeout_s=3, memory_mb=768, max_tests=max_tests
)
else:
final_answer = episode.final_answer or ""
extracted = ""
reason = parse_error or episode.end_reason or "missing_final_action"
verdict = {
"passed": False,
"passed_tests": 0,
"total_tests": min(max_tests, len(problem.hidden_tests)),
"error": "format_error: " + str(reason),
}
safe = safe_verdict(verdict)
record = {
"schema_version": 1,
"record_type": "rollout",
"run_id": run_id,
"timestamp_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"phase": phase,
"stage": stage,
"k": k,
"group": group,
"candidate_attempt": candidate_attempt,
"sample": sample_idx,
"problem_id": problem.problem_id,
"source": problem.source,
"sampling": {"temperature": 1.0, "top_p": 0.95, "max_new_tokens": max_new_tokens},
"prompt_sha256": prompt_hash,
"prompt_chars": len(first_user_msg),
"completion_raw": completion,
"assistant_content": THINK_PREFIX + completion,
"stop_seen": stop_seen,
"tool_type": tool_type,
"parse_error": parse_error,
"final_answer": final_answer,
"extracted_code": extracted,
"verdict": safe,
}
trace.rollout(record)
samples.append({
"completion": completion,
"valid_final": valid_final,
"verdict": verdict,
"safe_verdict": safe,
"code": extracted,
"tool_type": tool_type,
})
print(phase, problem.problem_id, sample_idx + 1,
"pass" if verdict.get("passed") else verdict.get("error", "fail"), flush=True)
return samples
def evaluate(tag: str) -> dict:
details = []
pass_count = 0
total = 0
for p in eval_subset:
sample_results = []
samples = sample_episodes(p, eval_samples, phase=tag, max_tests=80)
for sample in samples:
verdict = sample["verdict"]
ok = bool(verdict["passed"])
pass_count += int(ok)
total += 1
sample_results.append({
"passed": ok,
"end": sample["safe_verdict"].get("error", "ok"),
"completion_chars": len(sample["completion"]),
"code_chars": len(sample["code"]),
"stop_seen": STOP_STRING in sample["completion"],
"tool_type": sample["tool_type"],
})
details.append({"problem_id": p.problem_id, "difficulty": p.difficulty, "samples": sample_results})
work_vol.commit()
res = {
"tag": tag,
"pass_at_1": pass_count / max(total, 1),
"passed_samples": pass_count,
"total_samples": total,
"eval_limit": eval_limit,
"samples_per_problem": eval_samples,
"temperature": 1.0,
"top_p": 0.95,
"max_new_tokens": max_new_tokens,
"tool_turn_limit": 1,
"details": details,
}
save_json(f"{tag}_results.json", res)
work_vol.commit()
print(tag, "pass@1", res["pass_at_1"], flush=True)
return res
def checkpoint(tag: str, state: dict):
"""Persist the small LoRA adapter after every recoverable unit of work."""
path = checkpoint_root / tag
if path.exists():
shutil.rmtree(path)
path.mkdir(parents=True)
model.save_pretrained(path)
tok.save_pretrained(path)
compact_json_dump(path / "state.json", state)
compact_json_dump(run_dir / "latest_state.json", {"checkpoint": str(path), **state})
work_vol.commit()
return str(path)
def train_completion(first_user_msg: str, completion: str, weight: float):
"""Backprop only a complete assistant final turn; never truncate silently."""
if abs(weight) < 1e-9:
return None, 0.0, "zero_weight"
stop_idx = completion.find(STOP_STRING)
if stop_idx == -1:
return None, 0.0, "missing_stop"
prompt = render_messages([{"role": "user", "content": first_user_msg}], add_generation_prompt=True)
prompt_ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids[0]
# Batched generate right-pads finished rows with eos/pad until the longest
# row stops; training on that tail drowns the real signal. The target is
# exactly the turn content through </answer> plus ONE terminating EOS.
target = completion[: stop_idx + len(STOP_STRING)]
if tok.eos_token:
target += tok.eos_token
full = tok(prompt + target, return_tensors="pt", add_special_tokens=False)
if full.input_ids.shape[1] > 8192:
return None, 0.0, "overlength"
input_ids = full.input_ids.to(model.device)
labels = input_ids.clone()
labels[:, : min(prompt_ids.numel(), labels.shape[1])] = -100
if (labels != -100).sum() == 0:
return None, 0.0, "empty_target"
out = model(input_ids=input_ids, labels=labels)
nll = float(out.loss.detach().cpu())
weighted_loss = out.loss * float(weight)
weighted_loss.backward()
return nll, float(weighted_loss.detach().cpu()), None
baseline = evaluate("baseline")
opt = torch.optim.AdamW((p for p in model.parameters() if p.requires_grad), lr=4e-5, betas=(0.9, 0.95))
print("verified Python-3 format warmup", flush=True)
model.train()
sft_log = []
sft_pairs = python3_verified_solutions(limit=sft_examples)
sft_time_floor = rl_stop_at - 25 * 60 # keep at least ~25 min of RL window
for sft_idx, (prob, sol) in enumerate(sft_pairs, start=1):
if time.time() > sft_time_floor:
print("time_guard: stopping SFT early at", sft_idx - 1, "examples", flush=True)
break
opt.zero_grad(set_to_none=True)
nll, loss, skipped = train_completion(
prob.first_user_message(max_turns=1), short_completion_for_sft(sol), 1.0
)
entry = {"index": sft_idx, "problem_id": prob.problem_id, "nll": nll, "loss": loss,
"skipped": skipped}
if skipped is None:
entry["grad_norm"] = float(clip_grad_norm_(model.parameters(), 1.0))
opt.step()
sft_log.append(entry)
print("sft", prob.problem_id, entry, flush=True)
if sft_idx % 24 == 0:
save_json("sft_log.json", sft_log)
checkpoint(f"sft_{sft_idx:04d}", {"phase": "sft", "completed_examples": sft_idx})
save_json("sft_log.json", sft_log)
sft_checkpoint = checkpoint("after_sft", {
"phase": "sft_complete",
"completed_examples": len(sft_log),
"verified_examples": len(sft_pairs),
})
print("loading disjoint RL curriculum", flush=True)
schedule = [1, 8, 1]
sft_ids = {p.problem_id for p, _ in sft_pairs}
train_problems = load_codecontest_train(
limit=max(72, train_groups_per_stage * len(schedule) * 8),
seed=20260709,
exclude_problem_ids=sft_ids,
)
print("loaded RL candidates", len(train_problems), flush=True)
stage_log = []
candidate_cursor = 0
# Split the remaining RL window across stages so the mandatory final k=1
# stage always gets its share even when earlier stages run long.
rl_start = time.time()
rl_window = max(rl_stop_at - rl_start, 0.0)
stage_shares = [0.30, 0.40, 0.30]
assert len(stage_shares) == len(schedule)
stage_deadline = [rl_start + rl_window * sum(stage_shares[: i + 1]) for i in range(len(schedule))]
print("rl window minutes", round(rl_window / 60, 1), flush=True)
for stage_idx, k in enumerate(schedule):
opt.param_groups[0]["lr"] = 2e-5 if k == 1 else 1e-5
for group_idx in range(train_groups_per_stage):
if time.time() > stage_deadline[stage_idx]:
print("time_guard: closing stage", stage_idx, "k", k, "after", group_idx, "groups", flush=True)
stage_log.append({"stage": stage_idx, "k": k, "group": group_idx,
"skipped": "stage_time_guard"})
save_json("training_stage_log.json", stage_log)
break
selected = None
attempts = []
for attempt_idx in range(1, 6):
if candidate_cursor >= len(train_problems) or time.time() > stage_deadline[stage_idx]:
break
p = train_problems[candidate_cursor]
candidate_cursor += 1
samples = sample_episodes(
p, rollouts, phase="rl_candidate", stage=stage_idx, k=k, group=group_idx,
candidate_attempt=attempt_idx,
)
rewards = [1.0 if sample["verdict"].get("passed") else 0.0 for sample in samples]
advantages = transform_rewards(np.array(rewards, dtype=np.float64), k)
eligible = bool(np.any(np.abs(advantages) > 1e-10))
attempt = {
"problem_id": p.problem_id,
"passes": int(sum(rewards)),
"rewards": rewards,
"eligible": eligible,
"advantages": [float(x) for x in advantages],
}
attempts.append(attempt)
trace.summary({
"schema_version": 1,
"record_type": "candidate_summary",
"run_id": run_id,
"stage": stage_idx,
"k": k,
"group": group_idx,
"candidate_attempt": attempt_idx,
**attempt,
})
if eligible:
selected = (p, samples, rewards, advantages)
break
if selected is None:
entry = {
"stage": stage_idx,
"k": k,
"group": group_idx,
"skipped": "no_pkpo_eligible_group",
"candidate_attempts": attempts,
}
stage_log.append(entry)
save_json("training_stage_log.json", stage_log)
checkpoint(f"stage{stage_idx}_group{group_idx}_skipped", {
"phase": "rl", "stage": stage_idx, "k": k, "group": group_idx,
"status": "skipped", "candidate_cursor": candidate_cursor,
})
print("stage skipped", entry, flush=True)
continue
p, samples, rewards, adv = selected
opt.zero_grad(set_to_none=True)
losses = []
nlls = []
skipped_samples = []
first_user_msg = p.first_user_message(max_turns=1)
for sample_idx, (sample, a) in enumerate(zip(samples, adv)):
if not sample["valid_final"]:
skipped_samples.append({"sample": sample_idx, "reason": "invalid_final"})
losses.append(0.0)
nlls.append(None)
continue
nll, loss, skipped = train_completion(first_user_msg, sample["completion"], float(a) / rollouts)
nlls.append(nll)
losses.append(loss)
if skipped not in (None, "zero_weight"):
skipped_samples.append({"sample": sample_idx, "reason": skipped})
if any(nll is not None for nll in nlls):
grad_norm = float(clip_grad_norm_(model.parameters(), 1.0))
opt.step()
else:
grad_norm = 0.0
entry = {
"stage": stage_idx,
"k": k,
"group": group_idx,
"problem_id": p.problem_id,
"rewards": rewards,
"advantages": [float(x) for x in adv],
"losses": losses,
"nlls": nlls,
"skipped_samples": skipped_samples,
"candidate_attempts": attempts,
"grad_norm": grad_norm,
}
stage_log.append(entry)
save_json("training_stage_log.json", stage_log)
ckpt = checkpoint(f"stage{stage_idx}_group{group_idx}", {
"phase": "rl", "stage": stage_idx, "k": k, "group": group_idx,
"problem_id": p.problem_id, "candidate_cursor": candidate_cursor,
"after_sft_checkpoint": sft_checkpoint,
})
entry["checkpoint"] = ckpt
save_json("training_stage_log.json", stage_log)
trace.summary({
"schema_version": 1,
"record_type": "training_group_summary",
"run_id": run_id,
**entry,
})
print("stage", entry, flush=True)
final = evaluate("final")
final_dir = Path(WORK_DIR) / "final_model"
if final_dir.exists():
shutil.rmtree(final_dir)
final_dir.mkdir(parents=True)
model.eval()
merged = model.merge_and_unload()
merged.config.use_cache = True
merged.save_pretrained(final_dir, safe_serialization=True, max_shard_size="4GB")
tok.save_pretrained(final_dir)
_write_generation_config(merged, tok, final_dir)
_copy_source_bundle(final_dir)
eval_dir = final_dir / "eval"
eval_dir.mkdir()
for name in ["baseline_results.json", "final_results.json", "eval_subset.json", "training_stage_log.json", "sft_log.json"]:
src = run_dir / name
if src.exists():
shutil.copyfile(src, eval_dir / name)
notes = (
f"Modal H100 pipeline elapsed {(time.time() - started) / 60:.1f} minutes. "
f"Run id {run_id}; eval subset size {eval_limit}, samples/problem {eval_samples}, "
f"rollouts/group {rollouts}; verified Python-3 SFT examples {len(sft_pairs)}. "
"Raw rollout traces remain in the private Modal work volume and are not published."
)
(final_dir / "README.md").write_text(_model_card(baseline, final, schedule, notes), encoding="utf-8")
HfApi().upload_folder(repo_id=HF_REPO, repo_type="model", folder_path=str(final_dir), commit_message="Upload merged PKPO run")
trace.close()
work_vol.commit()
return {
"run_id": run_id,
"baseline": baseline["pass_at_1"],
"final": final["pass_at_1"],
"model_dir": str(final_dir),
"trace_text": str(run_dir / "raw_rollouts.txt"),
}
@app.local_entrypoint()
def main(
eval_limit: int = 6,
eval_samples: int = 6,
train_groups_per_stage: int = 4,
rollouts: int = 12,
max_new_tokens: int = 1024,
sft_examples: int = 96,
finish_by_epoch: float = 0.0,
):
print("Spawning scaffold push (runs in parallel on CPU)...")
scaffold_call = push_scaffold.spawn()
print("Running GPU pipeline...")
print(train_eval_upload.remote(
eval_limit,
eval_samples,
train_groups_per_stage,
rollouts,
max_new_tokens,
sft_examples,
finish_by_epoch,
))
print("scaffold:", scaffold_call.get())

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long