初始化项目，由ModelHub XC社区提供模型
Model: laion/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified-70-8B Source: Original Platform
This commit is contained in:
ModelHub XC
41
.gitattributes
vendored
Normal file
41
.gitattributes
vendored
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/20260604_123153_reward_vs_steps.png filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/20260604_123153_turn_count_distribution.png filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_567548.out filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_567550.out filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_567551.out filter=lfs diff=lfs merge=lfs -text
|
||||||
18
README.md
Normal file
18
README.md
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
# a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified-70-8B
|
||||||
|
|
||||||
|
## Training Traces
|
||||||
|
|
||||||
|
Training-time Daytona/Harbor rollouts for this run are uploaded as
|
||||||
|
a companion dataset:
|
||||||
|
**[penfever/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified](https://huggingface.co/datasets/penfever/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified)**
|
||||||
|
|
||||||
|
The dataset contains the `last` episode of each trial (per
|
||||||
|
`make_and_upload_trace_dataset --episodes last`) — the same rollouts
|
||||||
|
the policy was trained on after rollback / truncation.
|
||||||
|
|
||||||
|
## Provenance
|
||||||
|
|
||||||
|
- Base (SFT) model: `laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink`
|
||||||
|
- RL dataset: `DCAgent/selfinstruct-naive-sandboxes-2-verified`
|
||||||
|
- Checkpoint: `global_step_70` (EMA-best over steps 1-80, 5-period EMA α=1/3)
|
||||||
|
- Training: SkyRL GRPO, 56 GPU / 14 nodes, fan-out K=4; chain 567548→567550→567551 (terminated early at step ~81 by user request)
|
||||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"</think>": 151668,
|
||||||
|
"</tool_call>": 151658,
|
||||||
|
"</tool_response>": 151666,
|
||||||
|
"<think>": 151667,
|
||||||
|
"<tool_call>": 151657,
|
||||||
|
"<tool_response>": 151665,
|
||||||
|
"<|box_end|>": 151649,
|
||||||
|
"<|box_start|>": 151648,
|
||||||
|
"<|endoftext|>": 151643,
|
||||||
|
"<|file_sep|>": 151664,
|
||||||
|
"<|fim_middle|>": 151660,
|
||||||
|
"<|fim_pad|>": 151662,
|
||||||
|
"<|fim_prefix|>": 151659,
|
||||||
|
"<|fim_suffix|>": 151661,
|
||||||
|
"<|im_end|>": 151645,
|
||||||
|
"<|im_start|>": 151644,
|
||||||
|
"<|image_pad|>": 151655,
|
||||||
|
"<|object_ref_end|>": 151647,
|
||||||
|
"<|object_ref_start|>": 151646,
|
||||||
|
"<|quad_end|>": 151651,
|
||||||
|
"<|quad_start|>": 151650,
|
||||||
|
"<|repo_name|>": 151663,
|
||||||
|
"<|video_pad|>": 151656,
|
||||||
|
"<|vision_end|>": 151653,
|
||||||
|
"<|vision_pad|>": 151654,
|
||||||
|
"<|vision_start|>": 151652
|
||||||
|
}
|
||||||
89
chat_template.jinja
Normal file
89
chat_template.jinja
Normal file
@@ -0,0 +1,89 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- messages[0].content + '\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||||
|
{%- for message in messages[::-1] %}
|
||||||
|
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||||
|
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
||||||
|
{%- set ns.multi_step_tool = false %}
|
||||||
|
{%- set ns.last_query_index = index %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if message.content is string %}
|
||||||
|
{%- set content = message.content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- set content = '' %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{%- set reasoning_content = '' %}
|
||||||
|
{%- if message.reasoning_content is string %}
|
||||||
|
{%- set reasoning_content = message.reasoning_content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- if '</think>' in content %}
|
||||||
|
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
||||||
|
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if loop.index0 > ns.last_query_index %}
|
||||||
|
{%- if loop.last or (not loop.last and reasoning_content) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if message.tool_calls %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if (loop.first and content) or (not loop.first) %}
|
||||||
|
{{- '\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if tool_call.function %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{%- if tool_call.arguments is string %}
|
||||||
|
{{- tool_call.arguments }}
|
||||||
|
{%- else %}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||||
|
{{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||||
|
{{- '<think>\n\n</think>\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
68
config.json
Normal file
68
config.json
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen3ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151645,
|
||||||
|
"head_dim": 128,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 12288,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 40960,
|
||||||
|
"max_window_layers": 36,
|
||||||
|
"model_type": "qwen3",
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_hidden_layers": 36,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 1000000,
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"transformers_version": "4.57.6",
|
||||||
|
"use_cache": false,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 151936
|
||||||
|
}
|
||||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
{
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151645,
|
||||||
|
151643
|
||||||
|
],
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"temperature": 0.6,
|
||||||
|
"top_k": 20,
|
||||||
|
"top_p": 0.95,
|
||||||
|
"transformers_version": "4.57.6"
|
||||||
|
}
|
||||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:1102de19a59a72c177af55f2e68d63f603d1468d35cb8c668b88ac9e614874c2
|
||||||
|
size 4902257696
|
||||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:954c746e5f48a76a7fe633ec908e5837dd07cb629e70b0fc3eeb1cee12f46478
|
||||||
|
size 4915960368
|
||||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:fbfe63aac1f237f4fb43cfb70d976121b5c7eb803a6eff6161e9b8b1a84246bd
|
||||||
|
size 4983068496
|
||||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:a1dd4c39285f9a033a836e303b8664ebd7e755c58f1cda3506e9d54b410c5339
|
||||||
|
size 1580230264
|
||||||
407
model.safetensors.index.json
Normal file
407
model.safetensors.index.json
Normal file
@@ -0,0 +1,407 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"total_parameters": 8190735360,
|
||||||
|
"total_size": 16381470720
|
||||||
|
},
|
||||||
|
"weight_map": {
|
||||||
|
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.norm.weight": "model-00004-of-00004.safetensors"
|
||||||
|
}
|
||||||
|
}
|
||||||
154
rl_config.json
Normal file
154
rl_config.json
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
{
|
||||||
|
"job_name": "a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified",
|
||||||
|
"experiments_dir": "/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified",
|
||||||
|
"cluster_name": "jupiter",
|
||||||
|
"skyrl_entrypoint": "examples.terminal_bench.entrypoints.main_tbench",
|
||||||
|
"skyrl_hydra_args": [
|
||||||
|
"+terminal_bench_config=terminal_bench",
|
||||||
|
"trainer.strategy=fsdp2",
|
||||||
|
"trainer.algorithm.advantage_estimator=rloo_n",
|
||||||
|
"trainer.algorithm.use_kl_loss=false",
|
||||||
|
"trainer.algorithm.kl_loss_coef=0.0",
|
||||||
|
"trainer.algorithm.eps_clip_low=0.2",
|
||||||
|
"trainer.algorithm.eps_clip_high=0.05",
|
||||||
|
"trainer.algorithm.loss_reduction=token_mean",
|
||||||
|
"trainer.epochs=2",
|
||||||
|
"trainer.max_steps=80",
|
||||||
|
"trainer.update_epochs_per_batch=1",
|
||||||
|
"trainer.train_batch_size=64",
|
||||||
|
"trainer.policy_mini_batch_size=64",
|
||||||
|
"trainer.eval_batch_size=64",
|
||||||
|
"trainer.micro_forward_batch_size_per_gpu=4",
|
||||||
|
"trainer.micro_train_batch_size_per_gpu=1",
|
||||||
|
"trainer.max_prompt_length=999999",
|
||||||
|
"trainer.eval_interval=999999",
|
||||||
|
"trainer.eval_before_train=false",
|
||||||
|
"trainer.ckpt_interval=2",
|
||||||
|
"trainer.resume_mode=latest",
|
||||||
|
"trainer.hf_save_interval=5",
|
||||||
|
"++trainer.hf_hub_repo_id=laion/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified",
|
||||||
|
"++trainer.hf_hub_private=false",
|
||||||
|
"++trainer.hf_hub_revision=main",
|
||||||
|
"++trainer.enable_db_registration=false",
|
||||||
|
"trainer.project_name=OpenThoughts-Agent",
|
||||||
|
"trainer.log_level=INFO",
|
||||||
|
"trainer.tracker_commit_each_step=true",
|
||||||
|
"trainer.logger=console",
|
||||||
|
"trainer.run_name=a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified",
|
||||||
|
"trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/checkpoints",
|
||||||
|
"trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/exports",
|
||||||
|
"trainer.policy.optimizer_config.lr=8e-6",
|
||||||
|
"trainer.policy.optimizer_config.weight_decay=0.0",
|
||||||
|
"trainer.policy.optimizer_config.adam_betas=[0.9,0.999]",
|
||||||
|
"trainer.policy.optimizer_config.max_grad_norm=0.9",
|
||||||
|
"trainer.policy.fsdp_config.cpu_offload=false",
|
||||||
|
"trainer.policy.fsdp_config.reshard_after_forward=true",
|
||||||
|
"trainer.policy.fsdp_config.fsdp_size=4",
|
||||||
|
"trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
|
||||||
|
"trainer.ref.fsdp_config.cpu_offload=false",
|
||||||
|
"trainer.ref.fsdp_config.reshard_after_forward=true",
|
||||||
|
"trainer.ref.fsdp_config.fsdp_size=4",
|
||||||
|
"trainer.placement.colocate_all=false",
|
||||||
|
"trainer.placement.policy_num_nodes=2",
|
||||||
|
"trainer.placement.ref_num_nodes=2",
|
||||||
|
"trainer.placement.policy_num_gpus_per_node=4",
|
||||||
|
"trainer.placement.ref_num_gpus_per_node=4",
|
||||||
|
"trainer.fully_async.max_staleness_steps=16",
|
||||||
|
"trainer.fully_async.num_parallel_generation_workers=338",
|
||||||
|
"generator.backend=vllm",
|
||||||
|
"generator.timeout_multiplier=1.0",
|
||||||
|
"generator.model_dtype=bfloat16",
|
||||||
|
"generator.inference_engine_tensor_parallel_size=1",
|
||||||
|
"generator.num_inference_engines=48",
|
||||||
|
"generator.n_samples_per_prompt=8",
|
||||||
|
"generator.eval_n_samples_per_prompt=8",
|
||||||
|
"generator.gpu_memory_utilization=0.75",
|
||||||
|
"generator.max_num_seqs=24",
|
||||||
|
"generator.max_num_batched_tokens=65536",
|
||||||
|
"generator.enable_prefix_caching=true",
|
||||||
|
"generator.enable_chunked_prefill=true",
|
||||||
|
"generator.run_engines_locally=true",
|
||||||
|
"generator.weight_sync_backend=nccl",
|
||||||
|
"generator.async_engine=true",
|
||||||
|
"generator.batched=false",
|
||||||
|
"generator.enable_http_endpoint=true",
|
||||||
|
"generator.enable_ray_prometheus_stats=false",
|
||||||
|
"generator.vllm_stats_interval=1",
|
||||||
|
"generator.append_eos_token_after_stop_str_in_multi_turn=true",
|
||||||
|
"generator.max_turns=999999",
|
||||||
|
"generator.sampling_params.max_generate_length=4096",
|
||||||
|
"generator.sampling_params.temperature=0.7",
|
||||||
|
"generator.sampling_params.top_p=0.95",
|
||||||
|
"generator.sampling_params.top_k=20",
|
||||||
|
"++generator.engine_init_kwargs.max_model_len=32768",
|
||||||
|
"++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2",
|
||||||
|
"++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
|
||||||
|
"data.train_data=[\"/e/scratch/jureap59/feuer1/tasks/selfinstruct-naive-sandboxes-2-verified\"]",
|
||||||
|
"data.val_data=[]",
|
||||||
|
"+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/trace_jobs",
|
||||||
|
"+terminal_bench_config.harbor.name=terminus-2",
|
||||||
|
"+terminal_bench_config.harbor.max_episodes=999999",
|
||||||
|
"+terminal_bench_config.harbor.enable_summarize=false",
|
||||||
|
"+terminal_bench_config.harbor.store_all_messages=true",
|
||||||
|
"+terminal_bench_config.harbor.trajectory_config.raw_content=true",
|
||||||
|
"+terminal_bench_config.harbor.enable_episode_logging=false",
|
||||||
|
"+terminal_bench_config.harbor.record_terminal_session=false",
|
||||||
|
"+terminal_bench_config.harbor.enable_pane_logging=false",
|
||||||
|
"+terminal_bench_config.harbor.strict_json_parser=true",
|
||||||
|
"+terminal_bench_config.harbor.interleaved_thinking=true",
|
||||||
|
"+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true",
|
||||||
|
"+terminal_bench_config.harbor.override_timeout_sec=900",
|
||||||
|
"+terminal_bench_config.harbor.override_cpus=1",
|
||||||
|
"+terminal_bench_config.harbor.override_memory_mb=2048",
|
||||||
|
"+terminal_bench_config.harbor.override_storage_mb=2048",
|
||||||
|
"+terminal_bench_config.harbor.auto_snapshot=true",
|
||||||
|
"+terminal_bench_config.harbor.verifier_override_timeout_sec=120",
|
||||||
|
"+terminal_bench_config.harbor.max_retries=3",
|
||||||
|
"+terminal_bench_config.harbor.min_wait_sec=60.0",
|
||||||
|
"+terminal_bench_config.harbor.max_wait_sec=600.0",
|
||||||
|
"+terminal_bench_config.harbor.wait_multiplier=2.0",
|
||||||
|
"+terminal_bench_config.harbor.exclude_exceptions=[\"VerifierTimeoutError\",\"VerifierRuntimeError\",\"RewardFileNotFoundError\",\"RewardFileEmptyError\",\"VerifierOutputParseError\"]",
|
||||||
|
"+terminal_bench_config.harbor.n_concurrent_trials=675",
|
||||||
|
"+terminal_bench_config.harbor.log_level=INFO",
|
||||||
|
"+terminal_bench_config.harbor.enable_reward_shaping=false",
|
||||||
|
"+terminal_bench_config.harbor.enable_error_classification=true",
|
||||||
|
"+terminal_bench_config.harbor.mask_exceptions=[\"DaytonaError\",\"EnvironmentStartTimeoutError\",\"NetworkError\",\"ConnectionError\",\"RewardFileNotFoundError\",\"RewardFileEmptyError\",\"AgentEnvironmentTimeoutError\",\"ContextLengthExceededError\"]",
|
||||||
|
"+terminal_bench_config.harbor.default_error_treatment=zero",
|
||||||
|
"+terminal_bench_config.harbor.passthrough_exceptions=[\"AgentTimeoutError\"]",
|
||||||
|
"+terminal_bench_config.harbor.zero_exceptions=[]",
|
||||||
|
"+terminal_bench_config.model_info.max_input_tokens=32000",
|
||||||
|
"+terminal_bench_config.model_info.max_output_tokens=4096",
|
||||||
|
"+terminal_bench_config.archiving.enabled=false",
|
||||||
|
"+terminal_bench_config.trace_upload.enabled=true",
|
||||||
|
"+terminal_bench_config.trace_upload.repo_org=DCAgent",
|
||||||
|
"+terminal_bench_config.trace_upload.episodes=last",
|
||||||
|
"+terminal_bench_config.trace_upload.dataset_type=SFT",
|
||||||
|
"+terminal_bench_config.trace_upload.cleanup=true"
|
||||||
|
],
|
||||||
|
"model_path": "/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
|
||||||
|
"train_data": [
|
||||||
|
"/e/scratch/jureap59/feuer1/tasks/selfinstruct-naive-sandboxes-2-verified"
|
||||||
|
],
|
||||||
|
"val_data": [],
|
||||||
|
"num_nodes": 14,
|
||||||
|
"gpus_per_node": 4,
|
||||||
|
"cpus_per_node": 288,
|
||||||
|
"tensor_parallel_size": 1,
|
||||||
|
"ray_port": 6379,
|
||||||
|
"master_port": 12345,
|
||||||
|
"checkpoints_dir": null,
|
||||||
|
"export_path": "/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/exports",
|
||||||
|
"needs_ssh_tunnel": true,
|
||||||
|
"needs_cuda_detection": false,
|
||||||
|
"pinggy_persistent_url": null,
|
||||||
|
"pinggy_token": null,
|
||||||
|
"agent_name": "terminus-2",
|
||||||
|
"harbor_env": "daytona",
|
||||||
|
"proxychains_binary": null,
|
||||||
|
"ray_object_store_gb": 40.0,
|
||||||
|
"trace_upload_enabled": true,
|
||||||
|
"trace_upload_repo_org": "DCAgent",
|
||||||
|
"trace_upload_episodes": "last",
|
||||||
|
"trace_upload_dataset_type": "SFT",
|
||||||
|
"trace_upload_cleanup": true
|
||||||
|
}
|
||||||
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
240
tokenizer_config.json
Normal file
240
tokenizer_config.json
Normal file
@@ -0,0 +1,240 @@
|
|||||||
|
{
|
||||||
|
"add_bos_token": false,
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151643": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151644": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151645": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151646": {
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151647": {
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151648": {
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151649": {
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151650": {
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151651": {
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151652": {
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151653": {
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151654": {
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151655": {
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151656": {
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151657": {
|
||||||
|
"content": "<tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151658": {
|
||||||
|
"content": "</tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151659": {
|
||||||
|
"content": "<|fim_prefix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151660": {
|
||||||
|
"content": "<|fim_middle|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151661": {
|
||||||
|
"content": "<|fim_suffix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151662": {
|
||||||
|
"content": "<|fim_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151663": {
|
||||||
|
"content": "<|repo_name|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151664": {
|
||||||
|
"content": "<|file_sep|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151665": {
|
||||||
|
"content": "<tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151666": {
|
||||||
|
"content": "</tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151667": {
|
||||||
|
"content": "<think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151668": {
|
||||||
|
"content": "</think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|im_end|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"model_max_length": 32768,
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"padding_side": "left",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null
|
||||||
|
}
|
||||||
10
training_logs/20260604_123153_metrics_job_567548.csv
Normal file
10
training_logs/20260604_123153_metrics_job_567548.csv
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RewardFileNotFoundError,batch_errors/total_RewardFileNotFoundError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,timing/save_hf_model,batch_errors/avg_VerifierOutputParseError,batch_errors/total_VerifierOutputParseError,batch_errors/avg_DaytonaNotFoundError,batch_errors/total_DaytonaNotFoundError
|
||||||
|
0.0,0,64,512,0,0.0,0,0.0,3631.0469,3316.3012,3953.2569,25303,1,3243.0733,0.5059,-0.0087,0.0861,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.154,-0.0,0.0,1.0,0.0,0.0146,0.625,0.5059,10.7936,50.364,633.8085,26.1,857.9676,224.1591,0.1114,3.6373,28.9658,151.5911,180.8148,1358.0963,18.6186,151.7373,1155.0253,0,1,121,968,939,6,29,0.30578512396694213,37,,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,1,0.7969,0,0.7969,4822.3672,4358.4062,5286.3281,25740,1,3816.7518,0.5,-0.0131,0.1439,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1522,0.0,0.0,1.0,0.0,0.0203,0.6719,0.5,11.4775,50.4393,624.6882,27.2,857.9676,233.2794,0.0519,4.284,32.9178,189.316,222.548,833.5217,19.8811,189.578,586.8065,0,2,58,464,439,5,25,0.43103448275862066,25,38.5904,48.4066,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,2,1.3438,0,0.875,6234.4707,5933.786,6452.138,29898,1,5603.3019,0.4199,0.0118,0.1815,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1483,0.0,0.0,1.0,0.0,0.019,0.5938,0.4199,13.2484,50.647,630.5422,26.5,857.9676,227.4254,0.0614,4.581,59.1585,286.7886,346.2501,703.0723,19.4633,287.0298,332.7757,0,3,62,496,461,8,28,0.2903225806451613,18,,,0.11290322580645161,7.0,0.16129032258064516,10.0,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,3,1.7031,0,0.8438,6754.2012,6667.6059,6828.2464,30918,1,6199.201,0.4609,-0.0051,0.1719,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1546,0.0,0.0,1.0,0.0,0.0208,0.6406,0.4609,13.3757,50.8519,626.7498,26.9,857.9676,231.2178,0.0647,4.8313,51.8113,299.0531,351.1909,923.9643,19.7377,299.3145,548.2019,0,4,54,432,310,24,117,1.4259259259259258,77,0.0067,11.238,0.018518518518518517,1.0,1.0555555555555556,57.0,0.018518518518518517,1.0,0.07407407407407407,4.0,0.018518518518518517,1.0,0.05555555555555555,3.0,,,,,
|
||||||
|
0.0,0,64,512,4,2.6406,1,1.0,6037.8184,7489.2791,5303.55,31319,1,7148.2511,0.3359,0.0173,0.1494,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1177,-0.0041,0.0,1.0,0.0,0.0141,0.4531,0.3359,13.5831,51.1094,624.3487,27.2,857.9676,233.6189,0.0632,4.8082,51.5733,303.1749,355.0612,609.8127,19.9157,303.4245,230.0243,0,5,58,464,389,20,75,0.3793103448275862,22,,,,,1.0,58.0,,,,,0.05172413793103448,3.0,,,7.5975,0.13793103448275862,8.0,,
|
||||||
|
0.0,0,64,512,5,3.0469,1,1.0,6446.5898,7161.3189,6042.2324,31463,1,6726.1259,0.3613,0.0237,0.1225,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1422,-0.0,0.0,1.0,0.0,0.0174,0.5,0.3613,14.5126,51.5042,634.199,26.1,857.9676,223.7686,0.0641,4.8218,47.3073,311.2799,358.8908,528.5257,19.4415,311.5191,145.3683,0,6,55,440,378,14,46,0.2909090909090909,16,0.0069,9.3271,,,0.43636363636363634,24.0,,,0.10909090909090909,6.0,0.03636363636363636,2.0,0.14545454545454545,8.0,,,,0.14545454545454545,8.0
|
||||||
|
0.0,0,64,512,6,2.8438,1,1.0,5989.127,6446.3918,5759.824,30379,1,5927.6954,0.334,0.0071,0.2061,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1469,-0.0028,0.0,1.0,0.0,0.0169,0.4844,0.334,14.6947,51.7031,630.6147,26.5,857.9676,227.3529,0.0656,4.6023,46.2763,268.8931,315.4526,589.5032,20.6408,269.1104,248.8046,0,7,53,424,336,15,87,1.3018867924528301,69,,,,,0.4339622641509434,23.0,0.018867924528301886,1.0,0.018867924528301886,1.0,0.03773584905660377,2.0,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.8125,0,0.9375,5809.7773,6972.8086,5007.5545,30602,1,6322.6044,0.4082,0.0127,0.1764,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1395,-0.0,0.0,1.0,0.0,0.0182,0.5312,0.4082,14.9265,51.8948,629.9464,26.6,857.9676,228.0212,0.0612,5.115,50.7488,301.5099,352.5549,605.9063,19.7309,301.7446,228.5022,0,8,60,480,417,18,63,0.3,18,0.0077,9.0045,,,0.6333333333333333,38.0,,,,,0.11666666666666667,7.0,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.9688,0,0.9688,5550.0059,5062.7014,5740.6902,27434,1,5458.1306,0.2812,0.0007,0.1828,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.147,0.0,0.0,1.0,0.0,0.0205,0.4844,0.2812,14.893,51.924,626.8531,26.9,857.9676,231.1145,0.0634,4.1708,43.4535,247.6858,291.4212,687.7597,19.8574,247.9039,372.3071,0,9,57,456,384,18,70,0.5789473684210527,33,,,,,0.6491228070175439,37.0,,,,,0.10526315789473684,6.0,0.12280701754385964,7.0,,,,,
|
||||||
|
43
training_logs/20260604_123153_metrics_job_567550.csv
Normal file
43
training_logs/20260604_123153_metrics_job_567550.csv
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_RewardFileNotFoundError,batch_errors/total_RewardFileNotFoundError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_DaytonaNotFoundError,batch_errors/total_DaytonaNotFoundError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_RewardFileEmptyError,batch_errors/total_RewardFileEmptyError,batch_errors/avg_AddTestsDirError,batch_errors/total_AddTestsDirError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_OSError,batch_errors/total_OSError
|
||||||
|
0.0,0,64,512,7,1.625,0,0.5625,5094.9453,5220.4171,5029.7893,26213,1,4943.8339,0.3418,-0.0026,0.1557,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.153,0.0,0.0,1.0,0.0,0.0177,0.5556,0.3418,10.6846,50.2918,613.6667,28.5,857.9676,244.3008,0.1008,4.0497,40.2665,237.4547,278.0636,3374.1597,19.9528,237.6955,3072.0934,0,9,63,504,448,14,56,0.2857142857142857,18,0.5555555555555556,35,0.09523809523809523,6.0,0.031746031746031744,2.0,,,,,,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,1,0.7188,0,0.7188,6813.5703,6589.7525,6954.7038,31364,1,6199.8594,0.3867,0.0081,0.2078,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1451,0.0,0.0,1.0,0.0,0.0235,0.5938,0.3867,12.6901,50.5284,634.8915,26.0,857.9676,223.076,0.0696,4.7781,51.6722,295.8816,347.9268,1168.0124,20.0205,296.1843,795.2757,0,10,63,504,379,32,125,1.5555555555555556,98,0.23809523809523808,15,0.09523809523809523,6.0,,,9.5233,17.5722,6.5298,0.09523809523809523,6.0,0.015873015873015872,1.0,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,2,1.7344,1,1.0,6021.1484,7335.5163,5460.9861,31521,1,6627.1211,0.2988,0.0105,0.1794,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0,0.0,1.0,0.0,0.0178,0.4531,0.2988,14.0168,50.7668,629.5027,26.6,857.9676,228.4648,0.0642,4.7993,49.8186,302.0916,352.2082,507.8331,19.3212,302.3244,131.5017,0,11,56,448,322,27,113,1.3214285714285714,74,0.42857142857142855,24,0.07142857142857142,4.0,,,,,,0.30357142857142855,17.0,0.017857142857142856,1.0,0.08928571428571429,5.0,0.14285714285714285,8.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,3,2.2344,1,1.0,5205.8301,5815.7826,4926.0513,30274,1,5856.7327,0.3145,0.0052,0.192,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1282,0.0,0.0,1.0,0.0,0.0171,0.4844,0.3145,14.0573,50.8443,626.9561,26.9,857.9676,231.0115,0.0618,4.469,43.7187,271.5448,315.6215,516.3754,19.2857,271.8407,176.9956,0,12,59,472,412,10,47,0.3389830508474576,20,0.423728813559322,25,0.01694915254237288,1.0,,,0.0067,8.1886,,0.06779661016949153,4.0,,,0.03389830508474576,2.0,0.1864406779661017,11.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,4,2.2656,0,0.9844,5512.916,4847.4852,5840.7813,29622,1,5117.2995,0.3301,-0.0116,0.1489,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1537,-0.0016,0.0,1.0,0.0,0.0186,0.4688,0.3301,14.0911,50.8562,633.3426,26.2,857.9676,224.625,0.0627,4.4482,41.2335,241.7355,283.3967,610.8919,20.3642,242.1,302.6795,0,13,63,504,429,14,75,0.49206349206349204,31,0.6349206349206349,40,0.015873015873015872,1.0,,,,,,,,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.5,0,0.9219,5537.5039,5509.1908,5547.2388,29312,1,5324.5857,0.2559,0.0061,0.1565,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1449,-0.0,0.0,1.0,0.0,0.0173,0.4375,0.2559,14.1559,51.0214,629.3403,26.6,857.9676,228.6273,0.0613,4.4394,47.8807,251.6916,299.9619,838.9599,20.6381,252.0195,513.9169,0,14,59,472,380,18,89,0.6440677966101694,38,0.7966101694915254,47,0.1016949152542373,6.0,,,0.0073,8.3961,,0.03389830508474576,2.0,,,0.01694915254237288,1.0,0.03389830508474576,2.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.8281,1,1.0,5926.9746,6772.3189,5448.7217,30621,1,6058.2297,0.3613,0.0123,0.1417,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1363,-0.0,0.0,1.0,0.0,0.0151,0.5,0.3613,14.5139,51.3657,625.6211,27.1,857.9676,232.3465,0.0635,4.7605,53.283,282.2832,335.9143,643.0671,20.3593,282.5675,282.0282,0,15,62,496,426,16,68,0.5645161290322581,35,0.532258064516129,33,0.016129032258064516,1.0,0.03225806451612903,2.0,,,6.5721,0.016129032258064516,1.0,,,,,0.0967741935483871,6.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.7812,0,0.9844,5693.7383,5905.8841,5516.5699,30109,1,5394.2192,0.4551,0.0032,0.1188,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1362,-0.0028,0.0,1.0,0.0,0.0153,0.5625,0.4551,14.5472,51.3824,631.8962,26.3,857.9676,226.0714,0.0636,4.962,41.7927,245.157,287.3077,563.5944,19.3884,245.451,251.9322,0,16,55,440,373,15,60,0.7090909090909091,39,0.8363636363636363,46,0.01818181818181818,1.0,,,0.0071,7.7944,,0.03636363636363636,2.0,,,,,,,0.12727272727272726,7.0,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,3.1094,0,0.9531,5991.3418,6421.2284,5792.3657,30607,1,6045.6308,0.3164,0.0151,0.1259,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1354,-0.002,0.0,1.0,0.0,0.0168,0.4688,0.3164,14.6639,51.4677,628.2541,26.8,857.9676,229.7135,0.0634,4.6239,46.0697,288.584,335.0408,640.3003,21.2728,288.9073,279.3589,0,17,64,512,447,13,64,0.734375,47,0.234375,15,,,0.015625,1.0,,,,0.03125,2.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,8,2.9844,0,0.9531,5745.5977,5801.2898,5716.4256,31324,1,5631.7408,0.3438,-0.0059,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.145,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3438,14.9227,51.7915,634.3629,26.1,857.9676,223.6047,0.0641,4.7138,46.2768,255.4429,302.0771,762.5347,19.4478,255.7356,436.2918,0,18,61,488,391,23,89,0.9508196721311475,58,0.5409836065573771,33,0.01639344262295082,1.0,,,0.0066,8.8114,,,,,,0.13114754098360656,8.0,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,3.0,0,0.9531,5766.0996,7272.9571,5062.3238,30727,1,6005.4448,0.3184,-0.0053,0.175,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1368,0.0,0.0,1.0,0.0,0.0196,0.4688,0.3184,15.2079,52.0967,629.7617,26.6,857.9676,228.2059,0.0638,4.6698,46.7327,281.1661,328.289,836.6342,19.8892,281.4922,483.782,0,19,57,456,384,19,67,0.8421052631578947,48,0.2807017543859649,16,0.07017543859649122,4.0,0.08771929824561403,5.0,,,,0.03508771929824561,2.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.9375,0,0.9688,6294.1797,5124.1037,6713.1724,31096,1,6469.7493,0.2637,-0.0072,0.102,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1463,-0.0,0.0,1.0,0.0,0.0126,0.375,0.2637,15.1567,52.0453,626.5997,27.0,857.9676,231.3679,0.064,4.7232,47.7824,301.0734,349.1957,714.4801,19.1506,301.3489,341.4063,0,20,59,472,416,15,56,0.4067796610169492,24,0.6440677966101694,38,0.01694915254237288,1.0,,,0.0069,8.1983,6.0643,0.01694915254237288,1.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.8438,0,0.9688,6685.3047,6935.2396,6535.3438,30521,1,6310.5369,0.375,-0.0004,0.2167,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1433,0.0,0.0,1.0,0.0,0.0202,0.5781,0.375,15.1234,52.0566,624.2105,27.2,857.9676,233.7571,0.0659,4.7801,49.366,325.1266,374.9429,829.4379,18.844,325.5106,430.8666,0,21,59,472,386,24,83,1.0169491525423728,60,0.4406779661016949,26,0.05084745762711865,3.0,0.05084745762711865,3.0,,,,0.0847457627118644,5.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,3.2188,0,0.9844,6705.4805,8304.1943,5875.2878,30102,1,6467.5812,0.3418,0.0187,0.1562,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1426,-0.0023,0.0,1.0,0.0,0.0149,0.5,0.3418,15.2092,52.1088,633.8773,26.1,857.9676,224.0903,0.0619,4.578,57.091,320.4391,377.8433,763.5965,18.9387,320.6899,362.2325,0,22,61,488,388,24,99,0.7868852459016393,48,0.7704918032786885,47,0.06557377049180328,4.0,0.01639344262295082,1.0,0.0065,8.521,,0.11475409836065574,7.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.6562,0,0.9375,6630.8789,7731.1437,6098.287,29620,1,6953.1804,0.3262,0.0118,0.1969,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.0164,0.5156,0.3262,15.3535,52.4285,629.1158,26.7,857.9676,228.8517,0.0627,4.6368,49.5567,299.36,349.2749,1049.1967,19.6531,299.655,675.6279,0,23,57,456,345,23,111,1.1403508771929824,65,0.7719298245614035,44,0.05263157894736842,3.0,0.017543859649122806,1.0,,,,0.017543859649122806,1.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.7344,0,0.9375,5392.7832,7495.951,4577.7344,30573,1,5638.6051,0.2793,0.0003,0.144,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,0.0,0.0,1.0,0.0,0.0156,0.4219,0.2793,15.7614,52.8214,625.9011,27.0,857.9676,232.0665,0.0632,4.5896,41.1111,249.969,291.4409,704.2569,19.2507,250.2661,388.972,0,24,61,488,405,23,83,0.8852459016393442,54,0.3770491803278688,23,,,,,0.0075,8.5429,,0.09836065573770492,6.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.7031,0,0.9219,6406.502,6468.0984,6372.2401,29340,1,6506.3871,0.3574,0.0165,0.1367,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1401,0.0,0.0,1.0,0.0,0.0138,0.4844,0.3574,15.8598,53.0487,623.2285,27.4,857.9676,234.7391,0.0616,4.7813,52.9717,295.9893,349.3116,868.8734,19.3516,296.2778,495.425,0,25,62,496,400,24,80,0.8225806451612904,51,0.3870967741935484,24,0.0967741935483871,6.0,0.03225806451612903,2.0,,,6.535,0.03225806451612903,2.0,,,0.22580645161290322,14.0,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.5625,0,0.9688,6405.3945,7275.6353,5972.8187,30302,1,6438.8941,0.332,-0.0077,0.1465,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1377,-0.002,0.0,1.0,0.0,0.0135,0.4688,0.332,15.8798,53.1326,620.4645,27.7,857.9676,237.5031,0.0646,4.7998,53.7144,307.7867,361.8342,854.4198,20.2015,308.0548,467.581,0,26,57,456,362,19,93,0.6491228070175439,37,0.8771929824561403,50,0.10526315789473684,6.0,,,0.007,8.6985,,0.14035087719298245,8.0,,,,,,,0.017543859649122806,1.0,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.5625,1,1.0,5791.3965,7450.7455,5002.3689,28228,1,5710.9576,0.3223,0.0079,0.2159,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1355,0.0,0.0,1.0,0.0,0.0175,0.5,0.3223,15.8962,53.1489,618.6345,27.9,857.9676,239.3331,0.0587,4.2706,44.88,262.209,307.4009,590.0799,22.5095,262.4618,255.8951,0,27,60,480,415,20,60,0.45,27,0.38333333333333336,23,0.13333333333333333,8.0,0.016666666666666666,1.0,,,,0.03333333333333333,2.0,,,0.016666666666666666,1.0,,,0.05,3.0,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.9062,0,0.9688,7508.9941,6576.2874,7989.145,30784,1,6953.4419,0.3398,0.0043,0.1576,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1452,-0.0003,0.0,1.0,0.0,0.017,0.4844,0.3398,15.825,53.0919,616.7122,28.1,857.9676,241.2554,0.0646,4.8023,57.3471,341.847,399.5664,880.2184,18.9742,342.1543,456.8715,0,28,63,504,420,21,80,0.8412698412698413,53,0.36507936507936506,23,0.15873015873015872,10.0,0.047619047619047616,3.0,0.0069,8.6398,,0.031746031746031744,2.0,,,,,,,0.015873015873015872,1.0,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.7812,0,0.9531,6654.498,7316.7135,6279.8502,31394,1,6506.5919,0.3613,0.0096,0.1337,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1404,0.0,0.0,1.0,0.0,0.0162,0.5625,0.3613,15.9013,53.1658,614.1317,28.4,857.9676,243.8359,0.0636,4.839,48.4378,295.3489,344.1274,884.7489,19.1329,295.6257,516.6459,0,29,62,496,402,23,86,0.7741935483870968,48,0.6451612903225806,40,0.03225806451612903,2.0,0.016129032258064516,1.0,,,,0.016129032258064516,1.0,,,,,0.11290322580645161,7.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.5,0,0.9219,6475.6191,7192.9286,6125.3052,31134,1,6725.433,0.3281,0.0109,0.1896,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1312,0.0,0.0,1.0,0.0,0.0199,0.5156,0.3281,16.001,53.2318,611.9695,28.7,857.9676,245.9981,0.0673,4.7096,51.1201,311.19,362.6769,1129.64,19.6246,311.4892,742.6236,0,30,63,504,405,26,98,0.9523809523809523,60,0.4603174603174603,29,0.09523809523809523,6.0,,,0.0072,8.4626,6.0549,0.047619047619047616,3.0,,,,,,,0.015873015873015872,1.0,,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.2812,0,0.9531,6346.0625,7175.0743,6008.9918,30375,1,6325.3681,0.2891,-0.0015,0.1034,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0129,0.375,0.2891,16.219,53.4149,609.7183,28.9,857.9676,248.2493,0.0644,4.6585,49.8638,296.7068,346.9154,880.0104,19.5517,296.9867,508.8808,0,31,63,504,397,27,100,1.0158730158730158,64,0.4126984126984127,26,0.07936507936507936,5.0,,,,,,0.07936507936507936,5.0,,,,,0.09523809523809523,6.0,,,0.015873015873015872,1.0,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.5938,0,0.9688,5907.5469,7252.2229,5209.273,29747,1,6251.0608,0.3418,-0.005,0.1779,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1317,0.0,0.0,1.0,0.0,0.0177,0.5312,0.3418,16.2711,53.4052,607.4507,29.2,857.9676,250.5169,0.0626,4.4911,42.321,277.0021,319.666,799.5796,19.3098,277.2821,456.1088,0,32,60,480,371,23,104,0.95,57,0.7,42,0.1,6.0,,,0.0066,8.9378,,0.11666666666666667,7.0,,,,,0.08333333333333333,5.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.3906,0,0.9531,5799.2324,6060.1132,5681.7252,31737,1,6414.7137,0.3105,0.0079,0.1257,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1294,-0.002,0.0,1.0,0.0,0.0178,0.4688,0.3105,16.2997,53.5626,632.3168,26.3,857.9676,225.6508,0.0662,4.8155,53.5314,284.6735,338.5434,801.0326,19.4236,284.9454,438.2467,0,33,59,472,364,25,100,1.0508474576271187,62,0.4745762711864407,28,0.0847457627118644,5.0,0.01694915254237288,1.0,,,,0.1016949152542373,6.0,,,0.11864406779661017,7.0,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.6719,0,0.9531,6310.3008,7310.4972,5781.8388,29761,1,6110.0445,0.3457,0.0082,0.2142,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0033,0.0,1.0,0.0,0.0179,0.5312,0.3457,16.6979,53.8992,628.2749,26.8,857.9676,229.6927,0.0621,4.5025,50.7853,290.2551,341.3921,946.0027,19.3899,290.5443,580.7139,0,34,63,504,435,21,67,0.6825396825396826,43,0.31746031746031744,20,0.09523809523809523,6.0,0.031746031746031744,2.0,0.0073,8.4926,,,,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.8281,0,0.9531,6915.877,7142.4017,6717.5641,31178,1,5996.8799,0.4668,0.0027,0.2488,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1465,0.0,0.0,1.0,0.0,0.0205,0.6875,0.4668,16.7466,53.9885,624.4393,27.2,857.9676,233.5283,0.0654,5.2906,51.1954,306.1635,357.6744,848.0649,19.5558,306.4133,465.5401,0,35,59,472,397,19,69,0.5932203389830508,35,0.3728813559322034,22,0.13559322033898305,8.0,0.1016949152542373,6.0,,,6.2574,0.1016949152542373,6.0,,,,,0.01694915254237288,1.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.5156,0,0.8281,6071.5781,6369.9853,5873.9318,29647,1,5995.9806,0.3984,0.0309,0.1754,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1427,-0.0036,0.0,1.0,0.0,0.0172,0.5625,0.3984,16.6372,53.838,621.8133,27.5,857.9676,236.1543,0.063,4.5239,46.5181,276.1477,323.0187,924.8781,21.0895,276.4371,576.2418,0,36,63,504,422,19,79,0.9365079365079365,59,0.1746031746031746,11,0.06349206349206349,4.0,0.031746031746031744,2.0,0.0073,9.0132,,0.06349206349206349,4.0,,,,,,,,,,,0.015873015873015872,1.0,0.015873015873015872,1.0,,,,
|
||||||
|
0.0,0,64,512,5,2.625,0,0.9375,6053.6152,6609.8693,5762.244,29949,1,5973.1572,0.3438,-0.0022,0.1581,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1378,-0.0003,0.0,1.0,0.0,0.0201,0.5312,0.3438,16.8303,54.0954,619.2038,27.8,857.9676,238.7638,0.0621,4.5581,46.3663,282.1342,328.8628,883.0955,18.8513,282.434,530.8195,0,37,58,464,374,27,84,0.896551724137931,52,0.5862068965517241,34,0.20689655172413793,12.0,0.034482758620689655,2.0,,,,0.05172413793103448,3.0,,,0.06896551724137931,4.0,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.9062,0,0.9062,6101.7637,7142.1095,5429.3859,31890,1,6539.1202,0.3926,0.0027,0.1491,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0066,0.0,1.0,0.0,0.0133,0.5156,0.3926,16.8694,54.1906,616.2216,28.2,857.9676,241.746,0.0685,4.8885,48.6177,301.233,350.2252,1456.7333,20.4702,301.5387,1081.1455,0,38,62,496,448,18,43,0.5,31,0.11290322580645161,7,0.04838709677419355,3.0,0.016129032258064516,1.0,0.0071,8.2664,,0.016129032258064516,1.0,0.016129032258064516,1.0,0.04838709677419355,3.0,,,,,,,,,,,0.016129032258064516,1.0,,
|
||||||
|
0.0,0,64,512,5,2.625,0,0.9375,6800.1094,7008.201,6667.8083,29586,1,5671.6318,0.3887,0.0097,0.1379,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1503,-0.002,0.0,1.0,0.0,0.015,0.5312,0.3887,16.8346,54.2173,614.385,28.4,857.9676,243.5826,0.0628,4.5977,45.5648,283.358,329.2728,854.372,20.0914,283.6447,500.4064,0,39,62,496,407,21,86,1.0161290322580645,63,0.12903225806451613,8,0.08064516129032258,5.0,,,,,,0.1774193548387097,11.0,,,0.016129032258064516,1.0,0.016129032258064516,1.0,0.016129032258064516,1.0,,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.2969,0,0.8438,5812.7578,6822.4395,5366.2225,28547,1,5777.0387,0.3066,0.0021,0.1644,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.136,-0.0,0.0,1.0,0.0,0.0178,0.4531,0.3066,16.8475,54.2062,612.4843,28.6,857.9676,245.4833,0.0586,4.4393,42.9655,268.3647,311.628,893.5759,17.0362,268.6034,560.4691,0,40,48,384,283,30,83,0.9583333333333334,46,0.6458333333333334,31,0.08333333333333333,4.0,,,0.0068,396.8219,6.1042,0.625,30.0,,,0.08333333333333333,4.0,0.125,6.0,,,,,,,,,0.4791666666666667,23.0,,
|
||||||
|
0.0,0,64,512,5,2.75,0,0.8594,6251.3789,6817.3214,5974.9884,31729,1,7057.6721,0.3281,0.0461,0.1581,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1199,-0.0202,0.0,1.0,0.0,0.0154,0.4688,0.3281,17.2877,54.623,609.6594,28.9,857.9676,248.3082,0.0668,4.7741,53.9388,320.2493,374.5444,2041.8385,19.8279,320.5384,1642.6884,0,41,59,472,402,19,44,0.4067796610169492,24,0.288135593220339,17,0.03389830508474576,2.0,0.1016949152542373,6.0,,,,0.11864406779661017,7.0,0.03389830508474576,2.0,0.0847457627118644,5.0,,,,,,,,,,,0.23728813559322035,14.0,0.01694915254237288,1.0
|
||||||
|
0.0,0,64,512,6,2.1562,0,0.9062,5198.2734,5100.9904,5265.3762,30850,1,5154.8585,0.4082,0.0041,0.2199,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.139,-0.0062,0.0,1.0,0.0,0.0229,0.5938,0.4082,17.2743,54.6095,607.5067,29.2,857.9676,250.4609,0.0706,4.5927,38.4789,238.7629,277.6139,767.8562,19.0513,239.064,466.5943,0,42,63,504,448,11,43,0.3968253968253968,25,0.2222222222222222,14,,,,,0.0085,8.972,,0.06349206349206349,4.0,,,0.015873015873015872,1.0,0.07936507936507936,5.0,,,0.015873015873015872,1.0,,,,,0.09523809523809523,6.0,,
|
||||||
|
0.0,0,64,512,6,2.2812,0,0.9219,6603.6328,6320.4868,6769.3127,29924,1,5901.6823,0.3691,0.0059,0.0943,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1511,-0.0,0.0,1.0,0.0,0.0156,0.4531,0.3691,17.2621,54.5973,605.1696,29.5,857.9676,252.798,0.0627,4.7386,49.3034,289.1,338.7342,869.574,20.0024,289.3677,506.0951,0,43,63,504,388,26,112,0.8253968253968254,52,0.746031746031746,47,,,0.047619047619047616,3.0,,,,0.20634920634920634,13.0,,,,,,,,,,,,,,,,,0.015873015873015872,1.0
|
||||||
|
0.0,0,64,512,6,2.6719,0,0.9375,6097.0117,6883.791,5588.5145,30733,1,6180.0026,0.3926,0.003,0.1964,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1275,-0.0037,0.0,1.0,0.0,0.02,0.5781,0.3926,17.5231,54.6111,604.3675,29.6,857.9676,253.6001,0.072,4.7976,44.3964,285.5469,330.2905,783.7946,19.0298,285.8218,429.673,0,44,56,448,344,23,94,0.9642857142857143,54,0.6785714285714286,38,0.07142857142857142,4.0,0.017857142857142856,1.0,0.0063,8.3169,,0.05357142857142857,3.0,,,0.16071428571428573,9.0,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.5781,0,0.9531,6293.2793,6687.8072,6103.9971,31486,1,6498.3542,0.3242,0.0027,0.1589,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1235,-0.0007,0.0,1.0,0.0,0.0186,0.4844,0.3242,17.6409,54.684,601.6743,29.9,857.9676,256.2933,0.0656,4.846,47.9094,300.6953,348.9867,673.9404,19.8524,301.0114,300.2514,0,45,60,480,384,21,93,0.8,48,0.7166666666666667,43,0.03333333333333333,2.0,0.03333333333333333,2.0,,,6.1065,0.16666666666666666,10.0,,,,,0.03333333333333333,2.0,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.75,0,0.9375,5982.3438,6061.5407,5942.2794,31013,1,6331.3741,0.3359,0.0083,0.1586,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,-0.0007,0.0,1.0,0.0,0.0182,0.5312,0.3359,17.8517,54.9248,599.7339,30.1,857.9676,258.2337,0.0638,4.6266,45.5297,290.2921,336.1922,1024.0646,19.361,290.5982,663.8806,0,46,55,440,370,19,70,0.6181818181818182,34,0.8181818181818182,45,,,,,0.0065,107.2038,,0.07272727272727272,4.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.6719,0,0.9531,6900.8828,6491.8537,7093.6437,31520,1,6837.8518,0.3203,0.0046,0.1391,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1381,0.0,0.0,1.0,0.0,0.0162,0.4844,0.3203,17.6216,54.6812,598.9659,30.2,857.9676,259.0017,0.0794,4.8182,58.3023,322.2259,380.8975,1111.9881,19.6389,322.5152,706.6292,0,47,60,480,385,22,90,0.48333333333333334,29,0.8333333333333334,50,,,0.016666666666666666,1.0,,,,0.2,12.0,,,0.06666666666666667,4.0,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.375,0,0.8438,6583.623,7439.1768,6115.7825,31001,1,6334.7756,0.3535,-0.004,0.1772,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1309,-0.0013,0.0,1.0,0.0,0.0186,0.5156,0.3535,17.5547,54.6142,597.6776,30.3,857.9676,260.29,0.0646,4.8542,53.0938,306.4226,359.8881,928.1989,19.6921,306.7294,543.7599,0,48,58,464,396,13,55,0.1896551724137931,11,0.7241379310344828,42,,,0.017241379310344827,1.0,0.0073,8.3185,,0.06896551724137931,4.0,,,0.13793103448275862,8.0,0.05172413793103448,3.0,,,0.017241379310344827,1.0,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.7656,0,0.9531,6662.2363,7889.466,5836.0621,28624,1,5968.458,0.4023,0.0111,0.2309,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1378,-0.0011,0.0,1.0,0.0,0.0225,0.5938,0.4023,17.5219,54.582,597.1965,30.4,857.9676,260.7711,0.0628,4.5945,52.8577,284.1361,337.3518,759.6281,19.4761,284.4309,398.2013,0,49,64,512,418,22,91,0.734375,47,0.6875,44,0.078125,5.0,0.046875,3.0,,,,0.046875,3.0,,,,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.5781,0,0.9688,6234.9375,6473.3529,6148.7021,31740,1,6350.9234,0.2656,-0.0119,0.0852,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1381,0.0,0.0,1.0,0.0,0.0124,0.3438,0.2656,17.6574,54.7542,595.0074,30.6,857.9676,262.9602,0.0658,5.36,47.1592,284.0365,331.6008,773.9742,20.4465,284.3755,416.5621,0,50,42,336,293,13,43,0.4523809523809524,19,0.5714285714285714,24,,,,,0.0069,8.3248,6.1934,,,,,,,,,,,,,,,,,,,,
|
||||||
|
31
training_logs/20260604_123153_metrics_job_567551.csv
Normal file
31
training_logs/20260604_123153_metrics_job_567551.csv
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RewardFileNotFoundError,batch_errors/total_RewardFileNotFoundError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,timing/save_hf_model,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_VerifierOutputParseError,batch_errors/total_VerifierOutputParseError,batch_errors/avg_DaytonaNotFoundError,batch_errors/total_DaytonaNotFoundError
|
||||||
|
0.0,0,64,512,6,1.7812,0,0.6562,5990.6699,6322.5959,5686.0936,28123,1,5473.7808,0.4785,0.0087,0.1171,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1368,-0.0033,0.0,1.0,0.0,0.0174,0.6226,0.4785,12.3812,50.353,596.5785,30.5,857.9676,261.3891,0.1531,4.4122,51.3157,263.4737,315.2958,2184.4235,19.7425,263.8261,1844.9728,0,51,64,512,461,10,49,0.625,40,0.015625,1.0,0.25,16,0.03125,2.0,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,1,0.7812,0,0.7812,6731.9863,7178.8007,6166.5487,31193,1,6580.3321,0.5586,0.0038,0.1713,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.143,-0.0028,0.0,1.0,0.0,0.02,0.6875,0.5586,13.0269,50.5115,593.0834,30.9,857.9676,264.8842,0.0655,4.8546,61.4679,320.4413,382.2524,1651.923,19.4153,320.7179,1245.3831,0,52,59,472,386,22,86,0.4915254237288136,29,,,1.0,59,,,32.6834,41.116,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,2,1.4531,0,0.9688,6508.3203,6281.5808,6651.2962,30531,1,6289.6614,0.3867,-0.0019,0.123,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0142,0.5,0.3867,13.9249,50.6506,590.5195,31.2,857.9676,267.4481,0.0636,4.7459,45.9719,288.7953,335.1579,652.1002,19.5645,289.1214,292.6286,0,53,60,480,345,31,134,0.6333333333333333,38,0.03333333333333333,2.0,1.5166666666666666,91,,,,,0.016666666666666666,1.0,0.05,3.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,3,2.25,1,1.0,5810.2891,7441.8333,5095.3427,30242,1,6574.3825,0.3047,0.0096,0.2385,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1218,0.0,0.0,1.0,0.0,0.0211,0.5156,0.3047,14.0695,50.7521,590.0227,31.2,857.9676,267.9449,0.0622,4.5947,50.1011,300.1234,350.5934,560.9024,19.5021,300.4296,186.209,0,54,57,456,379,17,76,0.3684210526315789,21,0.10526315789473684,6.0,0.9298245614035088,53,,,0.0067,9.2903,0.017543859649122806,1.0,0.017543859649122806,1.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,4,2.4844,0,0.9531,5920.2324,6443.0893,5664.8837,30033,1,5739.4678,0.3281,0.0067,0.127,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1313,0.0,0.0,1.0,0.0,0.0168,0.4531,0.3281,14.2443,50.9048,588.7913,31.4,857.9676,269.1763,0.0612,5.0782,53.6346,271.6531,325.6797,683.268,19.8067,271.9836,332.7004,0,55,62,496,434,17,62,0.3387096774193548,21,0.04838709677419355,3.0,0.6774193548387096,42,,,,,,,,,7.0721,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.2812,0,0.9062,6120.8418,6407.7637,5873.5673,30008,1,6151.3952,0.4629,-0.0047,0.1712,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1397,-0.0,0.0,1.0,0.0,0.0151,0.5938,0.4629,14.2906,51.0547,633.2055,26.2,857.9676,224.7621,0.0629,4.6042,45.6063,296.6186,342.6071,854.4565,19.0306,296.9374,488.2113,0,56,64,512,422,22,87,0.734375,47,0.015625,1.0,0.578125,37,,,0.0067,9.5172,0.015625,1.0,,,,0.03125,2.0,0.015625,1.0,,,,
|
||||||
|
0.0,0,64,512,5,2.5781,0,0.9844,7322.8574,8407.4009,6459.0281,31653,1,6692.4401,0.4434,0.0133,0.1395,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.131,-0.0053,0.0,1.0,0.0,0.0166,0.5781,0.4434,14.4149,51.1216,629.6761,26.6,857.9676,228.2914,0.0687,5.4807,58.8815,329.3126,388.5892,854.4278,19.65,329.6382,440.7044,0,57,61,488,388,24,99,0.5737704918032787,35,,,1.1475409836065573,70,,,,,0.01639344262295082,1.0,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,3.0469,0,0.9688,6034.1055,6223.8895,5930.3263,29330,1,6569.5919,0.3535,0.0099,0.1348,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1272,-0.0006,0.0,1.0,0.0,0.0172,0.5156,0.3535,14.421,51.2411,632.4707,26.3,857.9676,225.4969,0.0756,4.5986,50.4083,317.2726,368.0298,785.544,20.2901,317.5455,392.6213,0,58,60,480,441,11,38,0.26666666666666666,16,,,0.4666666666666667,28,0.016666666666666666,1.0,0.0072,9.0284,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.7031,0,0.9844,6661.7168,7054.4919,6439.5046,31055,1,6324.187,0.3613,-0.002,0.1616,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1497,-0.0005,0.0,1.0,0.0,0.0227,0.5469,0.3613,15.0173,51.8337,629.3011,26.7,857.9676,228.6665,0.0655,4.7848,50.985,314.2571,365.6335,643.7273,20.2109,314.5827,253.0942,0,59,58,464,362,23,97,0.5517241379310345,32,0.10344827586206896,6.0,1.0,58,,,,,0.06896551724137931,4.0,0.017241379310344827,1.0,,0.017241379310344827,1.0,,,,,,
|
||||||
|
0.0,0,64,512,7,3.0781,0,0.9844,6317.8027,7111.7259,5821.2857,30479,1,6483.291,0.3848,0.0052,0.2215,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,0.0,0.0,1.0,0.0,0.0183,0.5469,0.3848,15.0261,51.9561,627.1725,26.9,857.9676,230.7951,0.0633,4.7623,52.6042,290.3792,343.2867,674.6483,21.1596,290.6188,305.4351,0,60,61,488,373,24,115,1.0491803278688525,64,0.06557377049180328,4.0,0.7704918032786885,47,,,0.0063,9.488,,,,,6.3751,,,,,,,,
|
||||||
|
0.0,0,64,512,6,3.125,1,1.0,6463.0605,7886.7006,5710.8687,30994,1,6880.4264,0.3457,0.0019,0.1625,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1193,0.0,0.0,1.0,0.0,0.0145,0.5156,0.3457,15.1388,52.027,625.0475,27.1,857.9676,232.92,0.0653,4.7961,53.4008,354.2283,408.0195,684.6355,19.3622,354.5529,252.4537,0,61,61,488,413,21,68,0.18032786885245902,11,,,0.9344262295081968,57,0.01639344262295082,1.0,,,0.03278688524590164,2.0,,,,0.06557377049180328,4.0,,,,,,
|
||||||
|
0.0,0,64,512,6,2.9375,0,0.9688,6648.5312,6675.8687,6631.293,29657,1,6232.3936,0.3867,-0.001,0.1137,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0142,0.5312,0.3867,15.158,52.1367,623.0032,27.4,857.9676,234.9644,0.0645,5.0387,50.7679,292.3912,343.5179,624.4476,21.7199,292.6853,254.166,0,62,63,504,449,15,46,0.12698412698412698,8,0.015873015873015872,1.0,0.5714285714285714,36,0.06349206349206349,4.0,0.0064,9.0101,0.07936507936507936,5.0,0.015873015873015872,1.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.8125,0,0.9062,6871.1211,7110.377,6738.0395,30986,1,6271.4665,0.3574,0.0086,0.1704,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1454,-0.0038,0.0,1.0,0.0,0.0191,0.5938,0.3574,15.3867,52.3629,620.4409,27.7,857.9676,237.5267,0.0654,4.8585,52.4883,316.0092,368.8701,868.6813,20.4287,316.3161,474.5195,0,63,63,504,415,22,86,0.42857142857142855,27,0.015873015873015872,1.0,0.8888888888888888,56,,,,,0.047619047619047616,3.0,0.047619047619047616,3.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.8125,0,0.9688,6867.2734,8067.7197,6336.3718,31346,1,7132.2744,0.3066,0.014,0.1409,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1309,-0.0,0.0,1.0,0.0,0.0143,0.4375,0.3066,15.383,52.3718,619.1424,27.8,857.9676,238.8252,0.0647,5.3526,58.7073,343.2849,402.3885,692.6827,21.0837,343.616,263.8521,0,64,62,496,435,19,60,0.3225806451612903,20,0.016129032258064516,1.0,0.6129032258064516,38,,,0.0068,8.8666,0.016129032258064516,1.0,0.016129032258064516,1.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.8125,1,1.0,6153.6172,7862.9684,5144.9938,31464,1,5836.2093,0.3711,0.0048,0.1809,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0,0.0,1.0,0.0,0.0227,0.5,0.3711,15.5993,52.6177,634.1627,26.1,857.9676,223.8049,0.0645,4.8537,50.2765,285.3918,336.0526,634.5957,22.2875,285.7112,271.3975,0,65,61,488,366,23,121,1.180327868852459,72,0.04918032786885246,3.0,0.7377049180327869,45,,,,,0.01639344262295082,1.0,0.01639344262295082,1.0,5.9537,,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.9531,0,0.9688,6439.0156,6111.65,6648.8654,29740,1,6685.0287,0.3906,-0.0145,0.1273,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0,0.0,1.0,0.0,0.0155,0.5469,0.3906,15.8226,52.9003,630.8162,26.5,857.9676,227.1514,0.0628,4.5991,57.2731,323.5507,381.1919,709.8547,20.4883,323.8556,303.571,0,66,60,480,435,12,44,0.36666666666666664,22,0.05,3.0,0.5666666666666667,34,,,0.0066,9.3084,0.016666666666666666,1.0,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,3.0625,1,1.0,6846.5332,6910.392,6785.5992,29213,1,6150.4072,0.4883,0.008,0.2377,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1313,0.0,0.0,1.0,0.0,0.0212,0.6875,0.4883,16.0647,53.2021,628.415,26.8,857.9676,229.5526,0.0627,4.6262,49.228,305.8525,355.4619,517.5303,22.0327,306.1709,135.4056,0,67,61,488,438,16,43,0.36065573770491804,22,,,0.3442622950819672,21,0.11475409836065574,7.0,,,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,3.2344,0,0.9844,7383.6035,8211.3237,6821.8393,29929,1,6141.4216,0.4043,0.0031,0.2121,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1473,0.0,0.0,1.0,0.0,0.025,0.6094,0.4043,16.173,53.2971,626.4578,27.0,857.9676,231.5098,0.0668,4.8139,51.8795,306.0463,358.2671,670.9587,22.6126,306.3205,285.261,0,68,60,480,418,13,62,0.35,21,0.08333333333333333,5.0,0.65,39,,,0.0075,9.2352,,,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,3.1094,0,0.9375,6849.9824,6946.125,6770.3214,31319,1,6263.3838,0.4531,-0.002,0.1978,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1369,-0.0,0.0,1.0,0.0,0.0205,0.6562,0.4531,16.3273,53.4486,624.0512,27.3,857.9676,233.9164,0.0686,4.9239,53.7165,303.0634,357.1172,730.7219,20.5135,303.3317,348.1627,0,69,57,456,354,22,100,0.8245614035087719,47,0.017543859649122806,1.0,1.0526315789473684,60,0.017543859649122806,1.0,,,0.017543859649122806,1.0,0.14035087719298245,8.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,3.1719,0,0.8906,5942.8418,6702.5523,5558.5176,29772,1,6202.3377,0.3359,-0.0023,0.118,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1266,-0.002,0.0,1.0,0.0,0.0136,0.5,0.3359,16.3538,53.356,622.1553,27.5,857.9676,235.8123,0.0651,4.4971,48.5204,287.6047,336.462,928.0517,20.97,287.8763,566.1177,0,70,61,488,398,22,89,0.6557377049180327,40,0.08196721311475409,5.0,0.7377049180327869,45,,,0.0063,8.9573,0.01639344262295082,1.0,,,6.1925,,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.7812,0,0.9375,6158.498,6986.8906,5661.4625,30577,1,5879.0702,0.375,0.0025,0.1169,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1303,0.0,0.0,1.0,0.0,0.0138,0.4688,0.375,16.3088,53.3733,619.9735,27.7,857.9676,237.9941,0.0645,4.8017,45.4262,268.4007,314.246,809.2009,20.4367,268.7549,469.7119,0,71,58,464,377,21,79,0.3793103448275862,22,,,1.0517241379310345,61,,,,,,,,,,,,,,0.13793103448275862,8.0,,
|
||||||
|
0.0,0,64,512,6,2.75,0,0.9688,6496.7656,7114.178,5968.8333,29148,1,6645.0121,0.4609,0.0151,0.1421,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1219,0.0,0.0,1.0,0.0,0.0186,0.6562,0.4609,16.312,53.3518,618.4863,27.9,857.9676,239.4813,0.0602,4.4786,50.5528,311.4958,362.4515,775.8533,20.601,311.8381,388.3179,0,72,61,488,359,26,129,1.0327868852459017,63,0.11475409836065574,7.0,0.9672131147540983,59,,,0.0066,9.0389,,,0.01639344262295082,1.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,7,2.8281,0,0.9688,6007.7578,6901.5911,5420.5469,31182,1,6450.2234,0.3965,0.0136,0.2237,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1235,0.0,0.0,1.0,0.0,0.0181,0.5625,0.3965,16.534,53.6315,616.6702,28.1,857.9676,241.2974,0.0668,5.1941,55.0841,299.7416,355.1541,736.8375,20.6845,300.0029,355.801,0,73,58,464,361,20,101,0.9482758620689655,55,0.08620689655172414,5.0,0.7931034482758621,46,0.034482758620689655,2.0,,,0.017241379310344827,1.0,0.034482758620689655,2.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.7344,0,0.9844,5930.1973,6072.1242,5869.7103,30745,1,6036.8087,0.2988,-0.0089,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1195,0.0,0.0,1.0,0.0,0.064,0.3594,0.2988,16.5898,53.7518,629.4752,26.6,857.9676,228.4924,0.0661,4.6933,50.044,292.5558,342.9545,639.5606,21.8612,292.844,270.0476,0,74,64,512,437,19,68,0.03125,2,0.109375,7.0,0.921875,59,,,0.0071,9.3935,0.0625,4.0,,,,,,,,,,0.046875,3.0
|
||||||
|
0.0,0,64,512,6,2.8906,0,0.9688,6931.8203,6564.4852,7112.8105,31582,1,6325.4492,0.3301,-0.0145,0.1116,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1352,-0.0011,0.0,1.0,0.0,0.0164,0.4531,0.3301,16.8138,53.9489,626.9986,26.9,857.9676,230.969,0.0663,4.9589,51.1396,308.4821,359.9248,987.0937,23.2754,308.7185,598.9303,0,75,64,512,434,21,77,0.1875,12,0.015625,1.0,0.96875,62,,,,,0.015625,1.0,0.03125,2.0,6.209,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.6875,0,0.9219,6552.0254,5927.6872,7116.0186,31827,1,6416.322,0.4746,-0.0091,0.1469,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1332,0.0,0.0,1.0,0.0,0.0181,0.5938,0.4746,17.0905,54.3295,624.7698,27.2,857.9676,233.1978,0.0662,4.888,47.7922,305.2241,353.352,1003.0464,20.8521,305.4933,623.9499,0,76,61,488,415,19,73,0.3770491803278688,23,,,0.9180327868852459,56,,,0.0073,9.5322,,,0.01639344262295082,1.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,6,2.3438,0,0.9062,6973.2559,7723.1186,6515.7925,31204,1,6409.6402,0.3789,0.0007,0.1955,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,-0.0,0.0,1.0,0.0,0.0179,0.5625,0.3789,17.2256,54.4399,628.907,26.7,857.9676,229.0605,0.0716,4.9191,53.9669,306.4151,360.706,827.6281,21.3515,306.6672,440.6474,0,77,58,464,347,27,106,0.7241379310344828,42,0.10344827586206896,6.0,1.293103448275862,75,0.10344827586206896,6.0,,,0.017241379310344827,1.0,0.017241379310344827,1.0,,,,,,,,0.06896551724137931,4.0
|
||||||
|
0.0,0,64,512,6,2.625,0,0.9219,5913.8125,7165.1304,5211.8537,31252,1,6425.6925,0.3594,0.0054,0.1736,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1127,-0.0016,0.0,1.0,0.0,0.0131,0.4688,0.3594,17.1172,54.3807,626.1709,27.0,857.9676,231.7967,0.0658,4.7719,53.2092,303.5102,357.0435,1027.6138,19.8591,303.7683,645.9345,0,78,59,472,339,26,132,0.7288135593220338,43,0.0847457627118644,5.0,1.4915254237288136,88,,,0.0071,13.3714,0.01694915254237288,1.0,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.1406,0,0.9219,5971.7539,7443.8,5271.7896,31359,1,6822.5791,0.3223,0.0002,0.1775,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1121,-0.001,0.0,1.0,0.0,0.0197,0.4688,0.3223,17.3101,54.5718,623.1119,27.4,857.9676,234.8557,0.0639,4.7264,50.7595,318.7277,369.8649,864.1247,20.4919,319.041,469.0373,0,79,61,488,393,22,92,0.5573770491803278,34,0.14754098360655737,9.0,0.7868852459016393,48,,,,,0.04918032786885246,3.0,0.01639344262295082,1.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,5,2.25,0,0.875,6524.4219,7517.6269,5923.5172,29828,1,6065.1914,0.377,0.0047,0.12,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1309,0.0,0.0,1.0,0.0,0.0175,0.5156,0.377,17.6478,54.8022,621.2485,27.6,857.9676,236.7191,0.0626,5.2654,46.0826,294.2125,340.6332,806.3176,22.4283,294.4877,437.9868,0,80,94,752,614,32,130,0.4787234042553192,45,0.02127659574468085,2.0,0.9680851063829787,91,0.010638297872340425,1.0,0.0075,9.6795,0.02127659574468085,2.0,0.010638297872340425,1.0,6.251,0.05319148936170213,5.0,,,,,,
|
||||||
|
396
training_logs/20260604_123153_metrics_report.md
Normal file
396
training_logs/20260604_123153_metrics_report.md
Normal file
@@ -0,0 +1,396 @@
|
|||||||
|
# SkyRL Training Metrics Analysis
|
||||||
|
|
||||||
|
Generated from 3 log files
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
| Log File | Total Steps | Metric Blocks | Final Reward (mean) | Final Reward (max) | Total Time (s) |
|
||||||
|
|----------|-------------|---------------|---------------------|-------------------|----------------|
|
||||||
|
| job_567548 | 9 | 9 | 0.4008 | 0.5059 | 6840.2 |
|
||||||
|
| job_567550 | 50 | 42 | 0.3437 | 0.4668 | 38753.5 |
|
||||||
|
| job_567551 | 80 | 30 | 0.3892 | 0.5586 | 25084.9 |
|
||||||
|
|
||||||
|
## Async Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:------------------------------|-----------:|---------:|------:|---------:|--------:|
|
||||||
|
| async/discard_rate | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| async/discarded_count | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| async/effective_batch_groups | 64 | 0 | 64 | 64 | 81 |
|
||||||
|
| async/effective_batch_samples | 512 | 0 | 512 | 512 | 81 |
|
||||||
|
| async/staleness_max | 5.48148 | 1.62874 | 0 | 8 | 81 |
|
||||||
|
| async/staleness_mean | 2.52044 | 0.596524 | 0 | 3.2344 | 81 |
|
||||||
|
| async/staleness_min | 0.135802 | 0.344713 | 0 | 1 | 81 |
|
||||||
|
| async/staleness_ratio | 0.921498 | 0.127923 | 0 | 1 | 81 |
|
||||||
|
|
||||||
|
## Generate Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:-------------------------------------|---------:|---------:|---------:|---------:|--------:|
|
||||||
|
| generate/avg_num_tokens | 6219.84 | 593.767 | 3631.05 | 7508.99 | 81 |
|
||||||
|
| generate/avg_tokens_non_zero_rewards | 6723.53 | 885.765 | 3316.3 | 8407.4 | 81 |
|
||||||
|
| generate/avg_tokens_zero_rewards | 5945.13 | 669.261 | 3953.26 | 7989.15 | 81 |
|
||||||
|
| generate/max_num_tokens | 30304.3 | 1297.57 | 25303 | 31890 | 81 |
|
||||||
|
| generate/min_num_tokens | 1 | 0 | 1 | 1 | 81 |
|
||||||
|
| generate/std_num_tokens | 6150.02 | 623.638 | 3243.07 | 7148.25 | 81 |
|
||||||
|
|
||||||
|
## Loss Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:----------------------------|-----------:|-----------:|--------:|-------:|--------:|
|
||||||
|
| loss/avg_final_rewards | 0.36687 | 0.0610739 | 0.2559 | 0.5586 | 81 |
|
||||||
|
| loss/avg_raw_advantages | 0.00432716 | 0.00984494 | -0.0145 | 0.0461 | 81 |
|
||||||
|
| loss/avg_raw_advantages_abs | 0.160136 | 0.0385764 | 0.0852 | 0.2488 | 81 |
|
||||||
|
|
||||||
|
## Policy Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:----------------------------|-------------:|------------:|--------:|-------:|--------:|
|
||||||
|
| policy/final_loss | -1.48148e-05 | 4.50309e-05 | -0.0003 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_max | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_mean | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_p99 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos00 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos10 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos20 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos30 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos40 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos50 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos60 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos70 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos80 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos90 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/n_tokens_dp_gt_10pct | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/n_tokens_dp_gt_1pct | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/n_tokens_dp_gt_50pct | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/policy_entropy | 0.135762 | 0.0098146 | 0.1121 | 0.1546 | 81 |
|
||||||
|
| policy/policy_loss | -0.00113086 | 0.00263399 | -0.0202 | 0 | 81 |
|
||||||
|
| policy/policy_lr | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/policy_update_steps | 1 | 0 | 1 | 1 | 81 |
|
||||||
|
| policy/ppo_clip_ratio | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/raw_grad_norm | 0.0180111 | 0.00588504 | 0.0124 | 0.064 | 81 |
|
||||||
|
|
||||||
|
## Reward Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:----------------------|---------:|----------:|-------:|-------:|--------:|
|
||||||
|
| reward/avg_pass_at_8 | 0.522458 | 0.0740283 | 0.3438 | 0.6875 | 81 |
|
||||||
|
| reward/avg_raw_reward | 0.36687 | 0.0610739 | 0.2559 | 0.5586 | 81 |
|
||||||
|
|
||||||
|
## System Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:------------------------|---------:|---------:|---------:|---------:|--------:|
|
||||||
|
| system/process_rss_gb | 15.5018 | 1.58876 | 10.6846 | 17.8517 | 81 |
|
||||||
|
| system/process_vms_gb | 52.6829 | 1.43311 | 50.2918 | 54.9248 | 81 |
|
||||||
|
| system/ram_available_gb | 620.096 | 12.2909 | 588.791 | 634.891 | 81 |
|
||||||
|
| system/ram_percent | 27.7247 | 1.43436 | 26 | 31.4 | 81 |
|
||||||
|
| system/ram_total_gb | 857.968 | 0 | 857.968 | 857.968 | 81 |
|
||||||
|
| system/ram_used_gb | 237.871 | 12.2909 | 223.076 | 269.176 | 81 |
|
||||||
|
|
||||||
|
## Timing Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:--------------------------------------|------------:|------------:|---------:|----------:|--------:|
|
||||||
|
| timing/compute_advantages_and_returns | 0.0666617 | 0.0122347 | 0.0519 | 0.1531 | 81 |
|
||||||
|
| timing/convert_to_training_input | 4.72614 | 0.280405 | 3.6373 | 5.4807 | 81 |
|
||||||
|
| timing/fwd_logprobs_values_reward | 49.3127 | 5.50948 | 28.9658 | 61.4679 | 81 |
|
||||||
|
| timing/policy_train | 290.478 | 30.592 | 151.591 | 354.228 | 81 |
|
||||||
|
| timing/run_training | 340.145 | 35.2621 | 180.815 | 408.019 | 81 |
|
||||||
|
| timing/step | 872.574 | 400.434 | 507.833 | 3374.16 | 81 |
|
||||||
|
| timing/sync_weights | 20.0706 | 1.03089 | 17.0362 | 23.2754 | 81 |
|
||||||
|
| timing/train_critic_and_policy | 290.765 | 30.6031 | 151.737 | 354.553 | 81 |
|
||||||
|
| timing/wait_for_generation_buffer | 507.629 | 406.806 | 131.502 | 3072.09 | 81 |
|
||||||
|
| timing/cleanup_old_checkpoints | 2.02637 | 7.98054 | 0.0063 | 38.5904 | 40 |
|
||||||
|
| timing/save_checkpoints | 23.1826 | 62.9803 | 7.7944 | 396.822 | 40 |
|
||||||
|
| timing/save_hf_model | 6.37928 | 0.425326 | 5.9537 | 7.5975 | 16 |
|
||||||
|
|
||||||
|
## Trainer Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:--------------------|--------:|--------:|------:|------:|--------:|
|
||||||
|
| trainer/epoch | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| trainer/global_step | 40.1111 | 23.3559 | 1 | 80 | 81 |
|
||||||
|
|
||||||
|
## Batch_Errors Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:----------------------------------------------|------------:|--------------:|------------:|------------:|--------:|
|
||||||
|
| batch_errors/total_batches | 60.9136 | 8.52379 | 42 | 121 | 81 |
|
||||||
|
| batch_errors/total_instances | 487.309 | 68.1903 | 336 | 968 | 81 |
|
||||||
|
| batch_errors/total_successful | 403.728 | 74.9498 | 283 | 939 | 81 |
|
||||||
|
| batch_errors/total_failed | 19.8642 | 5.64525 | 5 | 32 | 81 |
|
||||||
|
| batch_errors/total_masked | 79.2716 | 25.8404 | 25 | 134 | 81 |
|
||||||
|
| batch_errors/avg_RewardFileNotFoundError | 0.531055 | 0.271125 | 0.03125 | 1.42593 | 81 |
|
||||||
|
| batch_errors/total_RewardFileNotFoundError | 31.8272 | 15.4934 | 2 | 77 | 81 |
|
||||||
|
| batch_errors/avg_DaytonaValidationError | 0.0672515 | 0.0494668 | 0.0106383 | 0.186441 | 24 |
|
||||||
|
| batch_errors/total_DaytonaValidationError | 4 | 2.87417 | 1 | 11 | 24 |
|
||||||
|
| batch_errors/avg_ContextLengthExceededError | 0.779734 | 0.294334 | 0.16129 | 1.55556 | 79 |
|
||||||
|
| batch_errors/total_ContextLengthExceededError | 46.9873 | 18.3376 | 10 | 98 | 79 |
|
||||||
|
| batch_errors/avg_RuntimeError | 0.0324855 | 0.023494 | 0.015625 | 0.101695 | 46 |
|
||||||
|
| batch_errors/total_RuntimeError | 1.97826 | 1.40616 | 1 | 6 | 46 |
|
||||||
|
| batch_errors/avg_DaytonaError | 0.0612569 | 0.0447787 | 0.0106383 | 0.206897 | 52 |
|
||||||
|
| batch_errors/total_DaytonaError | 3.65385 | 2.65599 | 1 | 12 | 52 |
|
||||||
|
| batch_errors/avg_AgentTimeoutError | 0.0810606 | 0.0876512 | 0.015625 | 0.625 | 66 |
|
||||||
|
| batch_errors/total_AgentTimeoutError | 4.74242 | 4.54147 | 1 | 30 | 66 |
|
||||||
|
| batch_errors/avg_VerifierTimeoutError | 0.0785312 | 0.05595 | 0.015873 | 0.225806 | 23 |
|
||||||
|
| batch_errors/total_VerifierTimeoutError | 4.65217 | 3.2975 | 1 | 14 | 23 |
|
||||||
|
| batch_errors/avg_VerifierOutputParseError | 0.137931 | 0 | 0.137931 | 0.137931 | 2 |
|
||||||
|
| batch_errors/total_VerifierOutputParseError | 8 | 0 | 8 | 8 | 2 |
|
||||||
|
| batch_errors/avg_DaytonaNotFoundError | 0.0559985 | 0.0495709 | 0.015873 | 0.145455 | 9 |
|
||||||
|
| batch_errors/total_DaytonaNotFoundError | 3.22222 | 2.68225 | 1 | 8 | 9 |
|
||||||
|
| batch_errors/avg_InvalidChatHistory | 0.0198765 | 0.00788726 | 0.015625 | 0.0338983 | 5 |
|
||||||
|
| batch_errors/total_InvalidChatHistory | 1.2 | 0.447214 | 1 | 2 | 5 |
|
||||||
|
| batch_errors/avg_DaytonaAuthenticationError | 0.0163291 | 0.000790025 | 0.015873 | 0.0172414 | 3 |
|
||||||
|
| batch_errors/total_DaytonaAuthenticationError | 1 | 0 | 1 | 1 | 3 |
|
||||||
|
| batch_errors/avg_RewardFileEmptyError | 0.015873 | nan | 0.015873 | 0.015873 | 1 |
|
||||||
|
| batch_errors/total_RewardFileEmptyError | 1 | nan | 1 | 1 | 1 |
|
||||||
|
| batch_errors/avg_AddTestsDirError | 0.015873 | nan | 0.015873 | 0.015873 | 1 |
|
||||||
|
| batch_errors/total_AddTestsDirError | 1 | nan | 1 | 1 | 1 |
|
||||||
|
| batch_errors/avg_AgentSetupTimeoutError | 0.206955 | 0.203236 | 0.016129 | 0.479167 | 4 |
|
||||||
|
| batch_errors/total_AgentSetupTimeoutError | 11 | 9.62635 | 1 | 23 | 4 |
|
||||||
|
| batch_errors/avg_OSError | 0.0164111 | 0.000760944 | 0.015873 | 0.0169492 | 2 |
|
||||||
|
| batch_errors/total_OSError | 1 | 0 | 1 | 1 | 2 |
|
||||||
|
|
||||||
|
## Training Progression by Log
|
||||||
|
|
||||||
|
### job_567548
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 1 | 0.5059 | 0.6250 | 0.000000 | -0.0000 | 1358.1 | 1155.0 |
|
||||||
|
| 2 | 0.5000 | 0.6719 | 0.000000 | 0.0000 | 833.5 | 586.8 |
|
||||||
|
| 3 | 0.4199 | 0.5938 | 0.000000 | 0.0000 | 703.1 | 332.8 |
|
||||||
|
| 4 | 0.4609 | 0.6406 | 0.000000 | 0.0000 | 924.0 | 548.2 |
|
||||||
|
| 5 | 0.3359 | 0.4531 | 0.000000 | -0.0001 | 609.8 | 230.0 |
|
||||||
|
| 6 | 0.3613 | 0.5000 | 0.000000 | -0.0000 | 528.5 | 145.4 |
|
||||||
|
| 7 | 0.3340 | 0.4844 | 0.000000 | -0.0000 | 589.5 | 248.8 |
|
||||||
|
| 8 | 0.4082 | 0.5312 | 0.000000 | -0.0000 | 605.9 | 228.5 |
|
||||||
|
| 9 | 0.2812 | 0.4844 | 0.000000 | 0.0000 | 687.8 | 372.3 |
|
||||||
|
|
||||||
|
### job_567550
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 9 | 0.3418 | 0.5556 | 0.000000 | 0.0000 | 3374.2 | 3072.1 |
|
||||||
|
| 10 | 0.3867 | 0.5938 | 0.000000 | 0.0000 | 1168.0 | 795.3 |
|
||||||
|
| 11 | 0.2988 | 0.4531 | 0.000000 | -0.0000 | 507.8 | 131.5 |
|
||||||
|
| 12 | 0.3145 | 0.4844 | 0.000000 | 0.0000 | 516.4 | 177.0 |
|
||||||
|
| 13 | 0.3301 | 0.4688 | 0.000000 | -0.0000 | 610.9 | 302.7 |
|
||||||
|
| 14 | 0.2559 | 0.4375 | 0.000000 | -0.0000 | 839.0 | 513.9 |
|
||||||
|
| 15 | 0.3613 | 0.5000 | 0.000000 | -0.0000 | 643.1 | 282.0 |
|
||||||
|
| 16 | 0.4551 | 0.5625 | 0.000000 | -0.0000 | 563.6 | 251.9 |
|
||||||
|
| 17 | 0.3164 | 0.4688 | 0.000000 | -0.0000 | 640.3 | 279.4 |
|
||||||
|
| 18 | 0.3438 | 0.4531 | 0.000000 | 0.0000 | 762.5 | 436.3 |
|
||||||
|
| 19 | 0.3184 | 0.4688 | 0.000000 | 0.0000 | 836.6 | 483.8 |
|
||||||
|
| 20 | 0.2637 | 0.3750 | 0.000000 | -0.0000 | 714.5 | 341.4 |
|
||||||
|
| 21 | 0.3750 | 0.5781 | 0.000000 | 0.0000 | 829.4 | 430.9 |
|
||||||
|
| 22 | 0.3418 | 0.5000 | 0.000000 | -0.0000 | 763.6 | 362.2 |
|
||||||
|
| 23 | 0.3262 | 0.5156 | 0.000000 | 0.0000 | 1049.2 | 675.6 |
|
||||||
|
| 24 | 0.2793 | 0.4219 | 0.000000 | 0.0000 | 704.3 | 389.0 |
|
||||||
|
| 25 | 0.3574 | 0.4844 | 0.000000 | 0.0000 | 868.9 | 495.4 |
|
||||||
|
| 26 | 0.3320 | 0.4688 | 0.000000 | -0.0000 | 854.4 | 467.6 |
|
||||||
|
| 27 | 0.3223 | 0.5000 | 0.000000 | 0.0000 | 590.1 | 255.9 |
|
||||||
|
| 28 | 0.3398 | 0.4844 | 0.000000 | -0.0000 | 880.2 | 456.9 |
|
||||||
|
| 29 | 0.3613 | 0.5625 | 0.000000 | 0.0000 | 884.7 | 516.6 |
|
||||||
|
| 30 | 0.3281 | 0.5156 | 0.000000 | 0.0000 | 1129.6 | 742.6 |
|
||||||
|
| 31 | 0.2891 | 0.3750 | 0.000000 | -0.0000 | 880.0 | 508.9 |
|
||||||
|
| 32 | 0.3418 | 0.5312 | 0.000000 | 0.0000 | 799.6 | 456.1 |
|
||||||
|
| 33 | 0.3105 | 0.4688 | 0.000000 | -0.0000 | 801.0 | 438.2 |
|
||||||
|
| 34 | 0.3457 | 0.5312 | 0.000000 | -0.0001 | 946.0 | 580.7 |
|
||||||
|
| 35 | 0.4668 | 0.6875 | 0.000000 | 0.0000 | 848.1 | 465.5 |
|
||||||
|
| 36 | 0.3984 | 0.5625 | 0.000000 | -0.0001 | 924.9 | 576.2 |
|
||||||
|
| 37 | 0.3438 | 0.5312 | 0.000000 | -0.0000 | 883.1 | 530.8 |
|
||||||
|
| 38 | 0.3926 | 0.5156 | 0.000000 | -0.0001 | 1456.7 | 1081.1 |
|
||||||
|
| 39 | 0.3887 | 0.5312 | 0.000000 | -0.0000 | 854.4 | 500.4 |
|
||||||
|
| 40 | 0.3066 | 0.4531 | 0.000000 | -0.0000 | 893.6 | 560.5 |
|
||||||
|
| 41 | 0.3281 | 0.4688 | 0.000000 | -0.0003 | 2041.8 | 1642.7 |
|
||||||
|
| 42 | 0.4082 | 0.5938 | 0.000000 | -0.0001 | 767.9 | 466.6 |
|
||||||
|
| 43 | 0.3691 | 0.4531 | 0.000000 | -0.0000 | 869.6 | 506.1 |
|
||||||
|
| 44 | 0.3926 | 0.5781 | 0.000000 | -0.0001 | 783.8 | 429.7 |
|
||||||
|
| 45 | 0.3242 | 0.4844 | 0.000000 | -0.0000 | 673.9 | 300.3 |
|
||||||
|
| 46 | 0.3359 | 0.5312 | 0.000000 | -0.0000 | 1024.1 | 663.9 |
|
||||||
|
| 47 | 0.3203 | 0.4844 | 0.000000 | 0.0000 | 1112.0 | 706.6 |
|
||||||
|
| 48 | 0.3535 | 0.5156 | 0.000000 | -0.0000 | 928.2 | 543.8 |
|
||||||
|
| 49 | 0.4023 | 0.5938 | 0.000000 | -0.0000 | 759.6 | 398.2 |
|
||||||
|
| 50 | 0.2656 | 0.3438 | 0.000000 | 0.0000 | 774.0 | 416.6 |
|
||||||
|
|
||||||
|
### job_567551
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 51 | 0.4785 | 0.6226 | 0.000000 | -0.0001 | 2184.4 | 1845.0 |
|
||||||
|
| 52 | 0.5586 | 0.6875 | 0.000000 | -0.0000 | 1651.9 | 1245.4 |
|
||||||
|
| 53 | 0.3867 | 0.5000 | 0.000000 | 0.0000 | 652.1 | 292.6 |
|
||||||
|
| 54 | 0.3047 | 0.5156 | 0.000000 | 0.0000 | 560.9 | 186.2 |
|
||||||
|
| 55 | 0.3281 | 0.4531 | 0.000000 | 0.0000 | 683.3 | 332.7 |
|
||||||
|
| 56 | 0.4629 | 0.5938 | 0.000000 | -0.0000 | 854.5 | 488.2 |
|
||||||
|
| 57 | 0.4434 | 0.5781 | 0.000000 | -0.0001 | 854.4 | 440.7 |
|
||||||
|
| 58 | 0.3535 | 0.5156 | 0.000000 | -0.0000 | 785.5 | 392.6 |
|
||||||
|
| 59 | 0.3613 | 0.5469 | 0.000000 | -0.0000 | 643.7 | 253.1 |
|
||||||
|
| 60 | 0.3848 | 0.5469 | 0.000000 | 0.0000 | 674.6 | 305.4 |
|
||||||
|
| 61 | 0.3457 | 0.5156 | 0.000000 | 0.0000 | 684.6 | 252.5 |
|
||||||
|
| 62 | 0.3867 | 0.5312 | 0.000000 | 0.0000 | 624.4 | 254.2 |
|
||||||
|
| 63 | 0.3574 | 0.5938 | 0.000000 | -0.0001 | 868.7 | 474.5 |
|
||||||
|
| 64 | 0.3066 | 0.4375 | 0.000000 | -0.0000 | 692.7 | 263.9 |
|
||||||
|
| 65 | 0.3711 | 0.5000 | 0.000000 | -0.0000 | 634.6 | 271.4 |
|
||||||
|
| 66 | 0.3906 | 0.5469 | 0.000000 | -0.0000 | 709.9 | 303.6 |
|
||||||
|
| 67 | 0.4883 | 0.6875 | 0.000000 | 0.0000 | 517.5 | 135.4 |
|
||||||
|
| 68 | 0.4043 | 0.6094 | 0.000000 | 0.0000 | 671.0 | 285.3 |
|
||||||
|
| 69 | 0.4531 | 0.6562 | 0.000000 | -0.0000 | 730.7 | 348.2 |
|
||||||
|
| 70 | 0.3359 | 0.5000 | 0.000000 | -0.0000 | 928.1 | 566.1 |
|
||||||
|
| 71 | 0.3750 | 0.4688 | 0.000000 | 0.0000 | 809.2 | 469.7 |
|
||||||
|
| 72 | 0.4609 | 0.6562 | 0.000000 | 0.0000 | 775.9 | 388.3 |
|
||||||
|
| 73 | 0.3965 | 0.5625 | 0.000000 | 0.0000 | 736.8 | 355.8 |
|
||||||
|
| 74 | 0.2988 | 0.3594 | 0.000000 | 0.0000 | 639.6 | 270.0 |
|
||||||
|
| 75 | 0.3301 | 0.4531 | 0.000000 | -0.0000 | 987.1 | 598.9 |
|
||||||
|
| 76 | 0.4746 | 0.5938 | 0.000000 | 0.0000 | 1003.0 | 623.9 |
|
||||||
|
| 77 | 0.3789 | 0.5625 | 0.000000 | -0.0000 | 827.6 | 440.6 |
|
||||||
|
| 78 | 0.3594 | 0.4688 | 0.000000 | -0.0000 | 1027.6 | 645.9 |
|
||||||
|
| 79 | 0.3223 | 0.4688 | 0.000000 | -0.0000 | 864.1 | 469.0 |
|
||||||
|
| 80 | 0.3770 | 0.5156 | 0.000000 | 0.0000 | 806.3 | 438.0 |
|
||||||
|
|
||||||
|
## Timing Analysis
|
||||||
|
|
||||||
|
### Average Time Breakdown (% of step time)
|
||||||
|
|
||||||
|
| Component | Avg % of Step Time |
|
||||||
|
|-----------|-------------------|
|
||||||
|
| wait_for_generation_buffer | 53.6% |
|
||||||
|
| run_training | 43.2% |
|
||||||
|
| train_critic_and_policy | 37.0% |
|
||||||
|
| policy_train | 36.9% |
|
||||||
|
| fwd_logprobs_values_reward | 6.3% |
|
||||||
|
| save_checkpoints | 2.7% |
|
||||||
|
| sync_weights | 2.6% |
|
||||||
|
| save_hf_model | 0.8% |
|
||||||
|
| convert_to_training_input | 0.6% |
|
||||||
|
| cleanup_old_checkpoints | 0.2% |
|
||||||
|
| compute_advantages_and_returns | 0.0% |
|
||||||
|
|
||||||
|
## Cross-Log Comparison
|
||||||
|
|
||||||
|
| Log | Avg Reward | Pass@8 | Step Time (s) | Gen Wait Time (s) | Avg Tokens | Staleness |
|
||||||
|
|-----|------|------|------|------|------|------|
|
||||||
|
| job_567548 | 0.4008 | 0.5538 | 760.0180 | 427.5351 | 5697.2672 | 2.0174 |
|
||||||
|
| job_567550 | 0.3437 | 0.5021 | 922.7027 | 562.6884 | 6171.1224 | 2.5595 |
|
||||||
|
| job_567551 | 0.3892 | 0.5416 | 836.1619 | 454.5744 | 6444.8237 | 2.6167 |
|
||||||
|
|
||||||
|
## vLLM Inference Engine Analysis
|
||||||
|
|
||||||
|
Metrics from vLLM stat loggers (V1LoggingStatLoggerFixed).
|
||||||
|
|
||||||
|
> **Note**: Ray deduplicates similar log messages with `[repeated Nx across cluster]`,
|
||||||
|
> so we typically capture stats from one engine per timestamp. The stats shown are
|
||||||
|
> **per-engine** values. Multiply by num_inference_engines for cluster-wide estimates.
|
||||||
|
|
||||||
|
### Summary by Log (Per-Engine Stats)
|
||||||
|
|
||||||
|
| Log | Avg Running/Engine | Avg Waiting/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % |
|
||||||
|
|-----|-------------------|-------------------|--------------------------|----------------|------------------|
|
||||||
|
| job_567548 | 8.7 | 0.0 | 383.7 tok/s | 36.5% | 91.9% |
|
||||||
|
| job_567550 | 7.6 | 0.0 | 347.5 tok/s | 33.3% | 92.4% |
|
||||||
|
| job_567551 | 9.0 | 0.0 | 399.0 tok/s | 39.7% | 91.1% |
|
||||||
|
|
||||||
|
### Utilization Analysis (Per-Engine)
|
||||||
|
|
||||||
|
Key indicators of inference engine utilization:
|
||||||
|
|
||||||
|
- **Running requests/engine**: Concurrent requests being processed by each engine
|
||||||
|
- **Waiting requests**: Requests queued (0 = engine not saturated, has spare capacity)
|
||||||
|
- **Generation throughput**: Decode tokens/sec per engine
|
||||||
|
- 8B model on H100 can do **1000+ tok/s** when saturated
|
||||||
|
- If seeing <300 tok/s with 0 waiting, engine is **starved for requests**
|
||||||
|
|
||||||
|
#### job_567548
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=8.7, max=22
|
||||||
|
- **Waiting requests**: avg=0.0, max=4
|
||||||
|
- **Generation throughput/engine**: avg=383.7 tok/s, max=790.8 tok/s
|
||||||
|
- **KV cache usage**: avg=36.5%
|
||||||
|
- **Prefix cache hit rate**: avg=91.9%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
#### job_567550
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=7.6, max=24
|
||||||
|
- **Waiting requests**: avg=0.0, max=10
|
||||||
|
- **Generation throughput/engine**: avg=347.5 tok/s, max=878.7 tok/s
|
||||||
|
- **KV cache usage**: avg=33.3%
|
||||||
|
- **Prefix cache hit rate**: avg=92.4%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
#### job_567551
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=9.0, max=24
|
||||||
|
- **Waiting requests**: avg=0.0, max=19
|
||||||
|
- **Generation throughput/engine**: avg=399.0 tok/s, max=851.4 tok/s
|
||||||
|
- **KV cache usage**: avg=39.7%
|
||||||
|
- **Prefix cache hit rate**: avg=91.1%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
## Trial-Level Analysis (from result.json)
|
||||||
|
|
||||||
|
Total trials parsed: 44526
|
||||||
|
|
||||||
|
### Turn Count Statistics
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| Mean | 11.5 |
|
||||||
|
| Median | 9.0 |
|
||||||
|
| Std | 8.2 |
|
||||||
|
| Min | 1 |
|
||||||
|
| Max | 116 |
|
||||||
|
| Count | 44464 |
|
||||||
|
|
||||||
|
### Exception Distribution
|
||||||
|
|
||||||
|
| Exception Type | Count | % |
|
||||||
|
|---------------|-------|---|
|
||||||
|
| No exception | 35187 | 79.0% |
|
||||||
|
| ContextLengthExceededError | 3811 | 8.6% |
|
||||||
|
| RewardFileNotFoundError | 2794 | 6.3% |
|
||||||
|
| AgentTimeoutError | 2128 | 4.8% |
|
||||||
|
| DaytonaError | 204 | 0.5% |
|
||||||
|
| VerifierTimeoutError | 112 | 0.3% |
|
||||||
|
| DaytonaValidationError | 103 | 0.2% |
|
||||||
|
| RuntimeError | 93 | 0.2% |
|
||||||
|
| AgentSetupTimeoutError | 44 | 0.1% |
|
||||||
|
| DaytonaNotFoundError | 29 | 0.1% |
|
||||||
|
| VerifierOutputParseError | 16 | 0.0% |
|
||||||
|
| DaytonaAuthenticationError | 3 | 0.0% |
|
||||||
|
| RewardFileEmptyError | 1 | 0.0% |
|
||||||
|
| AddTestsDirError | 1 | 0.0% |
|
||||||
|
|
||||||
|
### Turn Count by Exception Type
|
||||||
|
|
||||||
|
| Exception Type | Mean Turns | Median Turns | Count |
|
||||||
|
|---------------|-----------|-------------|-------|
|
||||||
|
| RewardFileEmptyError | 21.0 | 21.0 | 1 |
|
||||||
|
| AddTestsDirError | 19.0 | 19.0 | 1 |
|
||||||
|
| AgentTimeoutError | 17.8 | 11.0 | 2128 |
|
||||||
|
| ContextLengthExceededError | 16.4 | 13.0 | 3811 |
|
||||||
|
| VerifierTimeoutError | 13.9 | 11.5 | 112 |
|
||||||
|
| RewardFileNotFoundError | 13.2 | 12.0 | 2794 |
|
||||||
|
| DaytonaAuthenticationError | 12.7 | 7.0 | 3 |
|
||||||
|
| VerifierOutputParseError | 11.5 | 10.0 | 16 |
|
||||||
|
| DaytonaError | 11.2 | 8.0 | 186 |
|
||||||
|
| RuntimeError | 10.7 | 10.0 | 93 |
|
||||||
|
| No exception | 10.4 | 8.0 | 35187 |
|
||||||
|
| DaytonaValidationError | 8.5 | 7.0 | 103 |
|
||||||
|
| DaytonaNotFoundError | 8.0 | 6.0 | 29 |
|
||||||
|
|
||||||
|
### Turn Count by Outcome
|
||||||
|
|
||||||
|
| Outcome | Mean Turns | Median Turns | Count |
|
||||||
|
|---------|-----------|-------------|-------|
|
||||||
|
| Success | 9.8 | 8.0 | 16178 |
|
||||||
|
| Failure | 11.6 | 9.0 | 20820 |
|
||||||
|
|
||||||
|
### Reward Summary
|
||||||
|
|
||||||
|
- Mean reward: 0.4373
|
||||||
|
- Success rate: 43.7%
|
||||||
|
- Trials with reward data: 36998
|
||||||
|
|
||||||
82
training_logs/20260604_123153_metrics_table.csv
Normal file
82
training_logs/20260604_123153_metrics_table.csv
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
log_file,async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RewardFileNotFoundError,batch_errors/total_RewardFileNotFoundError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,timing/save_hf_model,batch_errors/avg_VerifierOutputParseError,batch_errors/total_VerifierOutputParseError,batch_errors/avg_DaytonaNotFoundError,batch_errors/total_DaytonaNotFoundError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_RewardFileEmptyError,batch_errors/total_RewardFileEmptyError,batch_errors/avg_AddTestsDirError,batch_errors/total_AddTestsDirError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_OSError,batch_errors/total_OSError,global_step
|
||||||
|
job_567548,0.0,0,64,512,0,0.0,0,0.0,3631.0469,3316.3012,3953.2569,25303,1,3243.0733,0.5059,-0.0087,0.0861,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.154,-0.0,0.0,1.0,0.0,0.0146,0.625,0.5059,10.7936,50.364,633.8085,26.1,857.9676,224.1591,0.1114,3.6373,28.9658,151.5911,180.8148,1358.0963,18.6186,151.7373,1155.0253,0,1,121,968,939,6,29,0.30578512396694213,37,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,1
|
||||||
|
job_567548,0.0,0,64,512,1,0.7969,0,0.7969,4822.3672,4358.4062,5286.3281,25740,1,3816.7518,0.5,-0.0131,0.1439,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1522,0.0,0.0,1.0,0.0,0.0203,0.6719,0.5,11.4775,50.4393,624.6882,27.2,857.9676,233.2794,0.0519,4.284,32.9178,189.316,222.548,833.5217,19.8811,189.578,586.8065,0,2,58,464,439,5,25,0.43103448275862066,25,38.5904,48.4066,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2
|
||||||
|
job_567548,0.0,0,64,512,2,1.3438,0,0.875,6234.4707,5933.786,6452.138,29898,1,5603.3019,0.4199,0.0118,0.1815,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1483,0.0,0.0,1.0,0.0,0.019,0.5938,0.4199,13.2484,50.647,630.5422,26.5,857.9676,227.4254,0.0614,4.581,59.1585,286.7886,346.2501,703.0723,19.4633,287.0298,332.7757,0,3,62,496,461,8,28,0.2903225806451613,18,,,0.11290322580645161,7.0,0.16129032258064516,10.0,,,,,,,,,,,,,,,,,,,,,,,,,,3
|
||||||
|
job_567548,0.0,0,64,512,3,1.7031,0,0.8438,6754.2012,6667.6059,6828.2464,30918,1,6199.201,0.4609,-0.0051,0.1719,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1546,0.0,0.0,1.0,0.0,0.0208,0.6406,0.4609,13.3757,50.8519,626.7498,26.9,857.9676,231.2178,0.0647,4.8313,51.8113,299.0531,351.1909,923.9643,19.7377,299.3145,548.2019,0,4,54,432,310,24,117,1.4259259259259258,77,0.0067,11.238,0.018518518518518517,1.0,1.0555555555555556,57.0,0.018518518518518517,1.0,0.07407407407407407,4.0,0.018518518518518517,1.0,0.05555555555555555,3.0,,,,,,,,,,,,,,,,,,4
|
||||||
|
job_567548,0.0,0,64,512,4,2.6406,1,1.0,6037.8184,7489.2791,5303.55,31319,1,7148.2511,0.3359,0.0173,0.1494,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1177,-0.0041,0.0,1.0,0.0,0.0141,0.4531,0.3359,13.5831,51.1094,624.3487,27.2,857.9676,233.6189,0.0632,4.8082,51.5733,303.1749,355.0612,609.8127,19.9157,303.4245,230.0243,0,5,58,464,389,20,75,0.3793103448275862,22,,,,,1.0,58.0,,,,,0.05172413793103448,3.0,,,7.5975,0.13793103448275862,8.0,,,,,,,,,,,,,,,5
|
||||||
|
job_567548,0.0,0,64,512,5,3.0469,1,1.0,6446.5898,7161.3189,6042.2324,31463,1,6726.1259,0.3613,0.0237,0.1225,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1422,-0.0,0.0,1.0,0.0,0.0174,0.5,0.3613,14.5126,51.5042,634.199,26.1,857.9676,223.7686,0.0641,4.8218,47.3073,311.2799,358.8908,528.5257,19.4415,311.5191,145.3683,0,6,55,440,378,14,46,0.2909090909090909,16,0.0069,9.3271,,,0.43636363636363634,24.0,,,0.10909090909090909,6.0,0.03636363636363636,2.0,0.14545454545454545,8.0,,,,0.14545454545454545,8.0,,,,,,,,,,,,,6
|
||||||
|
job_567548,0.0,0,64,512,6,2.8438,1,1.0,5989.127,6446.3918,5759.824,30379,1,5927.6954,0.334,0.0071,0.2061,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1469,-0.0028,0.0,1.0,0.0,0.0169,0.4844,0.334,14.6947,51.7031,630.6147,26.5,857.9676,227.3529,0.0656,4.6023,46.2763,268.8931,315.4526,589.5032,20.6408,269.1104,248.8046,0,7,53,424,336,15,87,1.3018867924528301,69,,,,,0.4339622641509434,23.0,0.018867924528301886,1.0,0.018867924528301886,1.0,0.03773584905660377,2.0,,,,,,,,,,,,,,,,,,,,7
|
||||||
|
job_567548,0.0,0,64,512,7,2.8125,0,0.9375,5809.7773,6972.8086,5007.5545,30602,1,6322.6044,0.4082,0.0127,0.1764,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1395,-0.0,0.0,1.0,0.0,0.0182,0.5312,0.4082,14.9265,51.8948,629.9464,26.6,857.9676,228.0212,0.0612,5.115,50.7488,301.5099,352.5549,605.9063,19.7309,301.7446,228.5022,0,8,60,480,417,18,63,0.3,18,0.0077,9.0045,,,0.6333333333333333,38.0,,,,,0.11666666666666667,7.0,,,,,,,,,,,,,,,,,,,,8
|
||||||
|
job_567548,0.0,0,64,512,7,2.9688,0,0.9688,5550.0059,5062.7014,5740.6902,27434,1,5458.1306,0.2812,0.0007,0.1828,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.147,0.0,0.0,1.0,0.0,0.0205,0.4844,0.2812,14.893,51.924,626.8531,26.9,857.9676,231.1145,0.0634,4.1708,43.4535,247.6858,291.4212,687.7597,19.8574,247.9039,372.3071,0,9,57,456,384,18,70,0.5789473684210527,33,,,,,0.6491228070175439,37.0,,,,,0.10526315789473684,6.0,0.12280701754385964,7.0,,,,,,,,,,,,,,,,,,9
|
||||||
|
job_567550,0.0,0,64,512,7,1.625,0,0.5625,5094.9453,5220.4171,5029.7893,26213,1,4943.8339,0.3418,-0.0026,0.1557,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.153,0.0,0.0,1.0,0.0,0.0177,0.5556,0.3418,10.6846,50.2918,613.6667,28.5,857.9676,244.3008,0.1008,4.0497,40.2665,237.4547,278.0636,3374.1597,19.9528,237.6955,3072.0934,0,9,63,504,448,14,56,0.5555555555555556,35,,,,,0.2857142857142857,18.0,0.031746031746031744,2.0,0.09523809523809523,6.0,,,,,,,,,,,,,,,,,,,,,,9
|
||||||
|
job_567550,0.0,0,64,512,1,0.7188,0,0.7188,6813.5703,6589.7525,6954.7038,31364,1,6199.8594,0.3867,0.0081,0.2078,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1451,0.0,0.0,1.0,0.0,0.0235,0.5938,0.3867,12.6901,50.5284,634.8915,26.0,857.9676,223.076,0.0696,4.7781,51.6722,295.8816,347.9268,1168.0124,20.0205,296.1843,795.2757,0,10,63,504,379,32,125,0.23809523809523808,15,9.5233,17.5722,,,1.5555555555555556,98.0,,,0.09523809523809523,6.0,0.09523809523809523,6.0,,,6.5298,,,,,0.015873015873015872,1.0,,,,,,,,,,,10
|
||||||
|
job_567550,0.0,0,64,512,2,1.7344,1,1.0,6021.1484,7335.5163,5460.9861,31521,1,6627.1211,0.2988,0.0105,0.1794,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0,0.0,1.0,0.0,0.0178,0.4531,0.2988,14.0168,50.7668,629.5027,26.6,857.9676,228.4648,0.0642,4.7993,49.8186,302.0916,352.2082,507.8331,19.3212,302.3244,131.5017,0,11,56,448,322,27,113,0.42857142857142855,24,,,0.14285714285714285,8.0,1.3214285714285714,74.0,,,0.07142857142857142,4.0,0.30357142857142855,17.0,0.08928571428571429,5.0,,,,,,0.017857142857142856,1.0,,,,,,,,,,,11
|
||||||
|
job_567550,0.0,0,64,512,3,2.2344,1,1.0,5205.8301,5815.7826,4926.0513,30274,1,5856.7327,0.3145,0.0052,0.192,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1282,0.0,0.0,1.0,0.0,0.0171,0.4844,0.3145,14.0573,50.8443,626.9561,26.9,857.9676,231.0115,0.0618,4.469,43.7187,271.5448,315.6215,516.3754,19.2857,271.8407,176.9956,0,12,59,472,412,10,47,0.423728813559322,25,0.0067,8.1886,0.1864406779661017,11.0,0.3389830508474576,20.0,,,0.01694915254237288,1.0,0.06779661016949153,4.0,0.03389830508474576,2.0,,,,,,,,,,,,,,,,,,12
|
||||||
|
job_567550,0.0,0,64,512,4,2.2656,0,0.9844,5512.916,4847.4852,5840.7813,29622,1,5117.2995,0.3301,-0.0116,0.1489,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1537,-0.0016,0.0,1.0,0.0,0.0186,0.4688,0.3301,14.0911,50.8562,633.3426,26.2,857.9676,224.625,0.0627,4.4482,41.2335,241.7355,283.3967,610.8919,20.3642,242.1,302.6795,0,13,63,504,429,14,75,0.6349206349206349,40,,,,,0.49206349206349204,31.0,,,0.015873015873015872,1.0,,,,,,,,,,,,,,,,,,,,,,13
|
||||||
|
job_567550,0.0,0,64,512,5,2.5,0,0.9219,5537.5039,5509.1908,5547.2388,29312,1,5324.5857,0.2559,0.0061,0.1565,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1449,-0.0,0.0,1.0,0.0,0.0173,0.4375,0.2559,14.1559,51.0214,629.3403,26.6,857.9676,228.6273,0.0613,4.4394,47.8807,251.6916,299.9619,838.9599,20.6381,252.0195,513.9169,0,14,59,472,380,18,89,0.7966101694915254,47,0.0073,8.3961,0.03389830508474576,2.0,0.6440677966101694,38.0,,,0.1016949152542373,6.0,0.03389830508474576,2.0,0.01694915254237288,1.0,,,,,,,,,,,,,,,,,,14
|
||||||
|
job_567550,0.0,0,64,512,6,2.8281,1,1.0,5926.9746,6772.3189,5448.7217,30621,1,6058.2297,0.3613,0.0123,0.1417,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1363,-0.0,0.0,1.0,0.0,0.0151,0.5,0.3613,14.5139,51.3657,625.6211,27.1,857.9676,232.3465,0.0635,4.7605,53.283,282.2832,335.9143,643.0671,20.3593,282.5675,282.0282,0,15,62,496,426,16,68,0.532258064516129,33,,,0.0967741935483871,6.0,0.5645161290322581,35.0,0.03225806451612903,2.0,0.016129032258064516,1.0,0.016129032258064516,1.0,,,6.5721,,,,,,,,,,,,,,,,,15
|
||||||
|
job_567550,0.0,0,64,512,6,2.7812,0,0.9844,5693.7383,5905.8841,5516.5699,30109,1,5394.2192,0.4551,0.0032,0.1188,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1362,-0.0028,0.0,1.0,0.0,0.0153,0.5625,0.4551,14.5472,51.3824,631.8962,26.3,857.9676,226.0714,0.0636,4.962,41.7927,245.157,287.3077,563.5944,19.3884,245.451,251.9322,0,16,55,440,373,15,60,0.8363636363636363,46,0.0071,7.7944,,,0.7090909090909091,39.0,,,0.01818181818181818,1.0,0.03636363636363636,2.0,,,,,,0.12727272727272726,7.0,,,,,,,,,,,,,16
|
||||||
|
job_567550,0.0,0,64,512,7,3.1094,0,0.9531,5991.3418,6421.2284,5792.3657,30607,1,6045.6308,0.3164,0.0151,0.1259,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1354,-0.002,0.0,1.0,0.0,0.0168,0.4688,0.3164,14.6639,51.4677,628.2541,26.8,857.9676,229.7135,0.0634,4.6239,46.0697,288.584,335.0408,640.3003,21.2728,288.9073,279.3589,0,17,64,512,447,13,64,0.234375,15,,,,,0.734375,47.0,0.015625,1.0,,,0.03125,2.0,,,,,,,,,,,,,,,,,,,,17
|
||||||
|
job_567550,0.0,0,64,512,8,2.9844,0,0.9531,5745.5977,5801.2898,5716.4256,31324,1,5631.7408,0.3438,-0.0059,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.145,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3438,14.9227,51.7915,634.3629,26.1,857.9676,223.6047,0.0641,4.7138,46.2768,255.4429,302.0771,762.5347,19.4478,255.7356,436.2918,0,18,61,488,391,23,89,0.5409836065573771,33,0.0066,8.8114,,,0.9508196721311475,58.0,,,0.01639344262295082,1.0,,,0.13114754098360656,8.0,,,,,,,,,,,,,,,,,,18
|
||||||
|
job_567550,0.0,0,64,512,7,3.0,0,0.9531,5766.0996,7272.9571,5062.3238,30727,1,6005.4448,0.3184,-0.0053,0.175,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1368,0.0,0.0,1.0,0.0,0.0196,0.4688,0.3184,15.2079,52.0967,629.7617,26.6,857.9676,228.2059,0.0638,4.6698,46.7327,281.1661,328.289,836.6342,19.8892,281.4922,483.782,0,19,57,456,384,19,67,0.2807017543859649,16,,,,,0.8421052631578947,48.0,0.08771929824561403,5.0,0.07017543859649122,4.0,0.03508771929824561,2.0,,,,,,,,,,,,,,,,,,,,19
|
||||||
|
job_567550,0.0,0,64,512,7,2.9375,0,0.9688,6294.1797,5124.1037,6713.1724,31096,1,6469.7493,0.2637,-0.0072,0.102,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1463,-0.0,0.0,1.0,0.0,0.0126,0.375,0.2637,15.1567,52.0453,626.5997,27.0,857.9676,231.3679,0.064,4.7232,47.7824,301.0734,349.1957,714.4801,19.1506,301.3489,341.4063,0,20,59,472,416,15,56,0.6440677966101694,38,0.0069,8.1983,,,0.4067796610169492,24.0,,,0.01694915254237288,1.0,0.01694915254237288,1.0,,,6.0643,,,,,,,,,,,,,,,,,20
|
||||||
|
job_567550,0.0,0,64,512,7,2.8438,0,0.9688,6685.3047,6935.2396,6535.3438,30521,1,6310.5369,0.375,-0.0004,0.2167,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1433,0.0,0.0,1.0,0.0,0.0202,0.5781,0.375,15.1234,52.0566,624.2105,27.2,857.9676,233.7571,0.0659,4.7801,49.366,325.1266,374.9429,829.4379,18.844,325.5106,430.8666,0,21,59,472,386,24,83,0.4406779661016949,26,,,,,1.0169491525423728,60.0,0.05084745762711865,3.0,0.05084745762711865,3.0,0.0847457627118644,5.0,,,,,,,,,,,,,,,,,,,,21
|
||||||
|
job_567550,0.0,0,64,512,7,3.2188,0,0.9844,6705.4805,8304.1943,5875.2878,30102,1,6467.5812,0.3418,0.0187,0.1562,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1426,-0.0023,0.0,1.0,0.0,0.0149,0.5,0.3418,15.2092,52.1088,633.8773,26.1,857.9676,224.0903,0.0619,4.578,57.091,320.4391,377.8433,763.5965,18.9387,320.6899,362.2325,0,22,61,488,388,24,99,0.7704918032786885,47,0.0065,8.521,,,0.7868852459016393,48.0,0.01639344262295082,1.0,0.06557377049180328,4.0,0.11475409836065574,7.0,,,,,,,,,,,,,,,,,,,,22
|
||||||
|
job_567550,0.0,0,64,512,6,2.6562,0,0.9375,6630.8789,7731.1437,6098.287,29620,1,6953.1804,0.3262,0.0118,0.1969,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.0164,0.5156,0.3262,15.3535,52.4285,629.1158,26.7,857.9676,228.8517,0.0627,4.6368,49.5567,299.36,349.2749,1049.1967,19.6531,299.655,675.6279,0,23,57,456,345,23,111,0.7719298245614035,44,,,,,1.1403508771929824,65.0,0.017543859649122806,1.0,0.05263157894736842,3.0,0.017543859649122806,1.0,,,,,,,,,,,,,,,,,,,,23
|
||||||
|
job_567550,0.0,0,64,512,6,2.7344,0,0.9375,5392.7832,7495.951,4577.7344,30573,1,5638.6051,0.2793,0.0003,0.144,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,0.0,0.0,1.0,0.0,0.0156,0.4219,0.2793,15.7614,52.8214,625.9011,27.0,857.9676,232.0665,0.0632,4.5896,41.1111,249.969,291.4409,704.2569,19.2507,250.2661,388.972,0,24,61,488,405,23,83,0.3770491803278688,23,0.0075,8.5429,,,0.8852459016393442,54.0,,,,,0.09836065573770492,6.0,,,,,,,,,,,,,,,,,,,,24
|
||||||
|
job_567550,0.0,0,64,512,6,2.7031,0,0.9219,6406.502,6468.0984,6372.2401,29340,1,6506.3871,0.3574,0.0165,0.1367,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1401,0.0,0.0,1.0,0.0,0.0138,0.4844,0.3574,15.8598,53.0487,623.2285,27.4,857.9676,234.7391,0.0616,4.7813,52.9717,295.9893,349.3116,868.8734,19.3516,296.2778,495.425,0,25,62,496,400,24,80,0.3870967741935484,24,,,,,0.8225806451612904,51.0,0.03225806451612903,2.0,0.0967741935483871,6.0,0.03225806451612903,2.0,0.22580645161290322,14.0,6.535,,,,,,,,,,,,,,,,,25
|
||||||
|
job_567550,0.0,0,64,512,6,2.5625,0,0.9688,6405.3945,7275.6353,5972.8187,30302,1,6438.8941,0.332,-0.0077,0.1465,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1377,-0.002,0.0,1.0,0.0,0.0135,0.4688,0.332,15.8798,53.1326,620.4645,27.7,857.9676,237.5031,0.0646,4.7998,53.7144,307.7867,361.8342,854.4198,20.2015,308.0548,467.581,0,26,57,456,362,19,93,0.8771929824561403,50,0.007,8.6985,,,0.6491228070175439,37.0,,,0.10526315789473684,6.0,0.14035087719298245,8.0,,,,,,0.017543859649122806,1.0,,,,,,,,,,,,,26
|
||||||
|
job_567550,0.0,0,64,512,6,2.5625,1,1.0,5791.3965,7450.7455,5002.3689,28228,1,5710.9576,0.3223,0.0079,0.2159,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1355,0.0,0.0,1.0,0.0,0.0175,0.5,0.3223,15.8962,53.1489,618.6345,27.9,857.9676,239.3331,0.0587,4.2706,44.88,262.209,307.4009,590.0799,22.5095,262.4618,255.8951,0,27,60,480,415,20,60,0.38333333333333336,23,,,,,0.45,27.0,0.016666666666666666,1.0,0.13333333333333333,8.0,0.03333333333333333,2.0,0.016666666666666666,1.0,,,,0.05,3.0,,,,,,,,,,,,,27
|
||||||
|
job_567550,0.0,0,64,512,6,2.9062,0,0.9688,7508.9941,6576.2874,7989.145,30784,1,6953.4419,0.3398,0.0043,0.1576,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1452,-0.0003,0.0,1.0,0.0,0.017,0.4844,0.3398,15.825,53.0919,616.7122,28.1,857.9676,241.2554,0.0646,4.8023,57.3471,341.847,399.5664,880.2184,18.9742,342.1543,456.8715,0,28,63,504,420,21,80,0.36507936507936506,23,0.0069,8.6398,,,0.8412698412698413,53.0,0.047619047619047616,3.0,0.15873015873015872,10.0,0.031746031746031744,2.0,,,,,,0.015873015873015872,1.0,,,,,,,,,,,,,28
|
||||||
|
job_567550,0.0,0,64,512,6,2.7812,0,0.9531,6654.498,7316.7135,6279.8502,31394,1,6506.5919,0.3613,0.0096,0.1337,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1404,0.0,0.0,1.0,0.0,0.0162,0.5625,0.3613,15.9013,53.1658,614.1317,28.4,857.9676,243.8359,0.0636,4.839,48.4378,295.3489,344.1274,884.7489,19.1329,295.6257,516.6459,0,29,62,496,402,23,86,0.6451612903225806,40,,,0.11290322580645161,7.0,0.7741935483870968,48.0,0.016129032258064516,1.0,0.03225806451612903,2.0,0.016129032258064516,1.0,,,,,,,,,,,,,,,,,,,,29
|
||||||
|
job_567550,0.0,0,64,512,6,2.5,0,0.9219,6475.6191,7192.9286,6125.3052,31134,1,6725.433,0.3281,0.0109,0.1896,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1312,0.0,0.0,1.0,0.0,0.0199,0.5156,0.3281,16.001,53.2318,611.9695,28.7,857.9676,245.9981,0.0673,4.7096,51.1201,311.19,362.6769,1129.64,19.6246,311.4892,742.6236,0,30,63,504,405,26,98,0.4603174603174603,29,0.0072,8.4626,,,0.9523809523809523,60.0,,,0.09523809523809523,6.0,0.047619047619047616,3.0,,,6.0549,,,0.015873015873015872,1.0,,,,,,,,,,,,,30
|
||||||
|
job_567550,0.0,0,64,512,5,2.2812,0,0.9531,6346.0625,7175.0743,6008.9918,30375,1,6325.3681,0.2891,-0.0015,0.1034,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0129,0.375,0.2891,16.219,53.4149,609.7183,28.9,857.9676,248.2493,0.0644,4.6585,49.8638,296.7068,346.9154,880.0104,19.5517,296.9867,508.8808,0,31,63,504,397,27,100,0.4126984126984127,26,,,0.09523809523809523,6.0,1.0158730158730158,64.0,,,0.07936507936507936,5.0,0.07936507936507936,5.0,,,,,,,,,,0.015873015873015872,1.0,,,,,,,,,31
|
||||||
|
job_567550,0.0,0,64,512,6,2.5938,0,0.9688,5907.5469,7252.2229,5209.273,29747,1,6251.0608,0.3418,-0.005,0.1779,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1317,0.0,0.0,1.0,0.0,0.0177,0.5312,0.3418,16.2711,53.4052,607.4507,29.2,857.9676,250.5169,0.0626,4.4911,42.321,277.0021,319.666,799.5796,19.3098,277.2821,456.1088,0,32,60,480,371,23,104,0.7,42,0.0066,8.9378,0.08333333333333333,5.0,0.95,57.0,,,0.1,6.0,0.11666666666666667,7.0,,,,,,,,,,,,,,,,,,,,32
|
||||||
|
job_567550,0.0,0,64,512,5,2.3906,0,0.9531,5799.2324,6060.1132,5681.7252,31737,1,6414.7137,0.3105,0.0079,0.1257,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1294,-0.002,0.0,1.0,0.0,0.0178,0.4688,0.3105,16.2997,53.5626,632.3168,26.3,857.9676,225.6508,0.0662,4.8155,53.5314,284.6735,338.5434,801.0326,19.4236,284.9454,438.2467,0,33,59,472,364,25,100,0.4745762711864407,28,,,,,1.0508474576271187,62.0,0.01694915254237288,1.0,0.0847457627118644,5.0,0.1016949152542373,6.0,0.11864406779661017,7.0,,,,,,,,,,,,,,,,,,33
|
||||||
|
job_567550,0.0,0,64,512,6,2.6719,0,0.9531,6310.3008,7310.4972,5781.8388,29761,1,6110.0445,0.3457,0.0082,0.2142,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0033,0.0,1.0,0.0,0.0179,0.5312,0.3457,16.6979,53.8992,628.2749,26.8,857.9676,229.6927,0.0621,4.5025,50.7853,290.2551,341.3921,946.0027,19.3899,290.5443,580.7139,0,34,63,504,435,21,67,0.31746031746031744,20,0.0073,8.4926,,,0.6825396825396826,43.0,0.031746031746031744,2.0,0.09523809523809523,6.0,,,,,,,,,,,,,,,,,,,,,,34
|
||||||
|
job_567550,0.0,0,64,512,5,2.8281,0,0.9531,6915.877,7142.4017,6717.5641,31178,1,5996.8799,0.4668,0.0027,0.2488,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1465,0.0,0.0,1.0,0.0,0.0205,0.6875,0.4668,16.7466,53.9885,624.4393,27.2,857.9676,233.5283,0.0654,5.2906,51.1954,306.1635,357.6744,848.0649,19.5558,306.4133,465.5401,0,35,59,472,397,19,69,0.3728813559322034,22,,,0.01694915254237288,1.0,0.5932203389830508,35.0,0.1016949152542373,6.0,0.13559322033898305,8.0,0.1016949152542373,6.0,,,6.2574,,,,,,,,,,,,,,,,,35
|
||||||
|
job_567550,0.0,0,64,512,6,2.5156,0,0.8281,6071.5781,6369.9853,5873.9318,29647,1,5995.9806,0.3984,0.0309,0.1754,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1427,-0.0036,0.0,1.0,0.0,0.0172,0.5625,0.3984,16.6372,53.838,621.8133,27.5,857.9676,236.1543,0.063,4.5239,46.5181,276.1477,323.0187,924.8781,21.0895,276.4371,576.2418,0,36,63,504,422,19,79,0.1746031746031746,11,0.0073,9.0132,,,0.9365079365079365,59.0,0.031746031746031744,2.0,0.06349206349206349,4.0,0.06349206349206349,4.0,,,,,,,,,,,,0.015873015873015872,1.0,0.015873015873015872,1.0,,,,,36
|
||||||
|
job_567550,0.0,0,64,512,5,2.625,0,0.9375,6053.6152,6609.8693,5762.244,29949,1,5973.1572,0.3438,-0.0022,0.1581,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1378,-0.0003,0.0,1.0,0.0,0.0201,0.5312,0.3438,16.8303,54.0954,619.2038,27.8,857.9676,238.7638,0.0621,4.5581,46.3663,282.1342,328.8628,883.0955,18.8513,282.434,530.8195,0,37,58,464,374,27,84,0.5862068965517241,34,,,,,0.896551724137931,52.0,0.034482758620689655,2.0,0.20689655172413793,12.0,0.05172413793103448,3.0,0.06896551724137931,4.0,,,,,,,,,,,,,,,,,,37
|
||||||
|
job_567550,0.0,0,64,512,5,2.9062,0,0.9062,6101.7637,7142.1095,5429.3859,31890,1,6539.1202,0.3926,0.0027,0.1491,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0066,0.0,1.0,0.0,0.0133,0.5156,0.3926,16.8694,54.1906,616.2216,28.2,857.9676,241.746,0.0685,4.8885,48.6177,301.233,350.2252,1456.7333,20.4702,301.5387,1081.1455,0,38,62,496,448,18,43,0.11290322580645161,7,0.0071,8.2664,,,0.5,31.0,0.016129032258064516,1.0,0.04838709677419355,3.0,0.016129032258064516,1.0,0.04838709677419355,3.0,,,,,,0.016129032258064516,1.0,,,,,,,0.016129032258064516,1.0,,,38
|
||||||
|
job_567550,0.0,0,64,512,5,2.625,0,0.9375,6800.1094,7008.201,6667.8083,29586,1,5671.6318,0.3887,0.0097,0.1379,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1503,-0.002,0.0,1.0,0.0,0.015,0.5312,0.3887,16.8346,54.2173,614.385,28.4,857.9676,243.5826,0.0628,4.5977,45.5648,283.358,329.2728,854.372,20.0914,283.6447,500.4064,0,39,62,496,407,21,86,0.12903225806451613,8,,,0.016129032258064516,1.0,1.0161290322580645,63.0,,,0.08064516129032258,5.0,0.1774193548387097,11.0,0.016129032258064516,1.0,,,,0.016129032258064516,1.0,,,,,,,,,,,,,39
|
||||||
|
job_567550,0.0,0,64,512,5,2.2969,0,0.8438,5812.7578,6822.4395,5366.2225,28547,1,5777.0387,0.3066,0.0021,0.1644,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.136,-0.0,0.0,1.0,0.0,0.0178,0.4531,0.3066,16.8475,54.2062,612.4843,28.6,857.9676,245.4833,0.0586,4.4393,42.9655,268.3647,311.628,893.5759,17.0362,268.6034,560.4691,0,40,48,384,283,30,83,0.6458333333333334,31,0.0068,396.8219,0.125,6.0,0.9583333333333334,46.0,,,0.08333333333333333,4.0,0.625,30.0,0.08333333333333333,4.0,6.1042,,,,,,,,,,,,,0.4791666666666667,23.0,,,40
|
||||||
|
job_567550,0.0,0,64,512,5,2.75,0,0.8594,6251.3789,6817.3214,5974.9884,31729,1,7057.6721,0.3281,0.0461,0.1581,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1199,-0.0202,0.0,1.0,0.0,0.0154,0.4688,0.3281,17.2877,54.623,609.6594,28.9,857.9676,248.3082,0.0668,4.7741,53.9388,320.2493,374.5444,2041.8385,19.8279,320.5384,1642.6884,0,41,59,472,402,19,44,0.288135593220339,17,,,,,0.4067796610169492,24.0,0.1016949152542373,6.0,0.03389830508474576,2.0,0.11864406779661017,7.0,0.0847457627118644,5.0,,,,,,0.03389830508474576,2.0,,,,,,,0.23728813559322035,14.0,0.01694915254237288,1.0,41
|
||||||
|
job_567550,0.0,0,64,512,6,2.1562,0,0.9062,5198.2734,5100.9904,5265.3762,30850,1,5154.8585,0.4082,0.0041,0.2199,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.139,-0.0062,0.0,1.0,0.0,0.0229,0.5938,0.4082,17.2743,54.6095,607.5067,29.2,857.9676,250.4609,0.0706,4.5927,38.4789,238.7629,277.6139,767.8562,19.0513,239.064,466.5943,0,42,63,504,448,11,43,0.2222222222222222,14,0.0085,8.972,0.07936507936507936,5.0,0.3968253968253968,25.0,,,,,0.06349206349206349,4.0,0.015873015873015872,1.0,,,,,,,,0.015873015873015872,1.0,,,,,0.09523809523809523,6.0,,,42
|
||||||
|
job_567550,0.0,0,64,512,6,2.2812,0,0.9219,6603.6328,6320.4868,6769.3127,29924,1,5901.6823,0.3691,0.0059,0.0943,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1511,-0.0,0.0,1.0,0.0,0.0156,0.4531,0.3691,17.2621,54.5973,605.1696,29.5,857.9676,252.798,0.0627,4.7386,49.3034,289.1,338.7342,869.574,20.0024,289.3677,506.0951,0,43,63,504,388,26,112,0.746031746031746,47,,,,,0.8253968253968254,52.0,0.047619047619047616,3.0,,,0.20634920634920634,13.0,,,,,,,,,,,,,,,,,,0.015873015873015872,1.0,43
|
||||||
|
job_567550,0.0,0,64,512,6,2.6719,0,0.9375,6097.0117,6883.791,5588.5145,30733,1,6180.0026,0.3926,0.003,0.1964,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1275,-0.0037,0.0,1.0,0.0,0.02,0.5781,0.3926,17.5231,54.6111,604.3675,29.6,857.9676,253.6001,0.072,4.7976,44.3964,285.5469,330.2905,783.7946,19.0298,285.8218,429.673,0,44,56,448,344,23,94,0.6785714285714286,38,0.0063,8.3169,,,0.9642857142857143,54.0,0.017857142857142856,1.0,0.07142857142857142,4.0,0.05357142857142857,3.0,0.16071428571428573,9.0,,,,,,,,,,,,,,,,,,44
|
||||||
|
job_567550,0.0,0,64,512,6,2.5781,0,0.9531,6293.2793,6687.8072,6103.9971,31486,1,6498.3542,0.3242,0.0027,0.1589,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1235,-0.0007,0.0,1.0,0.0,0.0186,0.4844,0.3242,17.6409,54.684,601.6743,29.9,857.9676,256.2933,0.0656,4.846,47.9094,300.6953,348.9867,673.9404,19.8524,301.0114,300.2514,0,45,60,480,384,21,93,0.7166666666666667,43,,,0.03333333333333333,2.0,0.8,48.0,0.03333333333333333,2.0,0.03333333333333333,2.0,0.16666666666666666,10.0,,,6.1065,,,,,,,,,,,,,,,,,45
|
||||||
|
job_567550,0.0,0,64,512,6,2.75,0,0.9375,5982.3438,6061.5407,5942.2794,31013,1,6331.3741,0.3359,0.0083,0.1586,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,-0.0007,0.0,1.0,0.0,0.0182,0.5312,0.3359,17.8517,54.9248,599.7339,30.1,857.9676,258.2337,0.0638,4.6266,45.5297,290.2921,336.1922,1024.0646,19.361,290.5982,663.8806,0,46,55,440,370,19,70,0.8181818181818182,45,0.0065,107.2038,,,0.6181818181818182,34.0,,,,,0.07272727272727272,4.0,,,,,,,,,,,,,,,,,,,,46
|
||||||
|
job_567550,0.0,0,64,512,6,2.6719,0,0.9531,6900.8828,6491.8537,7093.6437,31520,1,6837.8518,0.3203,0.0046,0.1391,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1381,0.0,0.0,1.0,0.0,0.0162,0.4844,0.3203,17.6216,54.6812,598.9659,30.2,857.9676,259.0017,0.0794,4.8182,58.3023,322.2259,380.8975,1111.9881,19.6389,322.5152,706.6292,0,47,60,480,385,22,90,0.8333333333333334,50,,,,,0.48333333333333334,29.0,0.016666666666666666,1.0,,,0.2,12.0,0.06666666666666667,4.0,,,,,,,,,,,,,,,,,,47
|
||||||
|
job_567550,0.0,0,64,512,6,2.375,0,0.8438,6583.623,7439.1768,6115.7825,31001,1,6334.7756,0.3535,-0.004,0.1772,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1309,-0.0013,0.0,1.0,0.0,0.0186,0.5156,0.3535,17.5547,54.6142,597.6776,30.3,857.9676,260.29,0.0646,4.8542,53.0938,306.4226,359.8881,928.1989,19.6921,306.7294,543.7599,0,48,58,464,396,13,55,0.7241379310344828,42,0.0073,8.3185,0.05172413793103448,3.0,0.1896551724137931,11.0,0.017241379310344827,1.0,,,0.06896551724137931,4.0,0.13793103448275862,8.0,,,,,,,,0.017241379310344827,1.0,,,,,,,,,48
|
||||||
|
job_567550,0.0,0,64,512,6,2.7656,0,0.9531,6662.2363,7889.466,5836.0621,28624,1,5968.458,0.4023,0.0111,0.2309,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1378,-0.0011,0.0,1.0,0.0,0.0225,0.5938,0.4023,17.5219,54.582,597.1965,30.4,857.9676,260.7711,0.0628,4.5945,52.8577,284.1361,337.3518,759.6281,19.4761,284.4309,398.2013,0,49,64,512,418,22,91,0.6875,44,,,,,0.734375,47.0,0.046875,3.0,0.078125,5.0,0.046875,3.0,,,,,,,,,,,,,,,,,,,,49
|
||||||
|
job_567550,0.0,0,64,512,6,2.5781,0,0.9688,6234.9375,6473.3529,6148.7021,31740,1,6350.9234,0.2656,-0.0119,0.0852,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1381,0.0,0.0,1.0,0.0,0.0124,0.3438,0.2656,17.6574,54.7542,595.0074,30.6,857.9676,262.9602,0.0658,5.36,47.1592,284.0365,331.6008,773.9742,20.4465,284.3755,416.5621,0,50,42,336,293,13,43,0.5714285714285714,24,0.0069,8.3248,,,0.4523809523809524,19.0,,,,,,,,,6.1934,,,,,,,,,,,,,,,,,50
|
||||||
|
job_567551,0.0,0,64,512,6,1.7812,0,0.6562,5990.6699,6322.5959,5686.0936,28123,1,5473.7808,0.4785,0.0087,0.1171,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1368,-0.0033,0.0,1.0,0.0,0.0174,0.6226,0.4785,12.3812,50.353,596.5785,30.5,857.9676,261.3891,0.1531,4.4122,51.3157,263.4737,315.2958,2184.4235,19.7425,263.8261,1844.9728,0,51,64,512,461,10,49,0.625,40,,,0.03125,2.0,0.25,16.0,,,,,0.015625,1.0,,,,,,,,,,,,,,,,,,,,51
|
||||||
|
job_567551,0.0,0,64,512,1,0.7812,0,0.7812,6731.9863,7178.8007,6166.5487,31193,1,6580.3321,0.5586,0.0038,0.1713,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.143,-0.0028,0.0,1.0,0.0,0.02,0.6875,0.5586,13.0269,50.5115,593.0834,30.9,857.9676,264.8842,0.0655,4.8546,61.4679,320.4413,382.2524,1651.923,19.4153,320.7179,1245.3831,0,52,59,472,386,22,86,0.4915254237288136,29,32.6834,41.116,,,1.0,59.0,,,,,,,,,,,,,,,,,,,,,,,,,,52
|
||||||
|
job_567551,0.0,0,64,512,2,1.4531,0,0.9688,6508.3203,6281.5808,6651.2962,30531,1,6289.6614,0.3867,-0.0019,0.123,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0142,0.5,0.3867,13.9249,50.6506,590.5195,31.2,857.9676,267.4481,0.0636,4.7459,45.9719,288.7953,335.1579,652.1002,19.5645,289.1214,292.6286,0,53,60,480,345,31,134,0.6333333333333333,38,,,,,1.5166666666666666,91.0,0.016666666666666666,1.0,0.05,3.0,0.03333333333333333,2.0,,,,,,,,,,,,,,,,,,,,53
|
||||||
|
job_567551,0.0,0,64,512,3,2.25,1,1.0,5810.2891,7441.8333,5095.3427,30242,1,6574.3825,0.3047,0.0096,0.2385,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1218,0.0,0.0,1.0,0.0,0.0211,0.5156,0.3047,14.0695,50.7521,590.0227,31.2,857.9676,267.9449,0.0622,4.5947,50.1011,300.1234,350.5934,560.9024,19.5021,300.4296,186.209,0,54,57,456,379,17,76,0.3684210526315789,21,0.0067,9.2903,,,0.9298245614035088,53.0,0.017543859649122806,1.0,0.017543859649122806,1.0,0.10526315789473684,6.0,,,,,,,,,,,,,,,,,,,,54
|
||||||
|
job_567551,0.0,0,64,512,4,2.4844,0,0.9531,5920.2324,6443.0893,5664.8837,30033,1,5739.4678,0.3281,0.0067,0.127,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1313,0.0,0.0,1.0,0.0,0.0168,0.4531,0.3281,14.2443,50.9048,588.7913,31.4,857.9676,269.1763,0.0612,5.0782,53.6346,271.6531,325.6797,683.268,19.8067,271.9836,332.7004,0,55,62,496,434,17,62,0.3387096774193548,21,,,,,0.6774193548387096,42.0,,,,,0.04838709677419355,3.0,,,7.0721,,,,,,,,,,,,,,,,,55
|
||||||
|
job_567551,0.0,0,64,512,5,2.2812,0,0.9062,6120.8418,6407.7637,5873.5673,30008,1,6151.3952,0.4629,-0.0047,0.1712,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1397,-0.0,0.0,1.0,0.0,0.0151,0.5938,0.4629,14.2906,51.0547,633.2055,26.2,857.9676,224.7621,0.0629,4.6042,45.6063,296.6186,342.6071,854.4565,19.0306,296.9374,488.2113,0,56,64,512,422,22,87,0.734375,47,0.0067,9.5172,,,0.578125,37.0,0.015625,1.0,,,0.015625,1.0,0.03125,2.0,,,,,,0.015625,1.0,,,,,,,,,,,56
|
||||||
|
job_567551,0.0,0,64,512,5,2.5781,0,0.9844,7322.8574,8407.4009,6459.0281,31653,1,6692.4401,0.4434,0.0133,0.1395,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.131,-0.0053,0.0,1.0,0.0,0.0166,0.5781,0.4434,14.4149,51.1216,629.6761,26.6,857.9676,228.2914,0.0687,5.4807,58.8815,329.3126,388.5892,854.4278,19.65,329.6382,440.7044,0,57,61,488,388,24,99,0.5737704918032787,35,,,,,1.1475409836065573,70.0,0.01639344262295082,1.0,,,,,,,,,,,,,,,,,,,,,,,,57
|
||||||
|
job_567551,0.0,0,64,512,6,3.0469,0,0.9688,6034.1055,6223.8895,5930.3263,29330,1,6569.5919,0.3535,0.0099,0.1348,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1272,-0.0006,0.0,1.0,0.0,0.0172,0.5156,0.3535,14.421,51.2411,632.4707,26.3,857.9676,225.4969,0.0756,4.5986,50.4083,317.2726,368.0298,785.544,20.2901,317.5455,392.6213,0,58,60,480,441,11,38,0.26666666666666666,16,0.0072,9.0284,0.016666666666666666,1.0,0.4666666666666667,28.0,,,,,,,,,,,,,,,,,,,,,,,,,,58
|
||||||
|
job_567551,0.0,0,64,512,6,2.7031,0,0.9844,6661.7168,7054.4919,6439.5046,31055,1,6324.187,0.3613,-0.002,0.1616,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1497,-0.0005,0.0,1.0,0.0,0.0227,0.5469,0.3613,15.0173,51.8337,629.3011,26.7,857.9676,228.6665,0.0655,4.7848,50.985,314.2571,365.6335,643.7273,20.2109,314.5827,253.0942,0,59,58,464,362,23,97,0.5517241379310345,32,,,,,1.0,58.0,0.06896551724137931,4.0,0.017241379310344827,1.0,0.10344827586206896,6.0,0.017241379310344827,1.0,,,,,,,,,,,,,,,,,,59
|
||||||
|
job_567551,0.0,0,64,512,7,3.0781,0,0.9844,6317.8027,7111.7259,5821.2857,30479,1,6483.291,0.3848,0.0052,0.2215,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,0.0,0.0,1.0,0.0,0.0183,0.5469,0.3848,15.0261,51.9561,627.1725,26.9,857.9676,230.7951,0.0633,4.7623,52.6042,290.3792,343.2867,674.6483,21.1596,290.6188,305.4351,0,60,61,488,373,24,115,1.0491803278688525,64,0.0063,9.488,,,0.7704918032786885,47.0,,,,,0.06557377049180328,4.0,,,6.3751,,,,,,,,,,,,,,,,,60
|
||||||
|
job_567551,0.0,0,64,512,6,3.125,1,1.0,6463.0605,7886.7006,5710.8687,30994,1,6880.4264,0.3457,0.0019,0.1625,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1193,0.0,0.0,1.0,0.0,0.0145,0.5156,0.3457,15.1388,52.027,625.0475,27.1,857.9676,232.92,0.0653,4.7961,53.4008,354.2283,408.0195,684.6355,19.3622,354.5529,252.4537,0,61,61,488,413,21,68,0.18032786885245902,11,,,0.01639344262295082,1.0,0.9344262295081968,57.0,0.03278688524590164,2.0,,,,,0.06557377049180328,4.0,,,,,,,,,,,,,,,,,,61
|
||||||
|
job_567551,0.0,0,64,512,6,2.9375,0,0.9688,6648.5312,6675.8687,6631.293,29657,1,6232.3936,0.3867,-0.001,0.1137,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0142,0.5312,0.3867,15.158,52.1367,623.0032,27.4,857.9676,234.9644,0.0645,5.0387,50.7679,292.3912,343.5179,624.4476,21.7199,292.6853,254.166,0,62,63,504,449,15,46,0.12698412698412698,8,0.0064,9.0101,0.06349206349206349,4.0,0.5714285714285714,36.0,0.07936507936507936,5.0,0.015873015873015872,1.0,0.015873015873015872,1.0,,,,,,,,,,,,,,,,,,,,62
|
||||||
|
job_567551,0.0,0,64,512,6,2.8125,0,0.9062,6871.1211,7110.377,6738.0395,30986,1,6271.4665,0.3574,0.0086,0.1704,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1454,-0.0038,0.0,1.0,0.0,0.0191,0.5938,0.3574,15.3867,52.3629,620.4409,27.7,857.9676,237.5267,0.0654,4.8585,52.4883,316.0092,368.8701,868.6813,20.4287,316.3161,474.5195,0,63,63,504,415,22,86,0.42857142857142855,27,,,,,0.8888888888888888,56.0,0.047619047619047616,3.0,0.047619047619047616,3.0,0.015873015873015872,1.0,,,,,,,,,,,,,,,,,,,,63
|
||||||
|
job_567551,0.0,0,64,512,7,2.8125,0,0.9688,6867.2734,8067.7197,6336.3718,31346,1,7132.2744,0.3066,0.014,0.1409,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1309,-0.0,0.0,1.0,0.0,0.0143,0.4375,0.3066,15.383,52.3718,619.1424,27.8,857.9676,238.8252,0.0647,5.3526,58.7073,343.2849,402.3885,692.6827,21.0837,343.616,263.8521,0,64,62,496,435,19,60,0.3225806451612903,20,0.0068,8.8666,,,0.6129032258064516,38.0,0.016129032258064516,1.0,0.016129032258064516,1.0,0.016129032258064516,1.0,,,,,,,,,,,,,,,,,,,,64
|
||||||
|
job_567551,0.0,0,64,512,7,2.8125,1,1.0,6153.6172,7862.9684,5144.9938,31464,1,5836.2093,0.3711,0.0048,0.1809,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0,0.0,1.0,0.0,0.0227,0.5,0.3711,15.5993,52.6177,634.1627,26.1,857.9676,223.8049,0.0645,4.8537,50.2765,285.3918,336.0526,634.5957,22.2875,285.7112,271.3975,0,65,61,488,366,23,121,1.180327868852459,72,,,,,0.7377049180327869,45.0,0.01639344262295082,1.0,0.01639344262295082,1.0,0.04918032786885246,3.0,,,5.9537,,,,,,,,,,,,,,,,,65
|
||||||
|
job_567551,0.0,0,64,512,7,2.9531,0,0.9688,6439.0156,6111.65,6648.8654,29740,1,6685.0287,0.3906,-0.0145,0.1273,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0,0.0,1.0,0.0,0.0155,0.5469,0.3906,15.8226,52.9003,630.8162,26.5,857.9676,227.1514,0.0628,4.5991,57.2731,323.5507,381.1919,709.8547,20.4883,323.8556,303.571,0,66,60,480,435,12,44,0.36666666666666664,22,0.0066,9.3084,,,0.5666666666666667,34.0,0.016666666666666666,1.0,,,0.05,3.0,,,,,,,,,,,,,,,,,,,,66
|
||||||
|
job_567551,0.0,0,64,512,7,3.0625,1,1.0,6846.5332,6910.392,6785.5992,29213,1,6150.4072,0.4883,0.008,0.2377,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1313,0.0,0.0,1.0,0.0,0.0212,0.6875,0.4883,16.0647,53.2021,628.415,26.8,857.9676,229.5526,0.0627,4.6262,49.228,305.8525,355.4619,517.5303,22.0327,306.1709,135.4056,0,67,61,488,438,16,43,0.36065573770491804,22,,,0.11475409836065574,7.0,0.3442622950819672,21.0,,,,,,,,,,,,,,,,,,,,,,,,,,67
|
||||||
|
job_567551,0.0,0,64,512,7,3.2344,0,0.9844,7383.6035,8211.3237,6821.8393,29929,1,6141.4216,0.4043,0.0031,0.2121,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1473,0.0,0.0,1.0,0.0,0.025,0.6094,0.4043,16.173,53.2971,626.4578,27.0,857.9676,231.5098,0.0668,4.8139,51.8795,306.0463,358.2671,670.9587,22.6126,306.3205,285.261,0,68,60,480,418,13,62,0.35,21,0.0075,9.2352,,,0.65,39.0,,,,,0.08333333333333333,5.0,,,,,,,,,,,,,,,,,,,,68
|
||||||
|
job_567551,0.0,0,64,512,7,3.1094,0,0.9375,6849.9824,6946.125,6770.3214,31319,1,6263.3838,0.4531,-0.002,0.1978,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1369,-0.0,0.0,1.0,0.0,0.0205,0.6562,0.4531,16.3273,53.4486,624.0512,27.3,857.9676,233.9164,0.0686,4.9239,53.7165,303.0634,357.1172,730.7219,20.5135,303.3317,348.1627,0,69,57,456,354,22,100,0.8245614035087719,47,,,0.017543859649122806,1.0,1.0526315789473684,60.0,0.017543859649122806,1.0,0.14035087719298245,8.0,0.017543859649122806,1.0,,,,,,,,,,,,,,,,,,,,69
|
||||||
|
job_567551,0.0,0,64,512,7,3.1719,0,0.8906,5942.8418,6702.5523,5558.5176,29772,1,6202.3377,0.3359,-0.0023,0.118,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1266,-0.002,0.0,1.0,0.0,0.0136,0.5,0.3359,16.3538,53.356,622.1553,27.5,857.9676,235.8123,0.0651,4.4971,48.5204,287.6047,336.462,928.0517,20.97,287.8763,566.1177,0,70,61,488,398,22,89,0.6557377049180327,40,0.0063,8.9573,,,0.7377049180327869,45.0,0.01639344262295082,1.0,,,0.08196721311475409,5.0,,,6.1925,,,,,,,,,,,,,,,,,70
|
||||||
|
job_567551,0.0,0,64,512,7,2.7812,0,0.9375,6158.498,6986.8906,5661.4625,30577,1,5879.0702,0.375,0.0025,0.1169,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1303,0.0,0.0,1.0,0.0,0.0138,0.4688,0.375,16.3088,53.3733,619.9735,27.7,857.9676,237.9941,0.0645,4.8017,45.4262,268.4007,314.246,809.2009,20.4367,268.7549,469.7119,0,71,58,464,377,21,79,0.3793103448275862,22,,,,,1.0517241379310345,61.0,,,,,,,,,,0.13793103448275862,8.0,,,,,,,,,,,,,,,71
|
||||||
|
job_567551,0.0,0,64,512,6,2.75,0,0.9688,6496.7656,7114.178,5968.8333,29148,1,6645.0121,0.4609,0.0151,0.1421,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1219,0.0,0.0,1.0,0.0,0.0186,0.6562,0.4609,16.312,53.3518,618.4863,27.9,857.9676,239.4813,0.0602,4.4786,50.5528,311.4958,362.4515,775.8533,20.601,311.8381,388.3179,0,72,61,488,359,26,129,1.0327868852459017,63,0.0066,9.0389,,,0.9672131147540983,59.0,,,0.01639344262295082,1.0,0.11475409836065574,7.0,,,,,,,,,,,,,,,,,,,,72
|
||||||
|
job_567551,0.0,0,64,512,7,2.8281,0,0.9688,6007.7578,6901.5911,5420.5469,31182,1,6450.2234,0.3965,0.0136,0.2237,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1235,0.0,0.0,1.0,0.0,0.0181,0.5625,0.3965,16.534,53.6315,616.6702,28.1,857.9676,241.2974,0.0668,5.1941,55.0841,299.7416,355.1541,736.8375,20.6845,300.0029,355.801,0,73,58,464,361,20,101,0.9482758620689655,55,,,0.034482758620689655,2.0,0.7931034482758621,46.0,0.017241379310344827,1.0,0.034482758620689655,2.0,0.08620689655172414,5.0,,,,,,,,,,,,,,,,,,,,73
|
||||||
|
job_567551,0.0,0,64,512,6,2.7344,0,0.9844,5930.1973,6072.1242,5869.7103,30745,1,6036.8087,0.2988,-0.0089,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1195,0.0,0.0,1.0,0.0,0.064,0.3594,0.2988,16.5898,53.7518,629.4752,26.6,857.9676,228.4924,0.0661,4.6933,50.044,292.5558,342.9545,639.5606,21.8612,292.844,270.0476,0,74,64,512,437,19,68,0.03125,2,0.0071,9.3935,,,0.921875,59.0,0.0625,4.0,,,0.109375,7.0,,,,,,0.046875,3.0,,,,,,,,,,,,,74
|
||||||
|
job_567551,0.0,0,64,512,6,2.8906,0,0.9688,6931.8203,6564.4852,7112.8105,31582,1,6325.4492,0.3301,-0.0145,0.1116,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1352,-0.0011,0.0,1.0,0.0,0.0164,0.4531,0.3301,16.8138,53.9489,626.9986,26.9,857.9676,230.969,0.0663,4.9589,51.1396,308.4821,359.9248,987.0937,23.2754,308.7185,598.9303,0,75,64,512,434,21,77,0.1875,12,,,,,0.96875,62.0,0.015625,1.0,0.03125,2.0,0.015625,1.0,,,6.209,,,,,,,,,,,,,,,,,75
|
||||||
|
job_567551,0.0,0,64,512,6,2.6875,0,0.9219,6552.0254,5927.6872,7116.0186,31827,1,6416.322,0.4746,-0.0091,0.1469,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1332,0.0,0.0,1.0,0.0,0.0181,0.5938,0.4746,17.0905,54.3295,624.7698,27.2,857.9676,233.1978,0.0662,4.888,47.7922,305.2241,353.352,1003.0464,20.8521,305.4933,623.9499,0,76,61,488,415,19,73,0.3770491803278688,23,0.0073,9.5322,,,0.9180327868852459,56.0,,,0.01639344262295082,1.0,,,,,,,,,,,,,,,,,,,,,,76
|
||||||
|
job_567551,0.0,0,64,512,6,2.3438,0,0.9062,6973.2559,7723.1186,6515.7925,31204,1,6409.6402,0.3789,0.0007,0.1955,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,-0.0,0.0,1.0,0.0,0.0179,0.5625,0.3789,17.2256,54.4399,628.907,26.7,857.9676,229.0605,0.0716,4.9191,53.9669,306.4151,360.706,827.6281,21.3515,306.6672,440.6474,0,77,58,464,347,27,106,0.7241379310344828,42,,,0.10344827586206896,6.0,1.293103448275862,75.0,0.017241379310344827,1.0,0.017241379310344827,1.0,0.10344827586206896,6.0,,,,,,0.06896551724137931,4.0,,,,,,,,,,,,,77
|
||||||
|
job_567551,0.0,0,64,512,6,2.625,0,0.9219,5913.8125,7165.1304,5211.8537,31252,1,6425.6925,0.3594,0.0054,0.1736,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1127,-0.0016,0.0,1.0,0.0,0.0131,0.4688,0.3594,17.1172,54.3807,626.1709,27.0,857.9676,231.7967,0.0658,4.7719,53.2092,303.5102,357.0435,1027.6138,19.8591,303.7683,645.9345,0,78,59,472,339,26,132,0.7288135593220338,43,0.0071,13.3714,,,1.4915254237288136,88.0,0.01694915254237288,1.0,,,0.0847457627118644,5.0,,,,,,,,,,,,,,,,,,,,78
|
||||||
|
job_567551,0.0,0,64,512,5,2.1406,0,0.9219,5971.7539,7443.8,5271.7896,31359,1,6822.5791,0.3223,0.0002,0.1775,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1121,-0.001,0.0,1.0,0.0,0.0197,0.4688,0.3223,17.3101,54.5718,623.1119,27.4,857.9676,234.8557,0.0639,4.7264,50.7595,318.7277,369.8649,864.1247,20.4919,319.041,469.0373,0,79,61,488,393,22,92,0.5573770491803278,34,,,,,0.7868852459016393,48.0,0.04918032786885246,3.0,0.01639344262295082,1.0,0.14754098360655737,9.0,,,,,,,,,,,,,,,,,,,,79
|
||||||
|
job_567551,0.0,0,64,512,5,2.25,0,0.875,6524.4219,7517.6269,5923.5172,29828,1,6065.1914,0.377,0.0047,0.12,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1309,0.0,0.0,1.0,0.0,0.0175,0.5156,0.377,17.6478,54.8022,621.2485,27.6,857.9676,236.7191,0.0626,5.2654,46.0826,294.2125,340.6332,806.3176,22.4283,294.4877,437.9868,0,80,94,752,614,32,130,0.4787234042553192,45,0.0075,9.6795,0.010638297872340425,1.0,0.9680851063829787,91.0,0.02127659574468085,2.0,0.010638297872340425,1.0,0.02127659574468085,2.0,0.05319148936170213,5.0,6.251,,,,,,,,,,,,,,,,,80
|
||||||
|
3
training_logs/20260604_123153_reward_vs_steps.png
Normal file
3
training_logs/20260604_123153_reward_vs_steps.png
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:437395a0c447c3a38a6ae52fcc4d713ba5d6f855e16feaa4ffdacd2a25d364bb
|
||||||
|
size 314226
|
||||||
44527
training_logs/20260604_123153_trial_results.csv
Normal file
44527
training_logs/20260604_123153_trial_results.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:a7321f7de533b0fa814d958b93539c833a1cfe0bb99007711fbda33042b7032e
|
||||||
|
size 155692
|
||||||
1544
training_logs/20260604_123153_vllm_metrics_job_567548.csv
Normal file
1544
training_logs/20260604_123153_vllm_metrics_job_567548.csv
Normal file
File diff suppressed because it is too large
Load Diff
7690
training_logs/20260604_123153_vllm_metrics_job_567550.csv
Normal file
7690
training_logs/20260604_123153_vllm_metrics_job_567550.csv
Normal file
File diff suppressed because it is too large
Load Diff
5201
training_logs/20260604_123153_vllm_metrics_job_567551.csv
Normal file
5201
training_logs/20260604_123153_vllm_metrics_job_567551.csv
Normal file
File diff suppressed because it is too large
Load Diff
14433
training_logs/20260604_123153_vllm_metrics_table.csv
Normal file
14433
training_logs/20260604_123153_vllm_metrics_table.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:1854c97db9af9221893ec445a0a9304382a42726e82b065eeefff779ac636324
|
||||||
|
size 18292816
|
||||||
@@ -0,0 +1,248 @@
|
|||||||
|
|
||||||
|
The following have been reloaded with a version change:
|
||||||
|
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||||
|
|
||||||
|
|
||||||
|
Lmod is automatically replacing "GCC/14.3.0" with
|
||||||
|
"nvidia-compilers/25.9-CUDA-13".
|
||||||
|
|
||||||
|
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||||
|
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||||||
|
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||||
|
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||||
|
[ray] RAY_TMPDIR=/tmp/ray/ray_567549
|
||||||
|
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_567549
|
||||||
|
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_567549
|
||||||
|
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||||||
|
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||||||
|
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||||||
|
[proxy] Tunnel port: 7003
|
||||||
|
[proxy] Node IP: 10.128.16.49 (workers will connect here)
|
||||||
|
[proxy] ✓ SSH tunnel started successfully
|
||||||
|
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_567549.conf
|
||||||
|
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||||||
|
[proxy] - External traffic (internet) → PROXY via tunnel
|
||||||
|
[proxy] ✓ Daytona timeout settings configured
|
||||||
|
[proxy] Testing proxy connectivity...
|
||||||
|
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_567549.conf
|
||||||
|
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||||||
|
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||||
|
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||||||
|
[proxy] ⚠ Tunnel not accessible at 10.128.16.49:7003 (workers may fail)
|
||||||
|
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||||||
|
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||||||
|
=== Universal RL Training Runner ===
|
||||||
|
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/configs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_rl_config.json
|
||||||
|
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||||||
|
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||||
|
Python version: Python 3.12.12
|
||||||
|
UV_USE_IO_URING: 0
|
||||||
|
Proxy: DISABLED (direct internet or not configured)
|
||||||
|
========================================
|
||||||
|
=== RLJobRunner: a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified ===
|
||||||
|
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
|
||||||
|
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
|
||||||
|
HF_TOKEN=****pDbg
|
||||||
|
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||||||
|
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||||||
|
Environment configured:
|
||||||
|
TENSOR_PARALLEL_SIZE=1
|
||||||
|
NUM_INFERENCE_ENGINES=56
|
||||||
|
POLICY_NUM_NODES=14
|
||||||
|
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
|
||||||
|
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||||||
|
Cleaning up existing Ray instances...
|
||||||
|
=== Starting Ray Cluster ===
|
||||||
|
Nodes: 14
|
||||||
|
GPUs per node: 4
|
||||||
|
CPUs per node: 288
|
||||||
|
Head node: jpbo-002-01 (10.128.16.49)
|
||||||
|
Ray port: 6379
|
||||||
|
============================
|
||||||
|
Starting Ray head on jpbo-002-01 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-002-01.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.49 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.49 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray head on jpbo-002-01
|
||||||
|
Starting Ray worker on jpbo-002-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-02.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 1 on jpbo-002-02
|
||||||
|
Starting Ray worker on jpbo-002-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-03.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 2 on jpbo-002-03
|
||||||
|
Starting Ray worker on jpbo-002-04 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-04.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.52 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.52 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 3 on jpbo-002-04
|
||||||
|
Starting Ray worker on jpbo-002-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-05.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.53 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.53 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 4 on jpbo-002-05
|
||||||
|
Starting Ray worker on jpbo-002-06 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-06.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.54 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.54 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 5 on jpbo-002-06
|
||||||
|
Starting Ray worker on jpbo-002-07 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-07.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.55 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.55 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 6 on jpbo-002-07
|
||||||
|
Starting Ray worker on jpbo-002-08 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-08.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.56 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-08 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.56 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 7 on jpbo-002-08
|
||||||
|
Starting Ray worker on jpbo-002-09 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-09.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.57 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-09 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.57 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 8 on jpbo-002-09
|
||||||
|
Starting Ray worker on jpbo-002-10 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-10.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.58 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-10 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.58 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 9 on jpbo-002-10
|
||||||
|
Starting Ray worker on jpbo-002-11 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-11.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.59 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-11 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.59 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 10 on jpbo-002-11
|
||||||
|
Starting Ray worker on jpbo-002-12 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-12.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.60 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.60 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 11 on jpbo-002-12
|
||||||
|
Starting Ray worker on jpbo-002-13 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-13.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.61 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-13 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.61 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 12 on jpbo-002-13
|
||||||
|
Starting Ray worker on jpbo-002-14 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-14.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.62 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-14 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.62 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 13 on jpbo-002-14
|
||||||
|
Waiting for cluster (56 GPUs, 14 nodes)...
|
||||||
|
Connecting to Ray at 10.128.16.49:6379 (expecting 14 nodes, 56.0 GPUs)
|
||||||
|
Ray connection established, polling for resources...
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
||||||
|
srun stderr: 2026-06-03 16:26:49,474 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.49:6379...
|
||||||
|
2026-06-03 16:26:49,483 INFO worker.py:2007 -- Connected to Ray cluster.
|
||||||
|
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||||||
|
warnings.warn(
|
||||||
|
[33m(raylet, ip=10.128.16.62)[0m [2026-06-03 16:27:17,835 E 835178 835178] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||||
|
[36m(pid=983379)[0m [2026-06-03 16:27:34,323 E 983379 984025] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||||
|
[36m(pid=983396)[0m [2026-06-03 16:27:42,172 E 983396 985096] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
||||||
|
[2026-06-03 16:27:45,463 E 983335 983376] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||||
|
[36m(pid=983680)[0m [2026-06-03 16:27:45,417 E 983680 997125] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
||||||
|
main()
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
||||||
|
wait_for_cluster(
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
||||||
|
raise TimeoutError(
|
||||||
|
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0)
|
||||||
|
srun: error: jpbo-002-01: task 0: Exited with exit code 1
|
||||||
|
|
||||||
|
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 599, in _wait_for_cluster
|
||||||
|
result = subprocess.run(
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
||||||
|
raise CalledProcessError(retcode, process.args,
|
||||||
|
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
||||||
|
|
||||||
|
The above exception was the direct cause of the following exception:
|
||||||
|
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 930, in run
|
||||||
|
training_exit_code = self._run_with_ray()
|
||||||
|
^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1171, in _run_with_ray
|
||||||
|
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 826, in __enter__
|
||||||
|
self.start()
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 366, in start
|
||||||
|
self._wait_for_cluster()
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 627, in _wait_for_cluster
|
||||||
|
raise RuntimeError(
|
||||||
|
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||||
|
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
||||||
|
Collecting Ray logs from worker jpbo-002-02...
|
||||||
|
Collecting Ray logs from worker jpbo-002-03...
|
||||||
|
Collecting Ray logs from worker jpbo-002-04...
|
||||||
|
Collecting Ray logs from worker jpbo-002-05...
|
||||||
|
Collecting Ray logs from worker jpbo-002-06...
|
||||||
|
Collecting Ray logs from worker jpbo-002-07...
|
||||||
|
Collecting Ray logs from worker jpbo-002-08...
|
||||||
|
Collecting Ray logs from worker jpbo-002-09...
|
||||||
|
Collecting Ray logs from worker jpbo-002-10...
|
||||||
|
Collecting Ray logs from worker jpbo-002-11...
|
||||||
|
WARNING: Failed to collect logs from jpbo-002-11
|
||||||
|
Collecting Ray logs from worker jpbo-002-12...
|
||||||
|
Collecting Ray logs from worker jpbo-002-13...
|
||||||
|
Collecting Ray logs from worker jpbo-002-14...
|
||||||
|
[RLJobRunner] Crash-time Ray log preservation complete.
|
||||||
|
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||||||
|
repo_id: DCAgent/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified
|
||||||
|
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified
|
||||||
|
episodes: last
|
||||||
|
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/logs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_trace_upload.log
|
||||||
|
[RLJobRunner] Waiting for trace upload to complete...
|
||||||
|
[RLJobRunner] Trace upload failed with exit code 1.
|
||||||
|
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/ray_logs/
|
||||||
|
Collecting Ray logs from worker jpbo-002-02...
|
||||||
|
Collecting Ray logs from worker jpbo-002-03...
|
||||||
|
Collecting Ray logs from worker jpbo-002-04...
|
||||||
|
Collecting Ray logs from worker jpbo-002-05...
|
||||||
|
Collecting Ray logs from worker jpbo-002-06...
|
||||||
|
Collecting Ray logs from worker jpbo-002-07...
|
||||||
|
Collecting Ray logs from worker jpbo-002-08...
|
||||||
|
Collecting Ray logs from worker jpbo-002-09...
|
||||||
|
Collecting Ray logs from worker jpbo-002-10...
|
||||||
|
Collecting Ray logs from worker jpbo-002-11...
|
||||||
|
WARNING: Failed to collect logs from jpbo-002-11
|
||||||
|
Collecting Ray logs from worker jpbo-002-12...
|
||||||
|
Collecting Ray logs from worker jpbo-002-13...
|
||||||
|
Collecting Ray logs from worker jpbo-002-14...
|
||||||
|
Ray log preservation complete
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:76495276d93d837ada0ceb92d9696d81231ddaefd2f71f0552a2b3e6239d85b7
|
||||||
|
size 79650713
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:e0846934d1d6eb349cfcc327f94c0de8e04201b88f51be2cb763588cf913098f
|
||||||
|
size 65858521
|
||||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user