初始化项目，由ModelHub XC社区提供模型
Model: penfever/a3-rl-SankalpKJ_swesmith-oracle-filtered-40-8B Source: Original Platform
This commit is contained in:
ModelHub XC
41
.gitattributes
vendored
Normal file
41
.gitattributes
vendored
Normal file
@@ -0,0 +1,41 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/20260611_223001_reward_vs_steps.png filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/20260611_223001_turn_count_distribution.png filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_665437.out filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_665440.out filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_665442.out filter=lfs diff=lfs merge=lfs -text
|
||||
36
README.md
Normal file
36
README.md
Normal file
@@ -0,0 +1,36 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model: laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink
|
||||
tags:
|
||||
- reinforcement-learning
|
||||
- rl
|
||||
- swe
|
||||
- agent
|
||||
- skyrl
|
||||
datasets:
|
||||
- SankalpKJ/swesmith-oracle-filtered
|
||||
---
|
||||
|
||||
# a3-rl-SankalpKJ_swesmith-oracle-filtered-40-8B
|
||||
|
||||
RL-tuned (SkyRL) agentic SWE model. This is the **global_step 40** checkpoint,
|
||||
selected as the best checkpoint by EMA reward (EMA reward 0.0877).
|
||||
|
||||
- **Base model:** [laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink](https://huggingface.co/laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink) (a Qwen3-8B SFT)
|
||||
- **Training dataset:** [SankalpKJ/swesmith-oracle-filtered](https://huggingface.co/datasets/SankalpKJ/swesmith-oracle-filtered)
|
||||
- **Architecture:** Qwen3-8B
|
||||
- **Training framework:** SkyRL (agentic RL, a3 series)
|
||||
|
||||
The RL training configuration is included in this repo as `rl_config.yaml`.
|
||||
Parsed training metrics and the raw training logs are in `training_logs/`.
|
||||
|
||||
## Training Traces
|
||||
|
||||
The full agentic rollout traces from this RL run are published as a dataset:
|
||||
|
||||
- **Traces:** [penfever/a3-rl-SankalpKJ_swesmith-oracle-filtered](https://huggingface.co/datasets/penfever/a3-rl-SankalpKJ_swesmith-oracle-filtered)
|
||||
|
||||
## Note
|
||||
|
||||
Published to the `penfever/` namespace pending a `laion/` write-role bump
|
||||
(may be re-homed to `laion/` later).
|
||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"</think>": 151668,
|
||||
"</tool_call>": 151658,
|
||||
"</tool_response>": 151666,
|
||||
"<think>": 151667,
|
||||
"<tool_call>": 151657,
|
||||
"<tool_response>": 151665,
|
||||
"<|box_end|>": 151649,
|
||||
"<|box_start|>": 151648,
|
||||
"<|endoftext|>": 151643,
|
||||
"<|file_sep|>": 151664,
|
||||
"<|fim_middle|>": 151660,
|
||||
"<|fim_pad|>": 151662,
|
||||
"<|fim_prefix|>": 151659,
|
||||
"<|fim_suffix|>": 151661,
|
||||
"<|im_end|>": 151645,
|
||||
"<|im_start|>": 151644,
|
||||
"<|image_pad|>": 151655,
|
||||
"<|object_ref_end|>": 151647,
|
||||
"<|object_ref_start|>": 151646,
|
||||
"<|quad_end|>": 151651,
|
||||
"<|quad_start|>": 151650,
|
||||
"<|repo_name|>": 151663,
|
||||
"<|video_pad|>": 151656,
|
||||
"<|vision_end|>": 151653,
|
||||
"<|vision_pad|>": 151654,
|
||||
"<|vision_start|>": 151652
|
||||
}
|
||||
89
chat_template.jinja
Normal file
89
chat_template.jinja
Normal file
@@ -0,0 +1,89 @@
|
||||
{%- if tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{- messages[0].content + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||
{%- else %}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if message.content is string %}
|
||||
{%- set content = message.content %}
|
||||
{%- else %}
|
||||
{%- set content = '' %}
|
||||
{%- endif %}
|
||||
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- else %}
|
||||
{%- if '</think>' in content %}
|
||||
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
||||
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if loop.index0 > ns.last_query_index %}
|
||||
{%- if loop.last or (not loop.last and reasoning_content) %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if (loop.first and content) or (not loop.first) %}
|
||||
{{- '\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.function %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": "' }}
|
||||
{{- tool_call.name }}
|
||||
{{- '", "arguments": ' }}
|
||||
{%- if tool_call.arguments is string %}
|
||||
{{- tool_call.arguments }}
|
||||
{%- else %}
|
||||
{{- tool_call.arguments | tojson }}
|
||||
{%- endif %}
|
||||
{{- '}\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
68
config.json
Normal file
68
config.json
Normal file
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Qwen3ForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 151645,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 12288,
|
||||
"layer_types": [
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention"
|
||||
],
|
||||
"max_position_embeddings": 40960,
|
||||
"max_window_layers": 36,
|
||||
"model_type": "qwen3",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 36,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": 151643,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 1000000,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "4.57.6",
|
||||
"use_cache": false,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 151936
|
||||
}
|
||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
151645,
|
||||
151643
|
||||
],
|
||||
"pad_token_id": 151643,
|
||||
"temperature": 0.6,
|
||||
"top_k": 20,
|
||||
"top_p": 0.95,
|
||||
"transformers_version": "4.57.6"
|
||||
}
|
||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fb9f654db3fe774f043227ef4c99ecad68ab4fda600ecf980a364147d22be07e
|
||||
size 4902257696
|
||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:34ecd4c02ecdfee60136927c5c0f3022efce910a8e129c590f254c8ff1cd1407
|
||||
size 4915960368
|
||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:daa569d31a9d1e03c2170dca2bb745fb2edc77a7896d651aefe0b5c50c1d76a1
|
||||
size 4983068496
|
||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1c6f283da72441d7038c9eae8a7d29a450cd4b15b4c3cbaab3e1a5a86aeeccdd
|
||||
size 1580230264
|
||||
407
model.safetensors.index.json
Normal file
407
model.safetensors.index.json
Normal file
@@ -0,0 +1,407 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_parameters": 8190735360,
|
||||
"total_size": 16381470720
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.norm.weight": "model-00004-of-00004.safetensors"
|
||||
}
|
||||
}
|
||||
156
rl_config.yaml
Normal file
156
rl_config.yaml
Normal file
@@ -0,0 +1,156 @@
|
||||
{
|
||||
"job_name": "a3-rl-SankalpKJ_swesmith-oracle-filtered",
|
||||
"experiments_dir": "/e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered",
|
||||
"cluster_name": "jupiter",
|
||||
"skyrl_entrypoint": "examples.terminal_bench.entrypoints.main_tbench",
|
||||
"skyrl_hydra_args": [
|
||||
"+terminal_bench_config=terminal_bench",
|
||||
"trainer.strategy=fsdp2",
|
||||
"trainer.algorithm.advantage_estimator=rloo_n",
|
||||
"trainer.algorithm.use_kl_loss=false",
|
||||
"trainer.algorithm.kl_loss_coef=0.0",
|
||||
"trainer.algorithm.eps_clip_low=0.2",
|
||||
"trainer.algorithm.eps_clip_high=0.05",
|
||||
"trainer.algorithm.loss_reduction=token_mean",
|
||||
"trainer.epochs=2",
|
||||
"trainer.max_steps=80",
|
||||
"trainer.update_epochs_per_batch=1",
|
||||
"trainer.train_batch_size=64",
|
||||
"trainer.policy_mini_batch_size=64",
|
||||
"trainer.eval_batch_size=64",
|
||||
"trainer.micro_forward_batch_size_per_gpu=4",
|
||||
"trainer.micro_train_batch_size_per_gpu=1",
|
||||
"trainer.max_prompt_length=999999",
|
||||
"trainer.eval_interval=999999",
|
||||
"trainer.eval_before_train=false",
|
||||
"trainer.ckpt_interval=2",
|
||||
"trainer.resume_mode=latest",
|
||||
"trainer.hf_save_interval=5",
|
||||
"++trainer.hf_hub_repo_id=laion/a3-rl-SankalpKJ_swesmith-oracle-filtered",
|
||||
"++trainer.hf_hub_private=false",
|
||||
"++trainer.hf_hub_revision=main",
|
||||
"++trainer.enable_db_registration=false",
|
||||
"trainer.project_name=OpenThoughts-Agent",
|
||||
"trainer.log_level=INFO",
|
||||
"trainer.tracker_commit_each_step=true",
|
||||
"trainer.logger=console",
|
||||
"trainer.run_name=a3-rl-SankalpKJ_swesmith-oracle-filtered",
|
||||
"trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/a3-rl-SankalpKJ_swesmith-oracle-filtered/checkpoints",
|
||||
"trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/a3-rl-SankalpKJ_swesmith-oracle-filtered/exports",
|
||||
"trainer.policy.optimizer_config.lr=8e-6",
|
||||
"trainer.policy.optimizer_config.weight_decay=0.0",
|
||||
"trainer.policy.optimizer_config.adam_betas=[0.9,0.999]",
|
||||
"trainer.policy.optimizer_config.max_grad_norm=0.9",
|
||||
"trainer.policy.fsdp_config.cpu_offload=false",
|
||||
"trainer.policy.fsdp_config.reshard_after_forward=true",
|
||||
"trainer.policy.fsdp_config.fsdp_size=4",
|
||||
"trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
|
||||
"trainer.ref.fsdp_config.cpu_offload=false",
|
||||
"trainer.ref.fsdp_config.reshard_after_forward=true",
|
||||
"trainer.ref.fsdp_config.fsdp_size=4",
|
||||
"trainer.placement.colocate_all=false",
|
||||
"trainer.placement.policy_num_nodes=2",
|
||||
"trainer.placement.ref_num_nodes=2",
|
||||
"trainer.placement.policy_num_gpus_per_node=4",
|
||||
"trainer.placement.ref_num_gpus_per_node=4",
|
||||
"trainer.fully_async.max_staleness_steps=16",
|
||||
"trainer.fully_async.num_parallel_generation_workers=338",
|
||||
"generator.backend=vllm",
|
||||
"generator.timeout_multiplier=1.0",
|
||||
"generator.model_dtype=bfloat16",
|
||||
"generator.inference_engine_tensor_parallel_size=1",
|
||||
"generator.num_inference_engines=48",
|
||||
"generator.n_samples_per_prompt=8",
|
||||
"generator.eval_n_samples_per_prompt=8",
|
||||
"generator.gpu_memory_utilization=0.75",
|
||||
"generator.max_num_seqs=24",
|
||||
"generator.max_num_batched_tokens=65536",
|
||||
"generator.enable_prefix_caching=true",
|
||||
"generator.enable_chunked_prefill=true",
|
||||
"generator.run_engines_locally=true",
|
||||
"generator.weight_sync_backend=nccl",
|
||||
"generator.async_engine=true",
|
||||
"generator.batched=false",
|
||||
"generator.enable_http_endpoint=true",
|
||||
"generator.enable_ray_prometheus_stats=false",
|
||||
"generator.vllm_stats_interval=1",
|
||||
"generator.append_eos_token_after_stop_str_in_multi_turn=true",
|
||||
"generator.max_turns=999999",
|
||||
"generator.sampling_params.max_generate_length=4096",
|
||||
"generator.sampling_params.temperature=0.7",
|
||||
"generator.sampling_params.top_p=0.95",
|
||||
"generator.sampling_params.top_k=20",
|
||||
"++generator.engine_init_kwargs.max_model_len=32768",
|
||||
"++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2",
|
||||
"++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
|
||||
"data.train_data=[\"/e/scratch/jureap59/feuer1/tasks/swesmith-oracle-filtered\"]",
|
||||
"data.val_data=[]",
|
||||
"+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/a3-rl-SankalpKJ_swesmith-oracle-filtered/trace_jobs",
|
||||
"+terminal_bench_config.harbor.name=terminus-2",
|
||||
"+terminal_bench_config.harbor.max_episodes=999999",
|
||||
"+terminal_bench_config.harbor.enable_summarize=false",
|
||||
"+terminal_bench_config.harbor.store_all_messages=true",
|
||||
"+terminal_bench_config.harbor.trajectory_config.raw_content=true",
|
||||
"+terminal_bench_config.harbor.enable_episode_logging=false",
|
||||
"+terminal_bench_config.harbor.record_terminal_session=false",
|
||||
"+terminal_bench_config.harbor.enable_pane_logging=false",
|
||||
"+terminal_bench_config.harbor.strict_json_parser=true",
|
||||
"+terminal_bench_config.harbor.interleaved_thinking=true",
|
||||
"+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true",
|
||||
"+terminal_bench_config.harbor.override_timeout_sec=900",
|
||||
"+terminal_bench_config.harbor.override_cpus=1",
|
||||
"+terminal_bench_config.harbor.override_memory_mb=2048",
|
||||
"+terminal_bench_config.harbor.override_storage_mb=2048",
|
||||
"+terminal_bench_config.harbor.auto_snapshot=true",
|
||||
"+terminal_bench_config.harbor.verifier_override_timeout_sec=120",
|
||||
"+terminal_bench_config.harbor.max_retries=3",
|
||||
"+terminal_bench_config.harbor.min_wait_sec=60.0",
|
||||
"+terminal_bench_config.harbor.max_wait_sec=600.0",
|
||||
"+terminal_bench_config.harbor.wait_multiplier=2.0",
|
||||
"+terminal_bench_config.harbor.exclude_exceptions=[\"VerifierTimeoutError\",\"VerifierRuntimeError\",\"RewardFileNotFoundError\",\"RewardFileEmptyError\",\"VerifierOutputParseError\"]",
|
||||
"+terminal_bench_config.harbor.n_concurrent_trials=675",
|
||||
"+terminal_bench_config.harbor.log_level=INFO",
|
||||
"+terminal_bench_config.harbor.enable_reward_shaping=false",
|
||||
"+terminal_bench_config.harbor.enable_error_classification=true",
|
||||
"+terminal_bench_config.harbor.mask_exceptions=[\"DaytonaError\",\"EnvironmentStartTimeoutError\",\"NetworkError\",\"ConnectionError\",\"RewardFileNotFoundError\",\"RewardFileEmptyError\",\"AgentEnvironmentTimeoutError\",\"ContextLengthExceededError\"]",
|
||||
"+terminal_bench_config.harbor.default_error_treatment=zero",
|
||||
"+terminal_bench_config.harbor.passthrough_exceptions=[\"AgentTimeoutError\"]",
|
||||
"+terminal_bench_config.harbor.zero_exceptions=[]",
|
||||
"+terminal_bench_config.model_info.max_input_tokens=32000",
|
||||
"+terminal_bench_config.model_info.max_output_tokens=4096",
|
||||
"+terminal_bench_config.archiving.enabled=false",
|
||||
"+terminal_bench_config.trace_upload.enabled=true",
|
||||
"+terminal_bench_config.trace_upload.repo_org=DCAgent",
|
||||
"+terminal_bench_config.trace_upload.episodes=last",
|
||||
"+terminal_bench_config.trace_upload.dataset_type=SFT",
|
||||
"+terminal_bench_config.trace_upload.cleanup=true"
|
||||
],
|
||||
"model_path": "/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
|
||||
"train_data": [
|
||||
"/e/scratch/jureap59/feuer1/tasks/swesmith-oracle-filtered"
|
||||
],
|
||||
"val_data": [],
|
||||
"num_nodes": 14,
|
||||
"gpus_per_node": 4,
|
||||
"cpus_per_node": 288,
|
||||
"tensor_parallel_size": 1,
|
||||
"ray_port": 6379,
|
||||
"master_port": 12345,
|
||||
"checkpoints_dir": null,
|
||||
"export_path": "/e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/a3-rl-SankalpKJ_swesmith-oracle-filtered/exports",
|
||||
"needs_ssh_tunnel": true,
|
||||
"needs_cuda_detection": false,
|
||||
"pinggy_persistent_url": null,
|
||||
"pinggy_token": null,
|
||||
"agent_name": "terminus-2",
|
||||
"harbor_env": "daytona",
|
||||
"proxychains_binary": null,
|
||||
"container_sif": null,
|
||||
"container_binds": [],
|
||||
"ray_object_store_gb": 40.0,
|
||||
"trace_upload_enabled": true,
|
||||
"trace_upload_repo_org": "DCAgent",
|
||||
"trace_upload_episodes": "last",
|
||||
"trace_upload_dataset_type": "SFT",
|
||||
"trace_upload_cleanup": true
|
||||
}
|
||||
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"eos_token": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
240
tokenizer_config.json
Normal file
240
tokenizer_config.json
Normal file
@@ -0,0 +1,240 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"151643": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151644": {
|
||||
"content": "<|im_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151645": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151646": {
|
||||
"content": "<|object_ref_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151647": {
|
||||
"content": "<|object_ref_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151648": {
|
||||
"content": "<|box_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151649": {
|
||||
"content": "<|box_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151650": {
|
||||
"content": "<|quad_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151651": {
|
||||
"content": "<|quad_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151652": {
|
||||
"content": "<|vision_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151653": {
|
||||
"content": "<|vision_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151654": {
|
||||
"content": "<|vision_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151655": {
|
||||
"content": "<|image_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151656": {
|
||||
"content": "<|video_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151657": {
|
||||
"content": "<tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151658": {
|
||||
"content": "</tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151659": {
|
||||
"content": "<|fim_prefix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151660": {
|
||||
"content": "<|fim_middle|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151661": {
|
||||
"content": "<|fim_suffix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151662": {
|
||||
"content": "<|fim_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151663": {
|
||||
"content": "<|repo_name|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151664": {
|
||||
"content": "<|file_sep|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151665": {
|
||||
"content": "<tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151666": {
|
||||
"content": "</tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151667": {
|
||||
"content": "<think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151668": {
|
||||
"content": "</think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"bos_token": null,
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|im_end|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 32768,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"padding_side": "left",
|
||||
"split_special_tokens": false,
|
||||
"tokenizer_class": "Qwen2Tokenizer",
|
||||
"unk_token": null
|
||||
}
|
||||
20
training_logs/20260611_223001_metrics_job_665437.csv
Normal file
20
training_logs/20260611_223001_metrics_job_665437.csv
Normal file
@@ -0,0 +1,20 @@
|
||||
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError
|
||||
0.0,0,64,512,0,0.0,0,0.0,11349.9688,22580.7742,10626.1538,30811,1,11492.5734,0.0605,0.0059,0.1379,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.074,0.0,0.0,1.0,0.0,0.0087,0.3125,0.0605,12.6198,50.7038,629.8245,26.6,857.9675,228.143,0.1719,5.8266,95.3526,522.8761,618.6733,5196.8333,18.3245,523.1484,4554.0085,0,1,126,1008,543,110,464,2.6031746031746033,328,1.119047619047619,141,0.015873015873015872,2.0,0.007936507936507936,1.0,,,,,,,
|
||||
0.0,0,64,512,1,0.9844,0,0.9844,11598.0488,23352.6071,10918.0331,31129,1,11634.256,0.0547,0.0019,0.104,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0748,0.0,0.0,1.0,0.0,0.0075,0.25,0.0547,13.7325,50.9559,625.2936,27.1,857.9675,232.6739,0.0713,5.7945,93.5595,504.3783,598.3009,2571.1786,18.8207,504.6697,1948.26,0,2,62,496,222,57,274,3.3548387096774195,208,1.1129032258064515,69,0.04838709677419355,3.0,,,72.3958,83.1661,,,,,
|
||||
0.0,0,64,512,2,1.6875,1,1.0,9616.832,23228.3871,8739.58,31095,1,11337.0222,0.0605,0.005,0.1221,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0647,0.0,0.0,1.0,0.0,0.0074,0.3125,0.0605,14.3281,51.0302,618.9982,27.9,857.9675,238.9693,0.072,5.266,81.9794,485.583,567.9016,1492.9544,18.2728,485.8498,901.5112,0,3,63,504,188,62,316,3.5873015873015874,226,1.507936507936508,95,,,,,,,,,,,
|
||||
0.0,0,64,512,3,1.75,1,1.0,8344.4121,21399.1556,7086.4604,31383,1,11438.0109,0.0879,-0.0103,0.1777,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0548,0.0,0.0,1.0,0.0,0.0093,0.4062,0.0879,14.506,51.261,614.4324,28.4,857.9675,243.5351,0.0742,5.254,75.7532,482.7177,558.8002,2604.307,17.6001,482.9725,2022.6498,0,4,62,496,202,60,294,2.774193548387097,172,2.0483870967741935,127,0.04838709677419355,3.0,,,0.0089,12.4763,,,,,
|
||||
0.0,0,64,512,3,1.5312,1,1.0,9046.1094,21562.6481,7570.3603,31325,1,11605.7658,0.1055,-0.0012,0.1564,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0612,0.0,0.0,1.0,0.0,0.008,0.4375,0.1055,14.6636,51.4144,610.7278,28.8,857.9675,247.2397,0.0742,5.3358,76.8331,494.4498,571.6047,2193.9348,18.0363,494.6971,1598.9545,0,5,62,496,141,61,348,3.532258064516129,219,2.370967741935484,147,0.016129032258064516,1.0,,,,,9.0806,0.11290322580645161,7.0,,
|
||||
0.0,0,64,512,2,1.4219,1,1.0,6122.6758,20356.6604,4479.1002,31353,1,10363.1029,0.1035,0.0073,0.184,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0426,-0.002,0.0,1.0,0.0,0.007,0.3438,0.1035,15.0129,51.3718,607.0053,29.3,857.9675,250.9622,0.0705,4.9823,67.0112,449.3728,516.7236,2005.3759,17.9401,449.6414,1465.7263,0,6,64,512,140,64,353,2.8125,180,2.671875,171,0.03125,2.0,,,0.0089,8.5757,,0.296875,19.0,,
|
||||
0.0,0,64,512,3,1.5312,1,1.0,5639.3535,20543.9556,4203.1499,31125,1,9774.0402,0.0879,0.0147,0.1893,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0426,-0.0034,0.0,1.0,0.0,0.0136,0.3906,0.0879,14.9615,51.363,603.5803,29.6,857.9675,254.3873,0.0697,4.66,52.0849,398.0136,450.4269,2106.1877,22.0007,398.272,1629.0969,0,7,63,504,75,63,429,6.158730158730159,388,0.7142857142857143,45,0.031746031746031744,2.0,,,,,,,,,
|
||||
0.0,0,64,512,2,1.5312,1,1.0,3320.0449,25125.6923,2751.9619,31029,1,8287.569,0.0254,-0.0009,0.0917,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.022,-0.0,0.0,1.0,0.0,0.0037,0.1562,0.0254,14.8983,51.2988,605.2846,29.5,857.9675,252.6829,0.0665,4.4222,37.9972,334.0714,372.3853,1636.3839,20.0642,334.3211,1239.5085,0,8,62,496,104,60,392,6.290322580645161,390,0.12903225806451613,8,,,,,0.0089,8.5925,,,,,
|
||||
0.0,0,64,512,2,1.4219,0,0.9219,4733.4922,20816.8929,3803.0475,30692,1,9672.5683,0.0547,-0.0003,0.0654,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0286,0.0,0.0,1.0,0.0,0.0036,0.2344,0.0547,14.8563,51.3198,602.4516,29.8,857.9675,255.5159,0.0649,4.5125,56.4992,420.1371,476.9557,1936.8703,18.4296,420.3912,1436.9688,0,9,65,520,105,64,414,6.292307692307692,409,0.07692307692307693,5,,,0.015384615384615385,1.0,,,,,,,
|
||||
0.0,0,64,512,3,1.2969,0,0.9688,4541.293,22735.3846,3567.9465,30723,1,9361.45,0.0508,0.0059,0.0708,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0294,-0.0,0.0,1.0,0.0,0.0038,0.1719,0.0508,14.8421,51.3412,624.4957,27.2,857.9675,233.4719,0.0644,4.4072,59.7586,406.8498,466.9292,1774.4441,18.6093,407.1058,1284.4944,0,10,63,504,90,63,414,6.4603174603174605,407,0.1111111111111111,7,,,,,0.0086,9.0343,6.9689,,,,
|
||||
0.0,0,64,512,3,1.2969,0,0.9531,4183.541,22342.2632,3483.712,30184,1,9316.038,0.0371,-0.0043,0.0994,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0253,0.0,0.0,1.0,0.0,0.0062,0.1562,0.0371,14.9382,51.4313,619.8862,27.7,857.9675,238.0814,0.0623,4.4976,54.8936,365.088,420.2978,1815.6224,19.5224,365.3414,1371.3005,0,11,65,520,100,62,420,6.446153846153846,419,0.015384615384615385,1,,,,,,,,,,,
|
||||
0.0,0,64,512,2,1.2188,0,0.9062,4316.6074,20771.1176,3751.503,30890,1,9198.6209,0.0332,0.0016,0.1317,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0268,0.0,0.0,1.0,0.0,0.0057,0.1875,0.0332,14.9169,51.3809,615.1475,28.3,857.9675,242.8201,0.0645,4.4862,56.4202,386.4744,443.2228,1963.7998,19.8882,386.7377,1496.198,0,12,63,504,129,60,374,5.904761904761905,372,0.015873015873015872,1,0.015873015873015872,1.0,,,0.0085,8.5349,,,,0.015873015873015872,1.0
|
||||
0.0,0,64,512,2,1.2188,0,0.9688,5632.6406,23748.3261,3844.397,31136,1,10271.1862,0.0898,-0.0018,0.1381,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0362,-0.0,0.0,1.0,0.0,0.0062,0.2656,0.0898,14.9247,51.4518,611.0772,28.8,857.9675,246.8903,0.0664,4.7589,69.6389,453.1235,523.1041,2030.7998,20.2669,453.398,1482.6651,0,13,63,504,85,62,419,6.634920634920635,418,0.015873015873015872,1,,,,,,,,,,,
|
||||
0.0,0,64,512,3,1.2344,0,0.9219,3788.7988,21801.4333,2667.6805,31002,1,8774.0077,0.0586,0.0024,0.156,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0251,0.0,0.0,1.0,0.0,0.0058,0.3125,0.0586,15.1205,51.7013,607.4822,29.2,857.9675,250.4854,0.0651,4.506,44.243,363.9317,408.5352,1839.41,20.9491,364.2267,1405.4149,0,14,65,520,113,64,407,6.184615384615385,402,0.07692307692307693,5,,,,,0.0087,8.6444,,,,,
|
||||
0.0,0,64,512,3,1.2812,0,0.9688,4575.3594,21459.3256,3027.3625,31012,1,9291.0922,0.084,-0.004,0.1396,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0315,0.0,0.0,1.0,0.0,0.005,0.3125,0.084,15.1204,51.7007,627.364,26.9,857.9675,230.6035,0.0675,4.5468,58.713,400.9189,459.961,2154.3582,18.5589,401.1802,1671.2874,0,15,64,512,91,64,421,6.328125,405,0.25,16,,,,,,,6.0659,,,,
|
||||
0.0,0,64,512,2,1.2812,0,0.9531,4289.8574,22854.8077,3296.6708,31292,1,9569.7061,0.0508,-0.0026,0.1121,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0242,0.0,0.0,1.0,0.0,0.0041,0.2344,0.0508,15.1272,51.7703,616.1652,28.2,857.9675,241.8024,0.0666,4.624,60.7552,426.8336,487.9325,1964.7559,19.134,427.1104,1453.0604,0,16,64,512,94,64,418,6.515625,417,0.015625,1,,,,,0.0093,8.6404,,,,,
|
||||
0.0,0,64,512,2,1.2656,0,0.9375,3983.7305,21573.5135,2613.5789,31181,1,8949.1562,0.0723,-0.0027,0.0771,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0252,0.0,0.0,1.0,0.0,0.0047,0.2812,0.0723,15.2764,51.8998,611.1328,28.8,857.9675,246.8347,0.0643,4.5877,52.6894,376.4956,429.5203,2037.089,23.5187,376.7662,1579.4581,0,17,63,504,85,63,419,6.603174603174603,416,0.047619047619047616,3,,,,,,,,,,,
|
||||
0.0,0,64,512,3,1.1094,0,0.9062,3694.9473,20101.1053,2379.6857,31220,1,8633.844,0.0742,0.0185,0.1882,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0271,0.0,0.0,1.0,0.0,0.0061,0.2969,0.0742,15.1888,51.8959,607.5901,29.2,857.9675,250.3774,0.0644,4.4667,49.5139,338.9372,388.7806,2029.453,19.3153,339.2019,1616.886,0,18,62,496,55,62,441,7.17741935483871,445,0.06451612903225806,4,,,,,0.0101,8.498,,,,,
|
||||
0.0,0,64,512,3,1.1562,0,0.9062,2383.0566,23984.7,1504.9411,31350,1,7274.8501,0.0391,0.0024,0.1832,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0149,0.0,0.0,1.0,0.0,0.0032,0.1719,0.0391,15.341,52.0204,604.2086,29.6,857.9675,253.759,0.0633,4.2026,38.1289,293.2502,331.6945,1965.399,18.4552,293.502,1611.0432,0,19,33,264,77,32,187,6.03030303030303,199,0.12121212121212122,4,,,,,,,,,,,
|
||||
|
20
training_logs/20260611_223001_metrics_job_665440.csv
Normal file
20
training_logs/20260611_223001_metrics_job_665440.csv
Normal file
@@ -0,0 +1,20 @@
|
||||
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_BadRequestError,batch_errors/total_BadRequestError
|
||||
0.0,0,64,512,3,0.2969,0,0.2031,3022.0449,22315.9,2237.7419,31165,1,7996.9892,0.0391,0.0031,0.0794,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0193,-0.0,0.0,1.0,0.0,0.0025,0.1667,0.0391,12.3074,50.4413,646.5037,24.6,857.9676,211.4639,0.1937,4.3836,44.3156,318.3362,363.1298,4563.2206,27.4975,318.619,4168.2093,0,19,63,504,114,63,389,6.0476190476190474,381,0.12698412698412698,8.0,0.015873015873015872,1.0,,,,,,,
|
||||
0.0,0,64,512,1,0.8125,0,0.8125,4230.1367,19613.1224,2602.1317,30906,1,8827.526,0.0957,0.0009,0.0974,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0321,-0.0,0.0,1.0,0.0,0.0041,0.3281,0.0957,12.5155,50.6252,635.8582,25.9,857.9676,222.1094,0.0746,4.5471,49.3255,368.1026,417.799,2729.1254,22.4617,368.3983,2284.3116,0,20,62,496,106,62,390,6.209677419354839,385,0.08064516129032258,5.0,,,57.2769,66.1821,6.5415,,,,
|
||||
0.0,0,64,512,2,1.25,0,0.9844,4219.9961,20052.2075,2391.8758,30440,1,8988.1183,0.1035,-0.0052,0.1336,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0288,-0.0,0.0,1.0,0.0,0.0041,0.4844,0.1035,12.7537,50.898,644.9331,24.8,857.9676,213.0345,0.0768,4.5436,47.9568,380.8666,429.2084,1786.6399,19.8241,381.1743,1333.0593,0,21,65,520,96,64,421,6.338461538461538,412,0.13846153846153847,9.0,,,,,,0.046153846153846156,3.0,,
|
||||
0.0,0,64,512,2,1.3281,0,0.9844,4372.1699,24130.0789,2788.2025,30885,1,9441.7563,0.0742,-0.0074,0.207,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0252,-0.0,0.0,1.0,0.0,0.0052,0.25,0.0742,14.2454,51.0239,633.0668,26.2,857.9676,224.9008,0.0921,4.5941,60.8765,413.3934,474.6959,1801.2744,22.7048,413.7268,1299.2727,0,22,62,496,77,62,419,6.709677419354839,416,0.04838709677419355,3.0,,,0.0107,9.0301,,,,,
|
||||
0.0,0,64,512,3,1.25,0,0.9062,3215.9902,21851.5,2056.1037,30955,1,8045.2464,0.0586,0.001,0.064,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0222,0.0,0.0,1.0,0.0,0.0023,0.2812,0.0586,14.4943,51.246,625.9828,27.0,857.9676,231.9848,0.0911,4.5094,41.2756,320.365,362.0259,2312.7157,22.3457,320.6588,1923.8274,0,23,62,496,79,61,412,6.596774193548387,409,0.04838709677419355,3.0,,,,,,0.06451612903225806,4.0,0.016129032258064516,1.0
|
||||
0.0,0,64,512,2,1.2344,0,0.9531,3154.5469,23362.32,2117.1869,31010,1,8107.5049,0.0488,0.0051,0.0922,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0208,0.0,0.0,1.0,0.0,0.003,0.2656,0.0488,14.6713,51.4477,620.8464,27.6,857.9676,237.1212,0.0711,4.4954,58.7083,341.94,401.0182,1958.7534,23.8914,342.2355,1529.3403,0,24,63,504,72,63,432,6.857142857142857,432,,,,,0.0106,8.6578,,,,,
|
||||
0.0,0,64,512,2,1.0781,0,0.9531,2907.0625,21687.931,1779.4327,31250,1,7636.6044,0.0566,-0.0094,0.0978,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0193,0.0,0.0,1.0,0.0,0.0031,0.2656,0.0566,14.7885,51.5618,616.0342,28.2,857.9676,241.9334,0.0993,4.4024,38.8754,289.1906,328.471,1989.6963,24.4896,289.4959,1632.3275,0,25,64,512,99,63,413,6.453125,413,,,,,,,6.2536,,,,
|
||||
0.0,0,64,512,2,1.2344,0,0.9375,3729.0488,21425.7838,2350.5663,31470,1,8410.0012,0.0723,0.0097,0.0629,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0279,0.0,0.0,1.0,0.0,0.0047,0.2656,0.0723,14.6334,51.5005,611.8091,28.7,857.9676,246.1584,0.0724,4.5488,54.3068,363.9403,418.6124,2378.7009,23.3033,364.2328,1932.2312,0,26,64,512,81,64,430,6.71875,430,,,0.015625,1.0,0.0106,8.6989,,,,,
|
||||
0.0,0,64,512,2,1.1562,0,0.9375,2768.5215,19034.7407,1862.9794,31316,1,7084.0848,0.0527,-0.007,0.0706,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0241,0.0,0.0,1.0,0.0,0.0033,0.3125,0.0527,14.639,51.476,606.791,29.3,857.9676,251.1766,0.0967,4.3642,38.7391,265.2822,304.404,1956.0438,22.7867,265.5676,1624.4803,0,27,64,512,104,64,406,6.328125,405,0.015625,1.0,,,,,,0.03125,2.0,,
|
||||
0.0,0,64,512,2,1.1562,0,0.9062,3543.5586,16543.7941,2618.8556,31037,1,7945.2767,0.0664,-0.034,0.1257,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0301,-0.0,0.0,1.0,0.0,0.0066,0.3125,0.0664,14.9626,51.8556,601.9171,29.8,857.9676,256.0505,0.0968,4.514,42.3834,329.7107,372.5014,2185.9777,23.2048,330.0208,1785.752,0,28,63,504,130,62,372,5.888888888888889,371,0.015873015873015872,1.0,0.015873015873015872,1.0,0.0114,8.8621,,0.015873015873015872,1.0,,
|
||||
0.0,0,64,512,3,1.3594,0,0.9688,5395.25,21022.9091,3514.4595,30429,1,9793.3681,0.1074,-0.0042,0.0913,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0358,0.0,0.0,1.0,0.0,0.0037,0.375,0.1074,15.0327,51.9991,598.9776,30.2,857.9676,258.99,0.073,4.672,65.2871,400.398,466.0685,2020.9027,22.9964,400.708,1527.1586,0,29,64,512,88,64,424,6.609375,423,0.015625,1.0,,,,,,,,,
|
||||
0.0,0,64,512,2,1.2812,0,0.9375,3439.0098,21578.2895,1984.8059,31430,1,8243.3078,0.0742,-0.0041,0.0935,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0245,0.0,0.0,1.0,0.0,0.0037,0.375,0.0742,15.2515,52.2041,595.3509,30.6,857.9676,262.6167,0.073,4.5383,41.9493,327.7638,370.0848,1990.2611,23.8074,328.0617,1591.8244,0,30,63,504,84,63,420,6.666666666666667,420,,,,,0.0094,8.8468,7.2216,,,,
|
||||
0.0,0,64,512,2,1.2188,0,0.9688,3080.7031,18103.9032,2112.4719,30682,1,7689.1389,0.0605,-0.0081,0.0685,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0239,0.0,0.0,1.0,0.0,0.0026,0.3125,0.0605,15.1851,52.2018,592.6693,30.9,857.9676,265.2983,0.0706,4.344,37.8148,304.6191,342.7822,2032.1933,21.4183,304.8963,1663.6431,0,31,65,520,69,65,451,6.938461538461539,451,,,,,,,,,,,
|
||||
0.0,0,64,512,2,1.1094,0,0.9219,2563.5566,20302.4333,1459.4772,30461,1,7192.4905,0.0586,-0.0031,0.0958,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0182,0.0,0.0,1.0,0.0,0.0024,0.2812,0.0586,15.3267,52.3829,589.8547,31.2,857.9676,268.1129,0.0907,4.3146,39.5432,286.4901,326.4528,1954.6703,21.1101,286.8183,1602.7878,0,32,64,512,77,64,435,6.796875,435,,,,,0.0096,8.9937,,,,,
|
||||
0.0,0,64,512,3,1.2031,0,0.9375,2973.4863,22900.6111,1466.3929,31035,1,7813.7646,0.0703,-0.0086,0.1611,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0216,0.0,0.0,1.0,0.0,0.0039,0.3125,0.0703,15.3817,52.5188,587.3306,31.5,857.9676,270.637,0.095,4.4646,42.5818,338.4778,381.4496,2037.4334,21.7785,338.7721,1629.7359,0,33,63,504,97,63,407,6.4603174603174605,407,,,,,,,,,,,
|
||||
0.0,0,64,512,2,1.2656,0,0.9688,3865.0098,22052.9545,2155.0321,31394,1,8598.5383,0.0859,0.0022,0.083,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0266,0.0,0.0,1.0,0.0,0.0036,0.2812,0.0859,15.3569,52.457,585.365,31.8,857.9676,272.6026,0.0733,4.5461,48.1633,363.6179,412.1646,2066.1355,25.7698,363.9272,1623.65,0,34,64,512,94,64,417,6.515625,417,,,0.015625,1.0,0.0086,8.6415,,,,,
|
||||
0.0,0,64,512,3,1.0625,0,0.8438,3385.8906,19282.4857,2219.4738,30976,1,7882.3298,0.0684,0.004,0.0497,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0283,0.0,0.0,1.0,0.0,0.003,0.3125,0.0684,15.4366,52.5654,583.2695,32.0,857.9676,274.6981,0.0964,4.5272,41.7149,294.8022,336.9157,2075.7662,27.2687,295.1037,1707.0507,0,35,62,496,103,61,393,6.338709677419355,393,,,,,,,7.2214,,,,
|
||||
0.0,0,64,512,2,1.1875,0,0.9375,4324.877,22049.9231,2321.1761,30678,1,8858.8066,0.1016,0.0046,0.155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0324,0.0,0.0,1.0,0.0,0.005,0.3281,0.1016,15.4926,52.6221,638.7303,25.6,857.9676,219.2373,0.105,5.0488,56.6453,374.4842,431.4993,2465.1235,23.9408,374.7484,2004.63,0,36,63,504,88,63,416,6.571428571428571,414,0.031746031746031744,2.0,,,0.0086,9.2812,,,,,
|
||||
0.0,0,64,512,2,1.125,0,0.9219,3393.8008,19400.5143,2219.304,30734,1,7995.2371,0.0684,-0.0075,0.0638,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0239,0.0,0.0,1.0,0.0,0.0027,0.4219,0.0684,15.3723,52.5737,631.1822,26.4,857.9676,226.7854,0.097,4.4162,39.207,318.2496,357.8584,2026.5443,20.4635,318.5539,1643.8015,0,37,16,128,21,16,107,6.0625,97,0.625,10.0,,,,,,,,,
|
||||
|
20
training_logs/20260611_223001_metrics_job_665442.csv
Normal file
20
training_logs/20260611_223001_metrics_job_665442.csv
Normal file
@@ -0,0 +1,20 @@
|
||||
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_EnvironmentStartTimeoutError,batch_errors/total_EnvironmentStartTimeoutError,timing/save_hf_model,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError
|
||||
0.0,0,64,512,2,0.3906,0,0.2656,3568.8223,21907.375,2014.7076,30407,1,8511.6759,0.0781,0.0064,0.1634,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0225,0.0,0.0,1.0,0.0,0.0057,0.4615,0.0781,13.1506,50.6884,595.8149,30.6,857.9676,262.1526,0.1993,4.4228,47.6739,348.1637,396.2853,4273.6039,21.8023,348.411,3851.0932,0,37,64,512,93,62,419,6.515625,417,0.03125,2.0,,,,,,,,,,,
|
||||
0.0,0,64,512,1,0.7969,0,0.7969,3505.293,19578.125,1842.5862,30440,1,8178.2537,0.0938,-0.005,0.0693,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0252,-0.0,0.0,1.0,0.0,0.0031,0.3906,0.0938,13.3729,50.9678,592.067,31.0,857.9676,265.9006,0.0741,4.7716,39.8009,285.4323,325.5524,2002.5702,22.0107,285.6765,1650.2284,0,38,60,480,87,59,392,6.516666666666667,391,,,11.067,19.6408,0.016666666666666666,1.0,0.016666666666666666,1.0,,,,,
|
||||
0.0,0,64,512,2,1.2656,1,1.0,3434.6777,19201.2703,2206.5432,30799,1,8150.2939,0.0723,0.0149,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0261,-0.0015,0.0,1.0,0.0,0.0044,0.2656,0.0723,13.4437,51.0087,589.7198,31.3,857.9676,268.2477,0.0988,4.5332,42.6766,320.8587,363.9235,1968.0577,21.1464,321.1474,1578.4515,0,39,65,520,108,63,412,6.323076923076923,411,0.015384615384615385,1.0,,,,,,,,,,,
|
||||
0.0,0,64,512,2,1.25,0,0.9688,4040.6562,20375.0652,2428.2468,30911,1,8528.7211,0.0898,0.0048,0.0217,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0286,0.0,0.0,1.0,0.0,0.0019,0.2656,0.0898,13.5471,51.202,586.9761,31.6,857.9676,270.9915,0.098,4.4569,55.8701,358.8919,415.1705,2163.5657,21.276,359.202,1722.6557,0,40,64,512,79,61,432,6.734375,431,0.015625,1.0,0.0091,8.3809,0.015625,1.0,,,7.1735,,,,
|
||||
0.0,0,64,512,2,1.2812,1,1.0,2970.6348,20093.5676,1636.8484,30923,1,7516.3108,0.0723,0.0064,0.1391,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0219,0.0,0.0,1.0,0.0,0.0039,0.2812,0.0723,13.4956,51.2211,584.994,31.8,857.9676,272.9736,0.0953,4.41,36.7656,283.113,320.2793,1606.033,22.9696,283.4176,1258.3687,0,41,63,504,44,63,459,7.333333333333333,462,0.07936507936507936,5.0,,,,,,,,0.015873015873015872,1.0,,
|
||||
0.0,0,64,512,2,1.1094,0,0.875,1634.5039,21094.6296,551.1567,30676,1,5948.5119,0.0527,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.012,0.0,0.0,1.0,0.0,0.0,0.1875,0.0527,13.5087,51.3518,582.3663,32.1,857.9676,275.6013,0.089,4.191,26.4419,202.4572,229.1957,2187.2531,27.9916,202.6644,1925.8702,0,42,62,496,93,62,403,6.5,403,,,0.0862,8.3963,,,,,,,,,
|
||||
0.0,0,64,512,3,1.1562,0,0.9531,3687.9473,21659.9787,1871.4194,31193,1,8273.0941,0.0918,0.0168,0.1209,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0287,0.0,0.0,1.0,0.0,0.0048,0.375,0.0918,14.9308,51.7346,578.3634,32.6,857.9676,279.6042,0.0956,4.4952,41.1548,304.5746,346.1633,2201.8784,22.719,304.9124,1828.4937,0,43,64,512,99,63,413,6.421875,411,0.03125,2.0,,,,,,,,,,,
|
||||
0.0,0,64,512,2,1.125,0,0.9688,3553.6445,22041.7308,2564.57,31011,1,8173.3769,0.0508,0.0086,0.0768,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0285,0.0,0.0,1.0,0.0,0.0033,0.25,0.0508,14.9455,51.8056,576.2864,32.8,857.9676,281.6812,0.0946,4.4365,48.5577,340.6085,389.5569,2153.5102,21.1603,340.9042,1738.3506,0,44,62,496,83,60,413,6.661290322580645,413,,,0.0088,8.5905,,,,,,,,,
|
||||
0.0,0,64,512,2,1.2031,0,0.9219,3265.6094,21753.6061,1991.9061,31046,1,7954.943,0.0645,0.0059,0.0779,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0234,0.0,0.0,1.0,0.0,0.0028,0.2656,0.0645,14.9641,51.8647,575.2365,33.0,857.9676,282.7311,0.084,4.3381,52.0341,323.9391,376.3528,1721.3545,21.9444,324.2342,1318.7133,0,45,64,512,75,62,437,6.828125,437,,,,,,,,,5.9288,,,,
|
||||
0.0,0,64,512,3,1.2812,0,0.9375,3055.5059,20803.6905,1469.4979,30660,1,7938.0939,0.082,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0215,0.0,0.0,1.0,0.0,0.0,0.3125,0.082,15.0172,51.9069,572.0082,33.3,857.9676,285.9594,0.0835,4.4024,49.0487,304.4824,353.9057,2110.5882,23.6305,304.7727,1728.645,0,46,64,512,84,64,428,6.6875,428,,,0.0086,8.8655,,,,,,,,,
|
||||
0.0,0,64,512,2,1.0938,0,0.9062,3168.8477,20393.0222,1509.1306,31277,1,7799.7708,0.0879,-0.0057,0.0317,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0239,-0.0,0.0,1.0,0.0,0.0016,0.3125,0.0879,15.0454,51.9489,571.0099,33.4,857.9676,286.9576,0.0723,4.5105,38.3069,301.5151,340.1996,1938.5631,20.5692,301.8199,1573.2792,0,47,62,496,70,62,426,6.854838709677419,425,0.016129032258064516,1.0,,,,,,,,,,,
|
||||
0.0,0,64,512,3,1.2344,0,0.8906,2789.0898,22593.3125,1468.8083,30387,1,7499.7521,0.0625,0.0054,0.0705,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0205,0.0,0.0,1.0,0.0,0.0024,0.2812,0.0625,15.1748,52.1106,569.3416,33.6,857.9676,288.626,0.0694,4.2181,44.0702,284.2664,328.6084,1934.7623,21.3574,284.4683,1580.5732,0,48,63,504,72,61,432,6.857142857142857,432,,,0.0082,8.8011,,,,,,,,,
|
||||
0.0,0,64,512,2,1.1094,0,0.875,2950.2188,22200.0571,1537.7568,30187,1,7764.5758,0.0684,-0.0038,0.0896,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0195,0.0,0.0,1.0,0.0,0.003,0.1875,0.0684,14.8967,52.1049,567.3762,33.9,857.9676,290.5914,0.109,4.5224,38.8775,267.5912,306.8889,1990.0106,24.6606,267.9018,1653.9126,0,49,64,512,74,63,438,6.84375,438,,,,,,,,,,,,,
|
||||
0.0,0,64,512,2,1.0,0,0.9062,2849.5039,20867.0,1846.468,30412,1,7502.0696,0.0527,0.0135,0.1585,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0201,0.0,0.0,1.0,0.0,0.0036,0.2812,0.0527,14.9523,52.1703,565.6843,34.1,857.9676,292.2833,0.0959,4.4261,47.8666,291.3579,339.5654,2067.6726,20.7362,291.6023,1702.9395,0,50,64,512,71,63,440,6.859375,439,0.015625,1.0,0.0085,8.1311,0.015625,1.0,,,6.4196,,,,
|
||||
0.0,0,64,512,2,1.0781,0,0.9219,2999.1387,23140.8261,2051.7791,30773,1,7861.7815,0.0449,0.0053,0.0746,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0196,0.0,0.0,1.0,0.0,0.0036,0.25,0.0449,15.1374,52.4918,564.2138,34.2,857.9676,293.7538,0.0765,4.3451,55.9953,320.3913,376.787,2055.8264,22.1486,320.7136,1652.5417,0,51,64,512,81,64,431,6.734375,431,,,,,,,,,,,,,
|
||||
0.0,0,64,512,2,1.2969,0,0.9688,3029.0996,18844.75,1974.7229,30513,1,7642.7286,0.0625,-0.0001,0.0309,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0222,0.0,0.0,1.0,0.0,0.002,0.2969,0.0625,15.1295,52.4838,563.5904,34.3,857.9676,294.3772,0.0858,4.4522,48.0109,302.477,350.8653,1952.0873,22.0891,302.768,1574.6726,0,52,63,504,104,61,400,6.349206349206349,400,,,0.008,8.1584,,,,,,,,,
|
||||
0.0,0,64,512,3,1.2812,0,0.9375,4303.8984,21060.6667,2328.2096,30373,1,9028.9972,0.1055,-0.0004,0.1114,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0283,0.0,0.0,1.0,0.0,0.004,0.3906,0.1055,15.2133,52.6156,565.0471,34.1,857.9676,292.9205,0.0674,4.6788,54.0249,372.0372,426.4382,1895.8166,20.0,372.3455,1444.6942,0,53,64,512,107,62,405,6.328125,405,,,,,,,,,,,,,
|
||||
0.0,0,64,512,3,1.2656,0,0.9531,4038.2031,20528.7193,1972.3582,30709,1,8486.9036,0.1113,-0.0066,0.1182,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0284,0.0,0.0,1.0,0.0,0.0041,0.4062,0.1113,15.3165,52.8245,563.9276,34.3,857.9676,294.04,0.0763,4.518,51.3008,338.4849,390.1408,2121.6477,22.5942,338.7631,1704.39,0,54,64,512,102,62,404,6.296875,403,0.015625,1.0,0.0089,8.1862,,,,,,0.09375,6.0,,
|
||||
0.0,0,64,512,2,1.2031,0,0.9688,4057.9336,20640.4583,2342.5,31296,1,8672.1578,0.0938,-0.0074,0.1057,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0288,0.0,0.0,1.0,0.0,0.0036,0.2969,0.0938,15.4248,52.9057,563.0505,34.4,857.9676,294.9171,0.0922,4.633,55.8487,379.3597,435.5265,2030.4286,24.1291,379.585,1566.1321,0,55,63,504,74,63,429,6.809523809523809,429,,,,,,,,,6.3546,,,0.015873015873015872,1.0
|
||||
|
352
training_logs/20260611_223001_metrics_report.md
Normal file
352
training_logs/20260611_223001_metrics_report.md
Normal file
@@ -0,0 +1,352 @@
|
||||
# SkyRL Training Metrics Analysis
|
||||
|
||||
Generated from 3 log files
|
||||
|
||||
## Overview
|
||||
|
||||
| Log File | Total Steps | Metric Blocks | Final Reward (mean) | Final Reward (max) | Total Time (s) |
|
||||
|----------|-------------|---------------|---------------------|-------------------|----------------|
|
||||
| job_665437 | 19 | 19 | 0.0648 | 0.1055 | 41319.2 |
|
||||
| job_665440 | 37 | 19 | 0.0717 | 0.1074 | 42331.2 |
|
||||
| job_665442 | 55 | 19 | 0.0757 | 0.1113 | 40375.2 |
|
||||
|
||||
## Async Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:------------------------------|-----------:|---------:|------:|-------:|--------:|
|
||||
| async/discard_rate | 0 | 0 | 0 | 0 | 57 |
|
||||
| async/discarded_count | 0 | 0 | 0 | 0 | 57 |
|
||||
| async/effective_batch_groups | 64 | 0 | 64 | 64 | 57 |
|
||||
| async/effective_batch_samples | 512 | 0 | 512 | 512 | 57 |
|
||||
| async/staleness_max | 2.24561 | 0.63473 | 0 | 3 | 57 |
|
||||
| async/staleness_mean | 1.17982 | 0.28367 | 0 | 1.75 | 57 |
|
||||
| async/staleness_min | 0.140351 | 0.350438 | 0 | 1 | 57 |
|
||||
| async/staleness_ratio | 0.899949 | 0.184242 | 0 | 1 | 57 |
|
||||
|
||||
## Generate Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:-------------------------------------|---------:|---------:|----------:|--------:|--------:|
|
||||
| generate/avg_num_tokens | 4239.45 | 2014.21 | 1634.5 | 11598 | 57 |
|
||||
| generate/avg_tokens_non_zero_rewards | 21330.3 | 1584.74 | 16543.8 | 25125.7 | 57 |
|
||||
| generate/avg_tokens_zero_rewards | 2950.57 | 2085.81 | 551.157 | 10918 | 57 |
|
||||
| generate/max_num_tokens | 30915.4 | 340.183 | 30184 | 31470 | 57 |
|
||||
| generate/min_num_tokens | 1 | 0 | 1 | 1 | 57 |
|
||||
| generate/std_num_tokens | 8670.65 | 1208.44 | 5948.51 | 11634.3 | 57 |
|
||||
|
||||
## Loss Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:----------------------------|----------:|-----------:|--------:|-------:|--------:|
|
||||
| loss/avg_final_rewards | 0.0707246 | 0.0209058 | 0.0254 | 0.1113 | 57 |
|
||||
| loss/avg_raw_advantages | 0.0005 | 0.00822275 | -0.034 | 0.0185 | 57 |
|
||||
| loss/avg_raw_advantages_abs | 0.104714 | 0.047809 | 0 | 0.207 | 57 |
|
||||
|
||||
## Policy Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:----------------------------|-------------:|------------:|--------:|-------:|--------:|
|
||||
| policy/final_loss | -1.75439e-06 | 1.32453e-05 | -0.0001 | 0 | 57 |
|
||||
| policy/log_ratio_abs_max | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_mean | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_p99 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos00 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos10 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos20 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos30 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos40 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos50 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos60 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos70 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos80 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/log_ratio_abs_pos90 | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/n_tokens_dp_gt_10pct | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/n_tokens_dp_gt_1pct | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/n_tokens_dp_gt_50pct | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/policy_entropy | 0.0292228 | 0.0129584 | 0.012 | 0.0748 | 57 |
|
||||
| policy/policy_loss | -0.000121053 | 0.000550564 | -0.0034 | 0 | 57 |
|
||||
| policy/policy_lr | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/policy_update_steps | 1 | 0 | 1 | 1 | 57 |
|
||||
| policy/ppo_clip_ratio | 0 | 0 | 0 | 0 | 57 |
|
||||
| policy/raw_grad_norm | 0.00433158 | 0.00227134 | 0 | 0.0136 | 57 |
|
||||
|
||||
## Reward Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:----------------------|----------:|----------:|-------:|-------:|--------:|
|
||||
| reward/avg_pass_at_8 | 0.296921 | 0.075816 | 0.1562 | 0.4844 | 57 |
|
||||
| reward/avg_raw_reward | 0.0707246 | 0.0209058 | 0.0254 | 0.1113 | 57 |
|
||||
|
||||
## System Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:------------------------|---------:|-------------:|---------:|---------:|--------:|
|
||||
| system/process_rss_gb | 14.6472 | 0.81941 | 12.3074 | 15.4926 | 57 |
|
||||
| system/process_vms_gb | 51.6898 | 0.603303 | 50.4413 | 52.9057 | 57 |
|
||||
| system/ram_available_gb | 600.626 | 23.2068 | 563.051 | 646.504 | 57 |
|
||||
| system/ram_percent | 29.993 | 2.70488 | 24.6 | 34.4 | 57 |
|
||||
| system/ram_total_gb | 857.968 | 4.75595e-05 | 857.967 | 857.968 | 57 |
|
||||
| system/ram_used_gb | 257.341 | 23.2068 | 211.464 | 294.917 | 57 |
|
||||
|
||||
## Timing Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:--------------------------------------|------------:|------------:|----------:|----------:|--------:|
|
||||
| timing/compute_advantages_and_returns | 0.085607 | 0.0277791 | 0.0623 | 0.1993 | 57 |
|
||||
| timing/convert_to_training_input | 4.59077 | 0.330639 | 4.191 | 5.8266 | 57 |
|
||||
| timing/fwd_logprobs_values_reward | 51.6811 | 13.6812 | 26.4419 | 95.3526 | 57 |
|
||||
| timing/policy_train | 354.974 | 66.1806 | 202.457 | 522.876 | 57 |
|
||||
| timing/run_training | 407.023 | 78.9918 | 229.196 | 618.673 | 57 |
|
||||
| timing/step | 2175.89 | 641.095 | 1492.95 | 5196.83 | 57 |
|
||||
| timing/sync_weights | 21.644 | 2.43342 | 17.6001 | 27.9916 | 57 |
|
||||
| timing/train_critic_and_policy | 355.255 | 66.178 | 202.664 | 523.148 | 57 |
|
||||
| timing/wait_for_generation_buffer | 1742.62 | 626.373 | 901.511 | 4554.01 | 57 |
|
||||
| timing/cleanup_old_checkpoints | 5.22361 | 17.4395 | 0.008 | 72.3958 | 27 |
|
||||
| timing/save_checkpoints | 14.0929 | 17.7591 | 8.1311 | 83.1661 | 27 |
|
||||
| timing/save_hf_model | 6.83909 | 0.878296 | 5.9288 | 9.0806 | 11 |
|
||||
|
||||
## Trainer Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:--------------------|-------:|--------:|------:|------:|--------:|
|
||||
| trainer/epoch | 0 | 0 | 0 | 0 | 57 |
|
||||
| trainer/global_step | 28 | 15.8238 | 1 | 55 | 57 |
|
||||
|
||||
## Batch_Errors Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:------------------------------------------------|------------:|--------------:|-------------:|-------------:|--------:|
|
||||
| batch_errors/total_batches | 63.0351 | 11.2931 | 16 | 126 | 57 |
|
||||
| batch_errors/total_instances | 504.281 | 90.3449 | 128 | 1008 | 57 |
|
||||
| batch_errors/total_successful | 103.825 | 67.826 | 21 | 543 | 57 |
|
||||
| batch_errors/total_failed | 61.8947 | 9.8664 | 16 | 110 | 57 |
|
||||
| batch_errors/total_masked | 399.491 | 60.3052 | 107 | 464 | 57 |
|
||||
| batch_errors/avg_ContextLengthExceededError | 6.16236 | 1.10149 | 2.60317 | 7.33333 | 57 |
|
||||
| batch_errors/total_ContextLengthExceededError | 384.86 | 79.3519 | 97 | 462 | 57 |
|
||||
| batch_errors/avg_DaytonaError | 0.374391 | 0.699578 | 0.0153846 | 2.67188 | 37 |
|
||||
| batch_errors/total_DaytonaError | 24.5405 | 47.4884 | 1 | 171 | 37 |
|
||||
| batch_errors/avg_AgentTimeoutError | 0.0296636 | 0.0145576 | 0.015873 | 0.0483871 | 7 |
|
||||
| batch_errors/total_AgentTimeoutError | 2 | 0.816497 | 1 | 3 | 7 |
|
||||
| batch_errors/avg_DaytonaValidationError | 0.0363421 | 0.0299898 | 0.00793651 | 0.09375 | 8 |
|
||||
| batch_errors/total_DaytonaValidationError | 2.375 | 1.84681 | 1 | 6 | 8 |
|
||||
| batch_errors/avg_DaytonaAuthenticationError | 0.204889 | 0.130088 | 0.112903 | 0.296875 | 2 |
|
||||
| batch_errors/total_DaytonaAuthenticationError | 13 | 8.48528 | 7 | 19 | 2 |
|
||||
| batch_errors/avg_RuntimeError | 0.0158482 | 0.000352746 | 0.015625 | 0.0166667 | 8 |
|
||||
| batch_errors/total_RuntimeError | 1 | 0 | 1 | 1 | 8 |
|
||||
| batch_errors/avg_BadRequestError | 0.016129 | nan | 0.016129 | 0.016129 | 1 |
|
||||
| batch_errors/total_BadRequestError | 1 | nan | 1 | 1 | 1 |
|
||||
| batch_errors/avg_EnvironmentStartTimeoutError | 0.0166667 | nan | 0.0166667 | 0.0166667 | 1 |
|
||||
| batch_errors/total_EnvironmentStartTimeoutError | 1 | nan | 1 | 1 | 1 |
|
||||
| batch_errors/avg_VerifierTimeoutError | 0.015873 | nan | 0.015873 | 0.015873 | 1 |
|
||||
| batch_errors/total_VerifierTimeoutError | 1 | nan | 1 | 1 | 1 |
|
||||
|
||||
## Training Progression by Log
|
||||
|
||||
### job_665437
|
||||
|
||||
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||
|------|--------|--------|-----|------|---------------|-------------|
|
||||
| 1 | 0.0605 | 0.3125 | 0.000000 | 0.0000 | 5196.8 | 4554.0 |
|
||||
| 2 | 0.0547 | 0.2500 | 0.000000 | 0.0000 | 2571.2 | 1948.3 |
|
||||
| 3 | 0.0605 | 0.3125 | 0.000000 | 0.0000 | 1493.0 | 901.5 |
|
||||
| 4 | 0.0879 | 0.4062 | 0.000000 | 0.0000 | 2604.3 | 2022.6 |
|
||||
| 5 | 0.1055 | 0.4375 | 0.000000 | 0.0000 | 2193.9 | 1599.0 |
|
||||
| 6 | 0.1035 | 0.3438 | 0.000000 | -0.0000 | 2005.4 | 1465.7 |
|
||||
| 7 | 0.0879 | 0.3906 | 0.000000 | -0.0001 | 2106.2 | 1629.1 |
|
||||
| 8 | 0.0254 | 0.1562 | 0.000000 | -0.0000 | 1636.4 | 1239.5 |
|
||||
| 9 | 0.0547 | 0.2344 | 0.000000 | 0.0000 | 1936.9 | 1437.0 |
|
||||
| 10 | 0.0508 | 0.1719 | 0.000000 | -0.0000 | 1774.4 | 1284.5 |
|
||||
| 11 | 0.0371 | 0.1562 | 0.000000 | 0.0000 | 1815.6 | 1371.3 |
|
||||
| 12 | 0.0332 | 0.1875 | 0.000000 | 0.0000 | 1963.8 | 1496.2 |
|
||||
| 13 | 0.0898 | 0.2656 | 0.000000 | -0.0000 | 2030.8 | 1482.7 |
|
||||
| 14 | 0.0586 | 0.3125 | 0.000000 | 0.0000 | 1839.4 | 1405.4 |
|
||||
| 15 | 0.0840 | 0.3125 | 0.000000 | 0.0000 | 2154.4 | 1671.3 |
|
||||
| 16 | 0.0508 | 0.2344 | 0.000000 | 0.0000 | 1964.8 | 1453.1 |
|
||||
| 17 | 0.0723 | 0.2812 | 0.000000 | 0.0000 | 2037.1 | 1579.5 |
|
||||
| 18 | 0.0742 | 0.2969 | 0.000000 | 0.0000 | 2029.5 | 1616.9 |
|
||||
| 19 | 0.0391 | 0.1719 | 0.000000 | 0.0000 | 1965.4 | 1611.0 |
|
||||
|
||||
### job_665440
|
||||
|
||||
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||
|------|--------|--------|-----|------|---------------|-------------|
|
||||
| 19 | 0.0391 | 0.1667 | 0.000000 | -0.0000 | 4563.2 | 4168.2 |
|
||||
| 20 | 0.0957 | 0.3281 | 0.000000 | -0.0000 | 2729.1 | 2284.3 |
|
||||
| 21 | 0.1035 | 0.4844 | 0.000000 | -0.0000 | 1786.6 | 1333.1 |
|
||||
| 22 | 0.0742 | 0.2500 | 0.000000 | -0.0000 | 1801.3 | 1299.3 |
|
||||
| 23 | 0.0586 | 0.2812 | 0.000000 | 0.0000 | 2312.7 | 1923.8 |
|
||||
| 24 | 0.0488 | 0.2656 | 0.000000 | 0.0000 | 1958.8 | 1529.3 |
|
||||
| 25 | 0.0566 | 0.2656 | 0.000000 | 0.0000 | 1989.7 | 1632.3 |
|
||||
| 26 | 0.0723 | 0.2656 | 0.000000 | 0.0000 | 2378.7 | 1932.2 |
|
||||
| 27 | 0.0527 | 0.3125 | 0.000000 | 0.0000 | 1956.0 | 1624.5 |
|
||||
| 28 | 0.0664 | 0.3125 | 0.000000 | -0.0000 | 2186.0 | 1785.8 |
|
||||
| 29 | 0.1074 | 0.3750 | 0.000000 | 0.0000 | 2020.9 | 1527.2 |
|
||||
| 30 | 0.0742 | 0.3750 | 0.000000 | 0.0000 | 1990.3 | 1591.8 |
|
||||
| 31 | 0.0605 | 0.3125 | 0.000000 | 0.0000 | 2032.2 | 1663.6 |
|
||||
| 32 | 0.0586 | 0.2812 | 0.000000 | 0.0000 | 1954.7 | 1602.8 |
|
||||
| 33 | 0.0703 | 0.3125 | 0.000000 | 0.0000 | 2037.4 | 1629.7 |
|
||||
| 34 | 0.0859 | 0.2812 | 0.000000 | 0.0000 | 2066.1 | 1623.7 |
|
||||
| 35 | 0.0684 | 0.3125 | 0.000000 | 0.0000 | 2075.8 | 1707.1 |
|
||||
| 36 | 0.1016 | 0.3281 | 0.000000 | 0.0000 | 2465.1 | 2004.6 |
|
||||
| 37 | 0.0684 | 0.4219 | 0.000000 | 0.0000 | 2026.5 | 1643.8 |
|
||||
|
||||
### job_665442
|
||||
|
||||
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||
|------|--------|--------|-----|------|---------------|-------------|
|
||||
| 37 | 0.0781 | 0.4615 | 0.000000 | 0.0000 | 4273.6 | 3851.1 |
|
||||
| 38 | 0.0938 | 0.3906 | 0.000000 | -0.0000 | 2002.6 | 1650.2 |
|
||||
| 39 | 0.0723 | 0.2656 | 0.000000 | -0.0000 | 1968.1 | 1578.5 |
|
||||
| 40 | 0.0898 | 0.2656 | 0.000000 | 0.0000 | 2163.6 | 1722.7 |
|
||||
| 41 | 0.0723 | 0.2812 | 0.000000 | 0.0000 | 1606.0 | 1258.4 |
|
||||
| 42 | 0.0527 | 0.1875 | 0.000000 | 0.0000 | 2187.3 | 1925.9 |
|
||||
| 43 | 0.0918 | 0.3750 | 0.000000 | 0.0000 | 2201.9 | 1828.5 |
|
||||
| 44 | 0.0508 | 0.2500 | 0.000000 | 0.0000 | 2153.5 | 1738.4 |
|
||||
| 45 | 0.0645 | 0.2656 | 0.000000 | 0.0000 | 1721.4 | 1318.7 |
|
||||
| 46 | 0.0820 | 0.3125 | 0.000000 | 0.0000 | 2110.6 | 1728.6 |
|
||||
| 47 | 0.0879 | 0.3125 | 0.000000 | -0.0000 | 1938.6 | 1573.3 |
|
||||
| 48 | 0.0625 | 0.2812 | 0.000000 | 0.0000 | 1934.8 | 1580.6 |
|
||||
| 49 | 0.0684 | 0.1875 | 0.000000 | 0.0000 | 1990.0 | 1653.9 |
|
||||
| 50 | 0.0527 | 0.2812 | 0.000000 | 0.0000 | 2067.7 | 1702.9 |
|
||||
| 51 | 0.0449 | 0.2500 | 0.000000 | 0.0000 | 2055.8 | 1652.5 |
|
||||
| 52 | 0.0625 | 0.2969 | 0.000000 | 0.0000 | 1952.1 | 1574.7 |
|
||||
| 53 | 0.1055 | 0.3906 | 0.000000 | 0.0000 | 1895.8 | 1444.7 |
|
||||
| 54 | 0.1113 | 0.4062 | 0.000000 | 0.0000 | 2121.6 | 1704.4 |
|
||||
| 55 | 0.0938 | 0.2969 | 0.000000 | 0.0000 | 2030.4 | 1566.1 |
|
||||
|
||||
## Timing Analysis
|
||||
|
||||
### Average Time Breakdown (% of step time)
|
||||
|
||||
| Component | Avg % of Step Time |
|
||||
|-----------|-------------------|
|
||||
| wait_for_generation_buffer | 79.2% |
|
||||
| run_training | 19.5% |
|
||||
| train_critic_and_policy | 17.0% |
|
||||
| policy_train | 17.0% |
|
||||
| fwd_logprobs_values_reward | 2.5% |
|
||||
| sync_weights | 1.0% |
|
||||
| save_checkpoints | 0.6% |
|
||||
| save_hf_model | 0.3% |
|
||||
| convert_to_training_input | 0.2% |
|
||||
| cleanup_old_checkpoints | 0.2% |
|
||||
| compute_advantages_and_returns | 0.0% |
|
||||
|
||||
## Cross-Log Comparison
|
||||
|
||||
| Log | Avg Reward | Pass@8 | Step Time (s) | Gen Wait Time (s) | Avg Tokens | Staleness |
|
||||
|-----|------|------|------|------|------|------|
|
||||
| job_665437 | 0.0648 | 0.2755 | 2174.6925 | 1672.0259 | 5850.5668 | 1.2747 |
|
||||
| job_665440 | 0.0717 | 0.3122 | 2227.9568 | 1816.1628 | 3557.0874 | 1.1373 |
|
||||
| job_665442 | 0.0757 | 0.3031 | 2125.0121 | 1739.6845 | 3310.6962 | 1.1275 |
|
||||
|
||||
## vLLM Inference Engine Analysis
|
||||
|
||||
Metrics from vLLM stat loggers (V1LoggingStatLoggerFixed).
|
||||
|
||||
> **Note**: Ray deduplicates similar log messages with `[repeated Nx across cluster]`,
|
||||
> so we typically capture stats from one engine per timestamp. The stats shown are
|
||||
> **per-engine** values. Multiply by num_inference_engines for cluster-wide estimates.
|
||||
|
||||
### Summary by Log (Per-Engine Stats)
|
||||
|
||||
| Log | Avg Running/Engine | Avg Waiting/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % |
|
||||
|-----|-------------------|-------------------|--------------------------|----------------|------------------|
|
||||
| job_665437 | 6.4 | 0.0 | 282.8 tok/s | 32.9% | 92.7% |
|
||||
| job_665440 | 7.9 | 0.1 | 340.1 tok/s | 41.0% | 87.9% |
|
||||
| job_665442 | 8.0 | 0.1 | 346.4 tok/s | 41.4% | 88.4% |
|
||||
|
||||
### Utilization Analysis (Per-Engine)
|
||||
|
||||
Key indicators of inference engine utilization:
|
||||
|
||||
- **Running requests/engine**: Concurrent requests being processed by each engine
|
||||
- **Waiting requests**: Requests queued (0 = engine not saturated, has spare capacity)
|
||||
- **Generation throughput**: Decode tokens/sec per engine
|
||||
- 8B model on H100 can do **1000+ tok/s** when saturated
|
||||
- If seeing <300 tok/s with 0 waiting, engine is **starved for requests**
|
||||
|
||||
#### job_665437
|
||||
|
||||
- **Running requests/engine**: avg=6.4, max=23
|
||||
- **Waiting requests**: avg=0.0, max=9
|
||||
- **Generation throughput/engine**: avg=282.8 tok/s, max=779.1 tok/s
|
||||
- **KV cache usage**: avg=32.9%
|
||||
- **Prefix cache hit rate**: avg=92.7%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
#### job_665440
|
||||
|
||||
- **Running requests/engine**: avg=7.9, max=24
|
||||
- **Waiting requests**: avg=0.1, max=19
|
||||
- **Generation throughput/engine**: avg=340.1 tok/s, max=741.5 tok/s
|
||||
- **KV cache usage**: avg=41.0%
|
||||
- **Prefix cache hit rate**: avg=87.9%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
#### job_665442
|
||||
|
||||
- **Running requests/engine**: avg=8.0, max=24
|
||||
- **Waiting requests**: avg=0.1, max=20
|
||||
- **Generation throughput/engine**: avg=346.4 tok/s, max=826.3 tok/s
|
||||
- **KV cache usage**: avg=41.4%
|
||||
- **Prefix cache hit rate**: avg=88.4%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
## Trial-Level Analysis (from result.json)
|
||||
|
||||
Total trials parsed: 30520
|
||||
|
||||
### Turn Count Statistics
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Mean | 27.6 |
|
||||
| Median | 26.0 |
|
||||
| Std | 14.8 |
|
||||
| Min | 1 |
|
||||
| Max | 149 |
|
||||
| Count | 30459 |
|
||||
|
||||
### Exception Distribution
|
||||
|
||||
| Exception Type | Count | % |
|
||||
|---------------|-------|---|
|
||||
| ContextLengthExceededError | 23182 | 76.0% |
|
||||
| No exception | 3900 | 12.8% |
|
||||
| AgentTimeoutError | 2448 | 8.0% |
|
||||
| DaytonaError | 934 | 3.1% |
|
||||
| DaytonaAuthenticationError | 26 | 0.1% |
|
||||
| DaytonaValidationError | 19 | 0.1% |
|
||||
| RuntimeError | 8 | 0.0% |
|
||||
| VerifierTimeoutError | 1 | 0.0% |
|
||||
| BadRequestError | 1 | 0.0% |
|
||||
| EnvironmentStartTimeoutError | 1 | 0.0% |
|
||||
|
||||
### Turn Count by Exception Type
|
||||
|
||||
| Exception Type | Mean Turns | Median Turns | Count |
|
||||
|---------------|-----------|-------------|-------|
|
||||
| BadRequestError | 51.0 | 51.0 | 1 |
|
||||
| VerifierTimeoutError | 38.0 | 38.0 | 1 |
|
||||
| No exception | 30.0 | 30.0 | 3900 |
|
||||
| RuntimeError | 28.9 | 30.5 | 8 |
|
||||
| ContextLengthExceededError | 27.9 | 26.0 | 23182 |
|
||||
| AgentTimeoutError | 26.9 | 23.0 | 2448 |
|
||||
| DaytonaValidationError | 20.2 | 19.0 | 19 |
|
||||
| DaytonaAuthenticationError | 13.7 | 8.0 | 26 |
|
||||
| DaytonaError | 13.7 | 11.0 | 874 |
|
||||
|
||||
### Turn Count by Outcome
|
||||
|
||||
| Outcome | Mean Turns | Median Turns | Count |
|
||||
|---------|-----------|-------------|-------|
|
||||
| Success | 28.7 | 28.0 | 2214 |
|
||||
| Failure | 28.8 | 28.0 | 4120 |
|
||||
|
||||
### Reward Summary
|
||||
|
||||
- Mean reward: 0.3495
|
||||
- Success rate: 35.0%
|
||||
- Trials with reward data: 6334
|
||||
|
||||
58
training_logs/20260611_223001_metrics_table.csv
Normal file
58
training_logs/20260611_223001_metrics_table.csv
Normal file
@@ -0,0 +1,58 @@
|
||||
log_file,async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_BadRequestError,batch_errors/total_BadRequestError,batch_errors/avg_EnvironmentStartTimeoutError,batch_errors/total_EnvironmentStartTimeoutError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,global_step
|
||||
job_665437,0.0,0,64,512,0,0.0,0,0.0,11349.9688,22580.7742,10626.1538,30811,1,11492.5734,0.0605,0.0059,0.1379,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.074,0.0,0.0,1.0,0.0,0.0087,0.3125,0.0605,12.6198,50.7038,629.8245,26.6,857.9675,228.143,0.1719,5.8266,95.3526,522.8761,618.6733,5196.8333,18.3245,523.1484,4554.0085,0,1,126,1008,543,110,464,2.6031746031746033,328,1.119047619047619,141.0,0.015873015873015872,2.0,0.007936507936507936,1.0,,,,,,,,,,,,,,1
|
||||
job_665437,0.0,0,64,512,1,0.9844,0,0.9844,11598.0488,23352.6071,10918.0331,31129,1,11634.256,0.0547,0.0019,0.104,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0748,0.0,0.0,1.0,0.0,0.0075,0.25,0.0547,13.7325,50.9559,625.2936,27.1,857.9675,232.6739,0.0713,5.7945,93.5595,504.3783,598.3009,2571.1786,18.8207,504.6697,1948.26,0,2,62,496,222,57,274,3.3548387096774195,208,1.1129032258064515,69.0,0.04838709677419355,3.0,,,72.3958,83.1661,,,,,,,,,,,,2
|
||||
job_665437,0.0,0,64,512,2,1.6875,1,1.0,9616.832,23228.3871,8739.58,31095,1,11337.0222,0.0605,0.005,0.1221,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0647,0.0,0.0,1.0,0.0,0.0074,0.3125,0.0605,14.3281,51.0302,618.9982,27.9,857.9675,238.9693,0.072,5.266,81.9794,485.583,567.9016,1492.9544,18.2728,485.8498,901.5112,0,3,63,504,188,62,316,3.5873015873015874,226,1.507936507936508,95.0,,,,,,,,,,,,,,,,,,3
|
||||
job_665437,0.0,0,64,512,3,1.75,1,1.0,8344.4121,21399.1556,7086.4604,31383,1,11438.0109,0.0879,-0.0103,0.1777,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0548,0.0,0.0,1.0,0.0,0.0093,0.4062,0.0879,14.506,51.261,614.4324,28.4,857.9675,243.5351,0.0742,5.254,75.7532,482.7177,558.8002,2604.307,17.6001,482.9725,2022.6498,0,4,62,496,202,60,294,2.774193548387097,172,2.0483870967741935,127.0,0.04838709677419355,3.0,,,0.0089,12.4763,,,,,,,,,,,,4
|
||||
job_665437,0.0,0,64,512,3,1.5312,1,1.0,9046.1094,21562.6481,7570.3603,31325,1,11605.7658,0.1055,-0.0012,0.1564,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0612,0.0,0.0,1.0,0.0,0.008,0.4375,0.1055,14.6636,51.4144,610.7278,28.8,857.9675,247.2397,0.0742,5.3358,76.8331,494.4498,571.6047,2193.9348,18.0363,494.6971,1598.9545,0,5,62,496,141,61,348,3.532258064516129,219,2.370967741935484,147.0,0.016129032258064516,1.0,,,,,9.0806,0.11290322580645161,7.0,,,,,,,,,5
|
||||
job_665437,0.0,0,64,512,2,1.4219,1,1.0,6122.6758,20356.6604,4479.1002,31353,1,10363.1029,0.1035,0.0073,0.184,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0426,-0.002,0.0,1.0,0.0,0.007,0.3438,0.1035,15.0129,51.3718,607.0053,29.3,857.9675,250.9622,0.0705,4.9823,67.0112,449.3728,516.7236,2005.3759,17.9401,449.6414,1465.7263,0,6,64,512,140,64,353,2.8125,180,2.671875,171.0,0.03125,2.0,,,0.0089,8.5757,,0.296875,19.0,,,,,,,,,6
|
||||
job_665437,0.0,0,64,512,3,1.5312,1,1.0,5639.3535,20543.9556,4203.1499,31125,1,9774.0402,0.0879,0.0147,0.1893,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0426,-0.0034,0.0,1.0,0.0,0.0136,0.3906,0.0879,14.9615,51.363,603.5803,29.6,857.9675,254.3873,0.0697,4.66,52.0849,398.0136,450.4269,2106.1877,22.0007,398.272,1629.0969,0,7,63,504,75,63,429,6.158730158730159,388,0.7142857142857143,45.0,0.031746031746031744,2.0,,,,,,,,,,,,,,,,7
|
||||
job_665437,0.0,0,64,512,2,1.5312,1,1.0,3320.0449,25125.6923,2751.9619,31029,1,8287.569,0.0254,-0.0009,0.0917,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.022,-0.0,0.0,1.0,0.0,0.0037,0.1562,0.0254,14.8983,51.2988,605.2846,29.5,857.9675,252.6829,0.0665,4.4222,37.9972,334.0714,372.3853,1636.3839,20.0642,334.3211,1239.5085,0,8,62,496,104,60,392,6.290322580645161,390,0.12903225806451613,8.0,,,,,0.0089,8.5925,,,,,,,,,,,,8
|
||||
job_665437,0.0,0,64,512,2,1.4219,0,0.9219,4733.4922,20816.8929,3803.0475,30692,1,9672.5683,0.0547,-0.0003,0.0654,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0286,0.0,0.0,1.0,0.0,0.0036,0.2344,0.0547,14.8563,51.3198,602.4516,29.8,857.9675,255.5159,0.0649,4.5125,56.4992,420.1371,476.9557,1936.8703,18.4296,420.3912,1436.9688,0,9,65,520,105,64,414,6.292307692307692,409,0.07692307692307693,5.0,,,0.015384615384615385,1.0,,,,,,,,,,,,,,9
|
||||
job_665437,0.0,0,64,512,3,1.2969,0,0.9688,4541.293,22735.3846,3567.9465,30723,1,9361.45,0.0508,0.0059,0.0708,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0294,-0.0,0.0,1.0,0.0,0.0038,0.1719,0.0508,14.8421,51.3412,624.4957,27.2,857.9675,233.4719,0.0644,4.4072,59.7586,406.8498,466.9292,1774.4441,18.6093,407.1058,1284.4944,0,10,63,504,90,63,414,6.4603174603174605,407,0.1111111111111111,7.0,,,,,0.0086,9.0343,6.9689,,,,,,,,,,,10
|
||||
job_665437,0.0,0,64,512,3,1.2969,0,0.9531,4183.541,22342.2632,3483.712,30184,1,9316.038,0.0371,-0.0043,0.0994,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0253,0.0,0.0,1.0,0.0,0.0062,0.1562,0.0371,14.9382,51.4313,619.8862,27.7,857.9675,238.0814,0.0623,4.4976,54.8936,365.088,420.2978,1815.6224,19.5224,365.3414,1371.3005,0,11,65,520,100,62,420,6.446153846153846,419,0.015384615384615385,1.0,,,,,,,,,,,,,,,,,,11
|
||||
job_665437,0.0,0,64,512,2,1.2188,0,0.9062,4316.6074,20771.1176,3751.503,30890,1,9198.6209,0.0332,0.0016,0.1317,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0268,0.0,0.0,1.0,0.0,0.0057,0.1875,0.0332,14.9169,51.3809,615.1475,28.3,857.9675,242.8201,0.0645,4.4862,56.4202,386.4744,443.2228,1963.7998,19.8882,386.7377,1496.198,0,12,63,504,129,60,374,5.904761904761905,372,0.015873015873015872,1.0,0.015873015873015872,1.0,,,0.0085,8.5349,,,,0.015873015873015872,1.0,,,,,,,12
|
||||
job_665437,0.0,0,64,512,2,1.2188,0,0.9688,5632.6406,23748.3261,3844.397,31136,1,10271.1862,0.0898,-0.0018,0.1381,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0362,-0.0,0.0,1.0,0.0,0.0062,0.2656,0.0898,14.9247,51.4518,611.0772,28.8,857.9675,246.8903,0.0664,4.7589,69.6389,453.1235,523.1041,2030.7998,20.2669,453.398,1482.6651,0,13,63,504,85,62,419,6.634920634920635,418,0.015873015873015872,1.0,,,,,,,,,,,,,,,,,,13
|
||||
job_665437,0.0,0,64,512,3,1.2344,0,0.9219,3788.7988,21801.4333,2667.6805,31002,1,8774.0077,0.0586,0.0024,0.156,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0251,0.0,0.0,1.0,0.0,0.0058,0.3125,0.0586,15.1205,51.7013,607.4822,29.2,857.9675,250.4854,0.0651,4.506,44.243,363.9317,408.5352,1839.41,20.9491,364.2267,1405.4149,0,14,65,520,113,64,407,6.184615384615385,402,0.07692307692307693,5.0,,,,,0.0087,8.6444,,,,,,,,,,,,14
|
||||
job_665437,0.0,0,64,512,3,1.2812,0,0.9688,4575.3594,21459.3256,3027.3625,31012,1,9291.0922,0.084,-0.004,0.1396,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0315,0.0,0.0,1.0,0.0,0.005,0.3125,0.084,15.1204,51.7007,627.364,26.9,857.9675,230.6035,0.0675,4.5468,58.713,400.9189,459.961,2154.3582,18.5589,401.1802,1671.2874,0,15,64,512,91,64,421,6.328125,405,0.25,16.0,,,,,,,6.0659,,,,,,,,,,,15
|
||||
job_665437,0.0,0,64,512,2,1.2812,0,0.9531,4289.8574,22854.8077,3296.6708,31292,1,9569.7061,0.0508,-0.0026,0.1121,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0242,0.0,0.0,1.0,0.0,0.0041,0.2344,0.0508,15.1272,51.7703,616.1652,28.2,857.9675,241.8024,0.0666,4.624,60.7552,426.8336,487.9325,1964.7559,19.134,427.1104,1453.0604,0,16,64,512,94,64,418,6.515625,417,0.015625,1.0,,,,,0.0093,8.6404,,,,,,,,,,,,16
|
||||
job_665437,0.0,0,64,512,2,1.2656,0,0.9375,3983.7305,21573.5135,2613.5789,31181,1,8949.1562,0.0723,-0.0027,0.0771,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0252,0.0,0.0,1.0,0.0,0.0047,0.2812,0.0723,15.2764,51.8998,611.1328,28.8,857.9675,246.8347,0.0643,4.5877,52.6894,376.4956,429.5203,2037.089,23.5187,376.7662,1579.4581,0,17,63,504,85,63,419,6.603174603174603,416,0.047619047619047616,3.0,,,,,,,,,,,,,,,,,,17
|
||||
job_665437,0.0,0,64,512,3,1.1094,0,0.9062,3694.9473,20101.1053,2379.6857,31220,1,8633.844,0.0742,0.0185,0.1882,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0271,0.0,0.0,1.0,0.0,0.0061,0.2969,0.0742,15.1888,51.8959,607.5901,29.2,857.9675,250.3774,0.0644,4.4667,49.5139,338.9372,388.7806,2029.453,19.3153,339.2019,1616.886,0,18,62,496,55,62,441,7.17741935483871,445,0.06451612903225806,4.0,,,,,0.0101,8.498,,,,,,,,,,,,18
|
||||
job_665437,0.0,0,64,512,3,1.1562,0,0.9062,2383.0566,23984.7,1504.9411,31350,1,7274.8501,0.0391,0.0024,0.1832,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0149,0.0,0.0,1.0,0.0,0.0032,0.1719,0.0391,15.341,52.0204,604.2086,29.6,857.9675,253.759,0.0633,4.2026,38.1289,293.2502,331.6945,1965.399,18.4552,293.502,1611.0432,0,19,33,264,77,32,187,6.03030303030303,199,0.12121212121212122,4.0,,,,,,,,,,,,,,,,,,19
|
||||
job_665440,0.0,0,64,512,3,0.2969,0,0.2031,3022.0449,22315.9,2237.7419,31165,1,7996.9892,0.0391,0.0031,0.0794,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0193,-0.0,0.0,1.0,0.0,0.0025,0.1667,0.0391,12.3074,50.4413,646.5037,24.6,857.9676,211.4639,0.1937,4.3836,44.3156,318.3362,363.1298,4563.2206,27.4975,318.619,4168.2093,0,19,63,504,114,63,389,6.0476190476190474,381,0.12698412698412698,8.0,,,,,,,,,,0.015873015873015872,1.0,,,,,,,19
|
||||
job_665440,0.0,0,64,512,1,0.8125,0,0.8125,4230.1367,19613.1224,2602.1317,30906,1,8827.526,0.0957,0.0009,0.0974,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0321,-0.0,0.0,1.0,0.0,0.0041,0.3281,0.0957,12.5155,50.6252,635.8582,25.9,857.9676,222.1094,0.0746,4.5471,49.3255,368.1026,417.799,2729.1254,22.4617,368.3983,2284.3116,0,20,62,496,106,62,390,6.209677419354839,385,0.08064516129032258,5.0,,,,,57.2769,66.1821,6.5415,,,,,,,,,,,20
|
||||
job_665440,0.0,0,64,512,2,1.25,0,0.9844,4219.9961,20052.2075,2391.8758,30440,1,8988.1183,0.1035,-0.0052,0.1336,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0288,-0.0,0.0,1.0,0.0,0.0041,0.4844,0.1035,12.7537,50.898,644.9331,24.8,857.9676,213.0345,0.0768,4.5436,47.9568,380.8666,429.2084,1786.6399,19.8241,381.1743,1333.0593,0,21,65,520,96,64,421,6.338461538461538,412,0.13846153846153847,9.0,,,0.046153846153846156,3.0,,,,,,,,,,,,,,21
|
||||
job_665440,0.0,0,64,512,2,1.3281,0,0.9844,4372.1699,24130.0789,2788.2025,30885,1,9441.7563,0.0742,-0.0074,0.207,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0252,-0.0,0.0,1.0,0.0,0.0052,0.25,0.0742,14.2454,51.0239,633.0668,26.2,857.9676,224.9008,0.0921,4.5941,60.8765,413.3934,474.6959,1801.2744,22.7048,413.7268,1299.2727,0,22,62,496,77,62,419,6.709677419354839,416,0.04838709677419355,3.0,,,,,0.0107,9.0301,,,,,,,,,,,,22
|
||||
job_665440,0.0,0,64,512,3,1.25,0,0.9062,3215.9902,21851.5,2056.1037,30955,1,8045.2464,0.0586,0.001,0.064,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0222,0.0,0.0,1.0,0.0,0.0023,0.2812,0.0586,14.4943,51.246,625.9828,27.0,857.9676,231.9848,0.0911,4.5094,41.2756,320.365,362.0259,2312.7157,22.3457,320.6588,1923.8274,0,23,62,496,79,61,412,6.596774193548387,409,0.04838709677419355,3.0,,,0.06451612903225806,4.0,,,,,,,,0.016129032258064516,1.0,,,,,23
|
||||
job_665440,0.0,0,64,512,2,1.2344,0,0.9531,3154.5469,23362.32,2117.1869,31010,1,8107.5049,0.0488,0.0051,0.0922,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0208,0.0,0.0,1.0,0.0,0.003,0.2656,0.0488,14.6713,51.4477,620.8464,27.6,857.9676,237.1212,0.0711,4.4954,58.7083,341.94,401.0182,1958.7534,23.8914,342.2355,1529.3403,0,24,63,504,72,63,432,6.857142857142857,432,,,,,,,0.0106,8.6578,,,,,,,,,,,,24
|
||||
job_665440,0.0,0,64,512,2,1.0781,0,0.9531,2907.0625,21687.931,1779.4327,31250,1,7636.6044,0.0566,-0.0094,0.0978,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0193,0.0,0.0,1.0,0.0,0.0031,0.2656,0.0566,14.7885,51.5618,616.0342,28.2,857.9676,241.9334,0.0993,4.4024,38.8754,289.1906,328.471,1989.6963,24.4896,289.4959,1632.3275,0,25,64,512,99,63,413,6.453125,413,,,,,,,,,6.2536,,,,,,,,,,,25
|
||||
job_665440,0.0,0,64,512,2,1.2344,0,0.9375,3729.0488,21425.7838,2350.5663,31470,1,8410.0012,0.0723,0.0097,0.0629,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0279,0.0,0.0,1.0,0.0,0.0047,0.2656,0.0723,14.6334,51.5005,611.8091,28.7,857.9676,246.1584,0.0724,4.5488,54.3068,363.9403,418.6124,2378.7009,23.3033,364.2328,1932.2312,0,26,64,512,81,64,430,6.71875,430,,,,,,,0.0106,8.6989,,,,0.015625,1.0,,,,,,,26
|
||||
job_665440,0.0,0,64,512,2,1.1562,0,0.9375,2768.5215,19034.7407,1862.9794,31316,1,7084.0848,0.0527,-0.007,0.0706,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0241,0.0,0.0,1.0,0.0,0.0033,0.3125,0.0527,14.639,51.476,606.791,29.3,857.9676,251.1766,0.0967,4.3642,38.7391,265.2822,304.404,1956.0438,22.7867,265.5676,1624.4803,0,27,64,512,104,64,406,6.328125,405,0.015625,1.0,,,0.03125,2.0,,,,,,,,,,,,,,27
|
||||
job_665440,0.0,0,64,512,2,1.1562,0,0.9062,3543.5586,16543.7941,2618.8556,31037,1,7945.2767,0.0664,-0.034,0.1257,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0301,-0.0,0.0,1.0,0.0,0.0066,0.3125,0.0664,14.9626,51.8556,601.9171,29.8,857.9676,256.0505,0.0968,4.514,42.3834,329.7107,372.5014,2185.9777,23.2048,330.0208,1785.752,0,28,63,504,130,62,372,5.888888888888889,371,0.015873015873015872,1.0,,,0.015873015873015872,1.0,0.0114,8.8621,,,,0.015873015873015872,1.0,,,,,,,28
|
||||
job_665440,0.0,0,64,512,3,1.3594,0,0.9688,5395.25,21022.9091,3514.4595,30429,1,9793.3681,0.1074,-0.0042,0.0913,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0358,0.0,0.0,1.0,0.0,0.0037,0.375,0.1074,15.0327,51.9991,598.9776,30.2,857.9676,258.99,0.073,4.672,65.2871,400.398,466.0685,2020.9027,22.9964,400.708,1527.1586,0,29,64,512,88,64,424,6.609375,423,0.015625,1.0,,,,,,,,,,,,,,,,,,29
|
||||
job_665440,0.0,0,64,512,2,1.2812,0,0.9375,3439.0098,21578.2895,1984.8059,31430,1,8243.3078,0.0742,-0.0041,0.0935,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0245,0.0,0.0,1.0,0.0,0.0037,0.375,0.0742,15.2515,52.2041,595.3509,30.6,857.9676,262.6167,0.073,4.5383,41.9493,327.7638,370.0848,1990.2611,23.8074,328.0617,1591.8244,0,30,63,504,84,63,420,6.666666666666667,420,,,,,,,0.0094,8.8468,7.2216,,,,,,,,,,,30
|
||||
job_665440,0.0,0,64,512,2,1.2188,0,0.9688,3080.7031,18103.9032,2112.4719,30682,1,7689.1389,0.0605,-0.0081,0.0685,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0239,0.0,0.0,1.0,0.0,0.0026,0.3125,0.0605,15.1851,52.2018,592.6693,30.9,857.9676,265.2983,0.0706,4.344,37.8148,304.6191,342.7822,2032.1933,21.4183,304.8963,1663.6431,0,31,65,520,69,65,451,6.938461538461539,451,,,,,,,,,,,,,,,,,,,,31
|
||||
job_665440,0.0,0,64,512,2,1.1094,0,0.9219,2563.5566,20302.4333,1459.4772,30461,1,7192.4905,0.0586,-0.0031,0.0958,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0182,0.0,0.0,1.0,0.0,0.0024,0.2812,0.0586,15.3267,52.3829,589.8547,31.2,857.9676,268.1129,0.0907,4.3146,39.5432,286.4901,326.4528,1954.6703,21.1101,286.8183,1602.7878,0,32,64,512,77,64,435,6.796875,435,,,,,,,0.0096,8.9937,,,,,,,,,,,,32
|
||||
job_665440,0.0,0,64,512,3,1.2031,0,0.9375,2973.4863,22900.6111,1466.3929,31035,1,7813.7646,0.0703,-0.0086,0.1611,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0216,0.0,0.0,1.0,0.0,0.0039,0.3125,0.0703,15.3817,52.5188,587.3306,31.5,857.9676,270.637,0.095,4.4646,42.5818,338.4778,381.4496,2037.4334,21.7785,338.7721,1629.7359,0,33,63,504,97,63,407,6.4603174603174605,407,,,,,,,,,,,,,,,,,,,,33
|
||||
job_665440,0.0,0,64,512,2,1.2656,0,0.9688,3865.0098,22052.9545,2155.0321,31394,1,8598.5383,0.0859,0.0022,0.083,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0266,0.0,0.0,1.0,0.0,0.0036,0.2812,0.0859,15.3569,52.457,585.365,31.8,857.9676,272.6026,0.0733,4.5461,48.1633,363.6179,412.1646,2066.1355,25.7698,363.9272,1623.65,0,34,64,512,94,64,417,6.515625,417,,,,,,,0.0086,8.6415,,,,0.015625,1.0,,,,,,,34
|
||||
job_665440,0.0,0,64,512,3,1.0625,0,0.8438,3385.8906,19282.4857,2219.4738,30976,1,7882.3298,0.0684,0.004,0.0497,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0283,0.0,0.0,1.0,0.0,0.003,0.3125,0.0684,15.4366,52.5654,583.2695,32.0,857.9676,274.6981,0.0964,4.5272,41.7149,294.8022,336.9157,2075.7662,27.2687,295.1037,1707.0507,0,35,62,496,103,61,393,6.338709677419355,393,,,,,,,,,7.2214,,,,,,,,,,,35
|
||||
job_665440,0.0,0,64,512,2,1.1875,0,0.9375,4324.877,22049.9231,2321.1761,30678,1,8858.8066,0.1016,0.0046,0.155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0324,0.0,0.0,1.0,0.0,0.005,0.3281,0.1016,15.4926,52.6221,638.7303,25.6,857.9676,219.2373,0.105,5.0488,56.6453,374.4842,431.4993,2465.1235,23.9408,374.7484,2004.63,0,36,63,504,88,63,416,6.571428571428571,414,0.031746031746031744,2.0,,,,,0.0086,9.2812,,,,,,,,,,,,36
|
||||
job_665440,0.0,0,64,512,2,1.125,0,0.9219,3393.8008,19400.5143,2219.304,30734,1,7995.2371,0.0684,-0.0075,0.0638,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0239,0.0,0.0,1.0,0.0,0.0027,0.4219,0.0684,15.3723,52.5737,631.1822,26.4,857.9676,226.7854,0.097,4.4162,39.207,318.2496,357.8584,2026.5443,20.4635,318.5539,1643.8015,0,37,16,128,21,16,107,6.0625,97,0.625,10.0,,,,,,,,,,,,,,,,,,37
|
||||
job_665442,0.0,0,64,512,2,0.3906,0,0.2656,3568.8223,21907.375,2014.7076,30407,1,8511.6759,0.0781,0.0064,0.1634,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0225,0.0,0.0,1.0,0.0,0.0057,0.4615,0.0781,13.1506,50.6884,595.8149,30.6,857.9676,262.1526,0.1993,4.4228,47.6739,348.1637,396.2853,4273.6039,21.8023,348.411,3851.0932,0,37,64,512,93,62,419,6.515625,417,0.03125,2.0,,,,,,,,,,,,,,,,,,37
|
||||
job_665442,0.0,0,64,512,1,0.7969,0,0.7969,3505.293,19578.125,1842.5862,30440,1,8178.2537,0.0938,-0.005,0.0693,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0252,-0.0,0.0,1.0,0.0,0.0031,0.3906,0.0938,13.3729,50.9678,592.067,31.0,857.9676,265.9006,0.0741,4.7716,39.8009,285.4323,325.5524,2002.5702,22.0107,285.6765,1650.2284,0,38,60,480,87,59,392,6.516666666666667,391,,,,,,,11.067,19.6408,,,,0.016666666666666666,1.0,,,0.016666666666666666,1.0,,,38
|
||||
job_665442,0.0,0,64,512,2,1.2656,1,1.0,3434.6777,19201.2703,2206.5432,30799,1,8150.2939,0.0723,0.0149,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0261,-0.0015,0.0,1.0,0.0,0.0044,0.2656,0.0723,13.4437,51.0087,589.7198,31.3,857.9676,268.2477,0.0988,4.5332,42.6766,320.8587,363.9235,1968.0577,21.1464,321.1474,1578.4515,0,39,65,520,108,63,412,6.323076923076923,411,0.015384615384615385,1.0,,,,,,,,,,,,,,,,,,39
|
||||
job_665442,0.0,0,64,512,2,1.25,0,0.9688,4040.6562,20375.0652,2428.2468,30911,1,8528.7211,0.0898,0.0048,0.0217,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0286,0.0,0.0,1.0,0.0,0.0019,0.2656,0.0898,13.5471,51.202,586.9761,31.6,857.9676,270.9915,0.098,4.4569,55.8701,358.8919,415.1705,2163.5657,21.276,359.202,1722.6557,0,40,64,512,79,61,432,6.734375,431,0.015625,1.0,,,,,0.0091,8.3809,7.1735,,,0.015625,1.0,,,,,,,40
|
||||
job_665442,0.0,0,64,512,2,1.2812,1,1.0,2970.6348,20093.5676,1636.8484,30923,1,7516.3108,0.0723,0.0064,0.1391,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0219,0.0,0.0,1.0,0.0,0.0039,0.2812,0.0723,13.4956,51.2211,584.994,31.8,857.9676,272.9736,0.0953,4.41,36.7656,283.113,320.2793,1606.033,22.9696,283.4176,1258.3687,0,41,63,504,44,63,459,7.333333333333333,462,0.07936507936507936,5.0,,,0.015873015873015872,1.0,,,,,,,,,,,,,,41
|
||||
job_665442,0.0,0,64,512,2,1.1094,0,0.875,1634.5039,21094.6296,551.1567,30676,1,5948.5119,0.0527,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.012,0.0,0.0,1.0,0.0,0.0,0.1875,0.0527,13.5087,51.3518,582.3663,32.1,857.9676,275.6013,0.089,4.191,26.4419,202.4572,229.1957,2187.2531,27.9916,202.6644,1925.8702,0,42,62,496,93,62,403,6.5,403,,,,,,,0.0862,8.3963,,,,,,,,,,,,42
|
||||
job_665442,0.0,0,64,512,3,1.1562,0,0.9531,3687.9473,21659.9787,1871.4194,31193,1,8273.0941,0.0918,0.0168,0.1209,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0287,0.0,0.0,1.0,0.0,0.0048,0.375,0.0918,14.9308,51.7346,578.3634,32.6,857.9676,279.6042,0.0956,4.4952,41.1548,304.5746,346.1633,2201.8784,22.719,304.9124,1828.4937,0,43,64,512,99,63,413,6.421875,411,0.03125,2.0,,,,,,,,,,,,,,,,,,43
|
||||
job_665442,0.0,0,64,512,2,1.125,0,0.9688,3553.6445,22041.7308,2564.57,31011,1,8173.3769,0.0508,0.0086,0.0768,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0285,0.0,0.0,1.0,0.0,0.0033,0.25,0.0508,14.9455,51.8056,576.2864,32.8,857.9676,281.6812,0.0946,4.4365,48.5577,340.6085,389.5569,2153.5102,21.1603,340.9042,1738.3506,0,44,62,496,83,60,413,6.661290322580645,413,,,,,,,0.0088,8.5905,,,,,,,,,,,,44
|
||||
job_665442,0.0,0,64,512,2,1.2031,0,0.9219,3265.6094,21753.6061,1991.9061,31046,1,7954.943,0.0645,0.0059,0.0779,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0234,0.0,0.0,1.0,0.0,0.0028,0.2656,0.0645,14.9641,51.8647,575.2365,33.0,857.9676,282.7311,0.084,4.3381,52.0341,323.9391,376.3528,1721.3545,21.9444,324.2342,1318.7133,0,45,64,512,75,62,437,6.828125,437,,,,,,,,,5.9288,,,,,,,,,,,45
|
||||
job_665442,0.0,0,64,512,3,1.2812,0,0.9375,3055.5059,20803.6905,1469.4979,30660,1,7938.0939,0.082,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0215,0.0,0.0,1.0,0.0,0.0,0.3125,0.082,15.0172,51.9069,572.0082,33.3,857.9676,285.9594,0.0835,4.4024,49.0487,304.4824,353.9057,2110.5882,23.6305,304.7727,1728.645,0,46,64,512,84,64,428,6.6875,428,,,,,,,0.0086,8.8655,,,,,,,,,,,,46
|
||||
job_665442,0.0,0,64,512,2,1.0938,0,0.9062,3168.8477,20393.0222,1509.1306,31277,1,7799.7708,0.0879,-0.0057,0.0317,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0239,-0.0,0.0,1.0,0.0,0.0016,0.3125,0.0879,15.0454,51.9489,571.0099,33.4,857.9676,286.9576,0.0723,4.5105,38.3069,301.5151,340.1996,1938.5631,20.5692,301.8199,1573.2792,0,47,62,496,70,62,426,6.854838709677419,425,0.016129032258064516,1.0,,,,,,,,,,,,,,,,,,47
|
||||
job_665442,0.0,0,64,512,3,1.2344,0,0.8906,2789.0898,22593.3125,1468.8083,30387,1,7499.7521,0.0625,0.0054,0.0705,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0205,0.0,0.0,1.0,0.0,0.0024,0.2812,0.0625,15.1748,52.1106,569.3416,33.6,857.9676,288.626,0.0694,4.2181,44.0702,284.2664,328.6084,1934.7623,21.3574,284.4683,1580.5732,0,48,63,504,72,61,432,6.857142857142857,432,,,,,,,0.0082,8.8011,,,,,,,,,,,,48
|
||||
job_665442,0.0,0,64,512,2,1.1094,0,0.875,2950.2188,22200.0571,1537.7568,30187,1,7764.5758,0.0684,-0.0038,0.0896,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0195,0.0,0.0,1.0,0.0,0.003,0.1875,0.0684,14.8967,52.1049,567.3762,33.9,857.9676,290.5914,0.109,4.5224,38.8775,267.5912,306.8889,1990.0106,24.6606,267.9018,1653.9126,0,49,64,512,74,63,438,6.84375,438,,,,,,,,,,,,,,,,,,,,49
|
||||
job_665442,0.0,0,64,512,2,1.0,0,0.9062,2849.5039,20867.0,1846.468,30412,1,7502.0696,0.0527,0.0135,0.1585,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0201,0.0,0.0,1.0,0.0,0.0036,0.2812,0.0527,14.9523,52.1703,565.6843,34.1,857.9676,292.2833,0.0959,4.4261,47.8666,291.3579,339.5654,2067.6726,20.7362,291.6023,1702.9395,0,50,64,512,71,63,440,6.859375,439,0.015625,1.0,,,,,0.0085,8.1311,6.4196,,,0.015625,1.0,,,,,,,50
|
||||
job_665442,0.0,0,64,512,2,1.0781,0,0.9219,2999.1387,23140.8261,2051.7791,30773,1,7861.7815,0.0449,0.0053,0.0746,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0196,0.0,0.0,1.0,0.0,0.0036,0.25,0.0449,15.1374,52.4918,564.2138,34.2,857.9676,293.7538,0.0765,4.3451,55.9953,320.3913,376.787,2055.8264,22.1486,320.7136,1652.5417,0,51,64,512,81,64,431,6.734375,431,,,,,,,,,,,,,,,,,,,,51
|
||||
job_665442,0.0,0,64,512,2,1.2969,0,0.9688,3029.0996,18844.75,1974.7229,30513,1,7642.7286,0.0625,-0.0001,0.0309,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0222,0.0,0.0,1.0,0.0,0.002,0.2969,0.0625,15.1295,52.4838,563.5904,34.3,857.9676,294.3772,0.0858,4.4522,48.0109,302.477,350.8653,1952.0873,22.0891,302.768,1574.6726,0,52,63,504,104,61,400,6.349206349206349,400,,,,,,,0.008,8.1584,,,,,,,,,,,,52
|
||||
job_665442,0.0,0,64,512,3,1.2812,0,0.9375,4303.8984,21060.6667,2328.2096,30373,1,9028.9972,0.1055,-0.0004,0.1114,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0283,0.0,0.0,1.0,0.0,0.004,0.3906,0.1055,15.2133,52.6156,565.0471,34.1,857.9676,292.9205,0.0674,4.6788,54.0249,372.0372,426.4382,1895.8166,20.0,372.3455,1444.6942,0,53,64,512,107,62,405,6.328125,405,,,,,,,,,,,,,,,,,,,,53
|
||||
job_665442,0.0,0,64,512,3,1.2656,0,0.9531,4038.2031,20528.7193,1972.3582,30709,1,8486.9036,0.1113,-0.0066,0.1182,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0284,0.0,0.0,1.0,0.0,0.0041,0.4062,0.1113,15.3165,52.8245,563.9276,34.3,857.9676,294.04,0.0763,4.518,51.3008,338.4849,390.1408,2121.6477,22.5942,338.7631,1704.39,0,54,64,512,102,62,404,6.296875,403,0.015625,1.0,,,0.09375,6.0,0.0089,8.1862,,,,,,,,,,,,54
|
||||
job_665442,0.0,0,64,512,2,1.2031,0,0.9688,4057.9336,20640.4583,2342.5,31296,1,8672.1578,0.0938,-0.0074,0.1057,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0288,0.0,0.0,1.0,0.0,0.0036,0.2969,0.0938,15.4248,52.9057,563.0505,34.4,857.9676,294.9171,0.0922,4.633,55.8487,379.3597,435.5265,2030.4286,24.1291,379.585,1566.1321,0,55,63,504,74,63,429,6.809523809523809,429,,,,,,,,,6.3546,,,,,,,,,0.015873015873015872,1.0,55
|
||||
|
3
training_logs/20260611_223001_reward_vs_steps.png
Normal file
3
training_logs/20260611_223001_reward_vs_steps.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:98ae1d9281aca621febb0cbca9b328f6231aae101736bf8cdcd9ce2b3b3cb82a
|
||||
size 245535
|
||||
30521
training_logs/20260611_223001_trial_results.csv
Normal file
30521
training_logs/20260611_223001_trial_results.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:31ad20f0255763b87a6a68e666eb557a231cd04995892dd7003218dd0fbf41c0
|
||||
size 139590
|
||||
8495
training_logs/20260611_223001_vllm_metrics_job_665437.csv
Normal file
8495
training_logs/20260611_223001_vllm_metrics_job_665437.csv
Normal file
File diff suppressed because it is too large
Load Diff
8499
training_logs/20260611_223001_vllm_metrics_job_665440.csv
Normal file
8499
training_logs/20260611_223001_vllm_metrics_job_665440.csv
Normal file
File diff suppressed because it is too large
Load Diff
8492
training_logs/20260611_223001_vllm_metrics_job_665442.csv
Normal file
8492
training_logs/20260611_223001_vllm_metrics_job_665442.csv
Normal file
File diff suppressed because it is too large
Load Diff
25484
training_logs/20260611_223001_vllm_metrics_table.csv
Normal file
25484
training_logs/20260611_223001_vllm_metrics_table.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e4ad3f040f265359c5e513bb1fda0e3dfeb2c42ba4c64c90ca3e815b92f7d6ff
|
||||
size 97083244
|
||||
@@ -0,0 +1,234 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_665438
|
||||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_665438
|
||||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_665438
|
||||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||||
[proxy] Tunnel port: 7003
|
||||
[proxy] Node IP: 10.128.32.1 (workers will connect here)
|
||||
[proxy] ✓ SSH tunnel started successfully
|
||||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_665438.conf
|
||||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||||
[proxy] ✓ Daytona timeout settings configured
|
||||
[proxy] Testing proxy connectivity...
|
||||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_665438.conf
|
||||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||||
[proxy] ⚠ Tunnel not accessible at 10.128.32.1:7003 (workers may fail)
|
||||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||||
=== Universal RL Training Runner ===
|
||||
Config: /e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/configs/a3-rl-SankalpKJ_swesmith-oracle-filtered_rl_config.json
|
||||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python version: Python 3.12.12
|
||||
UV_USE_IO_URING: 0
|
||||
Proxy: DISABLED (direct internet or not configured)
|
||||
========================================
|
||||
=== RLJobRunner: a3-rl-SankalpKJ_swesmith-oracle-filtered ===
|
||||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/wandb
|
||||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/wandb
|
||||
HF_TOKEN=****pDbg
|
||||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||||
Environment configured:
|
||||
TENSOR_PARALLEL_SIZE=1
|
||||
NUM_INFERENCE_ENGINES=56
|
||||
POLICY_NUM_NODES=14
|
||||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/wandb
|
||||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||||
Cleaning up existing Ray instances...
|
||||
=== Starting Ray Cluster ===
|
||||
Nodes: 14
|
||||
GPUs per node: 4
|
||||
CPUs per node: 288
|
||||
Head node: jpbo-041-01 (10.128.32.1)
|
||||
Ray port: 6379
|
||||
============================
|
||||
Starting Ray head on jpbo-041-01 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-01.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.1 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.1 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray head on jpbo-041-01
|
||||
Starting Ray worker on jpbo-041-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-05.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.5 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.5 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 1 on jpbo-041-05
|
||||
Starting Ray worker on jpbo-041-06 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-06.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.6 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.6 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 2 on jpbo-041-06
|
||||
Starting Ray worker on jpbo-041-07 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-07.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.7 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.7 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 3 on jpbo-041-07
|
||||
Starting Ray worker on jpbo-042-01 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-01.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.49 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.49 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 4 on jpbo-042-01
|
||||
Starting Ray worker on jpbo-042-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-02.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 5 on jpbo-042-02
|
||||
Starting Ray worker on jpbo-042-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-03.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 6 on jpbo-042-03
|
||||
Starting Ray worker on jpbo-042-06 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-06.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.54 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.54 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 7 on jpbo-042-06
|
||||
Starting Ray worker on jpbo-042-08 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-08.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.56 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-08 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.56 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 8 on jpbo-042-08
|
||||
Starting Ray worker on jpbo-042-11 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-11.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.59 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-11 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.59 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 9 on jpbo-042-11
|
||||
Starting Ray worker on jpbo-042-12 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-12.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.60 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.60 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 10 on jpbo-042-12
|
||||
Starting Ray worker on jpbo-042-13 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-13.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.61 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-13 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.61 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 11 on jpbo-042-13
|
||||
Starting Ray worker on jpbo-042-15 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-15.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.63 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-15 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.63 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 12 on jpbo-042-15
|
||||
Starting Ray worker on jpbo-042-16 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-042-16.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.64 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-042-16 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.1:6379 --node-ip-address=10.128.32.64 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 13 on jpbo-042-16
|
||||
Waiting for cluster (56 GPUs, 14 nodes)...
|
||||
Connecting to Ray at 10.128.32.1:6379 (expecting 14 nodes, 56.0 GPUs)
|
||||
[Ray wait] Connection attempt failed:
|
||||
[Ray wait] Retrying in 10s...
|
||||
srun stderr: 2026-06-09 10:05:27,649 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.1:6379...
|
||||
[2026-06-09 10:05:32,654 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:06:02,655 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:06:08,659 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:06:38,659 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:06:44,662 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:07:14,663 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:07:20,666 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:07:50,667 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:07:56,669 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:08:26,670 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:08:32,673 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:09:02,673 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:09:08,676 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:09:38,677 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:09:44,679 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:10:14,680 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:10:20,684 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:10:50,685 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:10:56,687 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:11:26,688 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:11:32,690 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:12:02,691 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:12:08,694 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:12:38,694 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:12:44,696 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:13:14,697 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:13:20,700 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:13:50,701 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:13:56,704 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:14:26,704 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:14:32,708 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:15:02,709 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:15:08,711 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:15:38,712 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:15:44,715 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:16:14,715 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:16:20,718 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:16:50,719 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
[2026-06-09 10:16:56,722 W 314444 314444] rpc_client.h:153: Failed to connect to GCS at address 10.128.32.1:6379 within 5 seconds.
|
||||
[2026-06-09 10:17:26,722 W 314444 314444] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
||||
main()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
||||
wait_for_cluster(
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 73, in wait_for_cluster
|
||||
raise TimeoutError(
|
||||
TimeoutError: Could not connect to Ray cluster at 10.128.32.1:6379 within 600 seconds
|
||||
srun: error: jpbo-041-01: task 0: Exited with exit code 1
|
||||
|
||||
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.1:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
|
||||
result = subprocess.run(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
||||
raise CalledProcessError(retcode, process.args,
|
||||
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.1:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1346, in run
|
||||
training_exit_code = self._run_with_ray()
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1593, in _run_with_ray
|
||||
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
|
||||
self.start()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
|
||||
self._wait_for_cluster()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
|
||||
raise RuntimeError(
|
||||
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.1:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/a3-rl-SankalpKJ_swesmith-oracle-filtered/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
||||
Collecting Ray logs from worker jpbo-041-05...
|
||||
Collecting Ray logs from worker jpbo-041-06...
|
||||
Collecting Ray logs from worker jpbo-041-07...
|
||||
Collecting Ray logs from worker jpbo-042-01...
|
||||
Collecting Ray logs from worker jpbo-042-02...
|
||||
Collecting Ray logs from worker jpbo-042-03...
|
||||
Collecting Ray logs from worker jpbo-042-06...
|
||||
WARNING: Failed to collect logs from jpbo-042-06
|
||||
Collecting Ray logs from worker jpbo-042-08...
|
||||
WARNING: Failed to collect logs from jpbo-042-08
|
||||
Collecting Ray logs from worker jpbo-042-11...
|
||||
WARNING: Failed to collect logs from jpbo-042-11
|
||||
Collecting Ray logs from worker jpbo-042-12...
|
||||
WARNING: Failed to collect logs from jpbo-042-12
|
||||
Collecting Ray logs from worker jpbo-042-13...
|
||||
WARNING: Failed to collect logs from jpbo-042-13
|
||||
Collecting Ray logs from worker jpbo-042-15...
|
||||
WARNING: Failed to collect logs from jpbo-042-15
|
||||
Collecting Ray logs from worker jpbo-042-16...
|
||||
WARNING: Failed to collect logs from jpbo-042-16
|
||||
[RLJobRunner] Crash-time Ray log preservation complete.
|
||||
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||||
repo_id: DCAgent/a3-rl-SankalpKJ_swesmith-oracle-filtered
|
||||
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/a3-rl-SankalpKJ_swesmith-oracle-filtered
|
||||
episodes: last
|
||||
log: /e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_trace_upload.log
|
||||
[RLJobRunner] Waiting for trace upload to complete...
|
||||
[RLJobRunner] Trace upload failed with exit code 1.
|
||||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-SankalpKJ_swesmith-oracle-filtered/ray_logs/
|
||||
Collecting Ray logs from worker jpbo-041-05...
|
||||
Collecting Ray logs from worker jpbo-041-06...
|
||||
Collecting Ray logs from worker jpbo-041-07...
|
||||
Collecting Ray logs from worker jpbo-042-01...
|
||||
Collecting Ray logs from worker jpbo-042-02...
|
||||
Collecting Ray logs from worker jpbo-042-03...
|
||||
Collecting Ray logs from worker jpbo-042-06...
|
||||
WARNING: Failed to collect logs from jpbo-042-06
|
||||
Collecting Ray logs from worker jpbo-042-08...
|
||||
WARNING: Failed to collect logs from jpbo-042-08
|
||||
Collecting Ray logs from worker jpbo-042-11...
|
||||
WARNING: Failed to collect logs from jpbo-042-11
|
||||
Collecting Ray logs from worker jpbo-042-12...
|
||||
WARNING: Failed to collect logs from jpbo-042-12
|
||||
Collecting Ray logs from worker jpbo-042-13...
|
||||
WARNING: Failed to collect logs from jpbo-042-13
|
||||
Collecting Ray logs from worker jpbo-042-15...
|
||||
WARNING: Failed to collect logs from jpbo-042-15
|
||||
Collecting Ray logs from worker jpbo-042-16...
|
||||
WARNING: Failed to collect logs from jpbo-042-16
|
||||
Ray log preservation complete
|
||||
1123
training_logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_665439.out
Normal file
1123
training_logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_665439.out
Normal file
File diff suppressed because one or more lines are too long
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a039d5d1a3c3a5458c724f5860495b7599aaa85d73b5e462112af99ec295e36f
|
||||
size 102197136
|
||||
2541
training_logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_665441.out
Normal file
2541
training_logs/a3-rl-SankalpKJ_swesmith-oracle-filtered_665441.out
Normal file
File diff suppressed because one or more lines are too long
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:054493524250ae3614396c2aa650727d28a5baf25032bba20295fae101d6e67a
|
||||
size 119128844
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job735237/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_735237
|
||||
/var/spool/slurmd/job735237/slurm_script: line 257: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job735238/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_735238
|
||||
/var/spool/slurmd/job735238/slurm_script: line 257: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job735239/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_735239
|
||||
/var/spool/slurmd/job735239/slurm_script: line 257: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job735240/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_735240
|
||||
/var/spool/slurmd/job735240/slurm_script: line 257: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job735241/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_735241
|
||||
/var/spool/slurmd/job735241/slurm_script: line 257: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user