初始化项目，由ModelHub XC社区提供模型
Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B Source: Original Platform
This commit is contained in:
ModelHub XC
42
.gitattributes
vendored
Normal file
42
.gitattributes
vendored
Normal file
@@ -0,0 +1,42 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/ablation-pymethods2test-seqmean-arm0_630124.out filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/ablation-pymethods2test-seqmean-arm0_630127.out filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/ablation-pymethods2test-seqmean-arm0_630130.out filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/ablation-pymethods2test-seqmean-arm0_661735.out filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/20260612_220124_reward_vs_steps.png filter=lfs diff=lfs merge=lfs -text
|
||||
training_logs/20260612_220124_turn_count_distribution.png filter=lfs diff=lfs merge=lfs -text
|
||||
35
README.md
Normal file
35
README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
base_model: laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink
|
||||
tags:
|
||||
- rl
|
||||
- skyrl
|
||||
- agentic
|
||||
- swe
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# ablation-pymethods2test-seqmean-arm0-30-8B
|
||||
|
||||
RL (SkyRL GRPO) checkpoint from the **sequence-mean / RLOO-n (arm0)** ablation of
|
||||
the a3-successor study. The policy loss uses `loss_reduction=sequence_mean` with
|
||||
the `advantage_estimator=rloo_n` (RLOO-n) estimator, contrasting with the
|
||||
token-mean reduction of the a3 series.
|
||||
|
||||
- **Base model:** [laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink](https://huggingface.co/laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink) (a Qwen3-8B SFT)
|
||||
- **Training dataset:** [DCAgent/exp_rpt_pymethods2test-large](https://huggingface.co/datasets/DCAgent/exp_rpt_pymethods2test-large)
|
||||
- **Checkpoint:** `global_step_30`.
|
||||
- **Training:** SkyRL GRPO, `hf_save_interval=5`, 14x GH200 nodes on JSC Jupiter.
|
||||
|
||||
The `rl_config.yaml` in this repo is the exact launch config used for reproducibility.
|
||||
This is the step-30 checkpoint of the same run that produced
|
||||
[laion/ablation-pymethods2test-seqmean-arm0-15-8B](https://huggingface.co/laion/ablation-pymethods2test-seqmean-arm0-15-8B) (step 15).
|
||||
|
||||
## Training Traces
|
||||
|
||||
Training-time Daytona/Harbor rollouts for this run are uploaded as
|
||||
a companion dataset:
|
||||
**[penfever/ablation-pymethods2test-seqmean-arm0](https://huggingface.co/datasets/penfever/ablation-pymethods2test-seqmean-arm0)**
|
||||
|
||||
The dataset contains the `last` episode of each trial (per
|
||||
`make_and_upload_trace_dataset --episodes last`) — the same rollouts
|
||||
the policy was trained on after rollback / truncation.
|
||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"</think>": 151668,
|
||||
"</tool_call>": 151658,
|
||||
"</tool_response>": 151666,
|
||||
"<think>": 151667,
|
||||
"<tool_call>": 151657,
|
||||
"<tool_response>": 151665,
|
||||
"<|box_end|>": 151649,
|
||||
"<|box_start|>": 151648,
|
||||
"<|endoftext|>": 151643,
|
||||
"<|file_sep|>": 151664,
|
||||
"<|fim_middle|>": 151660,
|
||||
"<|fim_pad|>": 151662,
|
||||
"<|fim_prefix|>": 151659,
|
||||
"<|fim_suffix|>": 151661,
|
||||
"<|im_end|>": 151645,
|
||||
"<|im_start|>": 151644,
|
||||
"<|image_pad|>": 151655,
|
||||
"<|object_ref_end|>": 151647,
|
||||
"<|object_ref_start|>": 151646,
|
||||
"<|quad_end|>": 151651,
|
||||
"<|quad_start|>": 151650,
|
||||
"<|repo_name|>": 151663,
|
||||
"<|video_pad|>": 151656,
|
||||
"<|vision_end|>": 151653,
|
||||
"<|vision_pad|>": 151654,
|
||||
"<|vision_start|>": 151652
|
||||
}
|
||||
89
chat_template.jinja
Normal file
89
chat_template.jinja
Normal file
@@ -0,0 +1,89 @@
|
||||
{%- if tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{- messages[0].content + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||
{%- else %}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if message.content is string %}
|
||||
{%- set content = message.content %}
|
||||
{%- else %}
|
||||
{%- set content = '' %}
|
||||
{%- endif %}
|
||||
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- else %}
|
||||
{%- if '</think>' in content %}
|
||||
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
||||
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if loop.index0 > ns.last_query_index %}
|
||||
{%- if loop.last or (not loop.last and reasoning_content) %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if (loop.first and content) or (not loop.first) %}
|
||||
{{- '\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.function %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": "' }}
|
||||
{{- tool_call.name }}
|
||||
{{- '", "arguments": ' }}
|
||||
{%- if tool_call.arguments is string %}
|
||||
{{- tool_call.arguments }}
|
||||
{%- else %}
|
||||
{{- tool_call.arguments | tojson }}
|
||||
{%- endif %}
|
||||
{{- '}\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
68
config.json
Normal file
68
config.json
Normal file
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Qwen3ForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 151645,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 12288,
|
||||
"layer_types": [
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention",
|
||||
"full_attention"
|
||||
],
|
||||
"max_position_embeddings": 40960,
|
||||
"max_window_layers": 36,
|
||||
"model_type": "qwen3",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 36,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": 151643,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 1000000,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "4.57.6",
|
||||
"use_cache": false,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 151936
|
||||
}
|
||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
151645,
|
||||
151643
|
||||
],
|
||||
"pad_token_id": 151643,
|
||||
"temperature": 0.6,
|
||||
"top_k": 20,
|
||||
"top_p": 0.95,
|
||||
"transformers_version": "4.57.6"
|
||||
}
|
||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e062073cba7d9c5d285d271a3c90696620184de04e88ef003272a97e519bbaff
|
||||
size 4902257696
|
||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:65824499eb643fd7de3bf2259146af45dfac1b225f1398af10e11af3e98dc893
|
||||
size 4915960368
|
||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e032b21dccfa31f92417ccf5f192293c0e6c2686cde62ed9721dd7c256a322e5
|
||||
size 4983068496
|
||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c16d43d5aae0d283caa2870bfbac05f5e0fdcab371bf675899f9d8ac7a16a25b
|
||||
size 1580230264
|
||||
407
model.safetensors.index.json
Normal file
407
model.safetensors.index.json
Normal file
@@ -0,0 +1,407 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_parameters": 8190735360,
|
||||
"total_size": 16381470720
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.norm.weight": "model-00004-of-00004.safetensors"
|
||||
}
|
||||
}
|
||||
321
rl_config.yaml
Normal file
321
rl_config.yaml
Normal file
@@ -0,0 +1,321 @@
|
||||
entrypoint: examples.terminal_bench.entrypoints.main_tbench
|
||||
|
||||
# Hydra config groups (+ prefix in CLI)
|
||||
config_groups:
|
||||
terminal_bench_config: terminal_bench
|
||||
|
||||
# Terminal bench / agentic environment settings
|
||||
terminal_bench:
|
||||
# trials_dir: Directory for Harbor trial artifacts (derived from experiments_dir if null)
|
||||
trials_dir: null
|
||||
|
||||
# Harbor configuration - schema-driven mapping to TrialConfig
|
||||
harbor:
|
||||
# Agent settings
|
||||
name: terminus-2
|
||||
max_episodes: 999999
|
||||
enable_summarize: false
|
||||
store_all_messages: true
|
||||
trajectory_config:
|
||||
raw_content: true
|
||||
enable_episode_logging: false
|
||||
record_terminal_session: false
|
||||
enable_pane_logging: false
|
||||
|
||||
# Strict JSON parser
|
||||
strict_json_parser: true
|
||||
|
||||
# Interleaved Thinking Settings
|
||||
interleaved_thinking: true
|
||||
extra_body:
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
# 2026-05-27: 1800 → 900. The boundary-hugging a3 datasets (stack-junit,
|
||||
# nemotron, methods2test) hit the 1800s agent-timeout wall on ~70-95% of
|
||||
# trials (GLM-4_7-swesmith can't finish these in 1800s), so every group
|
||||
# waited ~1800s for its slowest sample → generation buffer crawled (5/64
|
||||
# in 4h19) and chains never reached training step 1. Halving to 900s
|
||||
# ~2× the valid-group yield. Tradeoff: lower solve-rate on the hardest
|
||||
# tasks (they fail faster) — acceptable vs. TIMEOUT-at-step-0. Fast-task
|
||||
# chains (e2egit, median 7 turns / <900s) are unaffected. See
|
||||
# agent_logs/2026-05-27_rl_chains_stuck_step0.md. AgentTimeoutError stays
|
||||
# PASSTHROUGH (standing pref — partial trajectory+reward is meaningful).
|
||||
override_timeout_sec: 900
|
||||
|
||||
# Environment settings
|
||||
override_cpus: 1
|
||||
override_memory_mb: 2048
|
||||
override_storage_mb: 2048
|
||||
|
||||
# ==========================================================================
|
||||
# AUTO SNAPSHOT: Reduce Daytona rate limits with hash-based snapshot caching
|
||||
# ==========================================================================
|
||||
# When true, automatically creates a snapshot from the Dockerfile on first use,
|
||||
# then reuses it for all subsequent sandboxes with the same Dockerfile content.
|
||||
# Snapshots are named: harbor__<sha256[:12]>__snapshot
|
||||
auto_snapshot: true
|
||||
|
||||
# Verifier settings
|
||||
verifier_override_timeout_sec: 120
|
||||
|
||||
# Retry settings
|
||||
max_retries: 3
|
||||
min_wait_sec: 60.0
|
||||
max_wait_sec: 600.0
|
||||
wait_multiplier: 2.0
|
||||
|
||||
exclude_exceptions:
|
||||
- VerifierTimeoutError
|
||||
- VerifierRuntimeError
|
||||
- RewardFileNotFoundError
|
||||
- RewardFileEmptyError
|
||||
- VerifierOutputParseError
|
||||
|
||||
# 3x base (300 → 900): triple Daytona concurrency for higher throughput.
|
||||
# Mirrors ALCC/56GPU_base.yaml — per-job sandbox count; doesn't affect
|
||||
# the cross-cluster RUNNING-RL cap (≤ 6) which is about job count, not
|
||||
# sandboxes-per-job.
|
||||
# NOTE (2026-05-27): a temporary 900→500 walk-back was tried to mitigate
|
||||
# the FD-exhaustion / uv__epoll_ctl_prep SIGABRT but REVERTED — the real
|
||||
# fix is porting harbor's terminus-2 FD-taming features from
|
||||
# penfever/temp-override into penfever/otagent-latest (that branch had the
|
||||
# issue tamed at full concurrency). See agent_logs/2026-05-27_rl_chains_stuck_step0.md.
|
||||
# 2026-05-28: 900 → 675 (-25%). Fresh a3 chains (#12/#13) abort with a
|
||||
# driver SIGABRT that is NOT FD exhaustion (fd-monitor showed 1.6% of the
|
||||
# 131072 limit) — suspected host-memory OOM from too many concurrent
|
||||
# never-completing trials piling up resident state in skyrl_entrypoint on
|
||||
# hard datasets. Cutting concurrent trial count reduces that resident
|
||||
# memory. Paired with num_parallel_generation_workers -25%. See
|
||||
# agent_logs/2026-05-28_fresh_a3_chain_crash_not_fd.md.
|
||||
n_concurrent_trials: 675
|
||||
|
||||
# Logging settings
|
||||
log_level: INFO
|
||||
|
||||
# Reward shaping (disabled - binary rewards)
|
||||
enable_reward_shaping: false
|
||||
|
||||
# RLOO-N error classification
|
||||
enable_error_classification: true
|
||||
mask_exceptions:
|
||||
- DaytonaError
|
||||
- EnvironmentStartTimeoutError
|
||||
- NetworkError
|
||||
- ConnectionError
|
||||
- RewardFileNotFoundError
|
||||
- RewardFileEmptyError
|
||||
- AgentEnvironmentTimeoutError
|
||||
- ContextLengthExceededError
|
||||
default_error_treatment: zero
|
||||
# NOTE (2026-05-27): AgentTimeoutError / ContextLengthExceededError moved
|
||||
# OUT of passthrough_exceptions into zero_exceptions. Passthrough routed
|
||||
# these soft-limit trials through the *normal* (live-trajectory) path in
|
||||
# terminal_bench_generator._process_trial_result, keeping the trial's
|
||||
# dangling vLLM ObjectRefs + orphaned litellm async-callback closures
|
||||
# alive in the training batch. That fed Ray's distributed-refcount race
|
||||
# (reference_count.cc:1619 → SIGABRT / WorkerCrashedError) that killed the
|
||||
# nemotron-junit chain (litellm +1 32769 VLLMValidationError +
|
||||
# ContextLengthExceeded on near-budget prompts). Both mask + zero take the
|
||||
# EARLY-RETURN path (response_ids=[0]) which never consumes rollout_details,
|
||||
# so no dangling refs enter training. 2026-05-27: ContextLengthExceededError
|
||||
# moved to mask_exceptions (above) — excluded from the RLOO-N baseline, since
|
||||
# GLM-4_7-swesmith ~32k prompts overflow routinely and we don't want hard
|
||||
# zeros dragging the baseline. AgentTimeoutError stays PASSTHROUGH (partial trajectory+reward is
|
||||
# meaningful signal; infrequent vs ContextLengthExceeded so low refcount risk). Belt-and-suspenders w/ the _harbor_compat rollback_hook that
|
||||
# already truncates these trials' dangling prompt-without-response.
|
||||
passthrough_exceptions:
|
||||
- AgentTimeoutError
|
||||
zero_exceptions: []
|
||||
|
||||
# Model info for Harbor's hosted_vllm validation
|
||||
model_info:
|
||||
# Lowered 32767 -> 32000 (2026-05-27). Harbor's litellm token counter is
|
||||
# +2 below vLLM's actual BPE tokenization on near-budget prompts (litellm
|
||||
# +1 plus chat-template/special-token drift), so a 32767-capped prompt
|
||||
# tokenizes to 32769 at the vLLM serving layer and is rejected with
|
||||
# `VLLMValidationError: 32769 input tokens ... context length is only
|
||||
# 32768`. vLLM returns this as an ErrorResponse (handled, not an engine
|
||||
# crash), but harbor's litellm then retries the *deterministically*
|
||||
# over-budget request (max_retries=3, exp backoff) across n_concurrent
|
||||
# trials. On a boundary-heavy task distribution (e.g. nemotron-junit: 111+
|
||||
# such overflows per chain link) the accumulating open sockets exhaust the
|
||||
# skyrl_entrypoint actor's file descriptors -> libuv uv__epoll_ctl_prep
|
||||
# aborts -> SIGABRT in the uvloop event loop -> ray.WorkerCrashedError kills
|
||||
# the chain before training step 1 (job a3-rl ...nemotron-junit #11,
|
||||
# chain 521442-448). A 768-token buffer below max_model_len=32768 ensures
|
||||
# harbor truncates before vLLM ever rejects, eliminating the retry storm.
|
||||
# Cost is ~2.3% usable context for all a3 chains; chains that don't hug the
|
||||
# boundary (e2egit reached step 68 fine at 32767) are unaffected.
|
||||
max_input_tokens: 32000
|
||||
max_output_tokens: 4096
|
||||
|
||||
archiving:
|
||||
# Enable trial archiving callback
|
||||
enabled: false
|
||||
|
||||
# Post-training trace upload to HuggingFace
|
||||
trace_upload:
|
||||
enabled: true
|
||||
repo_org: DCAgent
|
||||
episodes: last
|
||||
dataset_type: SFT
|
||||
cleanup: true
|
||||
|
||||
# Trainer configuration
|
||||
trainer:
|
||||
strategy: fsdp2
|
||||
algorithm:
|
||||
advantage_estimator: rloo_n
|
||||
use_kl_loss: false
|
||||
kl_loss_coef: 0.0
|
||||
eps_clip_low: 0.2
|
||||
# eps_clip_high=0.05 mirrors 24GPU_base — midpoint between 0.2 default and
|
||||
# 0.01 tight. Engages on collapse-onset ratios without over-clamping
|
||||
# healthy updates. Asymmetric — only tightening upper bound.
|
||||
eps_clip_high: 0.05
|
||||
# A/B control arm0 = sequence_mean (the 1/|o_i| brevity-biased reduction).
|
||||
# NOTE: the a3 production 56GPU_base.yaml default is token_mean, NOT
|
||||
# sequence_mean — so the completed run a3-rl-...-pymethods2test-large-80-8B
|
||||
# is NOT a valid arm0 and cannot be reused. This explicit arm0 config
|
||||
# establishes the sequence_mean baseline the bias interpretation needs.
|
||||
loss_reduction: sequence_mean
|
||||
|
||||
# Training loop settings
|
||||
epochs: 2
|
||||
max_steps: 80
|
||||
update_epochs_per_batch: 1
|
||||
|
||||
# Batch sizes
|
||||
train_batch_size: 64
|
||||
policy_mini_batch_size: 64
|
||||
eval_batch_size: 64
|
||||
|
||||
# Micro batch sizes (micro1x4 variant)
|
||||
micro_forward_batch_size_per_gpu: 4
|
||||
micro_train_batch_size_per_gpu: 1
|
||||
|
||||
max_prompt_length: 999999
|
||||
|
||||
# Evaluation and checkpointing
|
||||
eval_interval: 999999
|
||||
eval_before_train: false
|
||||
# Resumable checkpointing
|
||||
ckpt_interval: 2
|
||||
resume_mode: latest
|
||||
# HF upload-ready checkpoints
|
||||
hf_save_interval: 5
|
||||
# HuggingFace Hub upload (set via CLI: trainer.hf_hub_repo_id=org/repo)
|
||||
hf_hub_repo_id: null
|
||||
hf_hub_private: false
|
||||
hf_hub_revision: main
|
||||
|
||||
# Database registration (auto-registers trained model to Supabase)
|
||||
# Requires KEYS env var pointing to Supabase credentials file
|
||||
enable_db_registration: false
|
||||
|
||||
# Logging
|
||||
project_name: OpenThoughts-Agent
|
||||
log_level: INFO
|
||||
tracker_commit_each_step: true
|
||||
logger: console
|
||||
|
||||
# Paths
|
||||
run_name: null
|
||||
ckpt_path: null
|
||||
export_path: null
|
||||
|
||||
# Policy optimizer
|
||||
# max_grad_norm=0.9 mirrors 24GPU_base — guardrail against grad-norm spikes
|
||||
# entering correlation-mode-collapse territory (>1.0). 0.9 is just above the
|
||||
# natural healthy peak observed on this dataset+base (‖g‖ peak ~0.81).
|
||||
policy:
|
||||
optimizer_config:
|
||||
lr: 8e-6
|
||||
weight_decay: 0.0
|
||||
adam_betas: [0.9, 0.999]
|
||||
max_grad_norm: 0.9
|
||||
fsdp_config:
|
||||
cpu_offload: false
|
||||
reshard_after_forward: true
|
||||
fsdp_size: 4
|
||||
# Reference model
|
||||
ref:
|
||||
fsdp_config:
|
||||
cpu_offload: false
|
||||
reshard_after_forward: true
|
||||
fsdp_size: 4
|
||||
|
||||
# Model placement (async training) - 8 shared GPUs for policy/ref
|
||||
placement:
|
||||
colocate_all: false
|
||||
policy_num_nodes: 2
|
||||
ref_num_nodes: 2
|
||||
policy_num_gpus_per_node: 4
|
||||
ref_num_gpus_per_node: 4
|
||||
|
||||
# Fully async generation (settings from v2_maxconcurrent)
|
||||
fully_async:
|
||||
max_staleness_steps: 16
|
||||
# Setting conservatively to 1 / 2 of total concurrency
|
||||
# 2026-05-28: 450 → 338 (-25%), paired with n_concurrent_trials 900→675,
|
||||
# to cut driver memory pressure (suspected OOM on fresh hard-dataset a3
|
||||
# chains). See agent_logs/2026-05-28_fresh_a3_chain_crash_not_fd.md.
|
||||
num_parallel_generation_workers: 338
|
||||
|
||||
# Generator configuration
|
||||
generator:
|
||||
backend: vllm
|
||||
timeout_multiplier: 1.0
|
||||
model_dtype: bfloat16
|
||||
|
||||
inference_engine_tensor_parallel_size: 1
|
||||
# 3x base (16 → 48): triple vLLM engines to lift gen throughput above training rate.
|
||||
# Goal: build a real surplus of completed groups so the trainer never waits on gen.
|
||||
# Layout: 48 engines × TP=1 + 8 GPUs for policy/ref = 56 GPUs total = 14 nodes.
|
||||
num_inference_engines: 48
|
||||
|
||||
n_samples_per_prompt: 8
|
||||
eval_n_samples_per_prompt: 8
|
||||
|
||||
# Jupiter-specific gpu_memory_utilization: 0.75 (vs ALCC's 0.85).
|
||||
# GH200's 96 GB HBM has more headroom than ALCC's A100-80, but Jupiter
|
||||
# also runs hosted_vllm in the same Ray cluster as the engines and we
|
||||
# keep 0.75 across all jupiter yamls for consistency / fragmentation
|
||||
# safety. Bump to 0.85 only if KV-cache pressure justifies it.
|
||||
gpu_memory_utilization: 0.75
|
||||
|
||||
max_num_seqs: 24
|
||||
# Jupiter-specific 65536 (vs ALCC's 16384) — matches the existing
|
||||
# 24GPU_base.yaml on Jupiter; larger batched-token budget keeps the
|
||||
# engine fed when many concurrent requests arrive in bursts.
|
||||
max_num_batched_tokens: 65536
|
||||
|
||||
enable_prefix_caching: true
|
||||
enable_chunked_prefill: true
|
||||
|
||||
run_engines_locally: true
|
||||
weight_sync_backend: nccl
|
||||
async_engine: true
|
||||
batched: false
|
||||
enable_http_endpoint: true
|
||||
enable_ray_prometheus_stats: false
|
||||
vllm_stats_interval: 1
|
||||
append_eos_token_after_stop_str_in_multi_turn: true
|
||||
max_turns: 999999
|
||||
|
||||
sampling_params:
|
||||
max_generate_length: 4096
|
||||
temperature: 0.7
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
|
||||
engine_init_kwargs:
|
||||
max_model_len: 32768
|
||||
# Interleaved thinking chat template: preserves <think> blocks on ALL
|
||||
# historical assistant turns (stock Qwen3 template strips them).
|
||||
custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
|
||||
|
||||
# Data paths
|
||||
data:
|
||||
train_data: []
|
||||
val_data: []
|
||||
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"eos_token": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
240
tokenizer_config.json
Normal file
240
tokenizer_config.json
Normal file
@@ -0,0 +1,240 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"151643": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151644": {
|
||||
"content": "<|im_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151645": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151646": {
|
||||
"content": "<|object_ref_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151647": {
|
||||
"content": "<|object_ref_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151648": {
|
||||
"content": "<|box_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151649": {
|
||||
"content": "<|box_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151650": {
|
||||
"content": "<|quad_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151651": {
|
||||
"content": "<|quad_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151652": {
|
||||
"content": "<|vision_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151653": {
|
||||
"content": "<|vision_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151654": {
|
||||
"content": "<|vision_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151655": {
|
||||
"content": "<|image_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151656": {
|
||||
"content": "<|video_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151657": {
|
||||
"content": "<tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151658": {
|
||||
"content": "</tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151659": {
|
||||
"content": "<|fim_prefix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151660": {
|
||||
"content": "<|fim_middle|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151661": {
|
||||
"content": "<|fim_suffix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151662": {
|
||||
"content": "<|fim_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151663": {
|
||||
"content": "<|repo_name|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151664": {
|
||||
"content": "<|file_sep|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151665": {
|
||||
"content": "<tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151666": {
|
||||
"content": "</tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151667": {
|
||||
"content": "<think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151668": {
|
||||
"content": "</think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"bos_token": null,
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|im_end|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 32768,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"padding_side": "left",
|
||||
"split_special_tokens": false,
|
||||
"tokenizer_class": "Qwen2Tokenizer",
|
||||
"unk_token": null
|
||||
}
|
||||
29
training_logs/20260612_220124_metrics_job_630124.csv
Normal file
29
training_logs/20260612_220124_metrics_job_630124.csv
Normal file
@@ -0,0 +1,29 @@
|
||||
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError
|
||||
0.0,0,64,512,0,0.0,0,0.0,4169.1133,3891.7792,5095.1271,29264,1263,2914.3086,0.7695,-0.0069,0.0744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1189,0.0,0.0,1.0,0.0,0.0105,0.8125,0.7695,11.6896,50.3563,578.7643,32.5,857.9675,279.2032,0.1941,4.3043,30.7211,146.6405,177.747,3371.9765,22.9409,146.8314,3166.9842,0,1,119,952,918,22,26,0.04201680672268908,5.0,0.025210084033613446,3.0,0.008403361344537815,1.0,0.10084033613445378,12,0.14285714285714285,17,,,,,,,,,
|
||||
0.0,0,64,512,1,0.8594,0,0.8594,5688.084,5824.2419,5589.5185,31885,1,5477.7731,0.4199,0.0464,0.1255,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1361,-0.005,0.0,1.0,0.0,0.0161,0.5781,0.4199,12.6834,50.5541,574.2212,33.1,857.9675,283.7463,0.0641,4.8042,39.1437,260.9703,300.4542,1718.0337,24.4369,261.2461,1388.3356,0,2,63,504,454,24,37,0.15873015873015872,10.0,0.047619047619047616,3.0,0.06349206349206349,4.0,0.36507936507936506,23,0.15873015873015872,10,14.2681,24.8089,,,,,,,
|
||||
0.0,0,64,512,2,1.5625,0,0.9375,5287.3867,5200.8475,5333.1104,31817,1,5758.917,0.3457,0.0073,0.0951,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,-0.0021,0.0,1.0,0.0,0.0171,0.4688,0.3457,12.9334,50.6717,572.7552,33.2,857.9675,285.2123,0.1003,4.7883,38.0045,267.5741,305.9524,1220.661,24.4652,267.8472,885.4504,0,3,62,496,468,14,23,0.08064516129032258,5.0,,,,,0.3064516129032258,19,0.0967741935483871,6,,,,,,,,,
|
||||
0.0,0,64,512,3,1.8438,0,0.875,5684.4512,4907.0708,6233.8,31815,1,5008.2154,0.4141,-0.0061,0.0833,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1411,-0.004,0.0,1.0,0.0,0.0121,0.4844,0.4141,13.7453,50.795,563.5295,34.3,857.9675,294.438,0.0673,4.9105,37.4051,234.7096,272.4655,1357.8902,20.825,234.9927,1059.6856,0,4,62,496,451,21,36,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.3709677419354839,23,0.22580645161290322,14,0.008,12.7181,,,,,,,
|
||||
0.0,0,64,512,4,1.875,0,0.9219,5039.3906,5546.9962,4499.0363,31714,1,4334.7369,0.5156,0.0318,0.1173,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1279,-0.0036,0.0,1.0,0.0,0.0141,0.625,0.5156,13.8122,50.8325,562.6003,34.4,857.9675,295.3672,0.0989,4.6928,39.1023,199.9551,239.4905,1224.4331,24.7234,200.2888,955.52,0,5,62,496,449,22,32,0.11290322580645161,7.0,0.12903225806451613,8.0,,,0.3387096774193548,21,0.1935483870967742,12,,,8.2216,,,,,,
|
||||
0.0,0,64,512,4,2.1719,0,0.9375,5591.9941,5830.8641,5431.1863,31798,1,4994.2618,0.4023,0.0066,0.0843,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0004,0.0,1.0,0.0,0.0111,0.4844,0.4023,13.8712,50.9265,561.6264,34.5,857.9675,296.3411,0.0653,4.9236,37.4489,233.3771,271.219,1281.0159,22.6774,233.7045,982.1918,0,6,62,496,435,23,51,0.12903225806451613,8.0,0.03225806451612903,2.0,0.04838709677419355,3.0,0.5161290322580645,32,0.25806451612903225,16,0.0067,13.924,,0.016129032258064516,1.0,,,,
|
||||
0.0,0,64,512,4,2.1094,0,0.9062,4851.3477,5205.5652,4562.4468,31817,1,4336.985,0.4492,0.0059,0.0777,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1271,-0.0014,0.0,1.0,0.0,0.0139,0.5469,0.4492,13.9109,50.9409,561.0771,34.6,857.9675,296.8904,0.0644,4.6898,33.9517,197.6421,231.9494,1559.2848,23.4677,197.9329,1299.1744,0,7,63,504,457,15,42,,,0.07936507936507936,5.0,0.031746031746031744,2.0,0.5555555555555556,35,0.07936507936507936,5,,,,,,,,,
|
||||
0.0,0,64,512,4,2.1094,0,0.9219,5596.0137,5715.5123,5487.2164,31840,1,5416.4656,0.4766,0.0199,0.145,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,0.0,0.0,1.0,0.0,0.0136,0.625,0.4766,13.9023,50.9439,561.6647,34.5,857.9675,296.3028,0.0672,4.797,43.3436,268.6347,312.3633,1610.273,22.6954,268.9523,1270.4134,0,8,62,496,457,18,38,,,,,0.016129032258064516,1.0,0.4032258064516129,25,0.20967741935483872,13,0.0071,12.1975,,,,0.016129032258064516,1.0,,
|
||||
0.0,0,64,512,4,1.9844,0,0.9531,5326.7715,5167.9342,5454.2887,31758,1,4691.8758,0.4453,0.0119,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0137,0.5469,0.4453,13.8936,50.9354,559.6754,34.8,857.9675,298.2921,0.0782,4.7117,38.0949,222.0535,260.5136,1321.2856,23.5801,222.3401,1032.476,0,9,61,488,449,18,38,,,0.01639344262295082,1.0,0.03278688524590164,2.0,0.4426229508196721,27,0.22950819672131148,14,,,,,,,,,
|
||||
0.0,0,64,512,3,1.8906,0,0.9062,4968.5547,4853.8377,5036.8131,31768,1,4721.9898,0.373,0.0139,0.118,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1359,0.0,0.0,1.0,0.0,0.0144,0.5312,0.373,13.9257,51.0073,560.3441,34.7,857.9675,297.6235,0.0714,4.8344,33.5456,220.0195,253.8928,1294.919,24.3901,220.2753,1011.7949,0,10,62,496,442,18,46,,,0.12903225806451613,8.0,0.016129032258064516,1.0,0.6451612903225806,40,0.12903225806451613,8,0.0084,14.0711,8.8277,,,,,,
|
||||
0.0,0,64,512,4,1.7812,0,0.9219,4462.5332,4532.996,4394.2385,31762,1,4298.9711,0.4922,0.0008,0.1662,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0056,0.0,1.0,0.0,0.0183,0.6562,0.4922,14.0333,51.1276,559.0244,34.8,857.9675,298.9431,0.0938,4.6549,32.2602,198.6454,231.2913,1475.0347,24.5599,198.9368,1214.5237,0,11,64,512,471,16,33,,,0.125,8.0,0.046875,3.0,0.328125,21,0.140625,9,,,,,,,,,
|
||||
0.0,0,64,512,4,2.0938,0,0.8906,5849.1094,5035.9944,6282.4461,31797,1,4813.1067,0.3477,-0.0072,0.0606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.013,0.4219,0.3477,14.0347,51.1288,558.5858,34.9,857.9675,299.3817,0.0732,5.2651,38.6448,224.9423,263.9456,1559.3505,25.6632,225.2273,1264.4721,0,12,57,456,426,14,30,,,,,0.07017543859649122,4.0,0.42105263157894735,24,0.14035087719298245,8,0.0084,14.0752,,,,,,,
|
||||
0.0,0,64,512,4,1.8125,0,0.875,4866.082,5038.6537,4692.1569,31760,1,4066.2403,0.502,-0.0034,0.1269,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1255,-0.0,0.0,1.0,0.0,0.0186,0.625,0.502,14.0677,51.1943,557.1813,35.1,857.9675,300.7862,0.0704,4.6745,31.6351,183.7078,215.6617,1431.9056,23.0843,183.9558,1188.48,0,13,64,512,450,21,49,,,0.203125,13.0,0.25,16.0,0.46875,30,0.09375,6,,,,,,,,,
|
||||
0.0,0,64,512,4,2.0469,0,0.9375,4926.5117,4839.1749,4993.9031,31751,1,4266.9039,0.4355,0.0028,0.0804,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1228,-0.0013,0.0,1.0,0.0,0.0162,0.5469,0.4355,14.322,51.506,557.3558,35.0,857.9675,300.6118,0.0972,4.7529,33.9188,193.7138,228.0319,1142.7788,24.0818,194.0154,885.9041,0,14,63,504,473,18,30,,,,,,,0.3333333333333333,21,0.14285714285714285,9,0.0084,13.3568,,,,,,0.015873015873015872,1.0
|
||||
0.0,0,64,512,4,2.0312,0,0.9219,5025.9766,4558.8924,5475.1648,31707,1,4544.2113,0.4902,0.0047,0.108,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1336,-0.0,0.0,1.0,0.0,0.0154,0.6094,0.4902,14.6948,51.8791,556.9661,35.1,857.9675,301.0014,0.0749,4.7275,31.8001,218.695,250.8643,1313.9526,22.5691,218.989,1035.7868,0,15,64,512,477,18,28,,,0.109375,7.0,,,0.296875,19,0.1875,12,,,8.2163,,,,,,
|
||||
0.0,0,64,512,4,1.9375,0,0.8906,5304.2988,4924.3203,5616.6655,31702,1,4477.752,0.4512,0.0092,0.1087,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1249,-0.0,0.0,1.0,0.0,0.0148,0.5781,0.4512,14.4315,51.6356,556.9907,35.1,857.9675,300.9769,0.1006,4.7795,36.8211,219.4484,256.6458,1308.8006,22.9796,219.7236,1024.3882,0,16,61,488,421,21,66,,,,,0.09836065573770492,6.0,0.8360655737704918,51,0.16393442622950818,10,0.0077,14.8915,,,,0.01639344262295082,1.0,,
|
||||
0.0,0,64,512,4,2.3594,0,0.9375,5391.4434,6104.0653,4938.3706,31758,1,5320.6865,0.3887,0.0223,0.0896,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1221,-0.002,0.0,1.0,0.0,0.0108,0.5156,0.3887,14.5796,51.7618,556.1977,35.2,857.9675,301.7698,0.0974,5.3524,38.0588,253.8113,292.2721,1220.8525,24.0951,254.1154,899.1259,0,17,57,456,419,17,35,,,0.017543859649122806,1.0,0.03508771929824561,2.0,0.38596491228070173,22,0.24561403508771928,14,,,,,,0.017543859649122806,1.0,,
|
||||
0.0,0,64,512,4,1.7188,0,0.875,5066.6953,5047.4764,5082.7455,31770,1,4672.0163,0.4551,0.0005,0.1098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1223,-0.0028,0.0,1.0,0.0,0.0154,0.5625,0.4551,14.7728,51.9543,556.2234,35.2,857.9675,301.7441,0.0991,4.843,39.5878,215.8858,255.888,1582.6579,22.1364,216.2007,1299.7832,0,18,60,480,437,18,41,,,0.03333333333333333,2.0,0.016666666666666666,1.0,0.5333333333333333,32,0.16666666666666666,10,0.0075,16.3173,,,,,,,
|
||||
0.0,0,64,512,4,1.8281,0,0.8906,5320.7246,4968.5291,5526.808,31825,1,4635.3488,0.3691,-0.0052,0.097,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1408,-0.0,0.0,1.0,0.0,0.0136,0.4531,0.3691,14.7764,51.9575,554.7101,35.3,857.9675,303.2574,0.0666,4.8474,33.322,217.0675,250.7206,1533.2708,23.3393,217.3317,1254.3583,0,19,62,496,453,17,43,,,,,,,0.6129032258064516,38,0.0967741935483871,6,,,,,,,,,
|
||||
0.0,0,64,512,4,1.6875,0,0.8906,4981.0215,5122.7233,4885.6275,31734,1,4443.4633,0.4023,-0.0006,0.0906,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,0.0,0.0,1.0,0.0,0.0168,0.5156,0.4023,14.7832,52.0205,554.9844,35.3,857.9675,302.9831,0.0665,4.7337,34.0494,210.2643,244.6153,1501.26,23.3047,210.499,1228.6018,0,20,63,504,461,18,38,,,0.047619047619047616,3.0,0.06349206349206349,4.0,0.5396825396825397,34,0.06349206349206349,4,0.0078,13.3388,8.0322,,,0.031746031746031744,2.0,,
|
||||
0.0,0,64,512,4,1.7969,0,0.8906,5089.5234,5337.2258,4948.1963,31834,1,4103.7515,0.3633,-0.0098,0.0985,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0055,0.0,1.0,0.0,0.0123,0.4688,0.3633,14.7559,51.9613,554.3621,35.4,857.9675,303.6055,0.0721,4.8151,32.0319,197.6414,230.0275,1384.6688,23.9344,197.9232,1125.887,0,21,61,488,432,21,46,,,0.13114754098360656,8.0,,,0.6229508196721312,38,0.13114754098360656,8,,,,,,0.03278688524590164,2.0,,
|
||||
0.0,0,64,512,3,1.5156,0,0.8281,4888.4629,4445.7686,5246.6855,31774,1,4638.3152,0.4473,0.0144,0.0967,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,0.0,0.0,1.0,0.0,0.0135,0.5781,0.4473,15.1406,52.3359,554.3356,35.4,857.9675,303.6319,0.0672,4.8409,37.4714,213.0209,250.8158,1341.0965,22.4572,213.2768,1062.9785,0,22,64,512,452,21,60,,,,,0.046875,3.0,0.734375,47,0.15625,10,0.0085,14.204,,,,,,,
|
||||
0.0,0,64,512,4,1.6094,0,0.8594,5010.1426,5339.5915,4775.4515,31738,1,5125.7203,0.416,0.0029,0.123,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1269,-0.0,0.0,1.0,0.0,0.0161,0.5781,0.416,15.1399,52.3352,554.2477,35.4,857.9675,303.7198,0.0666,4.7525,38.7094,241.3185,280.3783,1636.7829,23.2267,241.602,1328.421,0,23,61,488,423,17,65,,,,,0.04918032786885246,3.0,0.639344262295082,39,0.3770491803278688,23,,,,,,,,,
|
||||
0.0,0,64,512,3,1.5156,0,0.8281,4957.0039,5145.7238,4853.8066,31744,1,4792.2542,0.3535,-0.005,0.139,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1197,0.0,0.0,1.0,0.0,0.0118,0.4531,0.3535,15.2222,52.591,552.66,35.6,857.9675,305.3075,0.0667,4.8036,34.8997,225.7814,261.0063,1813.0727,21.5017,226.0394,1525.7569,0,24,53,424,405,10,17,,,0.1509433962264151,8.0,0.09433962264150944,5.0,0.1320754716981132,7,0.24528301886792453,13,0.0085,13.7903,,,,,,,
|
||||
0.0,0,64,512,4,1.8281,0,0.9062,4999.9961,4608.5735,5274.3821,31665,1,3727.2595,0.4121,0.0076,0.0996,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1278,-0.0,0.0,1.0,0.0,0.0129,0.4844,0.4121,15.4929,52.9026,551.8461,35.7,857.9675,306.1215,0.0644,4.6949,29.2144,181.8811,211.389,988.3901,25.1751,182.1098,747.1268,0,25,54,432,406,12,19,,,0.12962962962962962,7.0,0.07407407407407407,4.0,0.2777777777777778,15,0.1111111111111111,6,,,8.7088,,,,,,
|
||||
0.0,0,64,512,4,1.7344,0,0.875,4801.6133,4766.3344,4852.3476,31766,1,3813.5277,0.5898,-0.0032,0.1052,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,-0.0035,0.0,1.0,0.0,0.0164,0.6875,0.5898,15.8889,53.344,552.8224,35.6,857.9675,305.1451,0.0946,4.7486,31.8278,188.1649,220.5231,1031.4949,22.9306,188.6001,783.2847,0,26,59,472,410,26,50,,,0.11864406779661017,7.0,0.3050847457627119,18.0,0.3220338983050847,19,0.23728813559322035,14,0.009,12.6671,,,,0.288135593220339,17.0,,
|
||||
0.0,0,64,512,4,2.1875,0,0.9688,5055.6387,4951.9907,5130.67,31763,1,4942.405,0.4199,0.0045,0.1012,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1162,-0.01,0.0,1.0,0.0,0.0135,0.5469,0.4199,16.046,53.4671,552.5507,35.6,857.9675,305.4168,0.0766,4.8488,33.5052,219.951,253.8289,2130.9697,18.5315,220.2467,1853.7559,0,27,52,416,390,13,6,,,0.28846153846153844,15.0,,,0.2692307692307692,14,0.057692307692307696,3,,,,,,0.19230769230769232,10.0,0.019230769230769232,1.0
|
||||
0.0,0,64,512,4,1.6719,0,0.8281,4799.998,4474.8537,5214.7378,21478,1,3336.6768,0.5605,0.0081,0.0474,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1169,-0.0193,0.0,1.0,0.0,0.0089,0.625,0.5605,15.7877,53.2084,552.8907,35.6,857.9675,305.0768,0.0486,3.4809,28.5303,161.7902,190.6488,998.4337,25.9964,162.0695,778.3023,0,28,44,352,335,11,15,,,0.13636363636363635,6.0,0.022727272727272728,1.0,0.4772727272727273,21,0.45454545454545453,20,0.0082,11.9058,,,,0.022727272727272728,1.0,,
|
||||
|
29
training_logs/20260612_220124_metrics_job_630127.csv
Normal file
29
training_logs/20260612_220124_metrics_job_630127.csv
Normal file
@@ -0,0 +1,29 @@
|
||||
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError
|
||||
0.0,0,64,512,5,0.8125,0,0.2031,5219.4023,5279.0935,5148.4872,24019,1,3593.1869,0.543,0.0174,0.1093,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0059,0.0,1.0,0.0,0.0136,0.661,0.543,9.9914,50.291,594.6118,30.7,857.9675,263.3557,0.2443,3.7877,34.8635,181.2173,216.5936,3907.0864,25.2212,181.4851,3661.4836,0,29,54,432,397,20,35,0.18518518518518517,10,0.48148148148148145,26,0.05555555555555555,3.0,,,,,,,
|
||||
0.0,0,64,512,1,0.875,0,0.875,6119.2617,5371.9608,6614.2273,31787,1,5360.732,0.3984,0.0066,0.11,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1391,-0.0,0.0,1.0,0.0,0.0133,0.5156,0.3984,12.6683,50.5745,589.7981,31.3,857.9675,268.1694,0.0668,4.853,45.3325,249.4957,295.1659,1561.0798,22.8392,249.7659,1238.2155,0,30,51,408,360,18,48,0.19607843137254902,10,0.9803921568627451,50,0.0784313725490196,4.0,17.6986,25.6657,6.5413,0.0784313725490196,4.0,,
|
||||
0.0,0,64,512,2,1.7031,1,1.0,5416.0312,5191.9848,5493.8579,31782,1,5258.8542,0.2578,0.0188,0.1227,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1224,-0.0045,0.0,1.0,0.0,0.0132,0.3906,0.2578,13.0473,50.9899,588.6119,31.4,857.9675,269.3556,0.0915,4.9642,38.9391,242.4843,281.7563,558.6231,24.3077,242.7249,247.5902,0,31,62,496,450,16,38,0.03225806451612903,2,0.5806451612903226,36,,,,,,0.06451612903225806,4.0,,
|
||||
0.0,0,64,512,3,1.8438,0,0.8438,4876.5957,4321.0181,5298.5292,31788,1,4349.0068,0.4316,0.008,0.1024,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1299,-0.0033,0.0,1.0,0.0,0.0147,0.5469,0.4316,13.3226,51.1669,587.694,31.5,857.9675,270.2736,0.0969,4.6902,32.9398,184.8866,218.1917,1881.6998,22.4991,185.1545,1636.313,0,32,61,488,435,20,50,0.26229508196721313,16,0.5245901639344263,32,0.04918032786885246,3.0,0.0086,8.4163,,0.04918032786885246,3.0,,
|
||||
0.0,0,64,512,4,1.9688,0,0.8906,5456.2109,4808.7673,5878.0935,31806,1,4961.1718,0.3945,0.0083,0.1325,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,0.0,0.0,1.0,0.0,0.0126,0.5469,0.3945,13.4299,51.223,587.0558,31.6,857.9675,270.9117,0.067,4.9084,36.655,238.8678,275.8322,1516.0964,23.2669,239.1098,1212.085,0,33,62,496,436,21,55,0.27419354838709675,17,0.5806451612903226,36,0.03225806451612903,2.0,,,,0.08064516129032258,5.0,,
|
||||
0.0,0,64,512,4,2.125,0,0.9844,5502.4434,5212.8122,5708.7692,31765,1,5396.2041,0.416,-0.004,0.0835,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0,0.0,1.0,0.0,0.0137,0.5156,0.416,13.5508,51.3198,587.3987,31.5,857.9675,270.5688,0.0691,5.493,44.4908,256.2057,301.0644,1068.5266,22.5088,256.5041,739.4567,0,34,63,504,462,14,41,0.23809523809523808,15,0.3968253968253968,25,0.015873015873015872,1.0,0.0088,8.3216,,0.015873015873015872,1.0,,
|
||||
0.0,0,64,512,3,1.8594,0,0.9219,5384.9941,5343.019,5414.1821,31664,1,4412.7671,0.4102,0.0025,0.0519,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,-0.002,0.0,1.0,0.0,0.0142,0.5,0.4102,13.7969,51.3292,586.7628,31.6,857.9675,271.2047,0.0906,4.7764,33.1627,211.1102,244.5902,1384.0633,22.1442,211.3362,1112.5444,0,35,61,488,463,8,17,0.04918032786885246,3,0.26229508196721313,16,,,,,5.8995,0.13114754098360656,8.0,,
|
||||
0.0,0,64,512,4,1.9219,0,0.9062,5825.4473,5333.0212,6246.5072,31794,1,4498.7568,0.4609,0.0027,0.1742,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1384,-0.0,0.0,1.0,0.0,0.0171,0.6406,0.4609,13.8943,51.4631,586.2767,31.7,857.9675,271.6909,0.5695,4.8349,37.7488,223.605,262.1992,1464.7075,20.515,223.8804,1177.1535,0,36,64,512,471,15,40,0.109375,7,0.515625,33,,,0.009,7.9437,,0.015625,1.0,,
|
||||
0.0,0,64,512,4,1.8438,0,0.9062,5148.5137,4631.637,5834.55,31764,1,4648.1595,0.5703,-0.0029,0.1133,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,-0.0011,0.0,1.0,0.0,0.0167,0.6875,0.5703,14.5586,51.6496,584.9858,31.8,857.9675,272.9817,0.0919,4.8241,38.277,222.4597,261.0628,1077.7825,26.2516,222.6934,785.6361,0,37,61,488,451,15,35,0.14754098360655737,9,0.4262295081967213,26,0.09836065573770492,6.0,,,,0.03278688524590164,2.0,,
|
||||
0.0,0,64,512,4,2.1875,0,0.9375,5691.4238,5458.8718,5887.1691,31763,1,4985.511,0.457,0.0025,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1239,-0.0,0.0,1.0,0.0,0.0158,0.6094,0.457,14.4333,51.5906,584.4034,31.9,857.9675,273.5641,0.8661,4.8087,37.1183,229.9869,268.1954,1281.4976,24.0319,230.2106,984.4536,0,38,63,504,457,20,44,0.1111111111111111,7,0.5238095238095238,33,,,0.0082,8.3888,,0.047619047619047616,3.0,,
|
||||
0.0,0,64,512,4,1.875,0,0.9219,4852.4375,4700.9213,4963.0034,31721,1,3852.6075,0.4219,-0.0118,0.1369,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0025,0.0,1.0,0.0,0.0151,0.5625,0.4219,14.8096,51.9002,584.34,31.9,857.9675,273.6276,0.1019,4.7148,35.3281,188.2665,223.9309,1220.1543,22.2426,188.5005,969.2585,0,39,63,504,446,22,48,0.09523809523809523,6,0.6666666666666666,42,0.031746031746031744,2.0,,,,0.15873015873015872,10.0,,
|
||||
0.0,0,64,512,4,1.8906,0,0.8438,4717.9707,4944.8443,4511.4142,31770,1,4317.3591,0.4766,0.0072,0.086,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1232,0.0,0.0,1.0,0.0,0.0143,0.5938,0.4766,14.7138,51.7721,583.8667,31.9,857.9675,274.1008,0.0992,4.8622,31.9697,193.9941,226.3482,1608.4681,22.3175,194.2789,1354.9332,0,40,60,480,456,10,24,0.05,3,0.36666666666666664,22,0.03333333333333333,2.0,0.0074,8.3081,6.0239,,,,
|
||||
0.0,0,64,512,4,1.6875,0,0.9531,5037.7695,5312.8784,4764.8016,27138,1,3679.8466,0.498,-0.0001,0.0849,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0122,0.5781,0.498,14.5892,51.6475,583.7137,32.0,857.9675,274.2538,0.0824,4.1919,29.9247,176.4747,206.7121,1037.8229,24.3689,176.7045,802.5427,0,41,60,480,441,19,37,0.18333333333333332,11,0.43333333333333335,26,,,,,,0.016666666666666666,1.0,0.016666666666666666,1.0
|
||||
0.0,0,64,512,4,1.9844,0,0.8906,5756.3848,5835.0311,5708.8025,31761,1,4473.6908,0.377,0.0069,0.1087,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1346,0.0,0.0,1.0,0.0,0.013,0.4531,0.377,14.9636,52.1455,583.6249,32.0,857.9675,274.3426,0.0719,4.8088,36.9135,222.7902,260.0054,1385.1388,21.7016,223.0196,1098.6195,0,42,61,488,424,20,50,0.22950819672131148,14,0.5573770491803278,34,0.03278688524590164,2.0,0.0076,7.9965,,0.22950819672131148,14.0,,
|
||||
0.0,0,64,512,4,1.5781,0,0.8281,4886.5996,4211.4396,5344.823,31794,1,4658.4948,0.4043,0.003,0.0477,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.119,-0.0014,0.0,1.0,0.0,0.0109,0.4844,0.4043,14.9648,52.146,583.4162,32.0,857.9675,274.5513,0.0932,4.8883,35.3553,210.4627,246.1689,1473.9582,23.9821,210.7199,1198.9132,0,43,61,488,441,16,45,0.16393442622950818,10,0.5081967213114754,31,0.06557377049180328,4.0,,,,0.03278688524590164,2.0,,
|
||||
0.0,0,64,512,4,1.9375,0,0.9844,5006.252,5006.7137,5005.7938,31832,1,4198.283,0.498,0.0058,0.1318,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1302,-0.0013,0.0,1.0,0.0,0.0142,0.625,0.498,15.1405,52.1464,582.1508,32.1,857.9675,275.8167,0.0701,4.8376,37.1394,204.9028,242.3402,995.3073,21.6574,205.1302,726.4685,0,44,63,504,464,15,40,0.2698412698412698,17,0.3492063492063492,22,0.015873015873015872,1.0,0.0087,8.5306,,,,,
|
||||
0.0,0,64,512,4,1.9688,0,0.9531,5307.5059,4945.0996,5629.7934,31811,1,4502.6994,0.4707,0.0034,0.0645,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0087,0.5156,0.4707,15.2917,52.3331,581.6784,32.2,857.9675,276.2891,0.0713,4.8688,35.6411,219.1314,255.1244,1030.1018,22.2906,219.4115,747.8149,0,45,57,456,419,11,20,0.07017543859649122,4,0.3508771929824561,20,0.03508771929824561,2.0,,,7.1117,0.2982456140350877,17.0,,
|
||||
0.0,0,64,512,4,2.3281,0,0.9531,5162.7871,4802.6639,5490.6604,31759,1,4252.9044,0.4766,-0.0039,0.0939,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1296,0.0,0.0,1.0,0.0,0.0135,0.5625,0.4766,15.655,52.6444,580.4724,32.3,857.9675,277.4952,0.1001,4.9649,38.1605,200.0621,238.5392,1116.6096,24.6502,200.2781,848.4495,0,46,62,496,452,18,44,0.14516129032258066,9,0.5483870967741935,34,0.016129032258064516,1.0,0.0077,8.5628,,,,,
|
||||
0.0,0,64,512,4,2.1094,0,0.9375,5453.8691,5448.0192,5459.9522,31779,1,4423.535,0.5098,0.01,0.1256,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0129,0.625,0.5098,15.5344,52.5839,580.0443,32.4,857.9675,277.9232,0.0896,4.8374,35.3367,219.7689,255.4572,1436.5679,22.4966,220.0305,1153.7719,0,47,61,488,451,18,36,0.16393442622950818,10,0.3770491803278688,23,0.04918032786885246,3.0,,,,0.01639344262295082,1.0,,
|
||||
0.0,0,64,512,4,1.875,0,0.875,5230.4609,4627.1281,5626.8252,31716,1,4361.9157,0.3965,-0.0025,0.057,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0,0.0,1.0,0.0,0.0113,0.5,0.3965,15.6175,52.7105,579.3495,32.5,857.9675,278.618,0.0831,4.8056,35.7289,205.2303,241.3389,1388.9346,24.246,205.5265,1118.5394,0,48,63,504,448,25,55,0.1746031746031746,11,0.6666666666666666,42,0.031746031746031744,2.0,0.0078,8.0292,,0.015873015873015872,1.0,,
|
||||
0.0,0,64,512,4,1.9688,0,0.9062,5567.3223,3973.2919,6469.1437,31784,1,5829.7172,0.3613,-0.0074,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0168,0.4844,0.3613,15.5988,52.7747,578.7762,32.5,857.9675,279.1913,0.1107,4.8687,41.1551,269.7094,311.286,1674.2094,22.9805,270.0198,1335.0696,0,49,60,480,440,17,39,0.13333333333333333,8,0.5166666666666667,31,,,,,,0.016666666666666666,1.0,,
|
||||
0.0,0,64,512,3,1.6875,0,0.9375,5274.1113,4738.0175,5707.9117,31829,1,4398.882,0.4473,0.008,0.0793,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0132,0.5469,0.4473,15.6793,52.8362,578.3136,32.6,857.9675,279.6539,0.0972,4.7852,34.4386,207.8657,242.6571,1060.2084,23.4244,208.1209,789.3372,0,50,65,520,431,21,76,0.12307692307692308,8,0.9538461538461539,62,0.09230769230769231,6.0,0.0081,9.1305,5.9495,0.18461538461538463,12.0,0.015384615384615385,1.0
|
||||
0.0,0,64,512,4,1.9219,0,0.8906,5243.7461,5174.6447,5286.9619,31743,1,5188.684,0.3848,-0.0033,0.0945,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1207,0.0,0.0,1.0,0.0,0.0967,0.4531,0.3848,15.6188,52.7762,577.5959,32.7,857.9675,280.3716,0.076,4.7495,40.1813,238.4651,278.9729,1686.7408,24.1315,238.7151,1378.8824,0,51,62,496,446,22,49,0.20967741935483872,13,0.5645161290322581,35,0.016129032258064516,1.0,,,,0.016129032258064516,1.0,,
|
||||
0.0,0,64,512,4,1.7812,0,0.8438,5252.4961,5411.8429,5141.6921,31823,1,4683.711,0.4102,0.0062,0.0796,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.133,-0.002,0.0,1.0,0.0,0.0124,0.5312,0.4102,15.7651,52.9658,577.1581,32.7,857.9675,280.8094,0.0944,4.7678,38.5332,214.205,253.0737,1600.439,23.268,214.4456,1319.3232,0,52,61,488,451,14,37,0.08196721311475409,5,0.4918032786885246,30,0.03278688524590164,2.0,0.0082,8.7241,,,,,
|
||||
0.0,0,64,512,3,1.8125,0,0.9531,5107.9043,4684.3886,5450.6078,31689,1,4035.9699,0.4473,-0.0001,0.0667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.132,0.0,0.0,1.0,0.0,0.0132,0.5312,0.4473,15.7054,52.9061,577.4001,32.7,857.9675,280.5674,0.097,4.8708,33.9418,188.7929,223.0849,1082.9166,25.281,189.0458,829.6739,0,53,63,504,454,17,47,0.015873015873015872,1,0.6984126984126984,44,0.031746031746031744,2.0,,,,0.047619047619047616,3.0,,
|
||||
0.0,0,64,512,3,1.7969,0,0.875,5634.0664,5882.4221,5476.1661,31801,1,5198.4542,0.3887,0.0101,0.0932,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0144,0.5156,0.3887,15.8018,52.9791,576.4988,32.8,857.9675,281.4687,0.0964,4.9065,40.1938,233.8941,274.5002,1827.1928,24.3101,234.2096,1523.4687,0,54,61,488,434,19,52,0.13114754098360656,8,0.6557377049180327,40,0.08196721311475409,5.0,0.0084,8.6128,,0.03278688524590164,2.0,,
|
||||
0.0,0,64,512,4,1.9375,0,0.9531,5371.4082,5022.761,5604.2182,31785,1,4782.3049,0.4004,-0.0082,0.1177,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1222,-0.0007,0.0,1.0,0.0,0.0128,0.5156,0.4004,15.8948,53.0398,576.6206,32.8,857.9675,281.3469,0.0653,4.8129,41.537,228.2535,270.096,1483.9509,23.783,228.4932,1185.2547,0,55,62,496,429,19,64,0.0967741935483871,6,0.9032258064516129,56,0.04838709677419355,3.0,,,6.0259,0.04838709677419355,3.0,,
|
||||
0.0,0,64,512,4,1.6094,0,0.9062,4940.75,4904.8341,4965.9269,31836,1,4570.0348,0.4121,-0.0024,0.1498,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.124,-0.0,0.0,1.0,0.0,0.0148,0.5156,0.4121,16.2064,53.3552,575.7253,32.9,857.9675,282.2422,0.0989,4.7645,34.0253,209.3135,243.7093,1375.7686,22.7461,209.5845,1104.5439,0,56,59,472,390,33,69,0.288135593220339,17,0.5084745762711864,30,0.3728813559322034,22.0,0.0075,15.5646,,0.22033898305084745,13.0,,
|
||||
|
15
training_logs/20260612_220124_metrics_job_630130.csv
Normal file
15
training_logs/20260612_220124_metrics_job_630130.csv
Normal file
@@ -0,0 +1,15 @@
|
||||
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError
|
||||
0.0,0,64,512,4,0.3438,0,0.1406,2745.916,4536.7524,2283.9066,31842,1,3652.8031,0.2051,0.0705,0.1449,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0683,-0.0442,0.0,1.0,0.0,0.0137,0.3898,0.2051,12.7084,50.5586,566.71,33.9,857.9675,291.2575,0.1351,4.4486,104.946,152.1059,257.396,10105.4336,18.2174,152.3142,9825.3711,0,57,63,504,281,58,154,1.8253968253968254,115,0.19047619047619047,12.0,0.42857142857142855,27,0.8095238095238095,51.0,0.25396825396825395,16.0,0.031746031746031744,2.0,,,,,
|
||||
0.0,0,64,512,1,0.9844,0,0.9844,2990.8008,4339.4828,2595.7323,31794,1,4262.9787,0.2266,0.0705,0.1484,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,-0.0481,0.0,1.0,0.0,0.0181,0.5,0.2266,12.7156,50.6524,565.6756,34.1,857.9675,292.2919,0.0685,4.3634,31.3815,172.8736,204.58,3407.9472,17.5583,173.1294,3181.4385,0,58,59,472,191,58,157,2.3389830508474576,138,0.22033898305084745,13.0,0.3050847457627119,18,1.9491525423728813,115.0,0.2711864406779661,16.0,,,113.7448,123.1255,,,
|
||||
0.0,0,64,512,2,1.4531,0,0.9219,1842.3164,3830.7287,1172.5901,31730,1,3083.3057,0.252,0.1959,0.2355,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0529,-0.0935,0.0,1.0,0.0,0.0296,0.5,0.252,12.8444,50.7361,566.7198,33.9,857.9675,291.2477,0.0904,4.2018,17.8872,126.8601,145.0002,3636.1329,18.2801,127.022,3468.6454,0,59,59,472,191,58,160,2.559322033898305,151,0.23728813559322035,14.0,0.3728813559322034,22,1.8305084745762712,108.0,0.3728813559322034,22.0,,,,,,,
|
||||
0.0,0,64,512,3,1.5469,0,0.9531,1687.9023,3823.3853,1110.3151,19335,1,2744.5105,0.2129,0.1122,0.1362,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0511,-0.0532,0.0,1.0,0.0,0.0138,0.4531,0.2129,12.8585,50.7328,566.4686,34.0,857.9675,291.4989,0.0597,2.7178,15.1498,108.5325,123.9181,3855.0493,17.9434,108.7081,3710.4636,0,60,63,504,312,48,79,0.7936507936507936,50,0.14285714285714285,9.0,0.31746031746031744,20,1.746031746031746,110.0,0.047619047619047616,3.0,,,0.0099,8.8255,275.4291,,
|
||||
0.0,0,64,512,3,1.4844,0,0.875,2705.2773,3926.8809,1668.8989,31646,1,3012.6819,0.459,0.1462,0.1798,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0697,-0.0959,0.0,1.0,0.0,0.0179,0.7188,0.459,13.0881,50.9166,566.6464,34.0,857.9675,291.3212,0.0697,4.435,19.2157,123.2323,142.7038,1882.4268,20.2012,123.418,1715.083,0,61,62,496,397,29,58,0.3387096774193548,21,0.1935483870967742,12.0,0.4032258064516129,25,0.6290322580645161,39.0,0.03225806451612903,2.0,,,,,,0.016129032258064516,1.0
|
||||
0.0,0,64,512,3,1.5625,0,0.875,4372.2871,4678.6603,4051.208,31765,1,4031.5953,0.5117,0.0301,0.1117,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1006,-0.022,0.0,1.0,0.0,0.0107,0.6094,0.5117,13.2397,51.094,568.5527,33.7,857.9675,289.4149,0.0919,4.5973,32.0721,189.1022,221.4727,1318.3369,21.5119,189.3083,1070.747,0,62,62,496,467,17,21,0.03225806451612903,2,0.11290322580645161,7.0,0.24193548387096775,15,0.11290322580645161,7.0,0.016129032258064516,1.0,,,0.0085,8.9973,,0.016129032258064516,1.0
|
||||
0.0,0,64,512,4,1.8906,0,0.9688,5146.4805,4434.7366,5789.4312,31600,1,4146.9309,0.4746,0.0104,0.0669,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1238,-0.0056,0.0,1.0,0.0,0.0102,0.5469,0.4746,13.2111,51.0682,568.2429,33.8,857.9675,289.7247,0.0709,4.7406,35.0463,194.8197,230.1324,1359.0594,21.4195,195.0147,1102.761,0,63,63,504,448,20,51,0.07936507936507936,5,0.09523809523809523,6.0,0.746031746031746,47,0.07936507936507936,5.0,,,,,,,,,
|
||||
0.0,0,64,512,7,1.8906,0,0.875,5587.1855,5587.4615,5587.0158,31766,1,4900.2073,0.3809,0.0053,0.1109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1128,0.0,0.0,1.0,0.0,0.0114,0.4844,0.3809,13.5775,51.5483,569.7428,33.6,857.9675,288.2247,0.1075,4.8015,38.5633,225.6816,264.5674,1636.5882,22.1274,225.8961,1345.0836,0,64,62,496,422,26,68,0.04838709677419355,3,0.04838709677419355,3.0,1.0,62,0.0967741935483871,6.0,,,,,0.0081,8.5496,,0.06451612903225806,4.0
|
||||
0.0,0,64,512,4,1.9531,0,0.9375,4798.2617,4883.6444,4723.5128,31812,1,4722.6084,0.4668,0.0183,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,0.0,0.0,1.0,0.0,0.0168,0.6094,0.4668,13.718,51.5807,568.653,33.7,857.9675,289.3145,0.075,4.7303,35.8252,213.9135,250.0471,1536.8344,21.852,214.1464,1260.1995,0,65,61,488,434,20,44,0.06557377049180328,4,0.04918032786885246,3.0,0.7213114754098361,44,0.16393442622950818,10.0,,,,,,,5.9602,,
|
||||
0.0,0,64,512,3,1.6875,0,0.9062,4990.3906,4683.8621,5191.767,31741,1,4339.1846,0.3965,0.0113,0.1192,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1176,-0.0005,0.0,1.0,0.0,0.0138,0.5781,0.3965,13.9927,51.9113,567.4896,33.9,857.9675,290.4779,0.0707,4.8411,34.6825,205.3311,240.3191,1507.0204,22.8738,205.5654,1238.9803,0,66,64,512,451,19,50,0.03125,2,0.046875,3.0,0.75,48,0.171875,11.0,,,,,0.0085,8.6432,,,
|
||||
0.0,0,64,512,3,1.5938,0,0.8906,4687.6602,4780.5265,4614.2762,31720,1,3991.8123,0.4414,0.0147,0.1094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1043,-0.0031,0.0,1.0,0.0,0.0126,0.5625,0.4414,14.1397,52.1146,567.5696,33.8,857.9675,290.3979,0.0726,4.6609,30.5811,185.1371,216.0168,1571.263,21.9126,185.3626,1328.6676,0,67,58,464,422,16,42,0.034482758620689655,2,0.08620689655172414,5.0,0.6896551724137931,40,,,,,,,,,,,
|
||||
0.0,0,64,512,4,1.5469,0,0.8281,4902.5645,4448.6776,5228.5101,31770,1,4445.7836,0.418,-0.0081,0.0909,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1147,-0.0,0.0,1.0,0.0,0.0128,0.5312,0.418,14.1629,52.2518,569.6768,33.6,857.9675,288.2908,0.0962,4.7131,33.8447,209.0269,243.2635,1442.218,21.5536,209.322,1172.6792,0,68,64,512,471,16,37,0.046875,3,0.0625,4.0,0.46875,30,0.046875,3.0,0.015625,1.0,,,0.0082,8.6812,,,
|
||||
0.0,0,64,512,4,1.7344,0,0.875,4912.1777,4638.6631,5239.691,31749,1,4027.0232,0.5449,0.0247,0.126,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1083,-0.002,0.0,1.0,0.0,0.0111,0.6562,0.5449,14.5627,52.6307,568.1539,33.8,857.9675,289.8137,0.0922,4.7384,33.01,189.5539,222.8763,1413.2169,23.2675,189.7737,1162.3272,0,69,62,496,442,18,54,0.03225806451612903,2,,,0.8387096774193549,52,,,,,,,,,,,
|
||||
0.0,0,64,512,3,1.9219,0,0.9062,4989.8223,4749.7429,5156.7649,31793,1,4613.5261,0.4102,-0.0096,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1134,-0.0,0.0,1.0,0.0,0.0151,0.5312,0.4102,14.5696,52.6356,567.6034,33.8,857.9675,290.3641,0.0859,4.6794,34.9319,213.77,248.9891,1779.8183,22.218,213.9708,1503.9267,0,70,62,496,433,16,61,0.12903225806451613,8,0.03225806451612903,2.0,0.8387096774193549,52,0.03225806451612903,2.0,,,,,0.0081,8.4344,5.9241,,
|
||||
|
9
training_logs/20260612_220124_metrics_job_661735.csv
Normal file
9
training_logs/20260612_220124_metrics_job_661735.csv
Normal file
@@ -0,0 +1,9 @@
|
||||
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,timing/save_hf_model,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError
|
||||
0.0,0,64,512,3,0.3281,0,0.2188,4521.9473,4333.5193,4758.5198,31663,1,3365.2785,0.5566,0.0074,0.094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0,0.0,1.0,0.0,0.0131,0.5965,0.5566,12.5978,50.5919,579.345,32.5,857.9676,278.6226,0.1569,4.5133,32.1738,167.8055,200.36,3675.9345,21.9439,168.0287,3449.1169,0,71,58,464,375,29,88,1.4137931034482758,82,0.20689655172413793,12,0.017241379310344827,1.0,,,,,,,,,
|
||||
0.0,0,64,512,1,0.9375,0,0.9375,5361.2598,5767.993,5205.1622,31699,1,5207.4533,0.2773,-0.013,0.1323,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1151,0.0,0.0,1.0,0.0,0.017,0.4219,0.2773,12.8662,50.7994,576.8029,32.8,857.9676,281.1647,0.0648,4.6437,37.3367,229.2593,266.8776,1642.3033,25.4393,229.4755,1345.3366,0,72,56,448,364,24,80,1.3392857142857142,75,0.16071428571428573,9,0.07142857142857142,4.0,11.424,21.2849,0.16071428571428573,9.0,0.05357142857142857,3.0,,,
|
||||
0.0,0,64,512,2,1.6406,1,1.0,4303.7402,4841.9941,4036.1871,31778,1,4623.5614,0.332,0.0023,0.0825,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0977,0.0,0.0,1.0,0.0,0.0095,0.4062,0.332,13.201,51.0858,576.4416,32.8,857.9676,281.526,0.0649,4.6475,32.7604,195.3408,228.4104,875.421,23.4598,195.5846,618.8995,0,73,63,504,471,12,33,0.47619047619047616,30,0.07936507936507936,5,,,,,0.047619047619047616,3.0,,,,,
|
||||
0.0,0,64,512,3,1.6406,0,0.8906,5242.7773,4926.2055,5552.0154,31761,1,4678.9952,0.4941,0.0138,0.1243,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,-0.0,0.0,1.0,0.0,0.0145,0.5938,0.4941,13.4431,51.3293,575.8802,32.9,857.9676,282.0874,0.062,4.6498,37.0456,231.3918,268.7201,1733.9771,23.6881,231.6122,1436.9161,0,74,59,472,401,23,56,0.847457627118644,50,0.1864406779661017,11,0.2542372881355932,15.0,0.007,10.013,0.03389830508474576,2.0,0.05084745762711865,3.0,,,
|
||||
0.0,0,64,512,4,1.9844,1,1.0,4678.0137,4390.89,4862.0673,31720,1,4446.3943,0.3906,-0.0162,0.1412,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1014,-0.002,0.0,1.0,0.0,0.0143,0.5625,0.3906,13.5915,51.5202,575.6154,32.9,857.9676,282.3522,0.0636,4.5241,34.8828,208.4871,243.6766,1567.8023,22.6625,208.7299,1296.9223,0,75,61,488,422,25,66,0.7704918032786885,47,0.21311475409836064,13,,,,,0.06557377049180328,4.0,0.09836065573770492,6.0,6.424,,
|
||||
0.0,0,64,512,4,2.4219,2,1.0,4572.6895,4582.679,4568.0657,31799,1,4302.6602,0.3164,-0.0023,0.112,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.103,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3164,13.6858,51.5748,574.5665,33.0,857.9676,283.4011,0.0663,4.6913,30.5968,200.2044,231.0996,1415.5476,24.2091,200.4362,1155.5266,0,76,61,488,415,22,64,0.7868852459016393,48,0.21311475409836064,13,0.14754098360655737,9.0,0.0074,9.8798,0.04918032786885246,3.0,0.11475409836065574,7.0,,,
|
||||
0.0,0,64,512,4,3.0312,3,1.0,4597.7812,4086.6667,4944.6689,31759,1,4559.7375,0.4043,0.0013,0.0431,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0033,0.0,1.0,0.0,0.0068,0.4688,0.4043,14.3994,51.5634,573.7055,33.1,857.9676,284.2621,0.0642,4.5039,32.7244,196.2376,229.2847,1161.9454,21.4738,196.4958,906.6629,0,77,63,504,414,32,86,0.8095238095238095,51,0.4603174603174603,29,0.15873015873015872,10.0,,,,,0.09523809523809523,6.0,,0.015873015873015872,1.0
|
||||
0.0,0,64,512,4,4.0,4,1.0,5291.9141,6195.0086,5027.3712,28898,1,4117.8095,0.2266,0.0072,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,0.0,0.0,1.0,0.0,0.0105,0.375,0.2266,14.2961,51.5109,573.1121,33.2,857.9676,284.8555,0.1485,4.4428,33.4904,188.476,222.3217,1837.7946,17.1266,188.6825,1593.8676,0,78,67,536,466,24,64,0.6567164179104478,44,0.23880597014925373,16,0.08955223880597014,6.0,0.0588,10.9374,0.029850746268656716,2.0,0.029850746268656716,2.0,,,
|
||||
|
393
training_logs/20260612_220124_metrics_report.md
Normal file
393
training_logs/20260612_220124_metrics_report.md
Normal file
@@ -0,0 +1,393 @@
|
||||
# SkyRL Training Metrics Analysis
|
||||
|
||||
Generated from 4 log files
|
||||
|
||||
## Overview
|
||||
|
||||
| Log File | Total Steps | Metric Blocks | Final Reward (mean) | Final Reward (max) | Total Time (s) |
|
||||
|----------|-------------|---------------|---------------------|-------------------|----------------|
|
||||
| job_630124 | 28 | 28 | 0.4467 | 0.7695 | 41354.5 |
|
||||
| job_630127 | 56 | 28 | 0.4329 | 0.5703 | 40185.7 |
|
||||
| job_630130 | 70 | 14 | 0.3858 | 0.5449 | 36451.3 |
|
||||
| job_661735 | 78 | 8 | 0.3747 | 0.5566 | 13910.7 |
|
||||
|
||||
## Async Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:------------------------------|-----------:|---------:|------:|------:|--------:|
|
||||
| async/discard_rate | 0 | 0 | 0 | 0 | 78 |
|
||||
| async/discarded_count | 0 | 0 | 0 | 0 | 78 |
|
||||
| async/effective_batch_groups | 64 | 0 | 64 | 64 | 78 |
|
||||
| async/effective_batch_samples | 512 | 0 | 512 | 512 | 78 |
|
||||
| async/staleness_max | 3.52564 | 1.00291 | 0 | 7 | 78 |
|
||||
| async/staleness_mean | 1.76964 | 0.525328 | 0 | 4 | 78 |
|
||||
| async/staleness_min | 0.153846 | 0.625786 | 0 | 4 | 78 |
|
||||
| async/staleness_ratio | 0.872991 | 0.178479 | 0 | 1 | 78 |
|
||||
|
||||
## Generate Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:-------------------------------------|-----------:|---------:|---------:|---------:|--------:|
|
||||
| generate/avg_num_tokens | 4953.5 | 781.064 | 1687.9 | 6119.26 | 78 |
|
||||
| generate/avg_tokens_non_zero_rewards | 4928.65 | 524.786 | 3823.39 | 6195.01 | 78 |
|
||||
| generate/avg_tokens_zero_rewards | 5030.87 | 1001.93 | 1110.32 | 6614.23 | 78 |
|
||||
| generate/max_num_tokens | 31246.8 | 2087.2 | 19335 | 31885 | 78 |
|
||||
| generate/min_num_tokens | 17.1795 | 142.893 | 1 | 1263 | 78 |
|
||||
| generate/std_num_tokens | 4459.29 | 622.221 | 2744.51 | 5829.72 | 78 |
|
||||
|
||||
## Loss Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:----------------------------|----------:|----------:|--------:|-------:|--------:|
|
||||
| loss/avg_final_rewards | 0.423427 | 0.0901758 | 0.2051 | 0.7695 | 78 |
|
||||
| loss/avg_raw_advantages | 0.0121513 | 0.0324204 | -0.0162 | 0.1959 | 78 |
|
||||
| loss/avg_raw_advantages_abs | 0.106845 | 0.0320619 | 0.0431 | 0.2355 | 78 |
|
||||
|
||||
## Policy Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:----------------------------|-------------:|------------:|--------:|-------:|--------:|
|
||||
| policy/final_loss | -9.23077e-05 | 0.000277638 | -0.0015 | 0 | 78 |
|
||||
| policy/log_ratio_abs_max | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_mean | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_p99 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos00 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos10 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos20 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos30 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos40 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos50 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos60 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos70 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos80 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/log_ratio_abs_pos90 | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/n_tokens_dp_gt_10pct | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/n_tokens_dp_gt_1pct | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/n_tokens_dp_gt_50pct | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/policy_entropy | 0.119653 | 0.0179549 | 0.0511 | 0.1411 | 78 |
|
||||
| policy/policy_loss | -0.00595641 | 0.017477 | -0.0959 | 0 | 78 |
|
||||
| policy/policy_lr | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/policy_update_steps | 1 | 0 | 1 | 1 | 78 |
|
||||
| policy/ppo_clip_ratio | 0 | 0 | 0 | 0 | 78 |
|
||||
| policy/raw_grad_norm | 0.0149538 | 0.00982649 | 0.0068 | 0.0967 | 78 |
|
||||
|
||||
## Reward Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:----------------------|---------:|----------:|-------:|-------:|--------:|
|
||||
| reward/avg_pass_at_8 | 0.543153 | 0.079973 | 0.375 | 0.8125 | 78 |
|
||||
| reward/avg_raw_reward | 0.423427 | 0.0901758 | 0.2051 | 0.7695 | 78 |
|
||||
|
||||
## System Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:------------------------|---------:|-------------:|---------:|---------:|--------:|
|
||||
| system/process_rss_gb | 14.2314 | 1.13198 | 9.9914 | 16.2064 | 78 |
|
||||
| system/process_vms_gb | 51.7172 | 0.827814 | 50.291 | 53.4671 | 78 |
|
||||
| system/ram_available_gb | 570.794 | 11.4823 | 551.846 | 594.612 | 78 |
|
||||
| system/ram_percent | 33.4705 | 1.34049 | 30.7 | 35.7 | 78 |
|
||||
| system/ram_total_gb | 857.968 | 3.05352e-05 | 857.967 | 857.968 | 78 |
|
||||
| system/ram_used_gb | 287.174 | 11.4823 | 263.356 | 306.122 | 78 |
|
||||
|
||||
## Timing Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:--------------------------------------|------------:|------------:|---------:|-----------:|--------:|
|
||||
| timing/compute_advantages_and_returns | 0.102533 | 0.107328 | 0.0486 | 0.8661 | 78 |
|
||||
| timing/convert_to_training_input | 4.70007 | 0.352461 | 2.7178 | 5.493 | 78 |
|
||||
| timing/fwd_logprobs_values_reward | 35.7722 | 9.32356 | 15.1498 | 104.946 | 78 |
|
||||
| timing/policy_train | 207.902 | 31.034 | 108.532 | 269.709 | 78 |
|
||||
| timing/run_training | 244.034 | 34.7885 | 123.918 | 312.363 | 78 |
|
||||
| timing/step | 1691.05 | 1167.37 | 558.623 | 10105.4 | 78 |
|
||||
| timing/sync_weights | 22.7971 | 1.88627 | 17.1266 | 26.2516 | 78 |
|
||||
| timing/train_critic_and_policy | 208.159 | 31.0529 | 108.708 | 270.02 | 78 |
|
||||
| timing/wait_for_generation_buffer | 1419.52 | 1176.94 | 247.59 | 9825.37 | 78 |
|
||||
| timing/cleanup_old_checkpoints | 4.0377 | 18.4601 | 0.0067 | 113.745 | 39 |
|
||||
| timing/save_checkpoints | 14.6624 | 18.3571 | 7.9437 | 123.126 | 39 |
|
||||
| timing/save_hf_model | 24.8864 | 69.3195 | 5.8995 | 275.429 | 15 |
|
||||
|
||||
## Trainer Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:--------------------|-------:|--------:|------:|------:|--------:|
|
||||
| trainer/epoch | 0 | 0 | 0 | 0 | 78 |
|
||||
| trainer/global_step | 39.5 | 22.6605 | 1 | 78 | 78 |
|
||||
|
||||
## Batch_Errors Metrics
|
||||
|
||||
| | Mean | Std | Min | Max | Count |
|
||||
|:----------------------------------------------|------------:|-----------:|-------------:|-----------:|--------:|
|
||||
| batch_errors/total_batches | 61.5256 | 7.45676 | 44 | 119 | 78 |
|
||||
| batch_errors/total_instances | 492.205 | 59.6541 | 352 | 952 | 78 |
|
||||
| batch_errors/total_successful | 432.179 | 76.3362 | 191 | 918 | 78 |
|
||||
| batch_errors/total_failed | 20.7308 | 9.45897 | 8 | 58 | 78 |
|
||||
| batch_errors/total_masked | 49.3333 | 27.27 | 6 | 160 | 78 |
|
||||
| batch_errors/avg_RuntimeError | 0.10335 | 0.0403443 | 0.0420168 | 0.15873 | 6 |
|
||||
| batch_errors/total_RuntimeError | 6.83333 | 1.94079 | 5 | 10 | 6 |
|
||||
| batch_errors/avg_VerifierTimeoutError | 0.200918 | 0.399779 | 0.015625 | 1.94915 | 61 |
|
||||
| batch_errors/total_VerifierTimeoutError | 12.1639 | 24.1283 | 1 | 115 | 61 |
|
||||
| batch_errors/avg_AgentTimeoutError | 0.184673 | 0.486823 | 0.00840336 | 2.55932 | 62 |
|
||||
| batch_errors/total_AgentTimeoutError | 11.1613 | 29.1948 | 1 | 151 | 62 |
|
||||
| batch_errors/avg_InvalidChatHistory | 0.549849 | 0.239037 | 0.10084 | 1.41379 | 78 |
|
||||
| batch_errors/total_InvalidChatHistory | 33.5256 | 14.2343 | 7 | 82 | 78 |
|
||||
| batch_errors/avg_ContextLengthExceededError | 0.161599 | 0.087392 | 0.015873 | 0.460317 | 77 |
|
||||
| batch_errors/total_ContextLengthExceededError | 9.84416 | 5.14287 | 1 | 29 | 77 |
|
||||
| batch_errors/avg_DaytonaError | 0.0555526 | 0.0370314 | 0.016129 | 0.114754 | 10 |
|
||||
| batch_errors/total_DaytonaError | 3.4 | 2.27058 | 1 | 7 | 10 |
|
||||
| batch_errors/avg_AgentSetupTimeoutError | 0.0930756 | 0.121065 | 0.0153846 | 0.372881 | 18 |
|
||||
| batch_errors/total_AgentSetupTimeoutError | 5.5 | 7.15583 | 1 | 22 | 18 |
|
||||
| batch_errors/avg_DaytonaAuthenticationError | 0.0222833 | 0.0083652 | 0.015873 | 0.031746 | 3 |
|
||||
| batch_errors/total_DaytonaAuthenticationError | 1.33333 | 0.57735 | 1 | 2 | 3 |
|
||||
|
||||
## Training Progression by Log
|
||||
|
||||
### job_630124
|
||||
|
||||
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||
|------|--------|--------|-----|------|---------------|-------------|
|
||||
| 1 | 0.7695 | 0.8125 | 0.000000 | 0.0000 | 3372.0 | 3167.0 |
|
||||
| 2 | 0.4199 | 0.5781 | 0.000000 | -0.0001 | 1718.0 | 1388.3 |
|
||||
| 3 | 0.3457 | 0.4688 | 0.000000 | -0.0000 | 1220.7 | 885.5 |
|
||||
| 4 | 0.4141 | 0.4844 | 0.000000 | -0.0001 | 1357.9 | 1059.7 |
|
||||
| 5 | 0.5156 | 0.6250 | 0.000000 | -0.0001 | 1224.4 | 955.5 |
|
||||
| 6 | 0.4023 | 0.4844 | 0.000000 | -0.0000 | 1281.0 | 982.2 |
|
||||
| 7 | 0.4492 | 0.5469 | 0.000000 | -0.0000 | 1559.3 | 1299.2 |
|
||||
| 8 | 0.4766 | 0.6250 | 0.000000 | 0.0000 | 1610.3 | 1270.4 |
|
||||
| 9 | 0.4453 | 0.5469 | 0.000000 | 0.0000 | 1321.3 | 1032.5 |
|
||||
| 10 | 0.3730 | 0.5312 | 0.000000 | 0.0000 | 1294.9 | 1011.8 |
|
||||
| 11 | 0.4922 | 0.6562 | 0.000000 | -0.0001 | 1475.0 | 1214.5 |
|
||||
| 12 | 0.3477 | 0.4219 | 0.000000 | 0.0000 | 1559.4 | 1264.5 |
|
||||
| 13 | 0.5020 | 0.6250 | 0.000000 | -0.0000 | 1431.9 | 1188.5 |
|
||||
| 14 | 0.4355 | 0.5469 | 0.000000 | -0.0000 | 1142.8 | 885.9 |
|
||||
| 15 | 0.4902 | 0.6094 | 0.000000 | -0.0000 | 1314.0 | 1035.8 |
|
||||
| 16 | 0.4512 | 0.5781 | 0.000000 | -0.0000 | 1308.8 | 1024.4 |
|
||||
| 17 | 0.3887 | 0.5156 | 0.000000 | -0.0000 | 1220.9 | 899.1 |
|
||||
| 18 | 0.4551 | 0.5625 | 0.000000 | -0.0000 | 1582.7 | 1299.8 |
|
||||
| 19 | 0.3691 | 0.4531 | 0.000000 | -0.0000 | 1533.3 | 1254.4 |
|
||||
| 20 | 0.4023 | 0.5156 | 0.000000 | 0.0000 | 1501.3 | 1228.6 |
|
||||
| 21 | 0.3633 | 0.4688 | 0.000000 | -0.0001 | 1384.7 | 1125.9 |
|
||||
| 22 | 0.4473 | 0.5781 | 0.000000 | 0.0000 | 1341.1 | 1063.0 |
|
||||
| 23 | 0.4160 | 0.5781 | 0.000000 | -0.0000 | 1636.8 | 1328.4 |
|
||||
| 24 | 0.3535 | 0.4531 | 0.000000 | 0.0000 | 1813.1 | 1525.8 |
|
||||
| 25 | 0.4121 | 0.4844 | 0.000000 | -0.0000 | 988.4 | 747.1 |
|
||||
| 26 | 0.5898 | 0.6875 | 0.000000 | -0.0001 | 1031.5 | 783.3 |
|
||||
| 27 | 0.4199 | 0.5469 | 0.000000 | -0.0002 | 2131.0 | 1853.8 |
|
||||
| 28 | 0.5605 | 0.6250 | 0.000000 | -0.0003 | 998.4 | 778.3 |
|
||||
|
||||
### job_630127
|
||||
|
||||
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||
|------|--------|--------|-----|------|---------------|-------------|
|
||||
| 29 | 0.5430 | 0.6610 | 0.000000 | -0.0001 | 3907.1 | 3661.5 |
|
||||
| 30 | 0.3984 | 0.5156 | 0.000000 | -0.0000 | 1561.1 | 1238.2 |
|
||||
| 31 | 0.2578 | 0.3906 | 0.000000 | -0.0001 | 558.6 | 247.6 |
|
||||
| 32 | 0.4316 | 0.5469 | 0.000000 | -0.0001 | 1881.7 | 1636.3 |
|
||||
| 33 | 0.3945 | 0.5469 | 0.000000 | 0.0000 | 1516.1 | 1212.1 |
|
||||
| 34 | 0.4160 | 0.5156 | 0.000000 | -0.0000 | 1068.5 | 739.5 |
|
||||
| 35 | 0.4102 | 0.5000 | 0.000000 | -0.0000 | 1384.1 | 1112.5 |
|
||||
| 36 | 0.4609 | 0.6406 | 0.000000 | -0.0000 | 1464.7 | 1177.2 |
|
||||
| 37 | 0.5703 | 0.6875 | 0.000000 | -0.0000 | 1077.8 | 785.6 |
|
||||
| 38 | 0.4570 | 0.6094 | 0.000000 | -0.0000 | 1281.5 | 984.5 |
|
||||
| 39 | 0.4219 | 0.5625 | 0.000000 | -0.0000 | 1220.2 | 969.3 |
|
||||
| 40 | 0.4766 | 0.5938 | 0.000000 | 0.0000 | 1608.5 | 1354.9 |
|
||||
| 41 | 0.4980 | 0.5781 | 0.000000 | -0.0000 | 1037.8 | 802.5 |
|
||||
| 42 | 0.3770 | 0.4531 | 0.000000 | 0.0000 | 1385.1 | 1098.6 |
|
||||
| 43 | 0.4043 | 0.4844 | 0.000000 | -0.0000 | 1474.0 | 1198.9 |
|
||||
| 44 | 0.4980 | 0.6250 | 0.000000 | -0.0000 | 995.3 | 726.5 |
|
||||
| 45 | 0.4707 | 0.5156 | 0.000000 | 0.0000 | 1030.1 | 747.8 |
|
||||
| 46 | 0.4766 | 0.5625 | 0.000000 | 0.0000 | 1116.6 | 848.4 |
|
||||
| 47 | 0.5098 | 0.6250 | 0.000000 | 0.0000 | 1436.6 | 1153.8 |
|
||||
| 48 | 0.3965 | 0.5000 | 0.000000 | -0.0000 | 1388.9 | 1118.5 |
|
||||
| 49 | 0.3613 | 0.4844 | 0.000000 | 0.0000 | 1674.2 | 1335.1 |
|
||||
| 50 | 0.4473 | 0.5469 | 0.000000 | 0.0000 | 1060.2 | 789.3 |
|
||||
| 51 | 0.3848 | 0.4531 | 0.000000 | 0.0000 | 1686.7 | 1378.9 |
|
||||
| 52 | 0.4102 | 0.5312 | 0.000000 | -0.0000 | 1600.4 | 1319.3 |
|
||||
| 53 | 0.4473 | 0.5312 | 0.000000 | 0.0000 | 1082.9 | 829.7 |
|
||||
| 54 | 0.3887 | 0.5156 | 0.000000 | 0.0000 | 1827.2 | 1523.5 |
|
||||
| 55 | 0.4004 | 0.5156 | 0.000000 | -0.0000 | 1484.0 | 1185.3 |
|
||||
| 56 | 0.4121 | 0.5156 | 0.000000 | -0.0000 | 1375.8 | 1104.5 |
|
||||
|
||||
### job_630130
|
||||
|
||||
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||
|------|--------|--------|-----|------|---------------|-------------|
|
||||
| 57 | 0.2051 | 0.3898 | 0.000000 | -0.0007 | 10105.4 | 9825.4 |
|
||||
| 58 | 0.2266 | 0.5000 | 0.000000 | -0.0008 | 3407.9 | 3181.4 |
|
||||
| 59 | 0.2520 | 0.5000 | 0.000000 | -0.0015 | 3636.1 | 3468.6 |
|
||||
| 60 | 0.2129 | 0.4531 | 0.000000 | -0.0008 | 3855.0 | 3710.5 |
|
||||
| 61 | 0.4590 | 0.7188 | 0.000000 | -0.0015 | 1882.4 | 1715.1 |
|
||||
| 62 | 0.5117 | 0.6094 | 0.000000 | -0.0003 | 1318.3 | 1070.7 |
|
||||
| 63 | 0.4746 | 0.5469 | 0.000000 | -0.0001 | 1359.1 | 1102.8 |
|
||||
| 64 | 0.3809 | 0.4844 | 0.000000 | 0.0000 | 1636.6 | 1345.1 |
|
||||
| 65 | 0.4668 | 0.6094 | 0.000000 | 0.0000 | 1536.8 | 1260.2 |
|
||||
| 66 | 0.3965 | 0.5781 | 0.000000 | -0.0000 | 1507.0 | 1239.0 |
|
||||
| 67 | 0.4414 | 0.5625 | 0.000000 | -0.0000 | 1571.3 | 1328.7 |
|
||||
| 68 | 0.4180 | 0.5312 | 0.000000 | -0.0000 | 1442.2 | 1172.7 |
|
||||
| 69 | 0.5449 | 0.6562 | 0.000000 | -0.0000 | 1413.2 | 1162.3 |
|
||||
| 70 | 0.4102 | 0.5312 | 0.000000 | -0.0000 | 1779.8 | 1503.9 |
|
||||
|
||||
### job_661735
|
||||
|
||||
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||
|------|--------|--------|-----|------|---------------|-------------|
|
||||
| 71 | 0.5566 | 0.5965 | 0.000000 | -0.0000 | 3675.9 | 3449.1 |
|
||||
| 72 | 0.2773 | 0.4219 | 0.000000 | 0.0000 | 1642.3 | 1345.3 |
|
||||
| 73 | 0.3320 | 0.4062 | 0.000000 | 0.0000 | 875.4 | 618.9 |
|
||||
| 74 | 0.4941 | 0.5938 | 0.000000 | -0.0000 | 1734.0 | 1436.9 |
|
||||
| 75 | 0.3906 | 0.5625 | 0.000000 | -0.0000 | 1567.8 | 1296.9 |
|
||||
| 76 | 0.3164 | 0.4531 | 0.000000 | 0.0000 | 1415.5 | 1155.5 |
|
||||
| 77 | 0.4043 | 0.4688 | 0.000000 | -0.0001 | 1161.9 | 906.7 |
|
||||
| 78 | 0.2266 | 0.3750 | 0.000000 | 0.0000 | 1837.8 | 1593.9 |
|
||||
|
||||
## Timing Analysis
|
||||
|
||||
### Average Time Breakdown (% of step time)
|
||||
|
||||
| Component | Avg % of Step Time |
|
||||
|-----------|-------------------|
|
||||
| wait_for_generation_buffer | 80.8% |
|
||||
| run_training | 17.3% |
|
||||
| train_critic_and_policy | 14.8% |
|
||||
| policy_train | 14.8% |
|
||||
| fwd_logprobs_values_reward | 2.5% |
|
||||
| sync_weights | 1.6% |
|
||||
| save_hf_model | 1.0% |
|
||||
| save_checkpoints | 0.9% |
|
||||
| convert_to_training_input | 0.3% |
|
||||
| cleanup_old_checkpoints | 0.2% |
|
||||
| compute_advantages_and_returns | 0.0% |
|
||||
|
||||
## Cross-Log Comparison
|
||||
|
||||
| Log | Avg Reward | Pass@8 | Step Time (s) | Gen Wait Time (s) | Avg Tokens | Staleness |
|
||||
|-----|------|------|------|------|------|------|
|
||||
| job_630124 | 0.4467 | 0.5575 | 1476.9481 | 1198.3201 | 5107.1387 | 1.7701 |
|
||||
| job_630127 | 0.4329 | 0.5431 | 1435.2019 | 1152.8499 | 5301.5774 | 1.8175 |
|
||||
| job_630130 | 0.3858 | 0.5479 | 2603.6675 | 2363.3124 | 4025.6459 | 1.5424 |
|
||||
| job_661735 | 0.3747 | 0.4847 | 1738.8407 | 1475.4061 | 4821.2654 | 1.9980 |
|
||||
|
||||
## vLLM Inference Engine Analysis
|
||||
|
||||
Metrics from vLLM stat loggers (V1LoggingStatLoggerFixed).
|
||||
|
||||
> **Note**: Ray deduplicates similar log messages with `[repeated Nx across cluster]`,
|
||||
> so we typically capture stats from one engine per timestamp. The stats shown are
|
||||
> **per-engine** values. Multiply by num_inference_engines for cluster-wide estimates.
|
||||
|
||||
### Summary by Log (Per-Engine Stats)
|
||||
|
||||
| Log | Avg Running/Engine | Avg Waiting/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % |
|
||||
|-----|-------------------|-------------------|--------------------------|----------------|------------------|
|
||||
| job_630124 | 11.5 | 0.4 | 494.3 tok/s | 51.4% | 67.8% |
|
||||
| job_630127 | 12.0 | 0.6 | 510.3 tok/s | 53.0% | 69.6% |
|
||||
| job_630130 | 6.4 | 0.0 | 287.3 tok/s | 28.1% | 85.0% |
|
||||
| job_661734 | 8.1 | 0.0 | 344.0 tok/s | 34.5% | 83.8% |
|
||||
| job_661735 | 10.2 | 0.2 | 442.6 tok/s | 44.3% | 69.2% |
|
||||
|
||||
### Utilization Analysis (Per-Engine)
|
||||
|
||||
Key indicators of inference engine utilization:
|
||||
|
||||
- **Running requests/engine**: Concurrent requests being processed by each engine
|
||||
- **Waiting requests**: Requests queued (0 = engine not saturated, has spare capacity)
|
||||
- **Generation throughput**: Decode tokens/sec per engine
|
||||
- 8B model on H100 can do **1000+ tok/s** when saturated
|
||||
- If seeing <300 tok/s with 0 waiting, engine is **starved for requests**
|
||||
|
||||
#### job_630124
|
||||
|
||||
- **Running requests/engine**: avg=11.5, max=24
|
||||
- **Waiting requests**: avg=0.4, max=43
|
||||
- **Generation throughput/engine**: avg=494.3 tok/s, max=984.1 tok/s
|
||||
- **KV cache usage**: avg=51.4%
|
||||
- **Prefix cache hit rate**: avg=67.8%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
#### job_630127
|
||||
|
||||
- **Running requests/engine**: avg=12.0, max=24
|
||||
- **Waiting requests**: avg=0.6, max=41
|
||||
- **Generation throughput/engine**: avg=510.3 tok/s, max=1005.9 tok/s
|
||||
- **KV cache usage**: avg=53.0%
|
||||
- **Prefix cache hit rate**: avg=69.6%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
#### job_630130
|
||||
|
||||
- **Running requests/engine**: avg=6.4, max=24
|
||||
- **Waiting requests**: avg=0.0, max=19
|
||||
- **Generation throughput/engine**: avg=287.3 tok/s, max=952.9 tok/s
|
||||
- **KV cache usage**: avg=28.1%
|
||||
- **Prefix cache hit rate**: avg=85.0%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
#### job_661734
|
||||
|
||||
- **Running requests/engine**: avg=8.1, max=24
|
||||
- **Waiting requests**: avg=0.0, max=4
|
||||
- **Generation throughput/engine**: avg=344.0 tok/s, max=862.0 tok/s
|
||||
- **KV cache usage**: avg=34.5%
|
||||
- **Prefix cache hit rate**: avg=83.8%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
#### job_661735
|
||||
|
||||
- **Running requests/engine**: avg=10.2, max=24
|
||||
- **Waiting requests**: avg=0.2, max=18
|
||||
- **Generation throughput/engine**: avg=442.6 tok/s, max=962.9 tok/s
|
||||
- **KV cache usage**: avg=44.3%
|
||||
- **Prefix cache hit rate**: avg=69.2%
|
||||
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||
|
||||
## Trial-Level Analysis (from result.json)
|
||||
|
||||
Total trials parsed: 38123
|
||||
|
||||
### Turn Count Statistics
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Mean | 5.0 |
|
||||
| Median | 5.0 |
|
||||
| Std | 2.4 |
|
||||
| Min | 1 |
|
||||
| Max | 33 |
|
||||
| Count | 37976 |
|
||||
|
||||
### Exception Distribution
|
||||
|
||||
| Exception Type | Count | % |
|
||||
|---------------|-------|---|
|
||||
| No exception | 26113 | 68.5% |
|
||||
| AgentTimeoutError | 10372 | 27.2% |
|
||||
| VerifierTimeoutError | 802 | 2.1% |
|
||||
| ContextLengthExceededError | 679 | 1.8% |
|
||||
| AgentSetupTimeoutError | 105 | 0.3% |
|
||||
| RuntimeError | 41 | 0.1% |
|
||||
| DaytonaError | 7 | 0.0% |
|
||||
| DaytonaAuthenticationError | 4 | 0.0% |
|
||||
|
||||
### Turn Count by Exception Type
|
||||
|
||||
| Exception Type | Mean Turns | Median Turns | Count |
|
||||
|---------------|-----------|-------------|-------|
|
||||
| No exception | 5.6 | 5.0 | 26113 |
|
||||
| VerifierTimeoutError | 5.6 | 5.0 | 802 |
|
||||
| ContextLengthExceededError | 3.8 | 4.0 | 679 |
|
||||
| AgentTimeoutError | 3.4 | 3.0 | 10372 |
|
||||
| DaytonaError | 3.2 | 4.0 | 6 |
|
||||
| DaytonaAuthenticationError | 1.8 | 1.5 | 4 |
|
||||
|
||||
### Turn Count by Outcome
|
||||
|
||||
| Outcome | Mean Turns | Median Turns | Count |
|
||||
|---------|-----------|-------------|-------|
|
||||
| Success | 5.3 | 5.0 | 16218 |
|
||||
| Failure | 4.8 | 5.0 | 19449 |
|
||||
|
||||
### Reward Summary
|
||||
|
||||
- Mean reward: 0.4547
|
||||
- Success rate: 45.5%
|
||||
- Trials with reward data: 35667
|
||||
|
||||
79
training_logs/20260612_220124_metrics_table.csv
Normal file
79
training_logs/20260612_220124_metrics_table.csv
Normal file
@@ -0,0 +1,79 @@
|
||||
log_file,async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,global_step
|
||||
job_630124,0.0,0,64,512,0,0.0,0,0.0,4169.1133,3891.7792,5095.1271,29264,1263,2914.3086,0.7695,-0.0069,0.0744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1189,0.0,0.0,1.0,0.0,0.0105,0.8125,0.7695,11.6896,50.3563,578.7643,32.5,857.9675,279.2032,0.1941,4.3043,30.7211,146.6405,177.747,3371.9765,22.9409,146.8314,3166.9842,0,1,119,952,918,22,26,0.04201680672268908,5.0,0.025210084033613446,3.0,0.008403361344537815,1.0,0.10084033613445378,12,0.14285714285714285,17.0,,,,,,,,,,1
|
||||
job_630124,0.0,0,64,512,1,0.8594,0,0.8594,5688.084,5824.2419,5589.5185,31885,1,5477.7731,0.4199,0.0464,0.1255,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1361,-0.005,0.0,1.0,0.0,0.0161,0.5781,0.4199,12.6834,50.5541,574.2212,33.1,857.9675,283.7463,0.0641,4.8042,39.1437,260.9703,300.4542,1718.0337,24.4369,261.2461,1388.3356,0,2,63,504,454,24,37,0.15873015873015872,10.0,0.047619047619047616,3.0,0.06349206349206349,4.0,0.36507936507936506,23,0.15873015873015872,10.0,14.2681,24.8089,,,,,,,,2
|
||||
job_630124,0.0,0,64,512,2,1.5625,0,0.9375,5287.3867,5200.8475,5333.1104,31817,1,5758.917,0.3457,0.0073,0.0951,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,-0.0021,0.0,1.0,0.0,0.0171,0.4688,0.3457,12.9334,50.6717,572.7552,33.2,857.9675,285.2123,0.1003,4.7883,38.0045,267.5741,305.9524,1220.661,24.4652,267.8472,885.4504,0,3,62,496,468,14,23,0.08064516129032258,5.0,,,,,0.3064516129032258,19,0.0967741935483871,6.0,,,,,,,,,,3
|
||||
job_630124,0.0,0,64,512,3,1.8438,0,0.875,5684.4512,4907.0708,6233.8,31815,1,5008.2154,0.4141,-0.0061,0.0833,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1411,-0.004,0.0,1.0,0.0,0.0121,0.4844,0.4141,13.7453,50.795,563.5295,34.3,857.9675,294.438,0.0673,4.9105,37.4051,234.7096,272.4655,1357.8902,20.825,234.9927,1059.6856,0,4,62,496,451,21,36,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.3709677419354839,23,0.22580645161290322,14.0,0.008,12.7181,,,,,,,,4
|
||||
job_630124,0.0,0,64,512,4,1.875,0,0.9219,5039.3906,5546.9962,4499.0363,31714,1,4334.7369,0.5156,0.0318,0.1173,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1279,-0.0036,0.0,1.0,0.0,0.0141,0.625,0.5156,13.8122,50.8325,562.6003,34.4,857.9675,295.3672,0.0989,4.6928,39.1023,199.9551,239.4905,1224.4331,24.7234,200.2888,955.52,0,5,62,496,449,22,32,0.11290322580645161,7.0,0.12903225806451613,8.0,,,0.3387096774193548,21,0.1935483870967742,12.0,,,8.2216,,,,,,,5
|
||||
job_630124,0.0,0,64,512,4,2.1719,0,0.9375,5591.9941,5830.8641,5431.1863,31798,1,4994.2618,0.4023,0.0066,0.0843,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0004,0.0,1.0,0.0,0.0111,0.4844,0.4023,13.8712,50.9265,561.6264,34.5,857.9675,296.3411,0.0653,4.9236,37.4489,233.3771,271.219,1281.0159,22.6774,233.7045,982.1918,0,6,62,496,435,23,51,0.12903225806451613,8.0,0.03225806451612903,2.0,0.04838709677419355,3.0,0.5161290322580645,32,0.25806451612903225,16.0,0.0067,13.924,,0.016129032258064516,1.0,,,,,6
|
||||
job_630124,0.0,0,64,512,4,2.1094,0,0.9062,4851.3477,5205.5652,4562.4468,31817,1,4336.985,0.4492,0.0059,0.0777,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1271,-0.0014,0.0,1.0,0.0,0.0139,0.5469,0.4492,13.9109,50.9409,561.0771,34.6,857.9675,296.8904,0.0644,4.6898,33.9517,197.6421,231.9494,1559.2848,23.4677,197.9329,1299.1744,0,7,63,504,457,15,42,,,0.07936507936507936,5.0,0.031746031746031744,2.0,0.5555555555555556,35,0.07936507936507936,5.0,,,,,,,,,,7
|
||||
job_630124,0.0,0,64,512,4,2.1094,0,0.9219,5596.0137,5715.5123,5487.2164,31840,1,5416.4656,0.4766,0.0199,0.145,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,0.0,0.0,1.0,0.0,0.0136,0.625,0.4766,13.9023,50.9439,561.6647,34.5,857.9675,296.3028,0.0672,4.797,43.3436,268.6347,312.3633,1610.273,22.6954,268.9523,1270.4134,0,8,62,496,457,18,38,,,,,0.016129032258064516,1.0,0.4032258064516129,25,0.20967741935483872,13.0,0.0071,12.1975,,,,0.016129032258064516,1.0,,,8
|
||||
job_630124,0.0,0,64,512,4,1.9844,0,0.9531,5326.7715,5167.9342,5454.2887,31758,1,4691.8758,0.4453,0.0119,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0137,0.5469,0.4453,13.8936,50.9354,559.6754,34.8,857.9675,298.2921,0.0782,4.7117,38.0949,222.0535,260.5136,1321.2856,23.5801,222.3401,1032.476,0,9,61,488,449,18,38,,,0.01639344262295082,1.0,0.03278688524590164,2.0,0.4426229508196721,27,0.22950819672131148,14.0,,,,,,,,,,9
|
||||
job_630124,0.0,0,64,512,3,1.8906,0,0.9062,4968.5547,4853.8377,5036.8131,31768,1,4721.9898,0.373,0.0139,0.118,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1359,0.0,0.0,1.0,0.0,0.0144,0.5312,0.373,13.9257,51.0073,560.3441,34.7,857.9675,297.6235,0.0714,4.8344,33.5456,220.0195,253.8928,1294.919,24.3901,220.2753,1011.7949,0,10,62,496,442,18,46,,,0.12903225806451613,8.0,0.016129032258064516,1.0,0.6451612903225806,40,0.12903225806451613,8.0,0.0084,14.0711,8.8277,,,,,,,10
|
||||
job_630124,0.0,0,64,512,4,1.7812,0,0.9219,4462.5332,4532.996,4394.2385,31762,1,4298.9711,0.4922,0.0008,0.1662,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0056,0.0,1.0,0.0,0.0183,0.6562,0.4922,14.0333,51.1276,559.0244,34.8,857.9675,298.9431,0.0938,4.6549,32.2602,198.6454,231.2913,1475.0347,24.5599,198.9368,1214.5237,0,11,64,512,471,16,33,,,0.125,8.0,0.046875,3.0,0.328125,21,0.140625,9.0,,,,,,,,,,11
|
||||
job_630124,0.0,0,64,512,4,2.0938,0,0.8906,5849.1094,5035.9944,6282.4461,31797,1,4813.1067,0.3477,-0.0072,0.0606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.013,0.4219,0.3477,14.0347,51.1288,558.5858,34.9,857.9675,299.3817,0.0732,5.2651,38.6448,224.9423,263.9456,1559.3505,25.6632,225.2273,1264.4721,0,12,57,456,426,14,30,,,,,0.07017543859649122,4.0,0.42105263157894735,24,0.14035087719298245,8.0,0.0084,14.0752,,,,,,,,12
|
||||
job_630124,0.0,0,64,512,4,1.8125,0,0.875,4866.082,5038.6537,4692.1569,31760,1,4066.2403,0.502,-0.0034,0.1269,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1255,-0.0,0.0,1.0,0.0,0.0186,0.625,0.502,14.0677,51.1943,557.1813,35.1,857.9675,300.7862,0.0704,4.6745,31.6351,183.7078,215.6617,1431.9056,23.0843,183.9558,1188.48,0,13,64,512,450,21,49,,,0.203125,13.0,0.25,16.0,0.46875,30,0.09375,6.0,,,,,,,,,,13
|
||||
job_630124,0.0,0,64,512,4,2.0469,0,0.9375,4926.5117,4839.1749,4993.9031,31751,1,4266.9039,0.4355,0.0028,0.0804,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1228,-0.0013,0.0,1.0,0.0,0.0162,0.5469,0.4355,14.322,51.506,557.3558,35.0,857.9675,300.6118,0.0972,4.7529,33.9188,193.7138,228.0319,1142.7788,24.0818,194.0154,885.9041,0,14,63,504,473,18,30,,,,,,,0.3333333333333333,21,0.14285714285714285,9.0,0.0084,13.3568,,,,,,0.015873015873015872,1.0,14
|
||||
job_630124,0.0,0,64,512,4,2.0312,0,0.9219,5025.9766,4558.8924,5475.1648,31707,1,4544.2113,0.4902,0.0047,0.108,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1336,-0.0,0.0,1.0,0.0,0.0154,0.6094,0.4902,14.6948,51.8791,556.9661,35.1,857.9675,301.0014,0.0749,4.7275,31.8001,218.695,250.8643,1313.9526,22.5691,218.989,1035.7868,0,15,64,512,477,18,28,,,0.109375,7.0,,,0.296875,19,0.1875,12.0,,,8.2163,,,,,,,15
|
||||
job_630124,0.0,0,64,512,4,1.9375,0,0.8906,5304.2988,4924.3203,5616.6655,31702,1,4477.752,0.4512,0.0092,0.1087,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1249,-0.0,0.0,1.0,0.0,0.0148,0.5781,0.4512,14.4315,51.6356,556.9907,35.1,857.9675,300.9769,0.1006,4.7795,36.8211,219.4484,256.6458,1308.8006,22.9796,219.7236,1024.3882,0,16,61,488,421,21,66,,,,,0.09836065573770492,6.0,0.8360655737704918,51,0.16393442622950818,10.0,0.0077,14.8915,,,,0.01639344262295082,1.0,,,16
|
||||
job_630124,0.0,0,64,512,4,2.3594,0,0.9375,5391.4434,6104.0653,4938.3706,31758,1,5320.6865,0.3887,0.0223,0.0896,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1221,-0.002,0.0,1.0,0.0,0.0108,0.5156,0.3887,14.5796,51.7618,556.1977,35.2,857.9675,301.7698,0.0974,5.3524,38.0588,253.8113,292.2721,1220.8525,24.0951,254.1154,899.1259,0,17,57,456,419,17,35,,,0.017543859649122806,1.0,0.03508771929824561,2.0,0.38596491228070173,22,0.24561403508771928,14.0,,,,,,0.017543859649122806,1.0,,,17
|
||||
job_630124,0.0,0,64,512,4,1.7188,0,0.875,5066.6953,5047.4764,5082.7455,31770,1,4672.0163,0.4551,0.0005,0.1098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1223,-0.0028,0.0,1.0,0.0,0.0154,0.5625,0.4551,14.7728,51.9543,556.2234,35.2,857.9675,301.7441,0.0991,4.843,39.5878,215.8858,255.888,1582.6579,22.1364,216.2007,1299.7832,0,18,60,480,437,18,41,,,0.03333333333333333,2.0,0.016666666666666666,1.0,0.5333333333333333,32,0.16666666666666666,10.0,0.0075,16.3173,,,,,,,,18
|
||||
job_630124,0.0,0,64,512,4,1.8281,0,0.8906,5320.7246,4968.5291,5526.808,31825,1,4635.3488,0.3691,-0.0052,0.097,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1408,-0.0,0.0,1.0,0.0,0.0136,0.4531,0.3691,14.7764,51.9575,554.7101,35.3,857.9675,303.2574,0.0666,4.8474,33.322,217.0675,250.7206,1533.2708,23.3393,217.3317,1254.3583,0,19,62,496,453,17,43,,,,,,,0.6129032258064516,38,0.0967741935483871,6.0,,,,,,,,,,19
|
||||
job_630124,0.0,0,64,512,4,1.6875,0,0.8906,4981.0215,5122.7233,4885.6275,31734,1,4443.4633,0.4023,-0.0006,0.0906,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,0.0,0.0,1.0,0.0,0.0168,0.5156,0.4023,14.7832,52.0205,554.9844,35.3,857.9675,302.9831,0.0665,4.7337,34.0494,210.2643,244.6153,1501.26,23.3047,210.499,1228.6018,0,20,63,504,461,18,38,,,0.047619047619047616,3.0,0.06349206349206349,4.0,0.5396825396825397,34,0.06349206349206349,4.0,0.0078,13.3388,8.0322,,,0.031746031746031744,2.0,,,20
|
||||
job_630124,0.0,0,64,512,4,1.7969,0,0.8906,5089.5234,5337.2258,4948.1963,31834,1,4103.7515,0.3633,-0.0098,0.0985,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0055,0.0,1.0,0.0,0.0123,0.4688,0.3633,14.7559,51.9613,554.3621,35.4,857.9675,303.6055,0.0721,4.8151,32.0319,197.6414,230.0275,1384.6688,23.9344,197.9232,1125.887,0,21,61,488,432,21,46,,,0.13114754098360656,8.0,,,0.6229508196721312,38,0.13114754098360656,8.0,,,,,,0.03278688524590164,2.0,,,21
|
||||
job_630124,0.0,0,64,512,3,1.5156,0,0.8281,4888.4629,4445.7686,5246.6855,31774,1,4638.3152,0.4473,0.0144,0.0967,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,0.0,0.0,1.0,0.0,0.0135,0.5781,0.4473,15.1406,52.3359,554.3356,35.4,857.9675,303.6319,0.0672,4.8409,37.4714,213.0209,250.8158,1341.0965,22.4572,213.2768,1062.9785,0,22,64,512,452,21,60,,,,,0.046875,3.0,0.734375,47,0.15625,10.0,0.0085,14.204,,,,,,,,22
|
||||
job_630124,0.0,0,64,512,4,1.6094,0,0.8594,5010.1426,5339.5915,4775.4515,31738,1,5125.7203,0.416,0.0029,0.123,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1269,-0.0,0.0,1.0,0.0,0.0161,0.5781,0.416,15.1399,52.3352,554.2477,35.4,857.9675,303.7198,0.0666,4.7525,38.7094,241.3185,280.3783,1636.7829,23.2267,241.602,1328.421,0,23,61,488,423,17,65,,,,,0.04918032786885246,3.0,0.639344262295082,39,0.3770491803278688,23.0,,,,,,,,,,23
|
||||
job_630124,0.0,0,64,512,3,1.5156,0,0.8281,4957.0039,5145.7238,4853.8066,31744,1,4792.2542,0.3535,-0.005,0.139,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1197,0.0,0.0,1.0,0.0,0.0118,0.4531,0.3535,15.2222,52.591,552.66,35.6,857.9675,305.3075,0.0667,4.8036,34.8997,225.7814,261.0063,1813.0727,21.5017,226.0394,1525.7569,0,24,53,424,405,10,17,,,0.1509433962264151,8.0,0.09433962264150944,5.0,0.1320754716981132,7,0.24528301886792453,13.0,0.0085,13.7903,,,,,,,,24
|
||||
job_630124,0.0,0,64,512,4,1.8281,0,0.9062,4999.9961,4608.5735,5274.3821,31665,1,3727.2595,0.4121,0.0076,0.0996,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1278,-0.0,0.0,1.0,0.0,0.0129,0.4844,0.4121,15.4929,52.9026,551.8461,35.7,857.9675,306.1215,0.0644,4.6949,29.2144,181.8811,211.389,988.3901,25.1751,182.1098,747.1268,0,25,54,432,406,12,19,,,0.12962962962962962,7.0,0.07407407407407407,4.0,0.2777777777777778,15,0.1111111111111111,6.0,,,8.7088,,,,,,,25
|
||||
job_630124,0.0,0,64,512,4,1.7344,0,0.875,4801.6133,4766.3344,4852.3476,31766,1,3813.5277,0.5898,-0.0032,0.1052,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,-0.0035,0.0,1.0,0.0,0.0164,0.6875,0.5898,15.8889,53.344,552.8224,35.6,857.9675,305.1451,0.0946,4.7486,31.8278,188.1649,220.5231,1031.4949,22.9306,188.6001,783.2847,0,26,59,472,410,26,50,,,0.11864406779661017,7.0,0.3050847457627119,18.0,0.3220338983050847,19,0.23728813559322035,14.0,0.009,12.6671,,,,0.288135593220339,17.0,,,26
|
||||
job_630124,0.0,0,64,512,4,2.1875,0,0.9688,5055.6387,4951.9907,5130.67,31763,1,4942.405,0.4199,0.0045,0.1012,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1162,-0.01,0.0,1.0,0.0,0.0135,0.5469,0.4199,16.046,53.4671,552.5507,35.6,857.9675,305.4168,0.0766,4.8488,33.5052,219.951,253.8289,2130.9697,18.5315,220.2467,1853.7559,0,27,52,416,390,13,6,,,0.28846153846153844,15.0,,,0.2692307692307692,14,0.057692307692307696,3.0,,,,,,0.19230769230769232,10.0,0.019230769230769232,1.0,27
|
||||
job_630124,0.0,0,64,512,4,1.6719,0,0.8281,4799.998,4474.8537,5214.7378,21478,1,3336.6768,0.5605,0.0081,0.0474,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1169,-0.0193,0.0,1.0,0.0,0.0089,0.625,0.5605,15.7877,53.2084,552.8907,35.6,857.9675,305.0768,0.0486,3.4809,28.5303,161.7902,190.6488,998.4337,25.9964,162.0695,778.3023,0,28,44,352,335,11,15,,,0.13636363636363635,6.0,0.022727272727272728,1.0,0.4772727272727273,21,0.45454545454545453,20.0,0.0082,11.9058,,,,0.022727272727272728,1.0,,,28
|
||||
job_630127,0.0,0,64,512,5,0.8125,0,0.2031,5219.4023,5279.0935,5148.4872,24019,1,3593.1869,0.543,0.0174,0.1093,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0059,0.0,1.0,0.0,0.0136,0.661,0.543,9.9914,50.291,594.6118,30.7,857.9675,263.3557,0.2443,3.7877,34.8635,181.2173,216.5936,3907.0864,25.2212,181.4851,3661.4836,0,29,54,432,397,20,35,,,,,0.05555555555555555,3.0,0.48148148148148145,26,0.18518518518518517,10.0,,,,,,,,,,29
|
||||
job_630127,0.0,0,64,512,1,0.875,0,0.875,6119.2617,5371.9608,6614.2273,31787,1,5360.732,0.3984,0.0066,0.11,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1391,-0.0,0.0,1.0,0.0,0.0133,0.5156,0.3984,12.6683,50.5745,589.7981,31.3,857.9675,268.1694,0.0668,4.853,45.3325,249.4957,295.1659,1561.0798,22.8392,249.7659,1238.2155,0,30,51,408,360,18,48,,,0.0784313725490196,4.0,0.0784313725490196,4.0,0.9803921568627451,50,0.19607843137254902,10.0,17.6986,25.6657,6.5413,,,,,,,30
|
||||
job_630127,0.0,0,64,512,2,1.7031,1,1.0,5416.0312,5191.9848,5493.8579,31782,1,5258.8542,0.2578,0.0188,0.1227,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1224,-0.0045,0.0,1.0,0.0,0.0132,0.3906,0.2578,13.0473,50.9899,588.6119,31.4,857.9675,269.3556,0.0915,4.9642,38.9391,242.4843,281.7563,558.6231,24.3077,242.7249,247.5902,0,31,62,496,450,16,38,,,0.06451612903225806,4.0,,,0.5806451612903226,36,0.03225806451612903,2.0,,,,,,,,,,31
|
||||
job_630127,0.0,0,64,512,3,1.8438,0,0.8438,4876.5957,4321.0181,5298.5292,31788,1,4349.0068,0.4316,0.008,0.1024,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1299,-0.0033,0.0,1.0,0.0,0.0147,0.5469,0.4316,13.3226,51.1669,587.694,31.5,857.9675,270.2736,0.0969,4.6902,32.9398,184.8866,218.1917,1881.6998,22.4991,185.1545,1636.313,0,32,61,488,435,20,50,,,0.04918032786885246,3.0,0.04918032786885246,3.0,0.5245901639344263,32,0.26229508196721313,16.0,0.0086,8.4163,,,,,,,,32
|
||||
job_630127,0.0,0,64,512,4,1.9688,0,0.8906,5456.2109,4808.7673,5878.0935,31806,1,4961.1718,0.3945,0.0083,0.1325,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,0.0,0.0,1.0,0.0,0.0126,0.5469,0.3945,13.4299,51.223,587.0558,31.6,857.9675,270.9117,0.067,4.9084,36.655,238.8678,275.8322,1516.0964,23.2669,239.1098,1212.085,0,33,62,496,436,21,55,,,0.08064516129032258,5.0,0.03225806451612903,2.0,0.5806451612903226,36,0.27419354838709675,17.0,,,,,,,,,,33
|
||||
job_630127,0.0,0,64,512,4,2.125,0,0.9844,5502.4434,5212.8122,5708.7692,31765,1,5396.2041,0.416,-0.004,0.0835,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0,0.0,1.0,0.0,0.0137,0.5156,0.416,13.5508,51.3198,587.3987,31.5,857.9675,270.5688,0.0691,5.493,44.4908,256.2057,301.0644,1068.5266,22.5088,256.5041,739.4567,0,34,63,504,462,14,41,,,0.015873015873015872,1.0,0.015873015873015872,1.0,0.3968253968253968,25,0.23809523809523808,15.0,0.0088,8.3216,,,,,,,,34
|
||||
job_630127,0.0,0,64,512,3,1.8594,0,0.9219,5384.9941,5343.019,5414.1821,31664,1,4412.7671,0.4102,0.0025,0.0519,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,-0.002,0.0,1.0,0.0,0.0142,0.5,0.4102,13.7969,51.3292,586.7628,31.6,857.9675,271.2047,0.0906,4.7764,33.1627,211.1102,244.5902,1384.0633,22.1442,211.3362,1112.5444,0,35,61,488,463,8,17,,,0.13114754098360656,8.0,,,0.26229508196721313,16,0.04918032786885246,3.0,,,5.8995,,,,,,,35
|
||||
job_630127,0.0,0,64,512,4,1.9219,0,0.9062,5825.4473,5333.0212,6246.5072,31794,1,4498.7568,0.4609,0.0027,0.1742,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1384,-0.0,0.0,1.0,0.0,0.0171,0.6406,0.4609,13.8943,51.4631,586.2767,31.7,857.9675,271.6909,0.5695,4.8349,37.7488,223.605,262.1992,1464.7075,20.515,223.8804,1177.1535,0,36,64,512,471,15,40,,,0.015625,1.0,,,0.515625,33,0.109375,7.0,0.009,7.9437,,,,,,,,36
|
||||
job_630127,0.0,0,64,512,4,1.8438,0,0.9062,5148.5137,4631.637,5834.55,31764,1,4648.1595,0.5703,-0.0029,0.1133,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,-0.0011,0.0,1.0,0.0,0.0167,0.6875,0.5703,14.5586,51.6496,584.9858,31.8,857.9675,272.9817,0.0919,4.8241,38.277,222.4597,261.0628,1077.7825,26.2516,222.6934,785.6361,0,37,61,488,451,15,35,,,0.03278688524590164,2.0,0.09836065573770492,6.0,0.4262295081967213,26,0.14754098360655737,9.0,,,,,,,,,,37
|
||||
job_630127,0.0,0,64,512,4,2.1875,0,0.9375,5691.4238,5458.8718,5887.1691,31763,1,4985.511,0.457,0.0025,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1239,-0.0,0.0,1.0,0.0,0.0158,0.6094,0.457,14.4333,51.5906,584.4034,31.9,857.9675,273.5641,0.8661,4.8087,37.1183,229.9869,268.1954,1281.4976,24.0319,230.2106,984.4536,0,38,63,504,457,20,44,,,0.047619047619047616,3.0,,,0.5238095238095238,33,0.1111111111111111,7.0,0.0082,8.3888,,,,,,,,38
|
||||
job_630127,0.0,0,64,512,4,1.875,0,0.9219,4852.4375,4700.9213,4963.0034,31721,1,3852.6075,0.4219,-0.0118,0.1369,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0025,0.0,1.0,0.0,0.0151,0.5625,0.4219,14.8096,51.9002,584.34,31.9,857.9675,273.6276,0.1019,4.7148,35.3281,188.2665,223.9309,1220.1543,22.2426,188.5005,969.2585,0,39,63,504,446,22,48,,,0.15873015873015872,10.0,0.031746031746031744,2.0,0.6666666666666666,42,0.09523809523809523,6.0,,,,,,,,,,39
|
||||
job_630127,0.0,0,64,512,4,1.8906,0,0.8438,4717.9707,4944.8443,4511.4142,31770,1,4317.3591,0.4766,0.0072,0.086,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1232,0.0,0.0,1.0,0.0,0.0143,0.5938,0.4766,14.7138,51.7721,583.8667,31.9,857.9675,274.1008,0.0992,4.8622,31.9697,193.9941,226.3482,1608.4681,22.3175,194.2789,1354.9332,0,40,60,480,456,10,24,,,,,0.03333333333333333,2.0,0.36666666666666664,22,0.05,3.0,0.0074,8.3081,6.0239,,,,,,,40
|
||||
job_630127,0.0,0,64,512,4,1.6875,0,0.9531,5037.7695,5312.8784,4764.8016,27138,1,3679.8466,0.498,-0.0001,0.0849,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0122,0.5781,0.498,14.5892,51.6475,583.7137,32.0,857.9675,274.2538,0.0824,4.1919,29.9247,176.4747,206.7121,1037.8229,24.3689,176.7045,802.5427,0,41,60,480,441,19,37,,,0.016666666666666666,1.0,,,0.43333333333333335,26,0.18333333333333332,11.0,,,,,,0.016666666666666666,1.0,,,41
|
||||
job_630127,0.0,0,64,512,4,1.9844,0,0.8906,5756.3848,5835.0311,5708.8025,31761,1,4473.6908,0.377,0.0069,0.1087,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1346,0.0,0.0,1.0,0.0,0.013,0.4531,0.377,14.9636,52.1455,583.6249,32.0,857.9675,274.3426,0.0719,4.8088,36.9135,222.7902,260.0054,1385.1388,21.7016,223.0196,1098.6195,0,42,61,488,424,20,50,,,0.22950819672131148,14.0,0.03278688524590164,2.0,0.5573770491803278,34,0.22950819672131148,14.0,0.0076,7.9965,,,,,,,,42
|
||||
job_630127,0.0,0,64,512,4,1.5781,0,0.8281,4886.5996,4211.4396,5344.823,31794,1,4658.4948,0.4043,0.003,0.0477,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.119,-0.0014,0.0,1.0,0.0,0.0109,0.4844,0.4043,14.9648,52.146,583.4162,32.0,857.9675,274.5513,0.0932,4.8883,35.3553,210.4627,246.1689,1473.9582,23.9821,210.7199,1198.9132,0,43,61,488,441,16,45,,,0.03278688524590164,2.0,0.06557377049180328,4.0,0.5081967213114754,31,0.16393442622950818,10.0,,,,,,,,,,43
|
||||
job_630127,0.0,0,64,512,4,1.9375,0,0.9844,5006.252,5006.7137,5005.7938,31832,1,4198.283,0.498,0.0058,0.1318,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1302,-0.0013,0.0,1.0,0.0,0.0142,0.625,0.498,15.1405,52.1464,582.1508,32.1,857.9675,275.8167,0.0701,4.8376,37.1394,204.9028,242.3402,995.3073,21.6574,205.1302,726.4685,0,44,63,504,464,15,40,,,,,0.015873015873015872,1.0,0.3492063492063492,22,0.2698412698412698,17.0,0.0087,8.5306,,,,,,,,44
|
||||
job_630127,0.0,0,64,512,4,1.9688,0,0.9531,5307.5059,4945.0996,5629.7934,31811,1,4502.6994,0.4707,0.0034,0.0645,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0087,0.5156,0.4707,15.2917,52.3331,581.6784,32.2,857.9675,276.2891,0.0713,4.8688,35.6411,219.1314,255.1244,1030.1018,22.2906,219.4115,747.8149,0,45,57,456,419,11,20,,,0.2982456140350877,17.0,0.03508771929824561,2.0,0.3508771929824561,20,0.07017543859649122,4.0,,,7.1117,,,,,,,45
|
||||
job_630127,0.0,0,64,512,4,2.3281,0,0.9531,5162.7871,4802.6639,5490.6604,31759,1,4252.9044,0.4766,-0.0039,0.0939,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1296,0.0,0.0,1.0,0.0,0.0135,0.5625,0.4766,15.655,52.6444,580.4724,32.3,857.9675,277.4952,0.1001,4.9649,38.1605,200.0621,238.5392,1116.6096,24.6502,200.2781,848.4495,0,46,62,496,452,18,44,,,,,0.016129032258064516,1.0,0.5483870967741935,34,0.14516129032258066,9.0,0.0077,8.5628,,,,,,,,46
|
||||
job_630127,0.0,0,64,512,4,2.1094,0,0.9375,5453.8691,5448.0192,5459.9522,31779,1,4423.535,0.5098,0.01,0.1256,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0129,0.625,0.5098,15.5344,52.5839,580.0443,32.4,857.9675,277.9232,0.0896,4.8374,35.3367,219.7689,255.4572,1436.5679,22.4966,220.0305,1153.7719,0,47,61,488,451,18,36,,,0.01639344262295082,1.0,0.04918032786885246,3.0,0.3770491803278688,23,0.16393442622950818,10.0,,,,,,,,,,47
|
||||
job_630127,0.0,0,64,512,4,1.875,0,0.875,5230.4609,4627.1281,5626.8252,31716,1,4361.9157,0.3965,-0.0025,0.057,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0,0.0,1.0,0.0,0.0113,0.5,0.3965,15.6175,52.7105,579.3495,32.5,857.9675,278.618,0.0831,4.8056,35.7289,205.2303,241.3389,1388.9346,24.246,205.5265,1118.5394,0,48,63,504,448,25,55,,,0.015873015873015872,1.0,0.031746031746031744,2.0,0.6666666666666666,42,0.1746031746031746,11.0,0.0078,8.0292,,,,,,,,48
|
||||
job_630127,0.0,0,64,512,4,1.9688,0,0.9062,5567.3223,3973.2919,6469.1437,31784,1,5829.7172,0.3613,-0.0074,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0168,0.4844,0.3613,15.5988,52.7747,578.7762,32.5,857.9675,279.1913,0.1107,4.8687,41.1551,269.7094,311.286,1674.2094,22.9805,270.0198,1335.0696,0,49,60,480,440,17,39,,,0.016666666666666666,1.0,,,0.5166666666666667,31,0.13333333333333333,8.0,,,,,,,,,,49
|
||||
job_630127,0.0,0,64,512,3,1.6875,0,0.9375,5274.1113,4738.0175,5707.9117,31829,1,4398.882,0.4473,0.008,0.0793,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0132,0.5469,0.4473,15.6793,52.8362,578.3136,32.6,857.9675,279.6539,0.0972,4.7852,34.4386,207.8657,242.6571,1060.2084,23.4244,208.1209,789.3372,0,50,65,520,431,21,76,,,0.18461538461538463,12.0,0.09230769230769231,6.0,0.9538461538461539,62,0.12307692307692308,8.0,0.0081,9.1305,5.9495,,,0.015384615384615385,1.0,,,50
|
||||
job_630127,0.0,0,64,512,4,1.9219,0,0.8906,5243.7461,5174.6447,5286.9619,31743,1,5188.684,0.3848,-0.0033,0.0945,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1207,0.0,0.0,1.0,0.0,0.0967,0.4531,0.3848,15.6188,52.7762,577.5959,32.7,857.9675,280.3716,0.076,4.7495,40.1813,238.4651,278.9729,1686.7408,24.1315,238.7151,1378.8824,0,51,62,496,446,22,49,,,0.016129032258064516,1.0,0.016129032258064516,1.0,0.5645161290322581,35,0.20967741935483872,13.0,,,,,,,,,,51
|
||||
job_630127,0.0,0,64,512,4,1.7812,0,0.8438,5252.4961,5411.8429,5141.6921,31823,1,4683.711,0.4102,0.0062,0.0796,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.133,-0.002,0.0,1.0,0.0,0.0124,0.5312,0.4102,15.7651,52.9658,577.1581,32.7,857.9675,280.8094,0.0944,4.7678,38.5332,214.205,253.0737,1600.439,23.268,214.4456,1319.3232,0,52,61,488,451,14,37,,,,,0.03278688524590164,2.0,0.4918032786885246,30,0.08196721311475409,5.0,0.0082,8.7241,,,,,,,,52
|
||||
job_630127,0.0,0,64,512,3,1.8125,0,0.9531,5107.9043,4684.3886,5450.6078,31689,1,4035.9699,0.4473,-0.0001,0.0667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.132,0.0,0.0,1.0,0.0,0.0132,0.5312,0.4473,15.7054,52.9061,577.4001,32.7,857.9675,280.5674,0.097,4.8708,33.9418,188.7929,223.0849,1082.9166,25.281,189.0458,829.6739,0,53,63,504,454,17,47,,,0.047619047619047616,3.0,0.031746031746031744,2.0,0.6984126984126984,44,0.015873015873015872,1.0,,,,,,,,,,53
|
||||
job_630127,0.0,0,64,512,3,1.7969,0,0.875,5634.0664,5882.4221,5476.1661,31801,1,5198.4542,0.3887,0.0101,0.0932,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0144,0.5156,0.3887,15.8018,52.9791,576.4988,32.8,857.9675,281.4687,0.0964,4.9065,40.1938,233.8941,274.5002,1827.1928,24.3101,234.2096,1523.4687,0,54,61,488,434,19,52,,,0.03278688524590164,2.0,0.08196721311475409,5.0,0.6557377049180327,40,0.13114754098360656,8.0,0.0084,8.6128,,,,,,,,54
|
||||
job_630127,0.0,0,64,512,4,1.9375,0,0.9531,5371.4082,5022.761,5604.2182,31785,1,4782.3049,0.4004,-0.0082,0.1177,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1222,-0.0007,0.0,1.0,0.0,0.0128,0.5156,0.4004,15.8948,53.0398,576.6206,32.8,857.9675,281.3469,0.0653,4.8129,41.537,228.2535,270.096,1483.9509,23.783,228.4932,1185.2547,0,55,62,496,429,19,64,,,0.04838709677419355,3.0,0.04838709677419355,3.0,0.9032258064516129,56,0.0967741935483871,6.0,,,6.0259,,,,,,,55
|
||||
job_630127,0.0,0,64,512,4,1.6094,0,0.9062,4940.75,4904.8341,4965.9269,31836,1,4570.0348,0.4121,-0.0024,0.1498,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.124,-0.0,0.0,1.0,0.0,0.0148,0.5156,0.4121,16.2064,53.3552,575.7253,32.9,857.9675,282.2422,0.0989,4.7645,34.0253,209.3135,243.7093,1375.7686,22.7461,209.5845,1104.5439,0,56,59,472,390,33,69,,,0.22033898305084745,13.0,0.3728813559322034,22.0,0.5084745762711864,30,0.288135593220339,17.0,0.0075,15.5646,,,,,,,,56
|
||||
job_630130,0.0,0,64,512,4,0.3438,0,0.1406,2745.916,4536.7524,2283.9066,31842,1,3652.8031,0.2051,0.0705,0.1449,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0683,-0.0442,0.0,1.0,0.0,0.0137,0.3898,0.2051,12.7084,50.5586,566.71,33.9,857.9675,291.2575,0.1351,4.4486,104.946,152.1059,257.396,10105.4336,18.2174,152.3142,9825.3711,0,57,63,504,281,58,154,,,0.8095238095238095,51.0,1.8253968253968254,115.0,0.42857142857142855,27,0.19047619047619047,12.0,,,,,,0.25396825396825395,16.0,0.031746031746031744,2.0,57
|
||||
job_630130,0.0,0,64,512,1,0.9844,0,0.9844,2990.8008,4339.4828,2595.7323,31794,1,4262.9787,0.2266,0.0705,0.1484,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,-0.0481,0.0,1.0,0.0,0.0181,0.5,0.2266,12.7156,50.6524,565.6756,34.1,857.9675,292.2919,0.0685,4.3634,31.3815,172.8736,204.58,3407.9472,17.5583,173.1294,3181.4385,0,58,59,472,191,58,157,,,1.9491525423728813,115.0,2.3389830508474576,138.0,0.3050847457627119,18,0.22033898305084745,13.0,113.7448,123.1255,,,,0.2711864406779661,16.0,,,58
|
||||
job_630130,0.0,0,64,512,2,1.4531,0,0.9219,1842.3164,3830.7287,1172.5901,31730,1,3083.3057,0.252,0.1959,0.2355,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0529,-0.0935,0.0,1.0,0.0,0.0296,0.5,0.252,12.8444,50.7361,566.7198,33.9,857.9675,291.2477,0.0904,4.2018,17.8872,126.8601,145.0002,3636.1329,18.2801,127.022,3468.6454,0,59,59,472,191,58,160,,,1.8305084745762712,108.0,2.559322033898305,151.0,0.3728813559322034,22,0.23728813559322035,14.0,,,,,,0.3728813559322034,22.0,,,59
|
||||
job_630130,0.0,0,64,512,3,1.5469,0,0.9531,1687.9023,3823.3853,1110.3151,19335,1,2744.5105,0.2129,0.1122,0.1362,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0511,-0.0532,0.0,1.0,0.0,0.0138,0.4531,0.2129,12.8585,50.7328,566.4686,34.0,857.9675,291.4989,0.0597,2.7178,15.1498,108.5325,123.9181,3855.0493,17.9434,108.7081,3710.4636,0,60,63,504,312,48,79,,,1.746031746031746,110.0,0.7936507936507936,50.0,0.31746031746031744,20,0.14285714285714285,9.0,0.0099,8.8255,275.4291,,,0.047619047619047616,3.0,,,60
|
||||
job_630130,0.0,0,64,512,3,1.4844,0,0.875,2705.2773,3926.8809,1668.8989,31646,1,3012.6819,0.459,0.1462,0.1798,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0697,-0.0959,0.0,1.0,0.0,0.0179,0.7188,0.459,13.0881,50.9166,566.6464,34.0,857.9675,291.3212,0.0697,4.435,19.2157,123.2323,142.7038,1882.4268,20.2012,123.418,1715.083,0,61,62,496,397,29,58,,,0.6290322580645161,39.0,0.3387096774193548,21.0,0.4032258064516129,25,0.1935483870967742,12.0,,,,0.016129032258064516,1.0,0.03225806451612903,2.0,,,61
|
||||
job_630130,0.0,0,64,512,3,1.5625,0,0.875,4372.2871,4678.6603,4051.208,31765,1,4031.5953,0.5117,0.0301,0.1117,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1006,-0.022,0.0,1.0,0.0,0.0107,0.6094,0.5117,13.2397,51.094,568.5527,33.7,857.9675,289.4149,0.0919,4.5973,32.0721,189.1022,221.4727,1318.3369,21.5119,189.3083,1070.747,0,62,62,496,467,17,21,,,0.11290322580645161,7.0,0.03225806451612903,2.0,0.24193548387096775,15,0.11290322580645161,7.0,0.0085,8.9973,,0.016129032258064516,1.0,0.016129032258064516,1.0,,,62
|
||||
job_630130,0.0,0,64,512,4,1.8906,0,0.9688,5146.4805,4434.7366,5789.4312,31600,1,4146.9309,0.4746,0.0104,0.0669,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1238,-0.0056,0.0,1.0,0.0,0.0102,0.5469,0.4746,13.2111,51.0682,568.2429,33.8,857.9675,289.7247,0.0709,4.7406,35.0463,194.8197,230.1324,1359.0594,21.4195,195.0147,1102.761,0,63,63,504,448,20,51,,,0.07936507936507936,5.0,0.07936507936507936,5.0,0.746031746031746,47,0.09523809523809523,6.0,,,,,,,,,,63
|
||||
job_630130,0.0,0,64,512,7,1.8906,0,0.875,5587.1855,5587.4615,5587.0158,31766,1,4900.2073,0.3809,0.0053,0.1109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1128,0.0,0.0,1.0,0.0,0.0114,0.4844,0.3809,13.5775,51.5483,569.7428,33.6,857.9675,288.2247,0.1075,4.8015,38.5633,225.6816,264.5674,1636.5882,22.1274,225.8961,1345.0836,0,64,62,496,422,26,68,,,0.0967741935483871,6.0,0.04838709677419355,3.0,1.0,62,0.04838709677419355,3.0,0.0081,8.5496,,0.06451612903225806,4.0,,,,,64
|
||||
job_630130,0.0,0,64,512,4,1.9531,0,0.9375,4798.2617,4883.6444,4723.5128,31812,1,4722.6084,0.4668,0.0183,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,0.0,0.0,1.0,0.0,0.0168,0.6094,0.4668,13.718,51.5807,568.653,33.7,857.9675,289.3145,0.075,4.7303,35.8252,213.9135,250.0471,1536.8344,21.852,214.1464,1260.1995,0,65,61,488,434,20,44,,,0.16393442622950818,10.0,0.06557377049180328,4.0,0.7213114754098361,44,0.04918032786885246,3.0,,,5.9602,,,,,,,65
|
||||
job_630130,0.0,0,64,512,3,1.6875,0,0.9062,4990.3906,4683.8621,5191.767,31741,1,4339.1846,0.3965,0.0113,0.1192,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1176,-0.0005,0.0,1.0,0.0,0.0138,0.5781,0.3965,13.9927,51.9113,567.4896,33.9,857.9675,290.4779,0.0707,4.8411,34.6825,205.3311,240.3191,1507.0204,22.8738,205.5654,1238.9803,0,66,64,512,451,19,50,,,0.171875,11.0,0.03125,2.0,0.75,48,0.046875,3.0,0.0085,8.6432,,,,,,,,66
|
||||
job_630130,0.0,0,64,512,3,1.5938,0,0.8906,4687.6602,4780.5265,4614.2762,31720,1,3991.8123,0.4414,0.0147,0.1094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1043,-0.0031,0.0,1.0,0.0,0.0126,0.5625,0.4414,14.1397,52.1146,567.5696,33.8,857.9675,290.3979,0.0726,4.6609,30.5811,185.1371,216.0168,1571.263,21.9126,185.3626,1328.6676,0,67,58,464,422,16,42,,,,,0.034482758620689655,2.0,0.6896551724137931,40,0.08620689655172414,5.0,,,,,,,,,,67
|
||||
job_630130,0.0,0,64,512,4,1.5469,0,0.8281,4902.5645,4448.6776,5228.5101,31770,1,4445.7836,0.418,-0.0081,0.0909,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1147,-0.0,0.0,1.0,0.0,0.0128,0.5312,0.418,14.1629,52.2518,569.6768,33.6,857.9675,288.2908,0.0962,4.7131,33.8447,209.0269,243.2635,1442.218,21.5536,209.322,1172.6792,0,68,64,512,471,16,37,,,0.046875,3.0,0.046875,3.0,0.46875,30,0.0625,4.0,0.0082,8.6812,,,,0.015625,1.0,,,68
|
||||
job_630130,0.0,0,64,512,4,1.7344,0,0.875,4912.1777,4638.6631,5239.691,31749,1,4027.0232,0.5449,0.0247,0.126,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1083,-0.002,0.0,1.0,0.0,0.0111,0.6562,0.5449,14.5627,52.6307,568.1539,33.8,857.9675,289.8137,0.0922,4.7384,33.01,189.5539,222.8763,1413.2169,23.2675,189.7737,1162.3272,0,69,62,496,442,18,54,,,,,0.03225806451612903,2.0,0.8387096774193549,52,,,,,,,,,,,,69
|
||||
job_630130,0.0,0,64,512,3,1.9219,0,0.9062,4989.8223,4749.7429,5156.7649,31793,1,4613.5261,0.4102,-0.0096,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1134,-0.0,0.0,1.0,0.0,0.0151,0.5312,0.4102,14.5696,52.6356,567.6034,33.8,857.9675,290.3641,0.0859,4.6794,34.9319,213.77,248.9891,1779.8183,22.218,213.9708,1503.9267,0,70,62,496,433,16,61,,,0.03225806451612903,2.0,0.12903225806451613,8.0,0.8387096774193549,52,0.03225806451612903,2.0,0.0081,8.4344,5.9241,,,,,,,70
|
||||
job_661735,0.0,0,64,512,3,0.3281,0,0.2188,4521.9473,4333.5193,4758.5198,31663,1,3365.2785,0.5566,0.0074,0.094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0,0.0,1.0,0.0,0.0131,0.5965,0.5566,12.5978,50.5919,579.345,32.5,857.9676,278.6226,0.1569,4.5133,32.1738,167.8055,200.36,3675.9345,21.9439,168.0287,3449.1169,0,71,58,464,375,29,88,,,0.017241379310344827,1.0,,,1.4137931034482758,82,0.20689655172413793,12.0,,,,,,,,,,71
|
||||
job_661735,0.0,0,64,512,1,0.9375,0,0.9375,5361.2598,5767.993,5205.1622,31699,1,5207.4533,0.2773,-0.013,0.1323,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1151,0.0,0.0,1.0,0.0,0.017,0.4219,0.2773,12.8662,50.7994,576.8029,32.8,857.9676,281.1647,0.0648,4.6437,37.3367,229.2593,266.8776,1642.3033,25.4393,229.4755,1345.3366,0,72,56,448,364,24,80,,,0.07142857142857142,4.0,0.16071428571428573,9.0,1.3392857142857142,75,0.16071428571428573,9.0,11.424,21.2849,,0.05357142857142857,3.0,,,,,72
|
||||
job_661735,0.0,0,64,512,2,1.6406,1,1.0,4303.7402,4841.9941,4036.1871,31778,1,4623.5614,0.332,0.0023,0.0825,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0977,0.0,0.0,1.0,0.0,0.0095,0.4062,0.332,13.201,51.0858,576.4416,32.8,857.9676,281.526,0.0649,4.6475,32.7604,195.3408,228.4104,875.421,23.4598,195.5846,618.8995,0,73,63,504,471,12,33,,,,,0.047619047619047616,3.0,0.47619047619047616,30,0.07936507936507936,5.0,,,,,,,,,,73
|
||||
job_661735,0.0,0,64,512,3,1.6406,0,0.8906,5242.7773,4926.2055,5552.0154,31761,1,4678.9952,0.4941,0.0138,0.1243,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,-0.0,0.0,1.0,0.0,0.0145,0.5938,0.4941,13.4431,51.3293,575.8802,32.9,857.9676,282.0874,0.062,4.6498,37.0456,231.3918,268.7201,1733.9771,23.6881,231.6122,1436.9161,0,74,59,472,401,23,56,,,0.2542372881355932,15.0,0.03389830508474576,2.0,0.847457627118644,50,0.1864406779661017,11.0,0.007,10.013,,0.05084745762711865,3.0,,,,,74
|
||||
job_661735,0.0,0,64,512,4,1.9844,1,1.0,4678.0137,4390.89,4862.0673,31720,1,4446.3943,0.3906,-0.0162,0.1412,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1014,-0.002,0.0,1.0,0.0,0.0143,0.5625,0.3906,13.5915,51.5202,575.6154,32.9,857.9676,282.3522,0.0636,4.5241,34.8828,208.4871,243.6766,1567.8023,22.6625,208.7299,1296.9223,0,75,61,488,422,25,66,,,,,0.06557377049180328,4.0,0.7704918032786885,47,0.21311475409836064,13.0,,,6.424,0.09836065573770492,6.0,,,,,75
|
||||
job_661735,0.0,0,64,512,4,2.4219,2,1.0,4572.6895,4582.679,4568.0657,31799,1,4302.6602,0.3164,-0.0023,0.112,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.103,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3164,13.6858,51.5748,574.5665,33.0,857.9676,283.4011,0.0663,4.6913,30.5968,200.2044,231.0996,1415.5476,24.2091,200.4362,1155.5266,0,76,61,488,415,22,64,,,0.14754098360655737,9.0,0.04918032786885246,3.0,0.7868852459016393,48,0.21311475409836064,13.0,0.0074,9.8798,,0.11475409836065574,7.0,,,,,76
|
||||
job_661735,0.0,0,64,512,4,3.0312,3,1.0,4597.7812,4086.6667,4944.6689,31759,1,4559.7375,0.4043,0.0013,0.0431,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0033,0.0,1.0,0.0,0.0068,0.4688,0.4043,14.3994,51.5634,573.7055,33.1,857.9676,284.2621,0.0642,4.5039,32.7244,196.2376,229.2847,1161.9454,21.4738,196.4958,906.6629,0,77,63,504,414,32,86,,,0.15873015873015872,10.0,,,0.8095238095238095,51,0.4603174603174603,29.0,,,,0.09523809523809523,6.0,0.015873015873015872,1.0,,,77
|
||||
job_661735,0.0,0,64,512,4,4.0,4,1.0,5291.9141,6195.0086,5027.3712,28898,1,4117.8095,0.2266,0.0072,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,0.0,0.0,1.0,0.0,0.0105,0.375,0.2266,14.2961,51.5109,573.1121,33.2,857.9676,284.8555,0.1485,4.4428,33.4904,188.476,222.3217,1837.7946,17.1266,188.6825,1593.8676,0,78,67,536,466,24,64,,,0.08955223880597014,6.0,0.029850746268656716,2.0,0.6567164179104478,44,0.23880597014925373,16.0,0.0588,10.9374,,0.029850746268656716,2.0,,,,,78
|
||||
|
3
training_logs/20260612_220124_reward_vs_steps.png
Normal file
3
training_logs/20260612_220124_reward_vs_steps.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b2f2c6fb565b5ad169b4da77e123d97bbf845a3e7025f2f6e114a19288fda4c6
|
||||
size 270221
|
||||
38124
training_logs/20260612_220124_trial_results.csv
Normal file
38124
training_logs/20260612_220124_trial_results.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b6c9df040dbcaa53e1db3173739c6a1463d1f0d798a4ec3ad56d41931760a7cc
|
||||
size 111916
|
||||
8507
training_logs/20260612_220124_vllm_metrics_job_630124.csv
Normal file
8507
training_logs/20260612_220124_vllm_metrics_job_630124.csv
Normal file
File diff suppressed because it is too large
Load Diff
8517
training_logs/20260612_220124_vllm_metrics_job_630127.csv
Normal file
8517
training_logs/20260612_220124_vllm_metrics_job_630127.csv
Normal file
File diff suppressed because it is too large
Load Diff
7403
training_logs/20260612_220124_vllm_metrics_job_630130.csv
Normal file
7403
training_logs/20260612_220124_vllm_metrics_job_630130.csv
Normal file
File diff suppressed because it is too large
Load Diff
1153
training_logs/20260612_220124_vllm_metrics_job_661734.csv
Normal file
1153
training_logs/20260612_220124_vllm_metrics_job_661734.csv
Normal file
File diff suppressed because it is too large
Load Diff
3110
training_logs/20260612_220124_vllm_metrics_job_661735.csv
Normal file
3110
training_logs/20260612_220124_vllm_metrics_job_661735.csv
Normal file
File diff suppressed because it is too large
Load Diff
28686
training_logs/20260612_220124_vllm_metrics_table.csv
Normal file
28686
training_logs/20260612_220124_vllm_metrics_table.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:57544589430cce1ff125aad5db8ab73a85bdbdec7dcaa61c6c4e48114e44a561
|
||||
size 44390800
|
||||
1552
training_logs/ablation-pymethods2test-seqmean-arm0_630125.out
Normal file
1552
training_logs/ablation-pymethods2test-seqmean-arm0_630125.out
Normal file
File diff suppressed because one or more lines are too long
1665
training_logs/ablation-pymethods2test-seqmean-arm0_630126.out
Normal file
1665
training_logs/ablation-pymethods2test-seqmean-arm0_630126.out
Normal file
File diff suppressed because one or more lines are too long
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:acba1fb03567f31a572dea981ac325fe2b4df0ecb5e9b3c49722f70599c5311e
|
||||
size 42935463
|
||||
370
training_logs/ablation-pymethods2test-seqmean-arm0_630128.out
Normal file
370
training_logs/ablation-pymethods2test-seqmean-arm0_630128.out
Normal file
@@ -0,0 +1,370 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_630128
|
||||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630128
|
||||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630128
|
||||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||||
[proxy] Tunnel port: 7003
|
||||
[proxy] Node IP: 10.128.32.33 (workers will connect here)
|
||||
[proxy] ✓ SSH tunnel started successfully
|
||||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
|
||||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||||
[proxy] ✓ Daytona timeout settings configured
|
||||
[proxy] Testing proxy connectivity...
|
||||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
|
||||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||||
[proxy] ⚠ Tunnel not accessible at 10.128.32.33:7003 (workers may fail)
|
||||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||||
=== Universal RL Training Runner ===
|
||||
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
|
||||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python version: Python 3.12.12
|
||||
UV_USE_IO_URING: 0
|
||||
Proxy: DISABLED (direct internet or not configured)
|
||||
========================================
|
||||
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
|
||||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
|
||||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
|
||||
HF_TOKEN=****pDbg
|
||||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||||
Environment configured:
|
||||
TENSOR_PARALLEL_SIZE=1
|
||||
NUM_INFERENCE_ENGINES=56
|
||||
POLICY_NUM_NODES=14
|
||||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
|
||||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||||
Cleaning up existing Ray instances...
|
||||
=== Starting Ray Cluster ===
|
||||
Nodes: 14
|
||||
GPUs per node: 4
|
||||
CPUs per node: 288
|
||||
Head node: jpbo-041-33 (10.128.32.33)
|
||||
Ray port: 6379
|
||||
============================
|
||||
Starting Ray head on jpbo-041-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-33.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray head on jpbo-041-33
|
||||
Starting Ray worker on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-34.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 1 on jpbo-041-34
|
||||
Starting Ray worker on jpbo-041-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-35.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 2 on jpbo-041-35
|
||||
Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 3 on jpbo-041-36
|
||||
Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 4 on jpbo-041-37
|
||||
Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 5 on jpbo-041-38
|
||||
Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 6 on jpbo-041-39
|
||||
Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 7 on jpbo-041-40
|
||||
Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 8 on jpbo-041-41
|
||||
Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 9 on jpbo-041-42
|
||||
Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 10 on jpbo-041-43
|
||||
Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 11 on jpbo-041-44
|
||||
Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 12 on jpbo-041-45
|
||||
Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 13 on jpbo-041-46
|
||||
Waiting for cluster (56 GPUs, 14 nodes)...
|
||||
Connecting to Ray at 10.128.32.33:6379 (expecting 14 nodes, 56.0 GPUs)
|
||||
Ray connection established, polling for resources...
|
||||
[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
||||
[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
|
||||
[Ray wait] nodes=3/14 GPUs=12.0/56.0 resources={'GPU': 12.0, 'CPU': 864.0, 'accelerator_type:GH200': 3.0, 'memory': 2273135755264.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 128849018880.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
||||
[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'CPU': 1152.0, 'accelerator_type:GH200': 4.0, 'memory': 3033402310656.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
||||
[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'memory': 3033402310656.0, 'accelerator_type:GH200': 4.0, 'CPU': 1152.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
|
||||
[Ray wait] nodes=5/14 GPUs=20.0/56.0 resources={'GPU': 20.0, 'memory': 3809847214080.0, 'accelerator_type:GH200': 5.0, 'CPU': 1440.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 214748364800.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
||||
[Ray wait] nodes=6/14 GPUs=24.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 4562133647360.0, 'accelerator_type:GH200': 6.0, 'CPU': 1728.0, 'GPU': 24.0, 'object_store_memory': 257698037760.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
||||
[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 6067020955648.0, 'accelerator_type:GH200': 8.0, 'CPU': 2304.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
||||
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
|
||||
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
|
||||
2026-06-07 14:35:10,656 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.33:6379...
|
||||
2026-06-07 14:35:10,666 INFO worker.py:2007 -- Connected to Ray cluster.
|
||||
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||||
warnings.warn(
|
||||
[33m(raylet, ip=10.128.32.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.43)[0m [2026-06-07 14:35:49,687 E 2691736 2691736] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[36m(pid=352159)[0m [2026-06-07 14:35:54,163 E 352159 352769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[36m(pid=352184)[0m [2026-06-07 14:36:00,280 E 352184 353748] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
||||
[33m(raylet, ip=10.128.32.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[2026-06-07 14:36:04,817 E 352114 352155] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.44)[0m [2026-06-07 14:36:16,668 E 3005389 3005389] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[36m(pid=352328)[0m [2026-06-07 14:36:04,815 E 352328 368299] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
||||
[33m(raylet, ip=10.128.32.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.42)[0m [2026-06-07 14:36:31,723 E 3108436 3108436] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.34)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.46)[0m [2026-06-07 14:36:47,990 E 2656361 2656361] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.40)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 3x across cluster][0m
|
||||
[33m(raylet, ip=10.128.32.34)[0m [2026-06-07 14:37:02,131 E 2733873 2733873] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.40)[0m [2026-06-07 14:37:06,688 E 2571074 2571074] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.36)[0m [2026-06-07 14:37:33,965 E 2625779 2625779] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.38)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.32.45)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 3x across cluster][0m
|
||||
[33m(raylet, ip=10.128.32.38)[0m [2026-06-07 14:38:06,213 E 2699635 2699635] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.32.45)[0m [2026-06-07 14:38:14,810 E 2825651 2825651] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
||||
main()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
||||
wait_for_cluster(
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
||||
raise TimeoutError(
|
||||
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=10, gpus=40.0)
|
||||
srun: error: jpbo-041-33: task 0: Exited with exit code 1
|
||||
|
||||
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
|
||||
result = subprocess.run(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
||||
raise CalledProcessError(retcode, process.args,
|
||||
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1285, in run
|
||||
training_exit_code = self._run_with_ray()
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1532, in _run_with_ray
|
||||
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
|
||||
self.start()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
|
||||
self._wait_for_cluster()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
|
||||
raise RuntimeError(
|
||||
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
||||
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
|
||||
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||||
repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
|
||||
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
|
||||
episodes: last
|
||||
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
|
||||
[RLJobRunner] Waiting for trace upload to complete...
|
||||
[RLJobRunner] Trace upload failed with exit code 1.
|
||||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
|
||||
Collecting Ray logs from worker jpbo-041-34...
|
||||
Collecting Ray logs from worker jpbo-041-35...
|
||||
Collecting Ray logs from worker jpbo-041-36...
|
||||
Collecting Ray logs from worker jpbo-041-37...
|
||||
Collecting Ray logs from worker jpbo-041-38...
|
||||
Collecting Ray logs from worker jpbo-041-39...
|
||||
Collecting Ray logs from worker jpbo-041-40...
|
||||
Collecting Ray logs from worker jpbo-041-41...
|
||||
Collecting Ray logs from worker jpbo-041-42...
|
||||
Collecting Ray logs from worker jpbo-041-43...
|
||||
Collecting Ray logs from worker jpbo-041-44...
|
||||
Collecting Ray logs from worker jpbo-041-45...
|
||||
Collecting Ray logs from worker jpbo-041-46...
|
||||
Ray log preservation complete
|
||||
2054
training_logs/ablation-pymethods2test-seqmean-arm0_630129.out
Normal file
2054
training_logs/ablation-pymethods2test-seqmean-arm0_630129.out
Normal file
File diff suppressed because one or more lines are too long
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ccadea7fa55f83615cb7841c06afa1a0bae0a0f8f21a8e874cf3b36a35660724
|
||||
size 23994623
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job657667/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_657667
|
||||
/var/spool/slurmd/job657667/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job657668/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_657668
|
||||
/var/spool/slurmd/job657668/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job657669/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_657669
|
||||
/var/spool/slurmd/job657669/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job657670/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_657670
|
||||
/var/spool/slurmd/job657670/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
@@ -0,0 +1,17 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
||||
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
||||
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
||||
Python executable:
|
||||
/var/spool/slurmd/job657671/slurm_script: line 128: python: command not found
|
||||
Python path check:
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_657671
|
||||
/var/spool/slurmd/job657671/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
||||
364
training_logs/ablation-pymethods2test-seqmean-arm0_661733.out
Normal file
364
training_logs/ablation-pymethods2test-seqmean-arm0_661733.out
Normal file
@@ -0,0 +1,364 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_661733
|
||||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_661733
|
||||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_661733
|
||||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||||
[proxy] Tunnel port: 7003
|
||||
[proxy] Node IP: 10.128.16.35 (workers will connect here)
|
||||
[proxy] ✓ SSH tunnel started successfully
|
||||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
|
||||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||||
[proxy] ✓ Daytona timeout settings configured
|
||||
[proxy] Testing proxy connectivity...
|
||||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
|
||||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||||
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
|
||||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||||
=== Universal RL Training Runner ===
|
||||
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
|
||||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python version: Python 3.12.12
|
||||
UV_USE_IO_URING: 0
|
||||
Proxy: DISABLED (direct internet or not configured)
|
||||
========================================
|
||||
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
|
||||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
|
||||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
|
||||
HF_TOKEN=****pDbg
|
||||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||||
Environment configured:
|
||||
TENSOR_PARALLEL_SIZE=1
|
||||
NUM_INFERENCE_ENGINES=56
|
||||
POLICY_NUM_NODES=14
|
||||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
|
||||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||||
Cleaning up existing Ray instances...
|
||||
=== Starting Ray Cluster ===
|
||||
Nodes: 14
|
||||
GPUs per node: 4
|
||||
CPUs per node: 288
|
||||
Head node: jpbo-001-35 (10.128.16.35)
|
||||
Ray port: 6379
|
||||
============================
|
||||
Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray head on jpbo-001-35
|
||||
Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 1 on jpbo-001-36
|
||||
Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 2 on jpbo-001-37
|
||||
Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 3 on jpbo-001-38
|
||||
Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 4 on jpbo-001-41
|
||||
Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 5 on jpbo-001-42
|
||||
Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 6 on jpbo-001-43
|
||||
Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 7 on jpbo-001-44
|
||||
Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 8 on jpbo-001-45
|
||||
Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 9 on jpbo-001-46
|
||||
Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 10 on jpbo-001-47
|
||||
Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 11 on jpbo-001-48
|
||||
Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 12 on jpbo-003-37
|
||||
Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
|
||||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||
Started Ray worker 13 on jpbo-003-39
|
||||
Waiting for cluster (56 GPUs, 14 nodes)...
|
||||
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
|
||||
Ray connection established, polling for resources...
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 40.0, 'memory': 7408839753728.0, 'accelerator_type:GH200': 10.0, 'object_store_memory': 429496729600.0, 'CPU': 2880.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=11/14 GPUs=44.0/56.0 resources={'GPU': 44.0, 'memory': 8175128150016.0, 'accelerator_type:GH200': 11.0, 'object_store_memory': 472446402560.0, 'node:10.128.16.135': 1.0, 'CPU': 3168.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
||||
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
|
||||
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
|
||||
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
|
||||
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
|
||||
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
|
||||
2026-06-08 13:34:49,641 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
|
||||
2026-06-08 13:34:49,647 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
||||
2026-06-08 13:34:49,649 INFO worker.py:2007 -- Connected to Ray cluster.
|
||||
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||||
warnings.warn(
|
||||
[36m(pid=1699456)[0m [2026-06-08 13:35:34,739 E 1699456 1700253] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[36m(pid=1699477)[0m [2026-06-08 13:35:42,369 E 1699477 1701156] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
||||
[2026-06-08 13:35:45,201 E 1699412 1699453] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.16.47)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[36m(pid=1699674)[0m [2026-06-08 13:35:45,144 E 1699674 1715593] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
||||
[33m(raylet, ip=10.128.16.133)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 5x across cluster][0m
|
||||
[33m(raylet, ip=10.128.16.47)[0m [2026-06-08 13:37:33,645 E 1598284 1598284] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.16.135)[0m [2026-06-08 13:37:44,343 E 3199851 3199851] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.16.133)[0m [2026-06-08 13:37:47,415 E 2150584 2150584] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||
[33m(raylet, ip=10.128.16.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[33m(raylet, ip=10.128.16.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
||||
main()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
||||
wait_for_cluster(
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
||||
raise TimeoutError(
|
||||
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0)
|
||||
srun: error: jpbo-001-35: task 0: Exited with exit code 1
|
||||
|
||||
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
|
||||
result = subprocess.run(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
||||
raise CalledProcessError(retcode, process.args,
|
||||
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1291, in run
|
||||
training_exit_code = self._run_with_ray()
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1538, in _run_with_ray
|
||||
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
|
||||
self.start()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
|
||||
self._wait_for_cluster()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
|
||||
raise RuntimeError(
|
||||
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
||||
Collecting Ray logs from worker jpbo-001-36...
|
||||
Collecting Ray logs from worker jpbo-001-37...
|
||||
Collecting Ray logs from worker jpbo-001-38...
|
||||
Collecting Ray logs from worker jpbo-001-41...
|
||||
Collecting Ray logs from worker jpbo-001-42...
|
||||
Collecting Ray logs from worker jpbo-001-43...
|
||||
Collecting Ray logs from worker jpbo-001-44...
|
||||
Collecting Ray logs from worker jpbo-001-45...
|
||||
Collecting Ray logs from worker jpbo-001-46...
|
||||
Collecting Ray logs from worker jpbo-001-47...
|
||||
Collecting Ray logs from worker jpbo-001-48...
|
||||
Collecting Ray logs from worker jpbo-003-37...
|
||||
Collecting Ray logs from worker jpbo-003-39...
|
||||
[RLJobRunner] Crash-time Ray log preservation complete.
|
||||
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/trace_jobs, skipping upload.
|
||||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ray_logs/
|
||||
Collecting Ray logs from worker jpbo-001-36...
|
||||
Collecting Ray logs from worker jpbo-001-37...
|
||||
Collecting Ray logs from worker jpbo-001-38...
|
||||
Collecting Ray logs from worker jpbo-001-41...
|
||||
Collecting Ray logs from worker jpbo-001-42...
|
||||
Collecting Ray logs from worker jpbo-001-43...
|
||||
Collecting Ray logs from worker jpbo-001-44...
|
||||
Collecting Ray logs from worker jpbo-001-45...
|
||||
Collecting Ray logs from worker jpbo-001-46...
|
||||
Collecting Ray logs from worker jpbo-001-47...
|
||||
Collecting Ray logs from worker jpbo-001-48...
|
||||
Collecting Ray logs from worker jpbo-003-37...
|
||||
Collecting Ray logs from worker jpbo-003-39...
|
||||
Ray log preservation complete
|
||||
37567
training_logs/ablation-pymethods2test-seqmean-arm0_661734.out
Normal file
37567
training_logs/ablation-pymethods2test-seqmean-arm0_661734.out
Normal file
File diff suppressed because one or more lines are too long
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ffc1dd737ba8b66df4987f88b45897fb347a2b159e9d578fd3f62ac19cd5a2f6
|
||||
size 14896573
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user