初始化项目，由ModelHub XC社区提供模型
Model: laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B Source: Original Platform
This commit is contained in:
ModelHub XC
42
.gitattributes
vendored
Normal file
42
.gitattributes
vendored
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/20260609_214045_reward_vs_steps.png filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/20260609_214045_turn_count_distribution.png filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653585.out filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653586.out filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653587.out filter=lfs diff=lfs merge=lfs -text
|
||||||
|
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653588.out filter=lfs diff=lfs merge=lfs -text
|
||||||
29
README.md
Normal file
29
README.md
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
base_model: laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink
|
||||||
|
tags:
|
||||||
|
- rl
|
||||||
|
- rloo
|
||||||
|
- terminal-bench
|
||||||
|
- openthoughts-agent
|
||||||
|
---
|
||||||
|
|
||||||
|
# ablation-pymethods2test-seqmean-arm0-tis (step 15)
|
||||||
|
|
||||||
|
RLOO length-bias A/B ablation, **arm0 = sequence_mean loss reduction WITH TIS**
|
||||||
|
(truncated importance sampling). Trained with SkyRL (RLOO-n, FSDP2) on
|
||||||
|
`DCAgent/exp_rpt_pymethods2test-large` from the Qwen3-8B pre-RL base
|
||||||
|
`laion/GLM-4_7-swesmith-...-fixthink`.
|
||||||
|
|
||||||
|
Checkpoint selected at **global_step 15** by trailing-5 EMA (alpha=1/3) of
|
||||||
|
`reward/avg_raw_reward` over the full training chain (cap step <= 80).
|
||||||
|
|
||||||
|
## Training Traces
|
||||||
|
|
||||||
|
Training-time Daytona/Harbor rollouts for this run are uploaded as
|
||||||
|
a companion dataset:
|
||||||
|
**[penfever/ablation-pymethods2test-seqmean-arm0-tis](https://huggingface.co/datasets/penfever/ablation-pymethods2test-seqmean-arm0-tis)**
|
||||||
|
|
||||||
|
The dataset contains the `last` episode of each trial (per
|
||||||
|
`make_and_upload_trace_dataset --episodes last`) — the same rollouts
|
||||||
|
the policy was trained on after rollback / truncation.
|
||||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"</think>": 151668,
|
||||||
|
"</tool_call>": 151658,
|
||||||
|
"</tool_response>": 151666,
|
||||||
|
"<think>": 151667,
|
||||||
|
"<tool_call>": 151657,
|
||||||
|
"<tool_response>": 151665,
|
||||||
|
"<|box_end|>": 151649,
|
||||||
|
"<|box_start|>": 151648,
|
||||||
|
"<|endoftext|>": 151643,
|
||||||
|
"<|file_sep|>": 151664,
|
||||||
|
"<|fim_middle|>": 151660,
|
||||||
|
"<|fim_pad|>": 151662,
|
||||||
|
"<|fim_prefix|>": 151659,
|
||||||
|
"<|fim_suffix|>": 151661,
|
||||||
|
"<|im_end|>": 151645,
|
||||||
|
"<|im_start|>": 151644,
|
||||||
|
"<|image_pad|>": 151655,
|
||||||
|
"<|object_ref_end|>": 151647,
|
||||||
|
"<|object_ref_start|>": 151646,
|
||||||
|
"<|quad_end|>": 151651,
|
||||||
|
"<|quad_start|>": 151650,
|
||||||
|
"<|repo_name|>": 151663,
|
||||||
|
"<|video_pad|>": 151656,
|
||||||
|
"<|vision_end|>": 151653,
|
||||||
|
"<|vision_pad|>": 151654,
|
||||||
|
"<|vision_start|>": 151652
|
||||||
|
}
|
||||||
89
chat_template.jinja
Normal file
89
chat_template.jinja
Normal file
@@ -0,0 +1,89 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- messages[0].content + '\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0].role == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||||
|
{%- for message in messages[::-1] %}
|
||||||
|
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||||
|
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
||||||
|
{%- set ns.multi_step_tool = false %}
|
||||||
|
{%- set ns.last_query_index = index %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if message.content is string %}
|
||||||
|
{%- set content = message.content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- set content = '' %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{%- set reasoning_content = '' %}
|
||||||
|
{%- if message.reasoning_content is string %}
|
||||||
|
{%- set reasoning_content = message.reasoning_content %}
|
||||||
|
{%- else %}
|
||||||
|
{%- if '</think>' in content %}
|
||||||
|
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
||||||
|
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if loop.index0 > ns.last_query_index %}
|
||||||
|
{%- if loop.last or (not loop.last and reasoning_content) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if message.tool_calls %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if (loop.first and content) or (not loop.first) %}
|
||||||
|
{{- '\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if tool_call.function %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{%- if tool_call.arguments is string %}
|
||||||
|
{{- tool_call.arguments }}
|
||||||
|
{%- else %}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||||
|
{{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||||
|
{{- '<think>\n\n</think>\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
68
config.json
Normal file
68
config.json
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen3ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151645,
|
||||||
|
"head_dim": 128,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 12288,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 40960,
|
||||||
|
"max_window_layers": 36,
|
||||||
|
"model_type": "qwen3",
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_hidden_layers": 36,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 1000000,
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"transformers_version": "4.57.6",
|
||||||
|
"use_cache": false,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 151936
|
||||||
|
}
|
||||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
{
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151645,
|
||||||
|
151643
|
||||||
|
],
|
||||||
|
"pad_token_id": 151643,
|
||||||
|
"temperature": 0.6,
|
||||||
|
"top_k": 20,
|
||||||
|
"top_p": 0.95,
|
||||||
|
"transformers_version": "4.57.6"
|
||||||
|
}
|
||||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:d64c4c111c0e674ca42193f6456087928359120312e518347c272e681f8e4705
|
||||||
|
size 4902257696
|
||||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:bfa8edcdfe585b27d4b7b59c38c548043499747d75c0ea2be592a25a7664f14f
|
||||||
|
size 4915960368
|
||||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:b38c647466a4fa3b2ce97422a578c425dcfecc481ce1644217a6e93ecff30609
|
||||||
|
size 4983068496
|
||||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:072f1769897157c462d2b178442faecbde237a70f9cf0a0e0ada54307656561d
|
||||||
|
size 1580230264
|
||||||
407
model.safetensors.index.json
Normal file
407
model.safetensors.index.json
Normal file
@@ -0,0 +1,407 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"total_parameters": 8190735360,
|
||||||
|
"total_size": 16381470720
|
||||||
|
},
|
||||||
|
"weight_map": {
|
||||||
|
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||||
|
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||||
|
"model.norm.weight": "model-00004-of-00004.safetensors"
|
||||||
|
}
|
||||||
|
}
|
||||||
332
rl_config.yaml
Normal file
332
rl_config.yaml
Normal file
@@ -0,0 +1,332 @@
|
|||||||
|
entrypoint: examples.terminal_bench.entrypoints.main_tbench
|
||||||
|
|
||||||
|
# Hydra config groups (+ prefix in CLI)
|
||||||
|
config_groups:
|
||||||
|
terminal_bench_config: terminal_bench
|
||||||
|
|
||||||
|
# Terminal bench / agentic environment settings
|
||||||
|
terminal_bench:
|
||||||
|
# trials_dir: Directory for Harbor trial artifacts (derived from experiments_dir if null)
|
||||||
|
trials_dir: null
|
||||||
|
|
||||||
|
# Harbor configuration - schema-driven mapping to TrialConfig
|
||||||
|
harbor:
|
||||||
|
# Agent settings
|
||||||
|
name: terminus-2
|
||||||
|
max_episodes: 999999
|
||||||
|
enable_summarize: false
|
||||||
|
store_all_messages: true
|
||||||
|
trajectory_config:
|
||||||
|
raw_content: true
|
||||||
|
enable_episode_logging: false
|
||||||
|
record_terminal_session: false
|
||||||
|
enable_pane_logging: false
|
||||||
|
|
||||||
|
# Strict JSON parser
|
||||||
|
strict_json_parser: true
|
||||||
|
|
||||||
|
# Interleaved Thinking Settings
|
||||||
|
interleaved_thinking: true
|
||||||
|
extra_body:
|
||||||
|
chat_template_kwargs:
|
||||||
|
enable_thinking: true
|
||||||
|
# 2026-05-27: 1800 → 900. The boundary-hugging a3 datasets (stack-junit,
|
||||||
|
# nemotron, methods2test) hit the 1800s agent-timeout wall on ~70-95% of
|
||||||
|
# trials (GLM-4_7-swesmith can't finish these in 1800s), so every group
|
||||||
|
# waited ~1800s for its slowest sample → generation buffer crawled (5/64
|
||||||
|
# in 4h19) and chains never reached training step 1. Halving to 900s
|
||||||
|
# ~2× the valid-group yield. Tradeoff: lower solve-rate on the hardest
|
||||||
|
# tasks (they fail faster) — acceptable vs. TIMEOUT-at-step-0. Fast-task
|
||||||
|
# chains (e2egit, median 7 turns / <900s) are unaffected. See
|
||||||
|
# agent_logs/2026-05-27_rl_chains_stuck_step0.md. AgentTimeoutError stays
|
||||||
|
# PASSTHROUGH (standing pref — partial trajectory+reward is meaningful).
|
||||||
|
override_timeout_sec: 900
|
||||||
|
|
||||||
|
# Environment settings
|
||||||
|
override_cpus: 1
|
||||||
|
override_memory_mb: 2048
|
||||||
|
override_storage_mb: 2048
|
||||||
|
|
||||||
|
# ==========================================================================
|
||||||
|
# AUTO SNAPSHOT: Reduce Daytona rate limits with hash-based snapshot caching
|
||||||
|
# ==========================================================================
|
||||||
|
# When true, automatically creates a snapshot from the Dockerfile on first use,
|
||||||
|
# then reuses it for all subsequent sandboxes with the same Dockerfile content.
|
||||||
|
# Snapshots are named: harbor__<sha256[:12]>__snapshot
|
||||||
|
auto_snapshot: true
|
||||||
|
|
||||||
|
# Verifier settings
|
||||||
|
verifier_override_timeout_sec: 120
|
||||||
|
|
||||||
|
# Retry settings
|
||||||
|
max_retries: 3
|
||||||
|
min_wait_sec: 60.0
|
||||||
|
max_wait_sec: 600.0
|
||||||
|
wait_multiplier: 2.0
|
||||||
|
|
||||||
|
exclude_exceptions:
|
||||||
|
- VerifierTimeoutError
|
||||||
|
- VerifierRuntimeError
|
||||||
|
- RewardFileNotFoundError
|
||||||
|
- RewardFileEmptyError
|
||||||
|
- VerifierOutputParseError
|
||||||
|
|
||||||
|
# 3x base (300 → 900): triple Daytona concurrency for higher throughput.
|
||||||
|
# Mirrors ALCC/56GPU_base.yaml — per-job sandbox count; doesn't affect
|
||||||
|
# the cross-cluster RUNNING-RL cap (≤ 6) which is about job count, not
|
||||||
|
# sandboxes-per-job.
|
||||||
|
# NOTE (2026-05-27): a temporary 900→500 walk-back was tried to mitigate
|
||||||
|
# the FD-exhaustion / uv__epoll_ctl_prep SIGABRT but REVERTED — the real
|
||||||
|
# fix is porting harbor's terminus-2 FD-taming features from
|
||||||
|
# penfever/temp-override into penfever/otagent-latest (that branch had the
|
||||||
|
# issue tamed at full concurrency). See agent_logs/2026-05-27_rl_chains_stuck_step0.md.
|
||||||
|
# 2026-05-28: 900 → 675 (-25%). Fresh a3 chains (#12/#13) abort with a
|
||||||
|
# driver SIGABRT that is NOT FD exhaustion (fd-monitor showed 1.6% of the
|
||||||
|
# 131072 limit) — suspected host-memory OOM from too many concurrent
|
||||||
|
# never-completing trials piling up resident state in skyrl_entrypoint on
|
||||||
|
# hard datasets. Cutting concurrent trial count reduces that resident
|
||||||
|
# memory. Paired with num_parallel_generation_workers -25%. See
|
||||||
|
# agent_logs/2026-05-28_fresh_a3_chain_crash_not_fd.md.
|
||||||
|
n_concurrent_trials: 675
|
||||||
|
|
||||||
|
# Logging settings
|
||||||
|
log_level: INFO
|
||||||
|
|
||||||
|
# Reward shaping (disabled - binary rewards)
|
||||||
|
enable_reward_shaping: false
|
||||||
|
|
||||||
|
# TIS variant (2026-06-07): request + store per-token rollout logprobs from
|
||||||
|
# vLLM. REQUIRED for trainer.algorithm.use_tis (the trainer asserts non-null
|
||||||
|
# rollout_logprobs when use_tis=true). Only difference vs the TIS-off base.
|
||||||
|
collect_rollout_details: true
|
||||||
|
|
||||||
|
# RLOO-N error classification
|
||||||
|
enable_error_classification: true
|
||||||
|
mask_exceptions:
|
||||||
|
- DaytonaError
|
||||||
|
- EnvironmentStartTimeoutError
|
||||||
|
- NetworkError
|
||||||
|
- ConnectionError
|
||||||
|
- RewardFileNotFoundError
|
||||||
|
- RewardFileEmptyError
|
||||||
|
- AgentEnvironmentTimeoutError
|
||||||
|
- ContextLengthExceededError
|
||||||
|
default_error_treatment: zero
|
||||||
|
# NOTE (2026-05-27): AgentTimeoutError / ContextLengthExceededError moved
|
||||||
|
# OUT of passthrough_exceptions into zero_exceptions. Passthrough routed
|
||||||
|
# these soft-limit trials through the *normal* (live-trajectory) path in
|
||||||
|
# terminal_bench_generator._process_trial_result, keeping the trial's
|
||||||
|
# dangling vLLM ObjectRefs + orphaned litellm async-callback closures
|
||||||
|
# alive in the training batch. That fed Ray's distributed-refcount race
|
||||||
|
# (reference_count.cc:1619 → SIGABRT / WorkerCrashedError) that killed the
|
||||||
|
# nemotron-junit chain (litellm +1 32769 VLLMValidationError +
|
||||||
|
# ContextLengthExceeded on near-budget prompts). Both mask + zero take the
|
||||||
|
# EARLY-RETURN path (response_ids=[0]) which never consumes rollout_details,
|
||||||
|
# so no dangling refs enter training. 2026-05-27: ContextLengthExceededError
|
||||||
|
# moved to mask_exceptions (above) — excluded from the RLOO-N baseline, since
|
||||||
|
# GLM-4_7-swesmith ~32k prompts overflow routinely and we don't want hard
|
||||||
|
# zeros dragging the baseline. AgentTimeoutError stays PASSTHROUGH (partial trajectory+reward is
|
||||||
|
# meaningful signal; infrequent vs ContextLengthExceeded so low refcount risk). Belt-and-suspenders w/ the _harbor_compat rollback_hook that
|
||||||
|
# already truncates these trials' dangling prompt-without-response.
|
||||||
|
passthrough_exceptions:
|
||||||
|
- AgentTimeoutError
|
||||||
|
zero_exceptions: []
|
||||||
|
|
||||||
|
# Model info for Harbor's hosted_vllm validation
|
||||||
|
model_info:
|
||||||
|
# Lowered 32767 -> 32000 (2026-05-27). Harbor's litellm token counter is
|
||||||
|
# +2 below vLLM's actual BPE tokenization on near-budget prompts (litellm
|
||||||
|
# +1 plus chat-template/special-token drift), so a 32767-capped prompt
|
||||||
|
# tokenizes to 32769 at the vLLM serving layer and is rejected with
|
||||||
|
# `VLLMValidationError: 32769 input tokens ... context length is only
|
||||||
|
# 32768`. vLLM returns this as an ErrorResponse (handled, not an engine
|
||||||
|
# crash), but harbor's litellm then retries the *deterministically*
|
||||||
|
# over-budget request (max_retries=3, exp backoff) across n_concurrent
|
||||||
|
# trials. On a boundary-heavy task distribution (e.g. nemotron-junit: 111+
|
||||||
|
# such overflows per chain link) the accumulating open sockets exhaust the
|
||||||
|
# skyrl_entrypoint actor's file descriptors -> libuv uv__epoll_ctl_prep
|
||||||
|
# aborts -> SIGABRT in the uvloop event loop -> ray.WorkerCrashedError kills
|
||||||
|
# the chain before training step 1 (job a3-rl ...nemotron-junit #11,
|
||||||
|
# chain 521442-448). A 768-token buffer below max_model_len=32768 ensures
|
||||||
|
# harbor truncates before vLLM ever rejects, eliminating the retry storm.
|
||||||
|
# Cost is ~2.3% usable context for all a3 chains; chains that don't hug the
|
||||||
|
# boundary (e2egit reached step 68 fine at 32767) are unaffected.
|
||||||
|
max_input_tokens: 32000
|
||||||
|
max_output_tokens: 4096
|
||||||
|
|
||||||
|
archiving:
|
||||||
|
# Enable trial archiving callback
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
# Post-training trace upload to HuggingFace
|
||||||
|
trace_upload:
|
||||||
|
enabled: true
|
||||||
|
repo_org: DCAgent
|
||||||
|
episodes: last
|
||||||
|
dataset_type: SFT
|
||||||
|
cleanup: true
|
||||||
|
|
||||||
|
# Trainer configuration
|
||||||
|
trainer:
|
||||||
|
strategy: fsdp2
|
||||||
|
algorithm:
|
||||||
|
advantage_estimator: rloo_n
|
||||||
|
use_kl_loss: false
|
||||||
|
kl_loss_coef: 0.0
|
||||||
|
eps_clip_low: 0.2
|
||||||
|
# eps_clip_high=0.05 mirrors 24GPU_base — midpoint between 0.2 default and
|
||||||
|
# 0.01 tight. Engages on collapse-onset ratios without over-clamping
|
||||||
|
# healthy updates. Asymmetric — only tightening upper bound.
|
||||||
|
eps_clip_high: 0.05
|
||||||
|
# A/B control arm0 = sequence_mean (the 1/|o_i| brevity-biased reduction).
|
||||||
|
# NOTE: the a3 production 56GPU_base.yaml default is token_mean, NOT
|
||||||
|
# sequence_mean — so the completed run a3-rl-...-pymethods2test-large-80-8B
|
||||||
|
# is NOT a valid arm0 and cannot be reused. This explicit arm0 config
|
||||||
|
# establishes the sequence_mean baseline the bias interpretation needs.
|
||||||
|
loss_reduction: sequence_mean
|
||||||
|
# TIS variant (2026-06-07): Truncated Importance Sampling. Bounds the
|
||||||
|
# off-policy ratio clamp(exp(old_log_probs - rollout_logprobs), max=cap).
|
||||||
|
# cap=2.0 is the standard truncation. REQUIRES collect_rollout_details=true
|
||||||
|
# (harbor block). Only difference vs the TIS-off base.
|
||||||
|
use_tis: true
|
||||||
|
tis_imp_ratio_cap: 2.0
|
||||||
|
|
||||||
|
# Training loop settings
|
||||||
|
epochs: 2
|
||||||
|
max_steps: 80
|
||||||
|
update_epochs_per_batch: 1
|
||||||
|
|
||||||
|
# Batch sizes
|
||||||
|
train_batch_size: 64
|
||||||
|
policy_mini_batch_size: 64
|
||||||
|
eval_batch_size: 64
|
||||||
|
|
||||||
|
# Micro batch sizes (micro1x4 variant)
|
||||||
|
micro_forward_batch_size_per_gpu: 4
|
||||||
|
micro_train_batch_size_per_gpu: 1
|
||||||
|
|
||||||
|
max_prompt_length: 999999
|
||||||
|
|
||||||
|
# Evaluation and checkpointing
|
||||||
|
eval_interval: 999999
|
||||||
|
eval_before_train: false
|
||||||
|
# Resumable checkpointing
|
||||||
|
ckpt_interval: 2
|
||||||
|
resume_mode: latest
|
||||||
|
# HF upload-ready checkpoints
|
||||||
|
hf_save_interval: 5
|
||||||
|
# HuggingFace Hub upload (set via CLI: trainer.hf_hub_repo_id=org/repo)
|
||||||
|
hf_hub_repo_id: null
|
||||||
|
hf_hub_private: false
|
||||||
|
hf_hub_revision: main
|
||||||
|
|
||||||
|
# Database registration (auto-registers trained model to Supabase)
|
||||||
|
# Requires KEYS env var pointing to Supabase credentials file
|
||||||
|
enable_db_registration: false
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
project_name: OpenThoughts-Agent
|
||||||
|
log_level: INFO
|
||||||
|
tracker_commit_each_step: true
|
||||||
|
logger: console
|
||||||
|
|
||||||
|
# Paths
|
||||||
|
run_name: null
|
||||||
|
ckpt_path: null
|
||||||
|
export_path: null
|
||||||
|
|
||||||
|
# Policy optimizer
|
||||||
|
# max_grad_norm=0.9 mirrors 24GPU_base — guardrail against grad-norm spikes
|
||||||
|
# entering correlation-mode-collapse territory (>1.0). 0.9 is just above the
|
||||||
|
# natural healthy peak observed on this dataset+base (‖g‖ peak ~0.81).
|
||||||
|
policy:
|
||||||
|
optimizer_config:
|
||||||
|
lr: 8e-6
|
||||||
|
weight_decay: 0.0
|
||||||
|
adam_betas: [0.9, 0.999]
|
||||||
|
max_grad_norm: 0.9
|
||||||
|
fsdp_config:
|
||||||
|
cpu_offload: false
|
||||||
|
reshard_after_forward: true
|
||||||
|
fsdp_size: 4
|
||||||
|
# Reference model
|
||||||
|
ref:
|
||||||
|
fsdp_config:
|
||||||
|
cpu_offload: false
|
||||||
|
reshard_after_forward: true
|
||||||
|
fsdp_size: 4
|
||||||
|
|
||||||
|
# Model placement (async training) - 8 shared GPUs for policy/ref
|
||||||
|
placement:
|
||||||
|
colocate_all: false
|
||||||
|
policy_num_nodes: 2
|
||||||
|
ref_num_nodes: 2
|
||||||
|
policy_num_gpus_per_node: 4
|
||||||
|
ref_num_gpus_per_node: 4
|
||||||
|
|
||||||
|
# Fully async generation (settings from v2_maxconcurrent)
|
||||||
|
fully_async:
|
||||||
|
max_staleness_steps: 16
|
||||||
|
# Setting conservatively to 1 / 2 of total concurrency
|
||||||
|
# 2026-05-28: 450 → 338 (-25%), paired with n_concurrent_trials 900→675,
|
||||||
|
# to cut driver memory pressure (suspected OOM on fresh hard-dataset a3
|
||||||
|
# chains). See agent_logs/2026-05-28_fresh_a3_chain_crash_not_fd.md.
|
||||||
|
num_parallel_generation_workers: 338
|
||||||
|
|
||||||
|
# Generator configuration
|
||||||
|
generator:
|
||||||
|
backend: vllm
|
||||||
|
timeout_multiplier: 1.0
|
||||||
|
model_dtype: bfloat16
|
||||||
|
|
||||||
|
inference_engine_tensor_parallel_size: 1
|
||||||
|
# 3x base (16 → 48): triple vLLM engines to lift gen throughput above training rate.
|
||||||
|
# Goal: build a real surplus of completed groups so the trainer never waits on gen.
|
||||||
|
# Layout: 48 engines × TP=1 + 8 GPUs for policy/ref = 56 GPUs total = 14 nodes.
|
||||||
|
num_inference_engines: 48
|
||||||
|
|
||||||
|
n_samples_per_prompt: 8
|
||||||
|
eval_n_samples_per_prompt: 8
|
||||||
|
|
||||||
|
# Jupiter-specific gpu_memory_utilization: 0.75 (vs ALCC's 0.85).
|
||||||
|
# GH200's 96 GB HBM has more headroom than ALCC's A100-80, but Jupiter
|
||||||
|
# also runs hosted_vllm in the same Ray cluster as the engines and we
|
||||||
|
# keep 0.75 across all jupiter yamls for consistency / fragmentation
|
||||||
|
# safety. Bump to 0.85 only if KV-cache pressure justifies it.
|
||||||
|
gpu_memory_utilization: 0.75
|
||||||
|
|
||||||
|
max_num_seqs: 24
|
||||||
|
# Jupiter-specific 65536 (vs ALCC's 16384) — matches the existing
|
||||||
|
# 24GPU_base.yaml on Jupiter; larger batched-token budget keeps the
|
||||||
|
# engine fed when many concurrent requests arrive in bursts.
|
||||||
|
max_num_batched_tokens: 65536
|
||||||
|
|
||||||
|
enable_prefix_caching: true
|
||||||
|
enable_chunked_prefill: true
|
||||||
|
|
||||||
|
run_engines_locally: true
|
||||||
|
weight_sync_backend: nccl
|
||||||
|
async_engine: true
|
||||||
|
batched: false
|
||||||
|
enable_http_endpoint: true
|
||||||
|
enable_ray_prometheus_stats: false
|
||||||
|
vllm_stats_interval: 1
|
||||||
|
append_eos_token_after_stop_str_in_multi_turn: true
|
||||||
|
max_turns: 999999
|
||||||
|
|
||||||
|
sampling_params:
|
||||||
|
max_generate_length: 4096
|
||||||
|
temperature: 0.7
|
||||||
|
top_p: 0.95
|
||||||
|
top_k: 20
|
||||||
|
|
||||||
|
engine_init_kwargs:
|
||||||
|
max_model_len: 32768
|
||||||
|
# Interleaved thinking chat template: preserves <think> blocks on ALL
|
||||||
|
# historical assistant turns (stock Qwen3 template strips them).
|
||||||
|
custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
|
||||||
|
|
||||||
|
# Data paths
|
||||||
|
data:
|
||||||
|
train_data: []
|
||||||
|
val_data: []
|
||||||
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
240
tokenizer_config.json
Normal file
240
tokenizer_config.json
Normal file
@@ -0,0 +1,240 @@
|
|||||||
|
{
|
||||||
|
"add_bos_token": false,
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151643": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151644": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151645": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151646": {
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151647": {
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151648": {
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151649": {
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151650": {
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151651": {
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151652": {
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151653": {
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151654": {
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151655": {
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151656": {
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151657": {
|
||||||
|
"content": "<tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151658": {
|
||||||
|
"content": "</tool_call>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151659": {
|
||||||
|
"content": "<|fim_prefix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151660": {
|
||||||
|
"content": "<|fim_middle|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151661": {
|
||||||
|
"content": "<|fim_suffix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151662": {
|
||||||
|
"content": "<|fim_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151663": {
|
||||||
|
"content": "<|repo_name|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151664": {
|
||||||
|
"content": "<|file_sep|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151665": {
|
||||||
|
"content": "<tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151666": {
|
||||||
|
"content": "</tool_response>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151667": {
|
||||||
|
"content": "<think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"151668": {
|
||||||
|
"content": "</think>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|im_end|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"model_max_length": 32768,
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"padding_side": "left",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null
|
||||||
|
}
|
||||||
15
training_logs/20260609_214045_metrics_job_653585.csv
Normal file
15
training_logs/20260609_214045_metrics_job_653585.csv
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,generate/tis/aligned_tokens,generate/tis/alignment_fail_count,generate/tis/exact_match_fraction,generate/tis/lcs_fallback_fraction,generate/tis/lcs_fallback_messages,generate/tis/unaligned_fraction,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,policy/rollout_train_prob_diff_mean,policy/rollout_train_prob_diff_std,policy/tis/imp_ratio_capped_fraction,policy/tis/imp_ratio_mean,policy/tis/log_ratio_abs_mean,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,tis/batch_skipped_no_logprobs,tis/skipped_fraction,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_Timeout,batch_errors/total_Timeout,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError
|
||||||
|
0.0,0,64,512,0,0.0,0,0.0,1665.6621,3021.6782,967.5947,14153,1,2057.7122,573938.0,33.0,0.9487,0.0,0.0,0.0513,0.3398,0.2099,0.2501,-0.0019,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0656,-0.1214,0.0,1.0,0.0,0.0309,425844310016.0,310867954499584.0,0.0,0.5258,0.022,0.6406,0.3398,7.5378,49.9637,630.257,26.5,857.9676,227.7106,0.0376,2.4443,199.3619,89.3179,288.8421,11028.1246,18.1382,89.4422,10718.6996,0.0,0.0,0,1,121,968,497,114,216,1.9090909090909092,231,1.6528925619834711,200.0,0.2231404958677686,27.0,0.15702479338842976,19,0.09090909090909091,11.0,0.024793388429752067,3.0,,,,,,,
|
||||||
|
0.0,0,64,512,1,1.0,1,1.0,2192.3809,5107.972,1422.0889,31541,1,3242.089,778056.0,90.0,0.8463,0.0,0.0,0.1537,0.209,0.14,0.236,-0.0009,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0639,-0.0569,0.0,1.0,0.0,0.0223,5135849414983680.0,3.8306663504564716e+18,0.0001,0.5252,0.0393,0.4844,0.209,13.664,50.6382,619.7108,27.8,857.9676,238.2568,0.0608,5.2456,18.5674,134.9773,153.7971,3675.2048,21.6594,135.1685,3494.5009,0.0,0.0,0,2,64,512,253,62,90,2.109375,135,1.25,80.0,0.5,32.0,0.109375,7,0.046875,3.0,0.03125,2.0,139.9582,149.9938,,,,,
|
||||||
|
0.0,0,64,512,2,1.6094,0,0.9844,1974.7832,3706.1466,1467.6162,19867,1,2776.8866,687353.0,78.0,0.904,0.0,0.0,0.096,0.2266,0.1618,0.2338,-0.0013,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0563,-0.0823,0.0,1.0,0.0,0.0252,5712403677315072.0,4.576328473257181e+18,0.0,0.4977,0.0341,0.5156,0.2266,13.514,50.458,618.5668,27.9,857.9676,239.4008,0.0426,3.6209,18.1097,113.5752,131.9041,3304.1227,19.8368,113.7514,3148.7583,0.0,0.0,0,3,63,504,278,59,93,1.7619047619047619,111,1.3015873015873016,82.0,0.31746031746031744,20.0,0.14285714285714285,9,0.031746031746031744,2.0,0.015873015873015872,1.0,,,0.015873015873015872,1.0,,,
|
||||||
|
0.0,0,64,512,3,1.7656,0,0.9375,1916.1016,3570.8227,1287.2183,26360,1,2567.8385,672216.0,119.0,0.8943,0.0,0.0,0.1057,0.2754,0.1458,0.2335,-0.0013,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0609,-0.081,0.0,1.0,0.0,0.0264,2.5092910058202726e+17,2.0297533524949546e+20,0.0,0.5539,0.0386,0.5312,0.2754,13.7533,50.7891,617.0518,28.1,857.9676,240.9158,0.0533,4.4556,16.0924,108.7123,125.0416,3238.4887,20.0299,108.8955,3088.9592,0.0,0.0,0,4,59,472,401,29,13,1.0,59,0.15254237288135594,9.0,0.0847457627118644,5.0,0.01694915254237288,1,0.06779661016949153,4.0,,,0.0083,12.9332,,,,,
|
||||||
|
0.0,0,64,512,3,1.4531,0,0.9219,3224.291,3590.6844,2890.709,13914,1,2397.9485,1106110.0,103.0,0.9644,0.0,0.0,0.0356,0.4766,0.064,0.1543,-0.0006,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0788,-0.0412,0.0,1.0,0.0,0.017,2.1654929612682035e+18,2.119323756633866e+21,0.0,0.8649,0.0405,0.625,0.4766,13.7164,50.7537,614.8819,28.3,857.9676,243.0857,0.0323,2.8211,19.9051,118.6808,138.8115,1062.3118,23.3888,118.8737,897.2872,0.0,0.0,0,5,63,504,460,23,14,0.42857142857142855,27,0.06349206349206349,4.0,0.047619047619047616,3.0,0.09523809523809523,6,0.047619047619047616,3.0,,,,,,,16.893,0.015873015873015872,1.0
|
||||||
|
0.0,0,64,512,4,1.6406,0,0.8594,4239.8125,4633.2308,3790.4268,22031,1,3300.5798,1457215.0,112.0,0.9654,0.0,0.0,0.0346,0.5332,0.044,0.102,-0.0004,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0853,-0.025,0.0,1.0,0.0,0.0138,6.141298557442851e+18,5.070549340210866e+21,0.0,0.9245,0.0456,0.6562,0.5332,14.4743,51.5385,613.4079,28.5,857.9676,244.5597,0.0452,4.2242,25.388,159.3519,185.0213,1290.7407,28.6557,159.5877,1072.8381,0.0,0.0,0,6,58,464,445,11,12,0.1206896551724138,7,,,,,0.10344827586206896,6,0.034482758620689655,2.0,,,0.0083,10.3463,,,,0.06896551724137931,4.0
|
||||||
|
0.0,0,64,512,4,1.8906,0,0.8906,4397.6465,4166.2911,4597.0327,25556,1,3231.8595,1555192.0,95.0,0.9507,0.0,0.0,0.0493,0.4629,0.0201,0.1187,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0913,-0.0045,0.0,1.0,0.0,0.0167,4.5730908733082645e+20,5.0551306138646176e+23,0.0,0.9745,0.0461,0.5781,0.4629,14.6233,51.6414,612.7198,28.6,857.9676,245.2477,0.0504,4.7036,26.7455,163.7089,190.7489,2018.6125,34.5773,163.9526,1788.5789,0.0,0.0,0,7,60,480,458,10,12,0.15,9,,,0.016666666666666666,1.0,0.16666666666666666,10,0.016666666666666666,1.0,,,,,,,,0.016666666666666666,1.0
|
||||||
|
0.0,0,64,512,4,2.1406,0,0.9531,5137.9238,4980.2167,5241.5307,31761,1,4015.1526,1850673.0,87.0,0.9135,0.0,0.0,0.0865,0.3965,0.0072,0.0726,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0938,0.0003,0.0,1.0,0.0,0.0119,2.930747012383905e+21,3.2028701271085374e+24,0.0,0.9696,0.046,0.4688,0.3965,15.9055,52.3258,610.4556,28.8,857.9676,247.512,0.0614,6.5073,32.8918,190.3358,223.5354,1224.5185,28.8017,190.5818,965.6712,0.0,0.0,0,8,63,504,480,13,21,0.047619047619047616,3,0.07936507936507936,5.0,,,0.20634920634920634,13,,,,,0.0081,12.67,,,,0.047619047619047616,3.0
|
||||||
|
0.0,0,64,512,5,1.9219,0,0.9062,4859.3164,4603.0995,5038.9236,31797,1,4029.8322,1739932.0,52.0,0.8962,0.0,0.0,0.1038,0.4121,0.0093,0.1242,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0803,0.0001,0.0,1.0,0.0,0.0145,1.7394266914541902e+21,2.2925263334899206e+24,0.0,0.9646,0.0375,0.5469,0.4121,16.1924,52.5256,608.6286,29.1,857.9676,249.339,0.0627,5.7252,31.0932,188.5143,219.9568,1481.3095,32.107,188.8005,1223.5171,0.0,0.0,0,9,63,504,487,10,14,0.047619047619047616,3,0.015873015873015872,1.0,,,0.1746031746031746,11,,,,,,,,,,0.031746031746031744,2.0
|
||||||
|
0.0,0,64,512,3,1.6562,0,0.8594,4206.2637,3770.6644,4770.7848,31696,1,3098.8536,1466169.0,27.0,0.9684,0.0,0.0,0.0316,0.5645,-0.0016,0.1212,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0758,-0.0009,0.0,1.0,0.0,0.0186,2.486790661662018e+22,3.009927560391077e+25,0.0,0.9792,0.0338,0.7031,0.5645,16.1496,52.6445,607.3474,29.2,857.9676,250.6202,0.062,5.807,27.2616,157.0913,184.6351,1344.8255,28.0337,157.3111,1126.3452,0.0,0.0,0,10,62,496,465,10,25,0.0967741935483871,6,0.04838709677419355,3.0,,,0.3387096774193548,21,0.016129032258064516,1.0,,,0.0079,11.2982,,,6.1509,,
|
||||||
|
0.0,0,64,512,4,1.8125,0,0.8594,4341.25,4421.7844,4281.534,31768,1,3837.6661,1544551.0,26.0,0.9519,0.0,0.0,0.0481,0.4258,0.0172,0.1082,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0768,-0.0016,0.0,1.0,0.0,0.0145,2.3760572720076554e+18,2.479232720767365e+21,0.0,0.9513,0.0343,0.5312,0.4258,16.1263,52.6095,605.798,29.4,857.9676,252.1696,0.0603,5.7445,30.3901,180.1222,210.8445,1608.5564,32.131,180.3938,1359.8319,0.0,0.0,0,11,63,504,482,8,13,0.14285714285714285,9,0.031746031746031744,2.0,,,0.1746031746031746,11,,,,,,,,,,,
|
||||||
|
0.0,0,64,512,4,2.0156,0,0.9375,5171.2402,4363.3762,5697.6548,31752,1,3838.9164,1805076.0,21.0,0.972,0.0,0.0,0.028,0.3945,-0.0056,0.099,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0793,0.0001,0.0,1.0,0.0,0.0187,3.3199502774606234e+17,4.460413625381222e+20,0.0,0.9698,0.0322,0.5156,0.3945,16.5123,53.0591,604.8148,29.5,857.9676,253.1528,0.5752,5.7891,36.193,189.4182,226.4272,1418.9828,24.7579,189.6581,1162.0041,0.0,0.0,0,12,63,504,486,8,16,0.031746031746031744,2,0.1111111111111111,7.0,,,0.14285714285714285,9,,,,,0.0085,10.287,,,,,
|
||||||
|
0.0,0,64,512,3,1.5781,0,0.8594,4488.6309,4142.0406,4878.3651,31794,1,3769.576,1574682.0,18.0,0.937,0.0,0.0,0.063,0.5293,0.0116,0.1159,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0641,0.0001,0.0,1.0,0.0,0.0191,1.1232591437954967e+20,1.4095364312304327e+23,0.0,0.9755,0.0277,0.6719,0.5293,16.3773,52.9352,603.7007,29.6,857.9676,254.2668,0.0652,5.8155,29.0791,176.4044,205.7881,1174.4667,23.8728,176.6434,938.9854,0.0,0.0,0,13,63,504,462,18,34,0.12698412698412698,8,0.12698412698412698,8.0,,,0.36507936507936506,23,0.047619047619047616,3.0,,,,,,,,,
|
||||||
|
0.0,0,64,512,4,1.9688,0,0.9375,4663.1953,4560.4923,4726.3722,31674,1,3835.0487,1647855.0,26.0,0.9488,0.0,0.0,0.0512,0.3809,-0.0175,0.1338,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0707,0.0,0.0,1.0,0.0,0.0173,1.3211500909875513e+22,1.6953347554969734e+25,0.0,0.9296,0.0299,0.5,0.3809,16.7322,53.2489,601.7022,29.9,857.9676,256.2654,0.1012,5.8519,30.3985,178.5781,209.3207,1621.8309,26.8666,178.8196,1379.7871,0.0,0.0,0,14,64,512,491,8,20,0.015625,1,0.1875,12.0,,,0.109375,7,0.015625,1.0,,,0.0082,11.2278,,,,,
|
||||||
|
21
training_logs/20260609_214045_metrics_job_653586.csv
Normal file
21
training_logs/20260609_214045_metrics_job_653586.csv
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,generate/tis/aligned_tokens,generate/tis/alignment_fail_count,generate/tis/exact_match_fraction,generate/tis/lcs_fallback_fraction,generate/tis/lcs_fallback_messages,generate/tis/unaligned_fraction,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,policy/rollout_train_prob_diff_mean,policy/rollout_train_prob_diff_std,policy/tis/imp_ratio_capped_fraction,policy/tis/imp_ratio_mean,policy/tis/log_ratio_abs_mean,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/save_hf_model,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,tis/batch_skipped_no_logprobs,tis/skipped_fraction,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError
|
||||||
|
0.0,0,64,512,4,0.4375,0,0.1875,4367.6348,4450.258,4236.6061,20102,1,3081.0821,1530592.0,25.0,0.9745,0.0,0.0,0.0255,0.6133,-0.0018,0.082,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0643,-0.0,0.0,1.0,0.0,0.0116,1005131.5625,1241936512.0,0.0,0.9556,0.021,0.65,0.6133,9.9297,50.2901,562.6478,34.4,857.9675,295.3198,0.0524,3.9222,32.4786,156.8412,189.5694,6.8707,4387.0719,37.9376,157.0373,4155.6423,0.0,0.0,0,15,54,432,390,20,41,0.7962962962962963,43,0.037037037037037035,2.0,0.09259259259259259,5.0,0.018518518518518517,1.0,,,,,,,,
|
||||||
|
0.0,0,64,512,1,0.9531,0,0.9531,5236.6426,4392.4831,5489.4619,31722,1,5068.1073,1953529.0,45.0,0.8724,0.0,0.0,0.1276,0.2305,0.008,0.0679,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0736,-0.0004,0.0,1.0,0.0,0.012,2.912904544507658e+18,4.067950672816252e+21,0.0,0.908,0.0255,0.3438,0.2305,13.9738,50.8674,555.7899,35.2,857.9675,302.1776,0.0685,6.0938,36.9697,235.7047,272.9919,,940.3521,34.9952,235.9529,626.262,0.0,0.0,0,16,58,464,422,13,41,0.5517241379310345,32,0.1896551724137931,11.0,0.017241379310344827,1.0,0.017241379310344827,1.0,10.4479,18.917,,,,,,
|
||||||
|
0.0,0,64,512,2,1.6562,0,0.9531,4441.8926,4303.1646,4561.4509,31764,1,4231.7973,1624165.0,28.0,0.9089,0.0,0.0,0.0911,0.4629,-0.0268,0.1262,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0631,-0.0001,0.0,1.0,0.0,0.014,80980987609088.0,9.804779835568947e+16,0.0,0.919,0.0208,0.6094,0.4629,14.1574,51.0022,555.1819,35.3,857.9675,302.7856,0.8639,6.2006,35.3247,195.5062,231.9566,,866.8196,41.0458,195.7672,587.611,0.0,0.0,0,17,63,504,479,13,24,0.2857142857142857,18,0.047619047619047616,3.0,0.047619047619047616,3.0,0.015873015873015872,1.0,,,,,,,,
|
||||||
|
0.0,0,64,512,3,1.9844,0,0.8438,4984.2539,4197.7163,5522.4112,31647,1,4678.7713,1808744.0,41.0,0.8809,0.0,0.0,0.1191,0.4062,-0.0087,0.1018,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0679,-0.0,0.0,1.0,0.0,0.0172,7.39416158020567e+16,9.942431530797027e+19,0.0,0.9589,0.0221,0.5312,0.4062,14.2903,51.1974,554.0929,35.4,857.9675,303.8746,0.0899,6.4232,37.3252,220.9561,258.6391,,2168.7359,37.6553,221.2235,1866.0111,0.0,0.0,0,18,63,504,457,21,46,0.6190476190476191,39,0.047619047619047616,3.0,0.06349206349206349,4.0,0.015873015873015872,1.0,0.0074,8.3881,,,,,,
|
||||||
|
0.0,0,64,512,4,1.9062,0,0.8906,4700.6758,3937.2571,5231.5298,31734,1,4306.1606,1743000.0,35.0,0.9022,0.0,0.0,0.0978,0.4102,-0.02,0.1351,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0714,0.0,0.0,1.0,0.0,0.0168,6328167232962560.0,7.310935387570962e+18,0.0,0.9156,0.0245,0.5156,0.4102,14.5789,51.2928,552.8044,35.6,857.9675,305.1631,0.0891,6.021,34.5214,198.7713,233.6051,,1196.6914,32.8597,198.9941,924.198,0.0,0.0,0,19,62,496,459,12,35,0.5483870967741935,34,,,0.03225806451612903,2.0,0.03225806451612903,2.0,,,,,,,,
|
||||||
|
0.0,0,64,512,4,1.6562,0,0.8281,4695.9609,4342.2462,4913.5457,31736,1,4065.4442,1681502.0,28.0,0.9544,0.0,0.0,0.0456,0.3809,0.0048,0.0503,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0696,-0.0005,0.0,1.0,0.0,0.0122,1080837378408448.0,1.4011869695511101e+18,0.0,0.9341,0.022,0.4531,0.3809,14.7026,51.4866,551.9064,35.7,857.9675,306.0612,0.0871,6.1017,31.5548,187.9218,219.7869,5.8852,1351.9719,33.9437,188.1445,1092.1314,0.0,0.0,0,20,56,448,403,15,42,0.6964285714285714,39,0.07142857142857142,4.0,,,0.10714285714285714,6.0,0.0088,8.1422,,,,,,
|
||||||
|
0.0,0,64,512,4,1.9844,0,0.8906,4777.4453,4620.9898,4874.4873,31826,1,4055.0244,1707389.0,27.0,0.9601,0.0,0.0,0.0399,0.3828,0.0103,0.1078,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0684,-0.0062,0.0,1.0,0.0,0.0144,20477047209984.0,2.673594378433331e+16,0.0,0.9125,0.0217,0.5,0.3828,14.9075,51.7109,624.8141,27.2,857.9675,233.1534,0.0882,7.6651,35.0631,196.5691,231.9387,,1653.9283,32.7526,196.7865,1381.5656,0.0,0.0,0,21,56,448,403,15,44,0.9642857142857143,54,,,0.03571428571428571,2.0,0.017857142857142856,1.0,,,,,,,,
|
||||||
|
0.0,0,64,512,3,1.6406,0,0.8438,4256.166,4364.5684,4192.2019,19194,1,3380.6239,1525708.0,35.0,0.9617,0.0,0.0,0.0383,0.3711,0.0204,0.1066,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0682,-0.0021,0.0,1.0,0.0,0.0156,9.329387085443891e+16,1.1523618092078439e+20,0.0,0.8961,0.0225,0.5156,0.3711,14.5482,51.4926,619.0483,27.8,857.9675,238.9193,0.047,3.8528,29.6378,159.7489,189.6301,,1708.2028,35.8861,159.9448,1478.8283,0.0,0.0,0,22,61,488,396,27,75,0.639344262295082,39,0.6229508196721312,38.0,0.04918032786885246,3.0,0.36065573770491804,22.0,0.0083,8.0671,0.03278688524590164,2.0,0.01639344262295082,1.0,,
|
||||||
|
0.0,0,64,512,3,1.2656,0,0.7969,3544.1543,3840.6749,3276.2937,28297,1,3722.3558,1252311.0,10.0,0.991,0.0,0.0,0.009,0.4746,0.0158,0.0979,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0518,-0.0098,0.0,1.0,0.0,0.0149,7758122496.0,8681794568192.0,0.0,0.8013,0.0161,0.5781,0.4746,15.0082,51.8909,612.8306,28.6,857.9675,245.137,0.0827,5.0234,30.5594,167.256,198.1548,,2873.6125,18.8351,167.5122,2651.5938,0.0,0.0,0,23,63,504,245,58,161,0.6825396825396826,43,1.9841269841269842,125.0,0.015873015873015872,1.0,1.0952380952380953,69.0,,,0.4603174603174603,29.0,,,,
|
||||||
|
0.0,0,64,512,4,1.6875,0,0.9375,2838.584,5697.4839,2204.031,21845,1,4033.9811,998412.0,19.0,0.9595,0.0,0.0,0.0405,0.1816,0.0534,0.1319,-0.0005,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0375,-0.031,0.0,1.0,0.0,0.0176,796371091193856.0,7.9573848653457e+17,0.0,0.4832,0.012,0.375,0.1816,14.9013,51.7034,610.9744,28.8,857.9675,246.9931,0.0656,3.9288,26.3368,163.6413,190.2636,,4553.1826,19.7707,163.8606,4339.2146,0.0,0.0,0,24,63,504,198,59,150,0.31746031746031744,20,2.015873015873016,127.0,0.031746031746031744,2.0,2.1904761904761907,138.0,0.0083,30.3375,0.30158730158730157,19.0,0.015873015873015872,1.0,0.015873015873015872,1.0
|
||||||
|
0.0,0,64,512,3,1.5469,0,0.9375,1484.3555,3727.4578,1050.3753,15945,1,2355.9005,540011.0,7.0,0.9765,0.0,0.0,0.0235,0.1621,0.0976,0.1584,-0.0006,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0251,-0.0401,0.0,1.0,0.0,0.0162,175853157744640.0,1.2922646919590707e+17,0.0,0.4014,0.0083,0.3906,0.1621,14.7827,51.591,611.105,28.8,857.9675,246.8625,0.0412,2.8411,14.0718,97.9816,112.2645,6.1145,2911.5013,20.8742,98.1511,2775.5178,0.0,0.0,0,25,60,480,239,50,62,0.16666666666666666,10,0.7833333333333333,47.0,0.06666666666666667,4.0,2.7,162.0,,,0.35,21.0,0.016666666666666666,1.0,0.016666666666666666,1.0
|
||||||
|
0.0,0,64,512,3,1.4688,0,0.9062,1762.2266,3276.9356,775.2226,14084,1,2254.5374,611382.0,7.0,0.9867,0.0,0.0,0.0133,0.3945,0.2088,0.2489,-0.0017,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.028,-0.1098,0.0,1.0,0.0,0.0301,78101203648512.0,6.106804833275085e+16,0.0,0.5056,0.0092,0.6875,0.3945,15.0391,51.8378,607.4099,29.2,857.9675,250.5576,0.0393,2.7152,16.2956,95.0195,111.5402,,2183.5461,21.0421,95.2048,2048.2464,0.0,0.0,0,26,50,400,287,35,28,0.26,13,0.38,19.0,0.1,5.0,2.2,110.0,0.0076,18.4531,0.2,10.0,,,,
|
||||||
|
0.0,0,64,512,3,1.7344,0,0.9844,3225.4141,4233.3925,2501.5638,31725,1,3531.9698,1097953.0,6.0,0.9706,0.0,0.0,0.0294,0.418,0.0857,0.1298,-0.001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0417,-0.064,0.0,1.0,0.0,0.0185,1.7051105289310222e+21,1.7866703500134324e+24,0.0,0.7126,0.0141,0.5938,0.418,15.2845,52.0295,607.246,29.2,857.9675,250.7216,0.0933,6.0684,24.9326,145.5165,170.7634,,1456.934,22.7893,145.7369,1257.3089,0.0,0.0,0,27,63,504,327,51,68,0.3968253968253968,25,0.5396825396825397,34.0,0.12698412698412698,8.0,0.9523809523809523,60.0,,,0.7301587301587301,46.0,,,0.015873015873015872,1.0
|
||||||
|
0.0,0,64,512,4,2.0312,0,0.9062,3251.666,4554.6894,2653.9829,21699,1,3609.6329,1139572.0,15.0,0.9862,0.0,0.0,0.0138,0.3145,0.1116,0.1947,-0.0011,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0421,-0.0686,0.0,1.0,0.0,0.0196,3.3728,2506.6016,0.0,0.6563,0.0144,0.5312,0.3145,15.0491,51.7938,605.1166,29.5,857.9675,252.8509,0.0646,4.1197,22.7726,156.63,179.688,,3692.4206,19.6838,156.8504,3488.9217,0.0,0.0,0,28,62,496,313,47,111,0.3870967741935484,24,1.3548387096774193,84.0,0.0967741935483871,6.0,0.967741935483871,60.0,0.0078,10.6979,0.24193548387096775,15.0,,,0.03225806451612903,2.0
|
||||||
|
0.0,0,64,512,4,1.6719,0,0.9375,2571.3672,3743.3082,2043.4958,15866,1,2671.8445,889053.0,7.0,0.9918,0.0,0.0,0.0082,0.3105,0.0625,0.1191,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.036,-0.0436,0.0,1.0,0.0,0.0144,2088775.625,1969496832.0,0.0,0.6323,0.0129,0.4375,0.3105,15.0667,51.8913,603.0693,29.7,857.9675,254.8982,0.0526,3.2231,19.9907,119.1558,139.374,,2121.552,24.1526,119.3302,1954.7975,0.0,0.0,0,29,62,496,399,34,26,0.3064516129032258,19,0.1935483870967742,12.0,0.04838709677419355,3.0,1.032258064516129,64.0,,,0.16129032258064516,10.0,,,,
|
||||||
|
0.0,0,64,512,4,1.5938,0,0.8594,3329.0859,3448.2411,3212.6911,31752,1,3171.8807,1207059.0,16.0,0.9547,0.0,0.0,0.0453,0.4941,0.0633,0.1659,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0435,-0.0529,0.0,1.0,0.0,0.0237,166590859444224.0,1.8302711074127872e+17,0.0,0.7955,0.0163,0.7031,0.4941,15.2241,52.0995,601.8652,29.8,857.9675,256.1023,0.0917,5.5872,23.2574,143.9244,167.4704,6.1638,1800.873,28.7964,144.1208,1599.0148,0.0,0.0,0,30,61,488,408,28,51,0.7049180327868853,43,0.01639344262295082,1.0,0.11475409836065574,7.0,0.4426229508196721,27.0,0.0073,8.1466,0.01639344262295082,1.0,0.01639344262295082,1.0,,
|
||||||
|
0.0,0,64,512,4,1.7188,0,0.875,4176.25,4287.3301,4101.4706,22905,1,3442.4973,1477397.0,13.0,0.9695,0.0,0.0,0.0305,0.4023,0.0198,0.0959,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0487,-0.0138,0.0,1.0,0.0,0.0177,17930.9121,21793098.0,0.0,0.8509,0.0185,0.5312,0.4023,14.9604,51.8221,601.6973,29.9,857.9675,256.2702,0.0519,4.5315,30.1351,156.4327,186.8623,,1635.6999,31.1503,156.6733,1413.1506,0.0,0.0,0,31,62,496,427,25,63,1.0483870967741935,65,0.016129032258064516,1.0,0.016129032258064516,1.0,0.0967741935483871,6.0,,,,,,,0.04838709677419355,3.0
|
||||||
|
0.0,0,64,512,5,1.7031,0,0.875,4112.8223,4007.204,4181.0836,31560,1,3957.1982,1471823.0,15.0,0.9631,0.0,0.0,0.0369,0.3926,0.0092,0.1152,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0462,-0.0013,0.0,1.0,0.0,0.0137,113758040.0,137789751296.0,0.0,0.8584,0.0187,0.5156,0.3926,15.251,52.1536,599.0416,30.2,857.9675,258.9259,0.0694,5.7857,28.8593,179.0917,208.2315,,1896.1165,35.5573,179.3023,1646.5384,0.0,0.0,0,32,62,496,454,17,33,0.3870967741935484,24,0.08064516129032258,5.0,0.016129032258064516,1.0,0.14516129032258066,9.0,0.0086,11.4834,,,,,0.06451612903225806,4.0
|
||||||
|
0.0,0,64,512,3,1.6719,0,0.8906,3989.2363,3479.5046,4367.2007,23645,1,3195.8215,1377193.0,19.0,0.9806,0.0,0.0,0.0194,0.4258,0.0032,0.0398,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0433,-0.0003,0.0,1.0,0.0,0.0155,214550183936.0,251782945046528.0,0.0,0.9224,0.0178,0.4844,0.4258,15.0996,52.0021,598.8062,30.2,857.9675,259.1614,0.0535,4.9425,28.9784,147.4112,176.6704,,1164.9656,31.3944,147.638,951.9529,0.0,0.0,0,33,64,512,443,16,69,1.046875,67,0.015625,1.0,,,,,,,,,,,0.015625,1.0
|
||||||
|
0.0,0,64,512,5,1.7812,0,0.875,3690.3398,3317.2692,3896.0939,20066,1,3147.0993,1351091.0,13.0,0.9855,0.0,0.0,0.0145,0.3555,-0.0158,0.1224,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0489,0.0,0.0,1.0,0.0,0.0231,4346809.0,4922826240.0,0.0,0.8724,0.0195,0.4844,0.3555,15.2644,52.2489,596.6576,30.5,857.9675,261.3099,0.0489,4.2274,24.6053,147.5165,172.3903,,1643.8279,30.4621,147.7346,1436.743,0.0,0.0,0,34,18,144,123,6,21,1.0555555555555556,19,,,0.05555555555555555,1.0,,,0.0074,8.954,,,,,0.05555555555555555,1.0
|
||||||
|
27
training_logs/20260609_214045_metrics_job_653587.csv
Normal file
27
training_logs/20260609_214045_metrics_job_653587.csv
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,generate/tis/aligned_tokens,generate/tis/alignment_fail_count,generate/tis/exact_match_fraction,generate/tis/lcs_fallback_fraction,generate/tis/lcs_fallback_messages,generate/tis/unaligned_fraction,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,policy/rollout_train_prob_diff_mean,policy/rollout_train_prob_diff_std,policy/tis/imp_ratio_capped_fraction,policy/tis/imp_ratio_mean,policy/tis/log_ratio_abs_mean,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/save_hf_model,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,tis/batch_skipped_no_logprobs,tis/skipped_fraction,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError
|
||||||
|
0.0,0,64,512,3,0.1719,0,0.0938,5270.0117,4869.6905,5658.0154,24181,1,4003.1029,1780122.0,10.0,0.987,0.0,0.0,0.013,0.4922,0.0054,0.0974,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0471,-0.0,0.0,1.0,0.0,0.0164,2104.1606,2702316.25,0.0,0.9513,0.0147,0.6066,0.4922,11.2632,50.6464,591.5163,31.1,857.9675,266.4512,0.0717,5.2462,36.047,201.8628,238.2231,8.0617,4940.941,41.3945,202.1037,4656.0768,0.0,0.0,0,35,58,464,389,22,75,0.08620689655172414,5,1.0172413793103448,59,0.1206896551724138,7.0,0.017241379310344827,1.0,0.1206896551724138,7.0,,,,,,
|
||||||
|
0.0,0,64,512,1,1.0,1,1.0,4019.6211,3506.7433,4314.7231,31761,1,4137.767,1417266.0,17.0,0.9591,0.0,0.0,0.0409,0.3652,0.0001,0.0845,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0442,0.0,0.0,1.0,0.0,0.0192,973045248.0,849779032064.0,0.0,0.8509,0.0153,0.4844,0.3652,14.0512,51.1256,585.3741,31.8,857.9675,272.5934,0.0674,6.7554,29.7924,182.7126,212.8136,,608.8706,36.372,182.9532,352.9221,0.0,0.0,0,36,64,512,464,17,41,0.078125,5,0.5,32,0.03125,2.0,0.109375,7.0,0.03125,2.0,11.8491,21.5894,,,,
|
||||||
|
0.0,0,64,512,2,1.5781,0,0.8906,4050.4727,3962.2194,4155.3205,26248,1,3351.8207,1378599.0,12.0,0.9874,0.0,0.0,0.0126,0.543,0.0002,0.1003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0429,0.0001,0.0,1.0,0.0,0.0164,451543132274688.0,5.2744631064356454e+17,0.0,0.9114,0.0146,0.625,0.543,14.0529,51.2486,585.133,31.8,857.9675,272.8345,0.0763,5.0232,27.0398,157.8818,185.2402,,1221.9848,32.8949,158.1234,998.821,0.0,0.0,0,37,60,480,426,16,51,0.06666666666666667,4,0.8,48,0.06666666666666667,4.0,0.05,3.0,0.03333333333333333,2.0,,,,,,
|
||||||
|
0.0,0,64,512,3,1.7812,0,0.8906,4166.3809,3516.1595,4705.1357,31759,1,3839.5371,1482005.0,13.0,0.9562,0.0,0.0,0.0438,0.4531,-0.0024,0.1352,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0404,-0.0007,0.0,1.0,0.0,0.019,5167771222540288.0,6.12676136445621e+18,0.0,0.8858,0.0136,0.5781,0.4531,14.5875,51.5708,583.2682,32.0,857.9675,274.6993,0.0874,6.1662,33.6069,178.474,212.3933,,1845.865,36.6567,178.6985,1590.6434,0.0,0.0,0,38,62,496,429,21,67,0.11290322580645161,7,0.8387096774193549,52,0.14516129032258066,9.0,,,0.016129032258064516,1.0,0.0087,9.8733,,,,
|
||||||
|
0.0,0,64,512,4,1.9531,0,0.8906,4455.7715,4539.2418,4379.7761,31568,1,4244.5861,1548541.0,19.0,0.9715,0.0,0.0,0.0285,0.4766,0.0051,0.0819,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0407,-0.0,0.0,1.0,0.0,0.017,4.4555619643529626e+19,5.5445188610444524e+22,0.0,0.8705,0.0132,0.5781,0.4766,14.9482,52.0431,582.4987,32.1,857.9675,275.4688,0.0874,6.7551,37.2039,192.8015,230.35,,1216.2695,35.0045,193.0581,944.1541,0.0,0.0,0,39,62,496,424,23,72,0.11290322580645161,7,0.7580645161290323,47,0.22580645161290322,14.0,,,0.06451612903225806,4.0,,,,,,
|
||||||
|
0.0,0,64,512,5,1.8906,0,0.8906,3978.8047,4426.8494,3586.5604,20630,1,3356.5378,1380244.0,18.0,0.9762,0.0,0.0,0.0238,0.4668,0.0008,0.0972,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0396,0.0,0.0,1.0,0.0,0.0151,8027375480602624.0,9.356849449931899e+18,0.0,0.8635,0.0152,0.5312,0.4668,14.7535,51.8218,581.7134,32.2,857.9675,276.2541,0.0558,4.3532,24.4683,152.7384,177.4882,6.7398,1338.8646,39.4864,152.9636,1117.5303,0.0,0.0,0,40,61,488,425,18,55,0.01639344262295082,1,0.7049180327868853,43,0.19672131147540983,12.0,0.13114754098360656,8.0,0.03278688524590164,2.0,0.0073,8.861,,,,
|
||||||
|
0.0,0,64,512,4,1.6719,0,0.8281,3751.0664,3454.8347,4004.3659,28890,1,3641.0091,1296617.0,12.0,0.9877,0.0,0.0,0.0123,0.4609,0.0022,0.08,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0406,-0.0,0.0,1.0,0.0,0.0169,1345461420032.0,1527438681047040.0,0.0,0.8763,0.0139,0.5781,0.4609,14.9705,51.9844,581.6381,32.2,857.9675,276.3294,0.0797,5.3066,30.2736,162.8792,193.4836,,1768.759,31.3917,163.1298,1538.5717,0.0,0.0,0,41,64,512,449,21,62,0.15625,10,0.703125,45,0.046875,3.0,0.015625,1.0,0.0625,4.0,,,,,,
|
||||||
|
0.0,0,64,512,4,1.9375,0,0.9062,4489.3984,4501.6872,4481.8391,31683,1,4385.293,1584898.0,26.0,0.939,0.0,0.0,0.061,0.3809,0.0119,0.0673,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0434,0.0,0.0,1.0,0.0,0.0124,602914881536.0,717373355065344.0,0.0,0.8819,0.0152,0.4375,0.3809,15.4638,52.4785,580.4658,32.3,857.9675,277.5018,0.0671,6.1951,32.6832,192.1724,225.2152,,1528.4712,36.6995,192.4644,1260.3568,0.0,0.0,0,42,62,496,445,20,45,0.14516129032258066,9,0.5645161290322581,35,,,0.0967741935483871,6.0,0.016129032258064516,1.0,0.0091,8.6068,,,,
|
||||||
|
0.0,0,64,512,3,1.7344,0,0.9062,4579.6543,4076.3868,4935.2967,31795,1,4209.4791,1582217.0,11.0,0.9742,0.0,0.0,0.0258,0.4141,0.003,0.0933,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0423,0.0,0.0,1.0,0.0,0.0148,1.2523732566881721e+22,1.575312744563639e+25,0.0,0.9056,0.0142,0.5,0.4141,15.3868,52.4961,580.9841,32.3,857.9675,276.9834,0.0969,5.8831,35.5495,203.7713,239.6873,,1429.6029,46.2261,204.0403,1137.7979,0.0,0.0,0,43,58,464,404,16,52,0.08620689655172414,5,0.7068965517241379,41,0.20689655172413793,12.0,0.13793103448275862,8.0,0.034482758620689655,2.0,,,,,,
|
||||||
|
0.0,0,64,512,4,1.7031,0,0.875,3711.0449,3450.7154,3979.6389,31858,1,3450.5205,1281843.0,21.0,0.9584,0.0,0.0,0.0416,0.5078,0.0026,0.0904,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0389,0.0001,0.0,1.0,0.0,0.0186,10689648640.0,12101714182144.0,0.0,0.8716,0.0135,0.6094,0.5078,15.393,52.5665,580.2031,32.4,857.9675,277.7644,0.0674,6.9397,27.3191,159.9392,187.542,,1259.6237,27.2463,160.1551,1037.8905,0.0,0.0,0,44,64,512,477,14,35,0.09375,6,0.25,16,0.125,8.0,,,0.03125,2.0,0.0088,9.1974,,,,
|
||||||
|
0.0,0,64,512,4,1.7969,0,0.9219,4316.8047,4018.2939,4590.7191,31802,1,3563.2553,1489843.0,18.0,0.9623,0.0,0.0,0.0377,0.4785,0.0011,0.0785,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0413,0.0,0.0,1.0,0.0,0.0143,1.399725969946154e+24,1.7084933114951544e+27,0.0,0.9425,0.0143,0.5312,0.4785,15.4081,52.5731,579.7631,32.4,857.9675,278.2045,0.0681,6.996,33.0651,177.8596,211.2338,6.2605,1402.8534,32.9762,178.1002,1151.6417,0.0,0.0,0,45,63,504,456,15,48,0.06349206349206349,4,0.5396825396825397,34,0.14285714285714285,9.0,,,0.015873015873015872,1.0,,,,,,
|
||||||
|
0.0,0,64,512,4,2.125,0,0.9688,4054.1738,3689.888,4427.0988,31767,1,3753.6733,1394041.0,18.0,0.9628,0.0,0.0,0.0372,0.5059,0.0001,0.1048,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0372,-0.0001,0.0,1.0,0.0,0.019,667826126848.0,788490430185472.0,0.0,0.9108,0.0129,0.5781,0.5059,15.491,52.6367,579.8985,32.4,857.9675,278.069,0.0677,6.2085,29.2944,171.0908,200.675,,1235.1016,35.891,171.3125,992.3206,0.0,0.0,0,46,63,504,465,16,39,0.06349206349206349,4,0.49206349206349204,31,0.1111111111111111,7.0,,,0.047619047619047616,3.0,0.0085,8.8618,,,,
|
||||||
|
0.0,0,64,512,4,2.0781,0,0.8906,4380.8105,4373.7847,4388.8992,31787,1,4245.6796,1564842.0,28.0,0.9236,0.0,0.0,0.0764,0.5352,0.0002,0.1101,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.04,0.0,0.0,1.0,0.0,0.0178,6668615680000.0,6459941124571136.0,0.0,0.917,0.0138,0.625,0.5352,15.551,52.6967,579.8947,32.4,857.9675,278.0728,0.0683,5.9577,30.8067,192.315,223.44,,1375.8745,34.2379,192.5645,1112.233,0.0,0.0,0,47,60,480,429,17,51,0.05,3,0.8166666666666667,49,0.06666666666666667,4.0,,,0.03333333333333333,2.0,,,,,,
|
||||||
|
0.0,0,64,512,4,1.7344,0,0.7812,3799.5566,3462.6914,4136.4219,31660,1,3327.2914,1342599.0,18.0,0.9759,0.0,0.0,0.0241,0.5,-0.0104,0.0643,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0401,0.0,0.0,1.0,0.0,0.0186,6386052608.0,7399555989504.0,0.0,0.8917,0.0134,0.5938,0.5,16.2215,53.1401,577.9128,32.6,857.9675,280.0547,0.0875,5.6471,29.4067,156.0174,185.712,,1598.9196,29.9299,156.2173,1377.6224,0.0,0.0,0,48,63,504,429,29,67,0.15873015873015872,10,0.6190476190476191,39,0.2698412698412698,17.0,0.12698412698412698,8.0,0.06349206349206349,4.0,0.0081,8.3558,,,,
|
||||||
|
0.0,0,64,512,4,1.8125,0,0.8594,3419.8555,3529.5067,3333.892,31764,1,3464.6841,1225041.0,9.0,0.9602,0.0,0.0,0.0398,0.4395,0.0036,0.0933,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0381,0.0001,0.0,1.0,0.0,0.0193,22766618624.0,22522718322688.0,0.0,0.8525,0.0126,0.5781,0.4395,16.5361,53.4547,577.4091,32.7,857.9675,280.5584,0.063,5.9879,28.4774,153.5356,182.3468,,1442.6214,29.7255,153.806,1224.5562,0.0,0.0,0,49,63,504,448,24,49,0.1111111111111111,7,0.5079365079365079,32,0.1111111111111111,7.0,0.1111111111111111,7.0,,,,,,,,
|
||||||
|
0.0,0,64,512,4,1.9375,0,0.8906,4705.582,3464.1697,5295.879,31742,1,4707.6528,1701349.0,45.0,0.9029,0.0,0.0,0.0971,0.3223,0.0002,0.0516,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0422,-0.0001,0.0,1.0,0.0,0.0187,464861233152.0,492378775355392.0,0.0,0.9011,0.0146,0.3906,0.3223,16.5455,53.464,576.9939,32.7,857.9675,280.9736,0.0903,7.0541,33.012,211.9908,245.3293,5.7884,1863.3175,39.9846,212.2265,1570.9436,0.0,0.0,0,50,62,496,411,24,78,0.11290322580645161,7,0.7903225806451613,49,0.24193548387096775,15.0,0.11290322580645161,7.0,0.11290322580645161,7.0,0.0085,8.5286,,,,
|
||||||
|
0.0,0,64,512,3,1.625,0,0.8906,3573.1074,3325.9396,3709.4242,21691,1,3523.2654,1255628.0,17.0,0.9817,0.0,0.0,0.0183,0.3555,-0.0051,0.0953,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0379,-0.0,0.0,1.0,0.0,0.0153,165468454912.0,185397279719424.0,0.0,0.8323,0.0132,0.4844,0.3555,16.3303,53.2484,577.5415,32.7,857.9675,280.426,0.0574,5.3966,29.4475,162.4351,192.1689,,1459.9757,33.9624,162.6636,1228.4414,0.0,0.0,0,51,64,512,420,29,92,0.171875,11,0.875,56,0.265625,17.0,,,0.140625,9.0,,,,,,
|
||||||
|
0.0,0,64,512,4,1.8438,0,0.9688,3976.1758,4500.2031,3661.7594,29854,1,3987.1647,1358843.0,25.0,0.9848,0.0,0.0,0.0152,0.375,0.0005,0.099,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0394,-0.0,0.0,1.0,0.0,0.0151,7481026.5,7046713344.0,0.0,0.8285,0.0148,0.4688,0.375,16.6743,53.6125,576.8345,32.8,857.9675,281.133,0.084,5.6133,29.8419,175.3867,205.5518,,1339.7774,38.5075,175.6254,1090.0995,0.0,0.0,0,52,63,504,433,23,61,0.015873015873015872,1,0.49206349206349204,31,0.38095238095238093,24.0,0.06349206349206349,4.0,0.09523809523809523,6.0,0.0072,8.6118,0.09523809523809523,6.0,,
|
||||||
|
0.0,0,64,512,3,1.7812,0,0.9062,3691.3457,3547.4033,3821.3755,22436,1,3372.4346,1290968.0,11.0,0.9861,0.0,0.0,0.0139,0.4746,-0.0042,0.0709,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0381,0.0,0.0,1.0,0.0,0.0124,1.2458670639324969e+23,1.4152924048125494e+26,0.0,0.8636,0.0132,0.5625,0.4746,16.4417,53.4303,577.7429,32.7,857.9675,280.2247,0.0685,5.4589,25.5377,153.0849,178.9488,,1705.9455,29.4538,153.3421,1492.0762,0.0,0.0,0,53,64,512,463,19,48,0.0625,4,0.5625,36,0.125,8.0,0.015625,1.0,,,,,,,,
|
||||||
|
0.0,0,64,512,4,1.7188,0,0.9062,3767.5215,3727.4593,3795.1551,18986,1,2995.1567,1342725.0,37.0,0.968,0.0,0.0,0.032,0.4082,0.0064,0.0877,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0444,-0.0,0.0,1.0,0.0,0.0185,327695663104.0,379507068370944.0,0.0,0.9092,0.0151,0.4844,0.4082,16.3465,53.3348,577.0278,32.7,857.9675,280.9397,0.0444,3.8435,25.2314,140.1255,165.6541,,1372.4083,33.4557,140.3778,1169.4509,0.0,0.0,0,54,62,496,426,22,70,0.04838709677419355,3,0.9354838709677419,58,0.06451612903225806,4.0,,,0.08064516129032258,5.0,0.0087,9.1348,,,,
|
||||||
|
0.0,0,64,512,4,2.0469,0,0.9219,4275.6836,3941.0486,4465.0031,31662,1,4375.7459,1463491.0,11.0,0.9714,0.0,0.0,0.0286,0.3613,-0.0046,0.1022,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0408,0.0,0.0,1.0,0.0,0.0216,4036787148161024.0,4.865830434606416e+18,0.0,0.8604,0.0139,0.4688,0.3613,16.6034,53.5885,576.6667,32.8,857.9675,281.3008,0.157,6.1578,33.4267,203.4849,237.3352,5.8658,1613.5996,28.3574,203.7511,1341.7459,0.0,0.0,0,55,61,488,424,21,57,0.04918032786885246,3,0.819672131147541,50,0.01639344262295082,1.0,0.11475409836065574,7.0,0.14754098360655737,9.0,,,,,,
|
||||||
|
0.0,0,64,512,4,1.7344,0,0.8906,4390.9277,3929.1513,4585.9,31767,1,4238.3757,1548471.0,22.0,0.9401,0.0,0.0,0.0599,0.2969,-0.001,0.0628,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,-0.0042,0.0,1.0,0.0,0.015,8617810003492864.0,1.0647384730359562e+19,0.0,0.8761,0.015,0.375,0.2969,16.7065,53.8438,576.3802,32.8,857.9675,281.5873,0.0907,6.1411,30.9645,191.9417,223.2241,,1633.3771,35.1631,192.1684,1368.8439,0.0,0.0,0,56,64,512,478,15,34,0.09375,6,0.390625,25,,,,,0.046875,3.0,0.0081,8.2677,,,,
|
||||||
|
0.0,0,64,512,4,1.5469,0,0.8281,4453.5156,4019.9141,4887.1172,31740,1,4404.608,1552609.0,34.0,0.9144,0.0,0.0,0.0856,0.5,0.0024,0.1244,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0436,-0.0001,0.0,1.0,0.0,0.0147,3.5539692025682697e+21,3.740553229295638e+24,0.0,0.9382,0.0158,0.6094,0.5,16.787,53.9058,575.7407,32.9,857.9675,282.2268,0.0908,6.1332,38.8151,208.6843,247.8062,,1501.615,35.5265,208.8997,1212.1417,0.0,0.0,0,57,61,488,455,11,29,0.03278688524590164,2,0.5081967213114754,31,0.01639344262295082,1.0,0.06557377049180328,4.0,0.08196721311475409,5.0,,,,,,
|
||||||
|
0.0,0,64,512,3,1.7031,0,0.875,3884.9863,4063.3161,3777.094,31844,1,3730.8638,1405568.0,16.0,0.9487,0.0,0.0,0.0513,0.377,0.0085,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0444,-0.0,0.0,1.0,0.0,0.0189,5.697833777259119e+21,6.752966200220271e+24,0.0,0.921,0.0151,0.5,0.377,17.2032,54.3492,574.8936,33.0,857.9675,283.0739,0.1129,6.2873,28.6923,178.963,207.988,,1448.7366,30.4954,179.1824,1203.9568,0.0,0.0,0,58,64,512,453,16,53,0.03125,2,0.671875,43,,,0.078125,5.0,0.125,8.0,0.0078,8.5897,0.015625,1.0,,
|
||||||
|
0.0,0,64,512,4,1.5938,0,0.875,3719.3281,3822.8142,3577.5139,31652,1,3793.566,1302863.0,9.0,0.9494,0.0,0.0,0.0506,0.5781,0.0061,0.1067,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0407,-0.0008,0.0,1.0,0.0,0.0169,2.916946366431232e+16,3.229335359815313e+19,0.0,0.8894,0.0141,0.6875,0.5781,17.3769,54.5306,573.5134,33.2,857.9675,284.4541,0.09,5.9757,31.3476,170.0586,201.7858,,1663.1725,33.9156,170.3478,1421.4882,0.0,0.0,0,59,62,496,460,15,35,0.04838709677419355,3,0.5,31,,,,,0.03225806451612903,2.0,,,,,0.016129032258064516,1.0
|
||||||
|
0.0,0,64,512,3,1.7188,0,0.875,4468.502,2738.5871,5586.5498,31799,1,4489.8129,1573926.0,34.0,0.9358,0.0,0.0,0.0642,0.3926,-0.0178,0.0646,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0453,-0.0,0.0,1.0,0.0,0.0139,5716752465920.0,7041144689000448.0,0.0,0.9321,0.0155,0.4688,0.3926,17.3024,54.4753,572.2839,33.3,857.9675,285.6836,0.108,6.3356,35.3958,205.7623,241.5432,6.2107,1507.1501,25.3052,206.039,1233.9571,0.0,0.0,0,60,62,496,439,20,57,0.11290322580645161,7,0.6935483870967742,43,,,,,0.08064516129032258,5.0,0.0079,8.517,,,,
|
||||||
|
21
training_logs/20260609_214045_metrics_job_653588.csv
Normal file
21
training_logs/20260609_214045_metrics_job_653588.csv
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,generate/tis/aligned_tokens,generate/tis/alignment_fail_count,generate/tis/exact_match_fraction,generate/tis/lcs_fallback_fraction,generate/tis/lcs_fallback_messages,generate/tis/unaligned_fraction,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,policy/rollout_train_prob_diff_mean,policy/rollout_train_prob_diff_std,policy/tis/imp_ratio_capped_fraction,policy/tis/imp_ratio_mean,policy/tis/log_ratio_abs_mean,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,tis/batch_skipped_no_logprobs,tis/skipped_fraction,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError
|
||||||
|
0.0,0,64,512,3,0.4219,0,0.2188,3685.4219,3527.8649,4019.75,16769,1,2307.0638,1257847.0,2.0,0.9993,0.0,0.0,0.0007,0.6797,-0.0006,0.0628,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0381,0.0,0.0,1.0,0.0,0.0162,1.0002,1.312,0.0,0.9678,0.0118,0.7544,0.6797,8.3562,50.2278,634.6806,26.0,857.9676,223.287,0.0717,3.4039,26.6535,126.6473,153.6151,4895.4874,32.7345,126.8502,4705.7335,0.0,0.0,0,61,54,432,360,23,69,1.2592592592592593,68,0.2222222222222222,12,0.1111111111111111,6,0.05555555555555555,3.0,,,,,
|
||||||
|
0.0,0,64,512,1,0.9219,0,0.9219,3966.3945,3777.5464,4071.4377,31765,1,4125.9274,1408431.0,14.0,0.9555,0.0,0.0,0.0445,0.3574,0.0054,0.0859,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0428,0.0,0.0,1.0,0.0,0.0312,4798878720.0,5626906804224.0,0.0,0.8313,0.0134,0.4688,0.3574,13.8742,50.7912,624.6771,27.2,857.9676,233.2905,0.0654,5.9309,30.7554,183.7737,214.8289,884.5667,32.4223,184.0075,631.377,0.0,0.0,0,62,56,448,392,17,55,0.7857142857142857,44,0.08928571428571429,5,0.14285714285714285,8,0.017857142857142856,1.0,8.4412,17.7924,,,
|
||||||
|
0.0,0,64,512,2,1.6094,0,0.9688,4484.8535,4127.1455,4739.6756,31775,1,4685.7602,1592891.0,27.0,0.914,0.0,0.0,0.086,0.416,0.0051,0.0481,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0438,0.0,0.0,1.0,0.0,0.014,1.49631712790315e+18,1.8779827487400106e+21,0.0,0.8914,0.0146,0.4531,0.416,14.0978,50.9457,622.3339,27.5,857.9676,235.6337,0.0902,5.9053,35.778,202.1171,238.2046,1021.0078,34.2374,202.3359,742.6567,0.0,0.0,0,63,65,520,475,16,39,0.4461538461538462,29,0.07692307692307693,5,0.07692307692307693,5,0.09230769230769231,6.0,,,,,
|
||||||
|
0.0,0,64,512,3,1.9219,0,0.8438,4086.5547,3298.7422,4704.1777,31791,1,3910.7476,1452474.0,19.0,0.9538,0.0,0.0,0.0462,0.4395,0.0004,0.0928,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0441,-0.0,0.0,1.0,0.0,0.0169,680327517306880.0,8.199202519226778e+17,0.0,0.9173,0.0141,0.5938,0.4395,14.9725,51.1755,619.0883,27.8,857.9676,238.8793,0.0605,5.8121,35.1041,187.1695,222.5884,1993.5726,35.8302,187.4233,1729.3387,0.0,0.0,0,64,60,480,427,15,53,0.75,45,0.15,9,0.05,3,,,0.0083,8.9351,,,
|
||||||
|
0.0,0,64,512,4,1.9531,0,0.8594,4482.5859,4188.5991,4726.175,31715,1,4433.6859,1614330.0,23.0,0.916,0.0,0.0,0.084,0.4531,0.0098,0.1061,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,0.0,0.0,1.0,0.0,0.0159,8.676132800661553e+23,8.132848488751659e+26,0.0,0.8934,0.0148,0.5781,0.4531,15.4079,51.6347,616.659,28.1,857.9676,241.3086,0.0896,5.6735,34.9766,204.9392,240.2083,1228.4685,32.6219,205.1416,949.9589,0.0,0.0,0,65,63,504,428,24,63,0.8253968253968254,52,0.12698412698412698,8,0.047619047619047616,3,0.20634920634920634,13.0,,,6.8083,,
|
||||||
|
0.0,0,64,512,5,1.9219,0,0.9062,4357.2324,4060.3786,4557.0752,31769,1,4668.8661,1580861.0,37.0,0.8957,0.0,0.0,0.1043,0.4023,0.0064,0.0877,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,-0.0002,0.0,1.0,0.0,0.0172,1.054274388152946e+23,1.2798084482887822e+26,0.0,0.8563,0.015,0.5312,0.4023,15.6097,51.7144,614.0408,28.4,857.9676,243.9268,0.0622,6.0018,33.4297,213.4844,247.2341,1499.3639,32.0364,213.7417,1214.0885,0.0,0.0,0,66,61,488,438,15,44,0.6065573770491803,37,0.09836065573770492,6,0.01639344262295082,1,0.09836065573770492,6.0,0.008,9.2733,,,
|
||||||
|
0.0,0,64,512,4,1.7344,0,0.8906,4419.9375,3553.4759,4918.4862,31735,1,4351.8887,1540086.0,31.0,0.9326,0.0,0.0,0.0674,0.3652,-0.0012,0.0879,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0436,-0.0053,0.0,1.0,0.0,0.0166,7106447872.0,7634057953280.0,0.0,0.9049,0.0147,0.4844,0.3652,15.652,51.8026,611.4824,28.7,857.9676,246.4852,0.0622,6.0487,33.6584,200.2845,234.3115,1811.2337,34.6927,200.5905,1536.1294,0.0,0.0,0,67,58,464,396,26,66,0.8275862068965517,48,0.25862068965517243,15,0.05172413793103448,3,0.034482758620689655,2.0,,,,0.05172413793103448,3.0
|
||||||
|
0.0,0,64,512,4,1.5156,0,0.8906,4066.9102,4048.2335,4078.5905,31837,1,4387.0666,1427586.0,25.0,0.9098,0.0,0.0,0.0902,0.3848,-0.0114,0.0895,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0438,-0.0,0.0,1.0,0.0,0.0141,3215390490492928.0,3.841650471617954e+18,0.0,0.8661,0.015,0.4688,0.3848,16.0492,52.3116,609.3674,29.0,857.9676,248.6002,0.0897,6.1724,31.0609,203.1937,234.6137,1248.5725,32.0514,203.4626,975.7293,0.0,0.0,0,68,63,504,455,13,46,0.6190476190476191,39,0.07936507936507936,5,0.031746031746031744,2,0.047619047619047616,3.0,0.0086,8.9362,,,
|
||||||
|
0.0,0,64,512,8,2.0156,0,0.9375,4388.7012,3827.7276,5121.5045,31658,1,3792.0489,1553676.0,18.0,0.9743,0.0,0.0,0.0257,0.5664,-0.0064,0.0802,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0449,0.0,0.0,1.0,0.0,0.0138,942792901656576.0,1.1535304966096814e+18,0.0,0.9097,0.0145,0.6406,0.5664,16.1733,52.4356,607.0387,29.2,857.9676,250.9289,0.1701,5.7865,33.4522,177.4228,211.3543,1399.7571,37.3866,177.7316,1145.224,0.0,0.0,0,69,62,496,433,24,58,0.5967741935483871,37,0.2903225806451613,18,0.03225806451612903,2,0.08064516129032258,5.0,,,,0.04838709677419355,3.0
|
||||||
|
0.0,0,64,512,3,1.5938,0,0.9219,4478.5273,4706.2548,4322.7138,31693,1,4388.6691,1567154.0,14.0,0.9596,0.0,0.0,0.0404,0.4062,0.0146,0.0753,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,-0.0017,0.0,1.0,0.0,0.0203,37698882502656.0,3.898363573436416e+16,0.0,0.8784,0.0142,0.5,0.4062,16.1904,52.4454,604.8707,29.5,857.9676,253.0969,0.0912,5.8848,33.2322,191.6949,225.2911,1712.3059,39.3967,191.9673,1441.7301,0.0,0.0,0,70,63,504,441,22,60,0.6349206349206349,40,0.14285714285714285,9,0.19047619047619047,12,0.047619047619047616,3.0,0.009,9.5302,6.3806,,
|
||||||
|
0.0,0,64,512,4,1.6094,0,0.8594,3870.6582,3993.4078,3804.6757,31660,1,3994.7037,1414538.0,14.0,0.939,0.0,0.0,0.061,0.3496,-0.014,0.1576,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0445,-0.0001,0.0,1.0,0.0,0.0232,6.827100124823203e+21,8.119763290842973e+24,0.0,0.8805,0.014,0.5,0.3496,16.2904,52.5068,603.2638,29.7,857.9676,254.7038,0.064,5.8584,27.7403,178.662,206.7592,1436.6431,32.111,178.9542,1191.9108,0.0,0.0,0,71,63,504,457,19,47,0.6507936507936508,41,0.06349206349206349,4,0.031746031746031744,2,,,,,,,
|
||||||
|
0.0,0,64,512,3,1.6094,0,0.875,4486.5332,3847.2383,5028.8953,31774,1,4668.9327,1632410.0,30.0,0.8736,0.0,0.0,0.1264,0.459,0.004,0.0411,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0437,0.0,0.0,1.0,0.0,0.0117,2.0079791588560077e+17,2.5626006126882246e+20,0.0,0.9135,0.0143,0.5,0.459,16.2309,52.5107,603.6477,29.6,857.9676,254.3199,0.063,5.8585,33.7104,209.3972,243.4676,1574.2016,33.0528,209.6937,1291.8188,0.0,0.0,0,72,60,480,436,12,43,0.55,33,0.11666666666666667,7,0.08333333333333333,5,0.016666666666666666,1.0,0.0102,9.398,,0.016666666666666666,1.0
|
||||||
|
0.0,0,64,512,3,1.625,0,0.9219,4030.5508,3747.5877,4257.7183,31764,1,3919.6801,1430904.0,31.0,0.9122,0.0,0.0,0.0878,0.4453,-0.0008,0.0867,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0396,-0.0001,0.0,1.0,0.0,0.0184,436822212608.0,400335680045056.0,0.0,0.9088,0.0133,0.5469,0.4453,16.4671,52.6346,602.5194,29.8,857.9676,255.4482,0.0953,5.9026,29.5781,177.2825,207.2711,1411.7807,48.8291,177.5971,1149.7701,0.0,0.0,0,73,64,512,477,18,30,0.375,24,0.046875,3,0.046875,3,0.078125,5.0,,,,,
|
||||||
|
0.0,0,64,512,6,1.9062,0,0.9375,4928.0742,5089.388,4838.3465,31829,1,4458.2535,1709041.0,22.0,0.9236,0.0,0.0,0.0764,0.3574,0.028,0.1375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0459,0.0,0.0,1.0,0.0,0.0152,30543561359360.0,3.860194628573594e+16,0.0,0.941,0.0148,0.4844,0.3574,16.4362,52.6385,601.1992,29.9,857.9676,256.7684,0.0942,6.2333,36.9222,215.6981,253.0054,1584.8429,35.8218,215.9884,1289.7748,0.0,0.0,0,74,60,480,413,18,57,0.9833333333333333,59,0.08333333333333333,5,0.016666666666666666,1,0.16666666666666666,10.0,0.0087,9.1809,,,
|
||||||
|
0.0,0,64,512,4,2.0469,1,1.0,4001.4727,3601.346,4281.9601,31741,1,3888.6649,1397275.0,23.0,0.9487,0.0,0.0,0.0513,0.4121,0.0048,0.063,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0436,-0.0026,0.0,1.0,0.0,0.0135,8.207850865374003e+16,9.702147778100068e+19,0.0,0.8586,0.0144,0.5312,0.4121,16.4568,52.7637,599.1791,30.2,857.9676,258.7885,0.0656,5.9013,30.0436,169.248,199.6035,1374.3084,36.1758,169.4935,1132.5895,0.0,0.0,0,75,63,504,422,22,64,0.8253968253968254,52,0.09523809523809523,6,0.1111111111111111,7,0.2857142857142857,18.0,,,6.5286,,
|
||||||
|
0.0,0,64,512,4,2.4219,2,1.0,4137.8809,3989.7778,4218.1777,31802,1,4719.6405,1531507.0,24.0,0.8609,0.0,0.0,0.1391,0.3516,0.0177,0.1025,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0403,-0.0055,0.0,1.0,0.0,0.0169,3.817858620540846e+17,4.6935484414353107e+20,0.0,0.8424,0.0134,0.4531,0.3516,16.7703,53.0803,597.8522,30.3,857.9676,260.1154,0.0901,7.0109,36.2508,203.9976,240.5622,1688.1237,39.0785,204.2208,1401.4078,0.0,0.0,0,76,62,496,434,20,62,0.8548387096774194,53,0.16129032258064516,10,0.016129032258064516,1,,,0.0106,9.297,,,
|
||||||
|
0.0,0,64,512,4,3.1875,3,1.0,4233.2773,3857.1,4420.2661,31765,1,4184.3433,1522291.0,17.0,0.9239,0.0,0.0,0.0761,0.332,0.0005,0.0567,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0415,-0.0,0.0,1.0,0.0,0.0102,2.0191042252585673e+22,2.49119404899192e+25,0.0,0.8799,0.0145,0.4219,0.332,16.9553,53.3296,595.7158,30.6,857.9676,262.2518,0.0855,5.9291,31.0768,185.4559,216.9091,1561.9809,30.9184,185.7463,1308.1743,0.0,0.0,0,77,62,496,395,33,95,0.9193548387096774,57,0.5806451612903226,36,0.04838709677419355,3,0.0967741935483871,6.0,,,,,
|
||||||
|
0.0,0,64,512,5,4.4219,4,1.0,4333.3574,4615.6172,4239.2708,31662,1,4448.1268,1530711.0,16.0,0.9692,0.0,0.0,0.0308,0.25,0.0094,0.0823,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0453,0.0,0.0,1.0,0.0,0.0122,1.5951538773913764e+21,1.973554746569748e+24,0.0,0.8063,0.0155,0.3594,0.25,17.0168,53.3903,595.507,30.6,857.9676,262.4606,0.0912,5.6163,32.4987,196.5511,229.4039,1882.9354,24.8707,196.8133,1622.983,0.0,0.0,0,78,96,768,690,25,71,0.53125,51,0.1875,18,0.020833333333333332,2,0.07291666666666667,7.0,0.0105,9.6184,,,
|
||||||
|
0.0,0,64,512,0,0.0,0,0.0,3256.0234,2826.6427,4613.9837,13362,1,1838.6673,1104019.0,5.0,0.996,0.0,0.0,0.004,0.7598,-0.0018,0.0763,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0334,-0.0011,0.0,1.0,0.0,0.0149,8603.9307,9039291.0,0.0,1.0023,0.0111,0.8125,0.7598,16.4131,52.7717,596.1255,30.5,857.9676,261.842,0.0432,2.9452,20.7324,109.5379,130.5083,2969.965,32.266,109.7322,2804.1924,0.0,0.0,1,79,61,488,428,22,60,0.819672131147541,50,0.19672131147540983,12,0.03278688524590164,2,,,,,,0.03278688524590164,2.0
|
||||||
|
0.0,0,64,512,1,0.7188,0,0.7188,4464.1445,4825.3981,4220.9477,31774,1,4860.2721,1603625.0,28.0,0.9096,0.0,0.0,0.0904,0.4023,0.0085,0.1165,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0434,-0.0001,0.0,1.0,0.0,0.0207,2.0403219462220677e+19,2.583743478900106e+22,0.0,0.8802,0.0142,0.5312,0.4023,17.2875,53.606,593.67,30.8,857.9676,264.2975,0.092,5.9528,32.0153,208.4403,240.7613,2243.0022,39.1203,208.6536,1957.1655,0.0,0.0,1,80,57,456,397,21,54,0.8070175438596491,46,0.08771929824561403,5,0.05263157894736842,3,0.08771929824561403,5.0,0.0085,10.2197,6.298,,
|
||||||
|
2
training_logs/20260609_214045_metrics_job_653589.csv
Normal file
2
training_logs/20260609_214045_metrics_job_653589.csv
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,generate/tis/aligned_tokens,generate/tis/alignment_fail_count,generate/tis/exact_match_fraction,generate/tis/lcs_fallback_fraction,generate/tis/lcs_fallback_messages,generate/tis/unaligned_fraction,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,policy/rollout_train_prob_diff_mean,policy/rollout_train_prob_diff_std,policy/tis/imp_ratio_capped_fraction,policy/tis/imp_ratio_mean,policy/tis/log_ratio_abs_mean,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/cleanup_old_checkpoints,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/save_checkpoints,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,tis/batch_skipped_no_logprobs,tis/skipped_fraction,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked
|
||||||
|
0.0,0,64,512,0,0.0,0,0.0,3458.0977,3170.4289,4497.3333,17781,1,2176.1485,1160818.0,8.0,0.9962,0.0,0.0,0.0038,0.7832,0.0044,0.0294,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.033,-0.002,0.0,1.0,0.0,0.0092,12.0706,11267.6396,0.0,1.0022,0.0104,0.7969,0.7832,8.6151,50.2076,591.2495,31.1,857.9676,266.7181,13.1363,0.0741,3.6039,30.4349,124.3234,155.0454,21.5484,3316.0311,31.9403,124.5359,3125.441,0.0,0.0,1,82,3,24,24,0,0
|
||||||
|
422
training_logs/20260609_214045_metrics_report.md
Normal file
422
training_logs/20260609_214045_metrics_report.md
Normal file
@@ -0,0 +1,422 @@
|
|||||||
|
# SkyRL Training Metrics Analysis
|
||||||
|
|
||||||
|
Generated from 5 log files
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
| Log File | Total Steps | Metric Blocks | Final Reward (mean) | Final Reward (max) | Total Time (s) |
|
||||||
|
|----------|-------------|---------------|---------------------|-------------------|----------------|
|
||||||
|
| job_653585 | 14 | 14 | 0.4019 | 0.5645 | 35492.1 |
|
||||||
|
| job_653586 | 34 | 20 | 0.3792 | 0.6133 | 42208.0 |
|
||||||
|
| job_653587 | 60 | 26 | 0.4408 | 0.5781 | 41323.7 |
|
||||||
|
| job_653588 | 80 | 20 | 0.4295 | 0.7598 | 35422.1 |
|
||||||
|
| job_653589 | 82 | 1 | 0.7832 | 0.7832 | 3316.0 |
|
||||||
|
|
||||||
|
## Async Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:------------------------------|-----------:|---------:|------:|---------:|--------:|
|
||||||
|
| async/discard_rate | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| async/discarded_count | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| async/effective_batch_groups | 64 | 0 | 64 | 64 | 81 |
|
||||||
|
| async/effective_batch_samples | 512 | 0 | 512 | 512 | 81 |
|
||||||
|
| async/staleness_max | 3.4321 | 1.28392 | 0 | 8 | 81 |
|
||||||
|
| async/staleness_mean | 1.65336 | 0.609775 | 0 | 4.4219 | 81 |
|
||||||
|
| async/staleness_min | 0.148148 | 0.614636 | 0 | 4 | 81 |
|
||||||
|
| async/staleness_ratio | 0.841627 | 0.223298 | 0 | 1 | 81 |
|
||||||
|
|
||||||
|
## Generate Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:-------------------------------------|----------------:|---------------:|------------:|----------------:|--------:|
|
||||||
|
| generate/avg_num_tokens | 3986.2 | 804.39 | 1484.36 | 5270.01 | 81 |
|
||||||
|
| generate/avg_tokens_non_zero_rewards | 3999.99 | 543.159 | 2738.59 | 5697.48 | 81 |
|
||||||
|
| generate/avg_tokens_zero_rewards | 4077.34 | 1100.09 | 775.223 | 5697.65 | 81 |
|
||||||
|
| generate/max_num_tokens | 28006.5 | 5837.94 | 13362 | 31858 | 81 |
|
||||||
|
| generate/min_num_tokens | 1 | 0 | 1 | 1 | 81 |
|
||||||
|
| generate/std_num_tokens | 3737.82 | 710.017 | 1838.67 | 5068.11 | 81 |
|
||||||
|
| generate/tis/aligned_tokens | 1.40771e+06 | 292189 | 540011 | 1.95353e+06 | 81 |
|
||||||
|
| generate/tis/alignment_fail_count | 27.6173 | 24.1674 | 2 | 119 | 81 |
|
||||||
|
| generate/tis/exact_match_fraction | 0.948404 | 0.0347641 | 0.8463 | 0.9993 | 81 |
|
||||||
|
| generate/tis/lcs_fallback_fraction | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| generate/tis/lcs_fallback_messages | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| generate/tis/unaligned_fraction | 0.0515963 | 0.0347641 | 0.0007 | 0.1537 | 81 |
|
||||||
|
|
||||||
|
## Loss Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:----------------------------|----------:|---------:|--------:|-------:|--------:|
|
||||||
|
| loss/avg_final_rewards | 0.420311 | 0.108774 | 0.1621 | 0.7832 | 81 |
|
||||||
|
| loss/avg_raw_advantages | 0.0198173 | 0.046916 | -0.0268 | 0.2099 | 81 |
|
||||||
|
| loss/avg_raw_advantages_abs | 0.10669 | 0.045766 | 0.0294 | 0.2501 | 81 |
|
||||||
|
|
||||||
|
## Policy Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:-------------------------------------|-------------:|------------:|--------:|------------:|--------:|
|
||||||
|
| policy/final_loss | -0.000169136 | 0.000402692 | -0.0019 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_max | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_mean | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_p99 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos00 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos10 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos20 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos30 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos40 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos50 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos60 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos70 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos80 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/log_ratio_abs_pos90 | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/n_tokens_dp_gt_10pct | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/n_tokens_dp_gt_1pct | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/n_tokens_dp_gt_50pct | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/policy_entropy | 0.0500136 | 0.0150327 | 0.0251 | 0.0938 | 81 |
|
||||||
|
| policy/policy_loss | -0.0109025 | 0.0256364 | -0.1214 | 0.0003 | 81 |
|
||||||
|
| policy/policy_lr | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/policy_update_steps | 1 | 0 | 1 | 1 | 81 |
|
||||||
|
| policy/ppo_clip_ratio | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| policy/raw_grad_norm | 0.017084 | 0.00422949 | 0.0092 | 0.0312 | 81 |
|
||||||
|
| policy/rollout_train_prob_diff_mean | 3.20105e+22 | 1.82246e+23 | 1.0002 | 1.39973e+24 | 81 |
|
||||||
|
| policy/rollout_train_prob_diff_std | 3.58965e+25 | 2.09602e+26 | 1.312 | 1.70849e+27 | 81 |
|
||||||
|
| policy/tis/imp_ratio_capped_fraction | 1.23457e-06 | 1.11111e-05 | 0 | 0.0001 | 81 |
|
||||||
|
| policy/tis/imp_ratio_mean | 0.857601 | 0.128126 | 0.4014 | 1.0023 | 81 |
|
||||||
|
| policy/tis/log_ratio_abs_mean | 0.0188667 | 0.00900744 | 0.0083 | 0.0461 | 81 |
|
||||||
|
|
||||||
|
## Reward Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:----------------------|---------:|----------:|-------:|-------:|--------:|
|
||||||
|
| reward/avg_pass_at_8 | 0.540064 | 0.0939115 | 0.3438 | 0.8125 | 81 |
|
||||||
|
| reward/avg_raw_reward | 0.420311 | 0.108774 | 0.1621 | 0.7832 | 81 |
|
||||||
|
|
||||||
|
## System Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:------------------------|---------:|-------------:|---------:|---------:|--------:|
|
||||||
|
| system/process_rss_gb | 15.1484 | 1.86533 | 7.5378 | 17.3769 | 81 |
|
||||||
|
| system/process_vms_gb | 52.1818 | 1.04916 | 49.9637 | 54.5306 | 81 |
|
||||||
|
| system/ram_available_gb | 595.168 | 19.2446 | 551.906 | 634.681 | 81 |
|
||||||
|
| system/ram_percent | 30.6296 | 2.24552 | 26 | 35.7 | 81 |
|
||||||
|
| system/ram_total_gb | 857.968 | 4.98454e-05 | 857.967 | 857.968 | 81 |
|
||||||
|
| system/ram_used_gb | 262.8 | 19.2446 | 223.287 | 306.061 | 81 |
|
||||||
|
|
||||||
|
## Timing Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:--------------------------------------|-------------:|------------:|---------:|-----------:|--------:|
|
||||||
|
| timing/compute_advantages_and_returns | 0.0897247 | 0.105922 | 0.0323 | 0.8639 | 81 |
|
||||||
|
| timing/convert_to_training_input | 5.42322 | 1.13896 | 2.4443 | 7.6651 | 81 |
|
||||||
|
| timing/fwd_logprobs_values_reward | 31.8642 | 19.5819 | 14.0718 | 199.362 | 81 |
|
||||||
|
| timing/policy_train | 171.947 | 31.4651 | 89.3179 | 235.705 | 81 |
|
||||||
|
| timing/run_training | 204.138 | 36.2341 | 111.54 | 288.842 | 81 |
|
||||||
|
| timing/step | 1947.68 | 1358.35 | 608.871 | 11028.1 | 81 |
|
||||||
|
| timing/sync_weights | 31.7567 | 6.30151 | 18.1382 | 48.8291 | 81 |
|
||||||
|
| timing/train_critic_and_policy | 172.183 | 31.4877 | 89.4422 | 235.953 | 81 |
|
||||||
|
| timing/wait_for_generation_buffer | 1706.35 | 1363.3 | 352.922 | 10718.7 | 81 |
|
||||||
|
| timing/cleanup_old_checkpoints | 4.49109 | 21.9401 | 0.0072 | 139.958 | 41 |
|
||||||
|
| timing/save_checkpoints | 14.6602 | 22.1707 | 8.0671 | 149.994 | 41 |
|
||||||
|
| timing/save_hf_model | 7.06378 | 2.67737 | 5.7884 | 16.893 | 16 |
|
||||||
|
|
||||||
|
## Tis Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:------------------------------|-------:|------:|------:|------:|--------:|
|
||||||
|
| tis/batch_skipped_no_logprobs | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
| tis/skipped_fraction | 0 | 0 | 0 | 0 | 81 |
|
||||||
|
|
||||||
|
## Trainer Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:--------------------|----------:|----------:|------:|------:|--------:|
|
||||||
|
| trainer/epoch | 0.037037 | 0.190029 | 0 | 1 | 81 |
|
||||||
|
| trainer/global_step | 41.0123 | 23.5481 | 1 | 82 | 81 |
|
||||||
|
|
||||||
|
## Batch_Errors Metrics
|
||||||
|
|
||||||
|
| | Mean | Std | Min | Max | Count |
|
||||||
|
|:----------------------------------------------|------------:|------------:|-----------:|------------:|--------:|
|
||||||
|
| batch_errors/total_batches | 61.284 | 11.5198 | 3 | 121 | 81 |
|
||||||
|
| batch_errors/total_instances | 490.272 | 92.1584 | 24 | 968 | 81 |
|
||||||
|
| batch_errors/total_successful | 415.506 | 86.5525 | 24 | 690 | 81 |
|
||||||
|
| batch_errors/total_failed | 22.8889 | 15.9421 | 0 | 114 | 81 |
|
||||||
|
| batch_errors/total_masked | 53.6173 | 32.8792 | 0 | 216 | 81 |
|
||||||
|
| batch_errors/avg_VerifierTimeoutError | 0.367127 | 0.644218 | 0.015625 | 2.7 | 63 |
|
||||||
|
| batch_errors/total_VerifierTimeoutError | 24.1587 | 45.4107 | 1 | 231 | 63 |
|
||||||
|
| batch_errors/avg_AgentTimeoutError | 0.221598 | 0.442882 | 0.015625 | 2.01587 | 73 |
|
||||||
|
| batch_errors/total_AgentTimeoutError | 15.0685 | 33.7895 | 1 | 200 | 73 |
|
||||||
|
| batch_errors/avg_AgentSetupTimeoutError | 0.231264 | 0.20643 | 0.016129 | 0.730159 | 16 |
|
||||||
|
| batch_errors/total_AgentSetupTimeoutError | 15.125 | 13.4604 | 1 | 46 | 16 |
|
||||||
|
| batch_errors/avg_InvalidChatHistory | 0.573631 | 0.281376 | 0.0169492 | 1.25926 | 80 |
|
||||||
|
| batch_errors/total_InvalidChatHistory | 34.8 | 16.5073 | 1 | 68 | 80 |
|
||||||
|
| batch_errors/avg_ContextLengthExceededError | 0.091597 | 0.0813106 | 0.015625 | 0.580645 | 74 |
|
||||||
|
| batch_errors/total_ContextLengthExceededError | 5.72973 | 5.20615 | 1 | 36 | 74 |
|
||||||
|
| batch_errors/avg_DaytonaAuthenticationError | 0.0275673 | 0.0259453 | 0.015625 | 0.0952381 | 9 |
|
||||||
|
| batch_errors/total_DaytonaAuthenticationError | 1.88889 | 1.69148 | 1 | 6 | 9 |
|
||||||
|
| batch_errors/avg_Timeout | 0.015873 | nan | 0.015873 | 0.015873 | 1 |
|
||||||
|
| batch_errors/total_Timeout | 1 | nan | 1 | 1 | 1 |
|
||||||
|
| batch_errors/avg_DaytonaError | 0.0940174 | 0.08989 | 0.015625 | 0.380952 | 38 |
|
||||||
|
| batch_errors/total_DaytonaError | 5.76316 | 5.66847 | 1 | 24 | 38 |
|
||||||
|
|
||||||
|
## Training Progression by Log
|
||||||
|
|
||||||
|
### job_653585
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 1 | 0.3398 | 0.6406 | 0.000000 | -0.0019 | 11028.1 | 10718.7 |
|
||||||
|
| 2 | 0.2090 | 0.4844 | 0.000000 | -0.0009 | 3675.2 | 3494.5 |
|
||||||
|
| 3 | 0.2266 | 0.5156 | 0.000000 | -0.0013 | 3304.1 | 3148.8 |
|
||||||
|
| 4 | 0.2754 | 0.5312 | 0.000000 | -0.0013 | 3238.5 | 3089.0 |
|
||||||
|
| 5 | 0.4766 | 0.6250 | 0.000000 | -0.0006 | 1062.3 | 897.3 |
|
||||||
|
| 6 | 0.5332 | 0.6562 | 0.000000 | -0.0004 | 1290.7 | 1072.8 |
|
||||||
|
| 7 | 0.4629 | 0.5781 | 0.000000 | -0.0001 | 2018.6 | 1788.6 |
|
||||||
|
| 8 | 0.3965 | 0.4688 | 0.000000 | 0.0000 | 1224.5 | 965.7 |
|
||||||
|
| 9 | 0.4121 | 0.5469 | 0.000000 | 0.0000 | 1481.3 | 1223.5 |
|
||||||
|
| 10 | 0.5645 | 0.7031 | 0.000000 | -0.0000 | 1344.8 | 1126.3 |
|
||||||
|
| 11 | 0.4258 | 0.5312 | 0.000000 | -0.0000 | 1608.6 | 1359.8 |
|
||||||
|
| 12 | 0.3945 | 0.5156 | 0.000000 | 0.0000 | 1419.0 | 1162.0 |
|
||||||
|
| 13 | 0.5293 | 0.6719 | 0.000000 | 0.0000 | 1174.5 | 939.0 |
|
||||||
|
| 14 | 0.3809 | 0.5000 | 0.000000 | 0.0000 | 1621.8 | 1379.8 |
|
||||||
|
|
||||||
|
### job_653586
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 15 | 0.6133 | 0.6500 | 0.000000 | -0.0000 | 4387.1 | 4155.6 |
|
||||||
|
| 16 | 0.2305 | 0.3438 | 0.000000 | -0.0000 | 940.4 | 626.3 |
|
||||||
|
| 17 | 0.4629 | 0.6094 | 0.000000 | -0.0000 | 866.8 | 587.6 |
|
||||||
|
| 18 | 0.4062 | 0.5312 | 0.000000 | -0.0000 | 2168.7 | 1866.0 |
|
||||||
|
| 19 | 0.4102 | 0.5156 | 0.000000 | 0.0000 | 1196.7 | 924.2 |
|
||||||
|
| 20 | 0.3809 | 0.4531 | 0.000000 | -0.0000 | 1352.0 | 1092.1 |
|
||||||
|
| 21 | 0.3828 | 0.5000 | 0.000000 | -0.0001 | 1653.9 | 1381.6 |
|
||||||
|
| 22 | 0.3711 | 0.5156 | 0.000000 | -0.0000 | 1708.2 | 1478.8 |
|
||||||
|
| 23 | 0.4746 | 0.5781 | 0.000000 | -0.0002 | 2873.6 | 2651.6 |
|
||||||
|
| 24 | 0.1816 | 0.3750 | 0.000000 | -0.0005 | 4553.2 | 4339.2 |
|
||||||
|
| 25 | 0.1621 | 0.3906 | 0.000000 | -0.0006 | 2911.5 | 2775.5 |
|
||||||
|
| 26 | 0.3945 | 0.6875 | 0.000000 | -0.0017 | 2183.5 | 2048.2 |
|
||||||
|
| 27 | 0.4180 | 0.5938 | 0.000000 | -0.0010 | 1456.9 | 1257.3 |
|
||||||
|
| 28 | 0.3145 | 0.5312 | 0.000000 | -0.0011 | 3692.4 | 3488.9 |
|
||||||
|
| 29 | 0.3105 | 0.4375 | 0.000000 | -0.0007 | 2121.6 | 1954.8 |
|
||||||
|
| 30 | 0.4941 | 0.7031 | 0.000000 | -0.0008 | 1800.9 | 1599.0 |
|
||||||
|
| 31 | 0.4023 | 0.5312 | 0.000000 | -0.0002 | 1635.7 | 1413.2 |
|
||||||
|
| 32 | 0.3926 | 0.5156 | 0.000000 | -0.0000 | 1896.1 | 1646.5 |
|
||||||
|
| 33 | 0.4258 | 0.4844 | 0.000000 | -0.0000 | 1165.0 | 952.0 |
|
||||||
|
| 34 | 0.3555 | 0.4844 | 0.000000 | 0.0000 | 1643.8 | 1436.7 |
|
||||||
|
|
||||||
|
### job_653587
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 35 | 0.4922 | 0.6066 | 0.000000 | -0.0000 | 4940.9 | 4656.1 |
|
||||||
|
| 36 | 0.3652 | 0.4844 | 0.000000 | 0.0000 | 608.9 | 352.9 |
|
||||||
|
| 37 | 0.5430 | 0.6250 | 0.000000 | 0.0000 | 1222.0 | 998.8 |
|
||||||
|
| 38 | 0.4531 | 0.5781 | 0.000000 | -0.0000 | 1845.9 | 1590.6 |
|
||||||
|
| 39 | 0.4766 | 0.5781 | 0.000000 | -0.0000 | 1216.3 | 944.2 |
|
||||||
|
| 40 | 0.4668 | 0.5312 | 0.000000 | 0.0000 | 1338.9 | 1117.5 |
|
||||||
|
| 41 | 0.4609 | 0.5781 | 0.000000 | -0.0000 | 1768.8 | 1538.6 |
|
||||||
|
| 42 | 0.3809 | 0.4375 | 0.000000 | 0.0000 | 1528.5 | 1260.4 |
|
||||||
|
| 43 | 0.4141 | 0.5000 | 0.000000 | 0.0000 | 1429.6 | 1137.8 |
|
||||||
|
| 44 | 0.5078 | 0.6094 | 0.000000 | 0.0000 | 1259.6 | 1037.9 |
|
||||||
|
| 45 | 0.4785 | 0.5312 | 0.000000 | 0.0000 | 1402.9 | 1151.6 |
|
||||||
|
| 46 | 0.5059 | 0.5781 | 0.000000 | -0.0000 | 1235.1 | 992.3 |
|
||||||
|
| 47 | 0.5352 | 0.6250 | 0.000000 | 0.0000 | 1375.9 | 1112.2 |
|
||||||
|
| 48 | 0.5000 | 0.5938 | 0.000000 | 0.0000 | 1598.9 | 1377.6 |
|
||||||
|
| 49 | 0.4395 | 0.5781 | 0.000000 | 0.0000 | 1442.6 | 1224.6 |
|
||||||
|
| 50 | 0.3223 | 0.3906 | 0.000000 | -0.0000 | 1863.3 | 1570.9 |
|
||||||
|
| 51 | 0.3555 | 0.4844 | 0.000000 | -0.0000 | 1460.0 | 1228.4 |
|
||||||
|
| 52 | 0.3750 | 0.4688 | 0.000000 | -0.0000 | 1339.8 | 1090.1 |
|
||||||
|
| 53 | 0.4746 | 0.5625 | 0.000000 | 0.0000 | 1705.9 | 1492.1 |
|
||||||
|
| 54 | 0.4082 | 0.4844 | 0.000000 | -0.0000 | 1372.4 | 1169.5 |
|
||||||
|
| 55 | 0.3613 | 0.4688 | 0.000000 | 0.0000 | 1613.6 | 1341.7 |
|
||||||
|
| 56 | 0.2969 | 0.3750 | 0.000000 | -0.0001 | 1633.4 | 1368.8 |
|
||||||
|
| 57 | 0.5000 | 0.6094 | 0.000000 | -0.0000 | 1501.6 | 1212.1 |
|
||||||
|
| 58 | 0.3770 | 0.5000 | 0.000000 | -0.0000 | 1448.7 | 1204.0 |
|
||||||
|
| 59 | 0.5781 | 0.6875 | 0.000000 | -0.0000 | 1663.2 | 1421.5 |
|
||||||
|
| 60 | 0.3926 | 0.4688 | 0.000000 | -0.0000 | 1507.2 | 1234.0 |
|
||||||
|
|
||||||
|
### job_653588
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 61 | 0.6797 | 0.7544 | 0.000000 | 0.0000 | 4895.5 | 4705.7 |
|
||||||
|
| 62 | 0.3574 | 0.4688 | 0.000000 | 0.0000 | 884.6 | 631.4 |
|
||||||
|
| 63 | 0.4160 | 0.4531 | 0.000000 | 0.0000 | 1021.0 | 742.7 |
|
||||||
|
| 64 | 0.4395 | 0.5938 | 0.000000 | -0.0000 | 1993.6 | 1729.3 |
|
||||||
|
| 65 | 0.4531 | 0.5781 | 0.000000 | 0.0000 | 1228.5 | 950.0 |
|
||||||
|
| 66 | 0.4023 | 0.5312 | 0.000000 | -0.0000 | 1499.4 | 1214.1 |
|
||||||
|
| 67 | 0.3652 | 0.4844 | 0.000000 | -0.0001 | 1811.2 | 1536.1 |
|
||||||
|
| 68 | 0.3848 | 0.4688 | 0.000000 | -0.0000 | 1248.6 | 975.7 |
|
||||||
|
| 69 | 0.5664 | 0.6406 | 0.000000 | 0.0000 | 1399.8 | 1145.2 |
|
||||||
|
| 70 | 0.4062 | 0.5000 | 0.000000 | -0.0000 | 1712.3 | 1441.7 |
|
||||||
|
| 71 | 0.3496 | 0.5000 | 0.000000 | -0.0000 | 1436.6 | 1191.9 |
|
||||||
|
| 72 | 0.4590 | 0.5000 | 0.000000 | 0.0000 | 1574.2 | 1291.8 |
|
||||||
|
| 73 | 0.4453 | 0.5469 | 0.000000 | -0.0000 | 1411.8 | 1149.8 |
|
||||||
|
| 74 | 0.3574 | 0.4844 | 0.000000 | 0.0000 | 1584.8 | 1289.8 |
|
||||||
|
| 75 | 0.4121 | 0.5312 | 0.000000 | -0.0000 | 1374.3 | 1132.6 |
|
||||||
|
| 76 | 0.3516 | 0.4531 | 0.000000 | -0.0001 | 1688.1 | 1401.4 |
|
||||||
|
| 77 | 0.3320 | 0.4219 | 0.000000 | -0.0000 | 1562.0 | 1308.2 |
|
||||||
|
| 78 | 0.2500 | 0.3594 | 0.000000 | 0.0000 | 1882.9 | 1623.0 |
|
||||||
|
| 79 | 0.7598 | 0.8125 | 0.000000 | -0.0000 | 2970.0 | 2804.2 |
|
||||||
|
| 80 | 0.4023 | 0.5312 | 0.000000 | -0.0000 | 2243.0 | 1957.2 |
|
||||||
|
|
||||||
|
### job_653589
|
||||||
|
|
||||||
|
| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
|
||||||
|
|------|--------|--------|-----|------|---------------|-------------|
|
||||||
|
| 82 | 0.7832 | 0.7969 | 0.000000 | -0.0000 | 3316.0 | 3125.4 |
|
||||||
|
|
||||||
|
## Timing Analysis
|
||||||
|
|
||||||
|
### Average Time Breakdown (% of step time)
|
||||||
|
|
||||||
|
| Component | Avg % of Step Time |
|
||||||
|
|-----------|-------------------|
|
||||||
|
| wait_for_generation_buffer | 84.5% |
|
||||||
|
| run_training | 13.1% |
|
||||||
|
| train_critic_and_policy | 11.2% |
|
||||||
|
| policy_train | 11.2% |
|
||||||
|
| sync_weights | 2.1% |
|
||||||
|
| fwd_logprobs_values_reward | 1.9% |
|
||||||
|
| save_checkpoints | 0.8% |
|
||||||
|
| save_hf_model | 0.4% |
|
||||||
|
| convert_to_training_input | 0.4% |
|
||||||
|
| cleanup_old_checkpoints | 0.2% |
|
||||||
|
| compute_advantages_and_returns | 0.0% |
|
||||||
|
|
||||||
|
## Cross-Log Comparison
|
||||||
|
|
||||||
|
| Log | Avg Reward | Pass@8 | Step Time (s) | Gen Wait Time (s) | Avg Tokens | Staleness |
|
||||||
|
|-----|------|------|------|------|------|------|
|
||||||
|
| job_653585 | 0.4019 | 0.5692 | 2535.1497 | 2311.8403 | 3748.4641 | 1.6038 |
|
||||||
|
| job_653586 | 0.3792 | 0.5216 | 2110.4003 | 1883.7625 | 3771.8087 | 1.6047 |
|
||||||
|
| job_653587 | 0.4408 | 0.5360 | 1589.3730 | 1339.4724 | 4128.8501 | 1.7007 |
|
||||||
|
| job_653588 | 0.4295 | 0.5307 | 1771.1060 | 1511.0877 | 4207.9546 | 1.7578 |
|
||||||
|
| job_653589 | 0.7832 | 0.7969 | 3316.0311 | 3125.4410 | 3458.0977 | 0.0000 |
|
||||||
|
|
||||||
|
## vLLM Inference Engine Analysis
|
||||||
|
|
||||||
|
Metrics from vLLM stat loggers (V1LoggingStatLoggerFixed).
|
||||||
|
|
||||||
|
> **Note**: Ray deduplicates similar log messages with `[repeated Nx across cluster]`,
|
||||||
|
> so we typically capture stats from one engine per timestamp. The stats shown are
|
||||||
|
> **per-engine** values. Multiply by num_inference_engines for cluster-wide estimates.
|
||||||
|
|
||||||
|
### Summary by Log (Per-Engine Stats)
|
||||||
|
|
||||||
|
| Log | Avg Running/Engine | Avg Waiting/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % |
|
||||||
|
|-----|-------------------|-------------------|--------------------------|----------------|------------------|
|
||||||
|
| job_653585 | 5.9 | 0.0 | 256.7 tok/s | 25.8% | 86.8% |
|
||||||
|
| job_653586 | 8.3 | 0.2 | 353.1 tok/s | 36.7% | 72.5% |
|
||||||
|
| job_653587 | 13.1 | 1.0 | 535.5 tok/s | 58.8% | 62.0% |
|
||||||
|
| job_653588 | 11.9 | 1.3 | 488.1 tok/s | 53.3% | 66.6% |
|
||||||
|
| job_653589 | 10.5 | 0.3 | 431.1 tok/s | 43.4% | 80.4% |
|
||||||
|
|
||||||
|
### Utilization Analysis (Per-Engine)
|
||||||
|
|
||||||
|
Key indicators of inference engine utilization:
|
||||||
|
|
||||||
|
- **Running requests/engine**: Concurrent requests being processed by each engine
|
||||||
|
- **Waiting requests**: Requests queued (0 = engine not saturated, has spare capacity)
|
||||||
|
- **Generation throughput**: Decode tokens/sec per engine
|
||||||
|
- 8B model on H100 can do **1000+ tok/s** when saturated
|
||||||
|
- If seeing <300 tok/s with 0 waiting, engine is **starved for requests**
|
||||||
|
|
||||||
|
#### job_653585
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=5.9, max=24
|
||||||
|
- **Waiting requests**: avg=0.0, max=10
|
||||||
|
- **Generation throughput/engine**: avg=256.7 tok/s, max=991.1 tok/s
|
||||||
|
- **KV cache usage**: avg=25.8%
|
||||||
|
- **Prefix cache hit rate**: avg=86.8%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
#### job_653586
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=8.3, max=24
|
||||||
|
- **Waiting requests**: avg=0.2, max=31
|
||||||
|
- **Generation throughput/engine**: avg=353.1 tok/s, max=961.0 tok/s
|
||||||
|
- **KV cache usage**: avg=36.7%
|
||||||
|
- **Prefix cache hit rate**: avg=72.5%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
#### job_653587
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=13.1, max=24
|
||||||
|
- **Waiting requests**: avg=1.0, max=35
|
||||||
|
- **Generation throughput/engine**: avg=535.5 tok/s, max=1039.7 tok/s
|
||||||
|
- **KV cache usage**: avg=58.8%
|
||||||
|
- **Prefix cache hit rate**: avg=62.0%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
#### job_653588
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=11.9, max=24
|
||||||
|
- **Waiting requests**: avg=1.3, max=37
|
||||||
|
- **Generation throughput/engine**: avg=488.1 tok/s, max=995.5 tok/s
|
||||||
|
- **KV cache usage**: avg=53.3%
|
||||||
|
- **Prefix cache hit rate**: avg=66.6%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
#### job_653589
|
||||||
|
|
||||||
|
- **Running requests/engine**: avg=10.5, max=24
|
||||||
|
- **Waiting requests**: avg=0.3, max=17
|
||||||
|
- **Generation throughput/engine**: avg=431.1 tok/s, max=943.7 tok/s
|
||||||
|
- **KV cache usage**: avg=43.4%
|
||||||
|
- **Prefix cache hit rate**: avg=80.4%
|
||||||
|
- ✅ **Well-utilized**: Engines saturated (waiting > 0)
|
||||||
|
|
||||||
|
## Trial-Level Analysis (from result.json)
|
||||||
|
|
||||||
|
Total trials parsed: 44755
|
||||||
|
|
||||||
|
### Turn Count Statistics
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| Mean | 4.9 |
|
||||||
|
| Median | 5.0 |
|
||||||
|
| Std | 2.3 |
|
||||||
|
| Min | 1 |
|
||||||
|
| Max | 70 |
|
||||||
|
| Count | 44394 |
|
||||||
|
|
||||||
|
### Exception Distribution
|
||||||
|
|
||||||
|
| Exception Type | Count | % |
|
||||||
|
|---------------|-------|---|
|
||||||
|
| No exception | 29160 | 65.2% |
|
||||||
|
| AgentTimeoutError | 13102 | 29.3% |
|
||||||
|
| VerifierTimeoutError | 1579 | 3.5% |
|
||||||
|
| ContextLengthExceededError | 428 | 1.0% |
|
||||||
|
| AgentSetupTimeoutError | 242 | 0.5% |
|
||||||
|
| DaytonaError | 226 | 0.5% |
|
||||||
|
| DaytonaAuthenticationError | 17 | 0.0% |
|
||||||
|
| Timeout | 1 | 0.0% |
|
||||||
|
|
||||||
|
### Turn Count by Exception Type
|
||||||
|
|
||||||
|
| Exception Type | Mean Turns | Median Turns | Count |
|
||||||
|
|---------------|-----------|-------------|-------|
|
||||||
|
| No exception | 5.5 | 5.0 | 29160 |
|
||||||
|
| VerifierTimeoutError | 5.3 | 5.0 | 1579 |
|
||||||
|
| Timeout | 5.0 | 5.0 | 1 |
|
||||||
|
| ContextLengthExceededError | 4.8 | 4.0 | 428 |
|
||||||
|
| AgentTimeoutError | 3.5 | 3.0 | 13102 |
|
||||||
|
| DaytonaError | 3.2 | 2.0 | 108 |
|
||||||
|
| DaytonaAuthenticationError | 3.0 | 3.0 | 16 |
|
||||||
|
|
||||||
|
### Turn Count by Outcome
|
||||||
|
|
||||||
|
| Outcome | Mean Turns | Median Turns | Count |
|
||||||
|
|---------|-----------|-------------|-------|
|
||||||
|
| Success | 5.1 | 5.0 | 18816 |
|
||||||
|
| Failure | 4.8 | 4.0 | 22337 |
|
||||||
|
|
||||||
|
### Reward Summary
|
||||||
|
|
||||||
|
- Mean reward: 0.4572
|
||||||
|
- Success rate: 45.7%
|
||||||
|
- Trials with reward data: 41153
|
||||||
|
|
||||||
82
training_logs/20260609_214045_metrics_table.csv
Normal file
82
training_logs/20260609_214045_metrics_table.csv
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
log_file,async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,generate/tis/aligned_tokens,generate/tis/alignment_fail_count,generate/tis/exact_match_fraction,generate/tis/lcs_fallback_fraction,generate/tis/lcs_fallback_messages,generate/tis/unaligned_fraction,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,policy/rollout_train_prob_diff_mean,policy/rollout_train_prob_diff_std,policy/tis/imp_ratio_capped_fraction,policy/tis/imp_ratio_mean,policy/tis/log_ratio_abs_mean,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,tis/batch_skipped_no_logprobs,tis/skipped_fraction,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_Timeout,batch_errors/total_Timeout,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,global_step
|
||||||
|
job_653585,0.0,0,64,512,0,0.0,0,0.0,1665.6621,3021.6782,967.5947,14153,1,2057.7122,573938.0,33.0,0.9487,0.0,0.0,0.0513,0.3398,0.2099,0.2501,-0.0019,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0656,-0.1214,0.0,1.0,0.0,0.0309,425844310016.0,310867954499584.0,0.0,0.5258,0.022,0.6406,0.3398,7.5378,49.9637,630.257,26.5,857.9676,227.7106,0.0376,2.4443,199.3619,89.3179,288.8421,11028.1246,18.1382,89.4422,10718.6996,0.0,0.0,0,1,121,968,497,114,216,1.9090909090909092,231.0,1.6528925619834711,200.0,0.2231404958677686,27.0,0.15702479338842976,19.0,0.09090909090909091,11.0,0.024793388429752067,3.0,,,,,,,,1
|
||||||
|
job_653585,0.0,0,64,512,1,1.0,1,1.0,2192.3809,5107.972,1422.0889,31541,1,3242.089,778056.0,90.0,0.8463,0.0,0.0,0.1537,0.209,0.14,0.236,-0.0009,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0639,-0.0569,0.0,1.0,0.0,0.0223,5135849414983680.0,3.8306663504564716e+18,0.0001,0.5252,0.0393,0.4844,0.209,13.664,50.6382,619.7108,27.8,857.9676,238.2568,0.0608,5.2456,18.5674,134.9773,153.7971,3675.2048,21.6594,135.1685,3494.5009,0.0,0.0,0,2,64,512,253,62,90,2.109375,135.0,1.25,80.0,0.5,32.0,0.109375,7.0,0.046875,3.0,0.03125,2.0,139.9582,149.9938,,,,,,2
|
||||||
|
job_653585,0.0,0,64,512,2,1.6094,0,0.9844,1974.7832,3706.1466,1467.6162,19867,1,2776.8866,687353.0,78.0,0.904,0.0,0.0,0.096,0.2266,0.1618,0.2338,-0.0013,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0563,-0.0823,0.0,1.0,0.0,0.0252,5712403677315072.0,4.576328473257181e+18,0.0,0.4977,0.0341,0.5156,0.2266,13.514,50.458,618.5668,27.9,857.9676,239.4008,0.0426,3.6209,18.1097,113.5752,131.9041,3304.1227,19.8368,113.7514,3148.7583,0.0,0.0,0,3,63,504,278,59,93,1.7619047619047619,111.0,1.3015873015873016,82.0,0.31746031746031744,20.0,0.14285714285714285,9.0,0.031746031746031744,2.0,0.015873015873015872,1.0,,,0.015873015873015872,1.0,,,,3
|
||||||
|
job_653585,0.0,0,64,512,3,1.7656,0,0.9375,1916.1016,3570.8227,1287.2183,26360,1,2567.8385,672216.0,119.0,0.8943,0.0,0.0,0.1057,0.2754,0.1458,0.2335,-0.0013,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0609,-0.081,0.0,1.0,0.0,0.0264,2.5092910058202726e+17,2.0297533524949546e+20,0.0,0.5539,0.0386,0.5312,0.2754,13.7533,50.7891,617.0518,28.1,857.9676,240.9158,0.0533,4.4556,16.0924,108.7123,125.0416,3238.4887,20.0299,108.8955,3088.9592,0.0,0.0,0,4,59,472,401,29,13,1.0,59.0,0.15254237288135594,9.0,0.0847457627118644,5.0,0.01694915254237288,1.0,0.06779661016949153,4.0,,,0.0083,12.9332,,,,,,4
|
||||||
|
job_653585,0.0,0,64,512,3,1.4531,0,0.9219,3224.291,3590.6844,2890.709,13914,1,2397.9485,1106110.0,103.0,0.9644,0.0,0.0,0.0356,0.4766,0.064,0.1543,-0.0006,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0788,-0.0412,0.0,1.0,0.0,0.017,2.1654929612682035e+18,2.119323756633866e+21,0.0,0.8649,0.0405,0.625,0.4766,13.7164,50.7537,614.8819,28.3,857.9676,243.0857,0.0323,2.8211,19.9051,118.6808,138.8115,1062.3118,23.3888,118.8737,897.2872,0.0,0.0,0,5,63,504,460,23,14,0.42857142857142855,27.0,0.06349206349206349,4.0,0.047619047619047616,3.0,0.09523809523809523,6.0,0.047619047619047616,3.0,,,,,,,16.893,0.015873015873015872,1.0,5
|
||||||
|
job_653585,0.0,0,64,512,4,1.6406,0,0.8594,4239.8125,4633.2308,3790.4268,22031,1,3300.5798,1457215.0,112.0,0.9654,0.0,0.0,0.0346,0.5332,0.044,0.102,-0.0004,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0853,-0.025,0.0,1.0,0.0,0.0138,6.141298557442851e+18,5.070549340210866e+21,0.0,0.9245,0.0456,0.6562,0.5332,14.4743,51.5385,613.4079,28.5,857.9676,244.5597,0.0452,4.2242,25.388,159.3519,185.0213,1290.7407,28.6557,159.5877,1072.8381,0.0,0.0,0,6,58,464,445,11,12,0.1206896551724138,7.0,,,,,0.10344827586206896,6.0,0.034482758620689655,2.0,,,0.0083,10.3463,,,,0.06896551724137931,4.0,6
|
||||||
|
job_653585,0.0,0,64,512,4,1.8906,0,0.8906,4397.6465,4166.2911,4597.0327,25556,1,3231.8595,1555192.0,95.0,0.9507,0.0,0.0,0.0493,0.4629,0.0201,0.1187,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0913,-0.0045,0.0,1.0,0.0,0.0167,4.5730908733082645e+20,5.0551306138646176e+23,0.0,0.9745,0.0461,0.5781,0.4629,14.6233,51.6414,612.7198,28.6,857.9676,245.2477,0.0504,4.7036,26.7455,163.7089,190.7489,2018.6125,34.5773,163.9526,1788.5789,0.0,0.0,0,7,60,480,458,10,12,0.15,9.0,,,0.016666666666666666,1.0,0.16666666666666666,10.0,0.016666666666666666,1.0,,,,,,,,0.016666666666666666,1.0,7
|
||||||
|
job_653585,0.0,0,64,512,4,2.1406,0,0.9531,5137.9238,4980.2167,5241.5307,31761,1,4015.1526,1850673.0,87.0,0.9135,0.0,0.0,0.0865,0.3965,0.0072,0.0726,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0938,0.0003,0.0,1.0,0.0,0.0119,2.930747012383905e+21,3.2028701271085374e+24,0.0,0.9696,0.046,0.4688,0.3965,15.9055,52.3258,610.4556,28.8,857.9676,247.512,0.0614,6.5073,32.8918,190.3358,223.5354,1224.5185,28.8017,190.5818,965.6712,0.0,0.0,0,8,63,504,480,13,21,0.047619047619047616,3.0,0.07936507936507936,5.0,,,0.20634920634920634,13.0,,,,,0.0081,12.67,,,,0.047619047619047616,3.0,8
|
||||||
|
job_653585,0.0,0,64,512,5,1.9219,0,0.9062,4859.3164,4603.0995,5038.9236,31797,1,4029.8322,1739932.0,52.0,0.8962,0.0,0.0,0.1038,0.4121,0.0093,0.1242,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0803,0.0001,0.0,1.0,0.0,0.0145,1.7394266914541902e+21,2.2925263334899206e+24,0.0,0.9646,0.0375,0.5469,0.4121,16.1924,52.5256,608.6286,29.1,857.9676,249.339,0.0627,5.7252,31.0932,188.5143,219.9568,1481.3095,32.107,188.8005,1223.5171,0.0,0.0,0,9,63,504,487,10,14,0.047619047619047616,3.0,0.015873015873015872,1.0,,,0.1746031746031746,11.0,,,,,,,,,,0.031746031746031744,2.0,9
|
||||||
|
job_653585,0.0,0,64,512,3,1.6562,0,0.8594,4206.2637,3770.6644,4770.7848,31696,1,3098.8536,1466169.0,27.0,0.9684,0.0,0.0,0.0316,0.5645,-0.0016,0.1212,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0758,-0.0009,0.0,1.0,0.0,0.0186,2.486790661662018e+22,3.009927560391077e+25,0.0,0.9792,0.0338,0.7031,0.5645,16.1496,52.6445,607.3474,29.2,857.9676,250.6202,0.062,5.807,27.2616,157.0913,184.6351,1344.8255,28.0337,157.3111,1126.3452,0.0,0.0,0,10,62,496,465,10,25,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.3387096774193548,21.0,0.016129032258064516,1.0,,,0.0079,11.2982,,,6.1509,,,10
|
||||||
|
job_653585,0.0,0,64,512,4,1.8125,0,0.8594,4341.25,4421.7844,4281.534,31768,1,3837.6661,1544551.0,26.0,0.9519,0.0,0.0,0.0481,0.4258,0.0172,0.1082,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0768,-0.0016,0.0,1.0,0.0,0.0145,2.3760572720076554e+18,2.479232720767365e+21,0.0,0.9513,0.0343,0.5312,0.4258,16.1263,52.6095,605.798,29.4,857.9676,252.1696,0.0603,5.7445,30.3901,180.1222,210.8445,1608.5564,32.131,180.3938,1359.8319,0.0,0.0,0,11,63,504,482,8,13,0.14285714285714285,9.0,0.031746031746031744,2.0,,,0.1746031746031746,11.0,,,,,,,,,,,,11
|
||||||
|
job_653585,0.0,0,64,512,4,2.0156,0,0.9375,5171.2402,4363.3762,5697.6548,31752,1,3838.9164,1805076.0,21.0,0.972,0.0,0.0,0.028,0.3945,-0.0056,0.099,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0793,0.0001,0.0,1.0,0.0,0.0187,3.3199502774606234e+17,4.460413625381222e+20,0.0,0.9698,0.0322,0.5156,0.3945,16.5123,53.0591,604.8148,29.5,857.9676,253.1528,0.5752,5.7891,36.193,189.4182,226.4272,1418.9828,24.7579,189.6581,1162.0041,0.0,0.0,0,12,63,504,486,8,16,0.031746031746031744,2.0,0.1111111111111111,7.0,,,0.14285714285714285,9.0,,,,,0.0085,10.287,,,,,,12
|
||||||
|
job_653585,0.0,0,64,512,3,1.5781,0,0.8594,4488.6309,4142.0406,4878.3651,31794,1,3769.576,1574682.0,18.0,0.937,0.0,0.0,0.063,0.5293,0.0116,0.1159,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0641,0.0001,0.0,1.0,0.0,0.0191,1.1232591437954967e+20,1.4095364312304327e+23,0.0,0.9755,0.0277,0.6719,0.5293,16.3773,52.9352,603.7007,29.6,857.9676,254.2668,0.0652,5.8155,29.0791,176.4044,205.7881,1174.4667,23.8728,176.6434,938.9854,0.0,0.0,0,13,63,504,462,18,34,0.12698412698412698,8.0,0.12698412698412698,8.0,,,0.36507936507936506,23.0,0.047619047619047616,3.0,,,,,,,,,,13
|
||||||
|
job_653585,0.0,0,64,512,4,1.9688,0,0.9375,4663.1953,4560.4923,4726.3722,31674,1,3835.0487,1647855.0,26.0,0.9488,0.0,0.0,0.0512,0.3809,-0.0175,0.1338,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0707,0.0,0.0,1.0,0.0,0.0173,1.3211500909875513e+22,1.6953347554969734e+25,0.0,0.9296,0.0299,0.5,0.3809,16.7322,53.2489,601.7022,29.9,857.9676,256.2654,0.1012,5.8519,30.3985,178.5781,209.3207,1621.8309,26.8666,178.8196,1379.7871,0.0,0.0,0,14,64,512,491,8,20,0.015625,1.0,0.1875,12.0,,,0.109375,7.0,0.015625,1.0,,,0.0082,11.2278,,,,,,14
|
||||||
|
job_653586,0.0,0,64,512,4,0.4375,0,0.1875,4367.6348,4450.258,4236.6061,20102,1,3081.0821,1530592.0,25.0,0.9745,0.0,0.0,0.0255,0.6133,-0.0018,0.082,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0643,-0.0,0.0,1.0,0.0,0.0116,1005131.5625,1241936512.0,0.0,0.9556,0.021,0.65,0.6133,9.9297,50.2901,562.6478,34.4,857.9675,295.3198,0.0524,3.9222,32.4786,156.8412,189.5694,4387.0719,37.9376,157.0373,4155.6423,0.0,0.0,0,15,54,432,390,20,41,0.018518518518518517,1.0,0.037037037037037035,2.0,,,0.7962962962962963,43.0,0.09259259259259259,5.0,,,,,,,6.8707,,,15
|
||||||
|
job_653586,0.0,0,64,512,1,0.9531,0,0.9531,5236.6426,4392.4831,5489.4619,31722,1,5068.1073,1953529.0,45.0,0.8724,0.0,0.0,0.1276,0.2305,0.008,0.0679,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0736,-0.0004,0.0,1.0,0.0,0.012,2.912904544507658e+18,4.067950672816252e+21,0.0,0.908,0.0255,0.3438,0.2305,13.9738,50.8674,555.7899,35.2,857.9675,302.1776,0.0685,6.0938,36.9697,235.7047,272.9919,940.3521,34.9952,235.9529,626.262,0.0,0.0,0,16,58,464,422,13,41,0.017241379310344827,1.0,0.1896551724137931,11.0,,,0.5517241379310345,32.0,0.017241379310344827,1.0,,,10.4479,18.917,,,,,,16
|
||||||
|
job_653586,0.0,0,64,512,2,1.6562,0,0.9531,4441.8926,4303.1646,4561.4509,31764,1,4231.7973,1624165.0,28.0,0.9089,0.0,0.0,0.0911,0.4629,-0.0268,0.1262,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0631,-0.0001,0.0,1.0,0.0,0.014,80980987609088.0,9.804779835568947e+16,0.0,0.919,0.0208,0.6094,0.4629,14.1574,51.0022,555.1819,35.3,857.9675,302.7856,0.8639,6.2006,35.3247,195.5062,231.9566,866.8196,41.0458,195.7672,587.611,0.0,0.0,0,17,63,504,479,13,24,0.015873015873015872,1.0,0.047619047619047616,3.0,,,0.2857142857142857,18.0,0.047619047619047616,3.0,,,,,,,,,,17
|
||||||
|
job_653586,0.0,0,64,512,3,1.9844,0,0.8438,4984.2539,4197.7163,5522.4112,31647,1,4678.7713,1808744.0,41.0,0.8809,0.0,0.0,0.1191,0.4062,-0.0087,0.1018,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0679,-0.0,0.0,1.0,0.0,0.0172,7.39416158020567e+16,9.942431530797027e+19,0.0,0.9589,0.0221,0.5312,0.4062,14.2903,51.1974,554.0929,35.4,857.9675,303.8746,0.0899,6.4232,37.3252,220.9561,258.6391,2168.7359,37.6553,221.2235,1866.0111,0.0,0.0,0,18,63,504,457,21,46,0.015873015873015872,1.0,0.047619047619047616,3.0,,,0.6190476190476191,39.0,0.06349206349206349,4.0,,,0.0074,8.3881,,,,,,18
|
||||||
|
job_653586,0.0,0,64,512,4,1.9062,0,0.8906,4700.6758,3937.2571,5231.5298,31734,1,4306.1606,1743000.0,35.0,0.9022,0.0,0.0,0.0978,0.4102,-0.02,0.1351,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0714,0.0,0.0,1.0,0.0,0.0168,6328167232962560.0,7.310935387570962e+18,0.0,0.9156,0.0245,0.5156,0.4102,14.5789,51.2928,552.8044,35.6,857.9675,305.1631,0.0891,6.021,34.5214,198.7713,233.6051,1196.6914,32.8597,198.9941,924.198,0.0,0.0,0,19,62,496,459,12,35,0.03225806451612903,2.0,,,,,0.5483870967741935,34.0,0.03225806451612903,2.0,,,,,,,,,,19
|
||||||
|
job_653586,0.0,0,64,512,4,1.6562,0,0.8281,4695.9609,4342.2462,4913.5457,31736,1,4065.4442,1681502.0,28.0,0.9544,0.0,0.0,0.0456,0.3809,0.0048,0.0503,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0696,-0.0005,0.0,1.0,0.0,0.0122,1080837378408448.0,1.4011869695511101e+18,0.0,0.9341,0.022,0.4531,0.3809,14.7026,51.4866,551.9064,35.7,857.9675,306.0612,0.0871,6.1017,31.5548,187.9218,219.7869,1351.9719,33.9437,188.1445,1092.1314,0.0,0.0,0,20,56,448,403,15,42,0.10714285714285714,6.0,0.07142857142857142,4.0,,,0.6964285714285714,39.0,,,,,0.0088,8.1422,,,5.8852,,,20
|
||||||
|
job_653586,0.0,0,64,512,4,1.9844,0,0.8906,4777.4453,4620.9898,4874.4873,31826,1,4055.0244,1707389.0,27.0,0.9601,0.0,0.0,0.0399,0.3828,0.0103,0.1078,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0684,-0.0062,0.0,1.0,0.0,0.0144,20477047209984.0,2.673594378433331e+16,0.0,0.9125,0.0217,0.5,0.3828,14.9075,51.7109,624.8141,27.2,857.9675,233.1534,0.0882,7.6651,35.0631,196.5691,231.9387,1653.9283,32.7526,196.7865,1381.5656,0.0,0.0,0,21,56,448,403,15,44,0.017857142857142856,1.0,,,,,0.9642857142857143,54.0,0.03571428571428571,2.0,,,,,,,,,,21
|
||||||
|
job_653586,0.0,0,64,512,3,1.6406,0,0.8438,4256.166,4364.5684,4192.2019,19194,1,3380.6239,1525708.0,35.0,0.9617,0.0,0.0,0.0383,0.3711,0.0204,0.1066,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0682,-0.0021,0.0,1.0,0.0,0.0156,9.329387085443891e+16,1.1523618092078439e+20,0.0,0.8961,0.0225,0.5156,0.3711,14.5482,51.4926,619.0483,27.8,857.9675,238.9193,0.047,3.8528,29.6378,159.7489,189.6301,1708.2028,35.8861,159.9448,1478.8283,0.0,0.0,0,22,61,488,396,27,75,0.36065573770491804,22.0,0.6229508196721312,38.0,0.03278688524590164,2.0,0.639344262295082,39.0,0.04918032786885246,3.0,0.01639344262295082,1.0,0.0083,8.0671,,,,,,22
|
||||||
|
job_653586,0.0,0,64,512,3,1.2656,0,0.7969,3544.1543,3840.6749,3276.2937,28297,1,3722.3558,1252311.0,10.0,0.991,0.0,0.0,0.009,0.4746,0.0158,0.0979,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0518,-0.0098,0.0,1.0,0.0,0.0149,7758122496.0,8681794568192.0,0.0,0.8013,0.0161,0.5781,0.4746,15.0082,51.8909,612.8306,28.6,857.9675,245.137,0.0827,5.0234,30.5594,167.256,198.1548,2873.6125,18.8351,167.5122,2651.5938,0.0,0.0,0,23,63,504,245,58,161,1.0952380952380953,69.0,1.9841269841269842,125.0,0.4603174603174603,29.0,0.6825396825396826,43.0,0.015873015873015872,1.0,,,,,,,,,,23
|
||||||
|
job_653586,0.0,0,64,512,4,1.6875,0,0.9375,2838.584,5697.4839,2204.031,21845,1,4033.9811,998412.0,19.0,0.9595,0.0,0.0,0.0405,0.1816,0.0534,0.1319,-0.0005,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0375,-0.031,0.0,1.0,0.0,0.0176,796371091193856.0,7.9573848653457e+17,0.0,0.4832,0.012,0.375,0.1816,14.9013,51.7034,610.9744,28.8,857.9675,246.9931,0.0656,3.9288,26.3368,163.6413,190.2636,4553.1826,19.7707,163.8606,4339.2146,0.0,0.0,0,24,63,504,198,59,150,2.1904761904761907,138.0,2.015873015873016,127.0,0.30158730158730157,19.0,0.31746031746031744,20.0,0.031746031746031744,2.0,0.015873015873015872,1.0,0.0083,30.3375,,,,0.015873015873015872,1.0,24
|
||||||
|
job_653586,0.0,0,64,512,3,1.5469,0,0.9375,1484.3555,3727.4578,1050.3753,15945,1,2355.9005,540011.0,7.0,0.9765,0.0,0.0,0.0235,0.1621,0.0976,0.1584,-0.0006,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0251,-0.0401,0.0,1.0,0.0,0.0162,175853157744640.0,1.2922646919590707e+17,0.0,0.4014,0.0083,0.3906,0.1621,14.7827,51.591,611.105,28.8,857.9675,246.8625,0.0412,2.8411,14.0718,97.9816,112.2645,2911.5013,20.8742,98.1511,2775.5178,0.0,0.0,0,25,60,480,239,50,62,2.7,162.0,0.7833333333333333,47.0,0.35,21.0,0.16666666666666666,10.0,0.06666666666666667,4.0,0.016666666666666666,1.0,,,,,6.1145,0.016666666666666666,1.0,25
|
||||||
|
job_653586,0.0,0,64,512,3,1.4688,0,0.9062,1762.2266,3276.9356,775.2226,14084,1,2254.5374,611382.0,7.0,0.9867,0.0,0.0,0.0133,0.3945,0.2088,0.2489,-0.0017,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.028,-0.1098,0.0,1.0,0.0,0.0301,78101203648512.0,6.106804833275085e+16,0.0,0.5056,0.0092,0.6875,0.3945,15.0391,51.8378,607.4099,29.2,857.9675,250.5576,0.0393,2.7152,16.2956,95.0195,111.5402,2183.5461,21.0421,95.2048,2048.2464,0.0,0.0,0,26,50,400,287,35,28,2.2,110.0,0.38,19.0,0.2,10.0,0.26,13.0,0.1,5.0,,,0.0076,18.4531,,,,,,26
|
||||||
|
job_653586,0.0,0,64,512,3,1.7344,0,0.9844,3225.4141,4233.3925,2501.5638,31725,1,3531.9698,1097953.0,6.0,0.9706,0.0,0.0,0.0294,0.418,0.0857,0.1298,-0.001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0417,-0.064,0.0,1.0,0.0,0.0185,1.7051105289310222e+21,1.7866703500134324e+24,0.0,0.7126,0.0141,0.5938,0.418,15.2845,52.0295,607.246,29.2,857.9675,250.7216,0.0933,6.0684,24.9326,145.5165,170.7634,1456.934,22.7893,145.7369,1257.3089,0.0,0.0,0,27,63,504,327,51,68,0.9523809523809523,60.0,0.5396825396825397,34.0,0.7301587301587301,46.0,0.3968253968253968,25.0,0.12698412698412698,8.0,,,,,,,,0.015873015873015872,1.0,27
|
||||||
|
job_653586,0.0,0,64,512,4,2.0312,0,0.9062,3251.666,4554.6894,2653.9829,21699,1,3609.6329,1139572.0,15.0,0.9862,0.0,0.0,0.0138,0.3145,0.1116,0.1947,-0.0011,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0421,-0.0686,0.0,1.0,0.0,0.0196,3.3728,2506.6016,0.0,0.6563,0.0144,0.5312,0.3145,15.0491,51.7938,605.1166,29.5,857.9675,252.8509,0.0646,4.1197,22.7726,156.63,179.688,3692.4206,19.6838,156.8504,3488.9217,0.0,0.0,0,28,62,496,313,47,111,0.967741935483871,60.0,1.3548387096774193,84.0,0.24193548387096775,15.0,0.3870967741935484,24.0,0.0967741935483871,6.0,,,0.0078,10.6979,,,,0.03225806451612903,2.0,28
|
||||||
|
job_653586,0.0,0,64,512,4,1.6719,0,0.9375,2571.3672,3743.3082,2043.4958,15866,1,2671.8445,889053.0,7.0,0.9918,0.0,0.0,0.0082,0.3105,0.0625,0.1191,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.036,-0.0436,0.0,1.0,0.0,0.0144,2088775.625,1969496832.0,0.0,0.6323,0.0129,0.4375,0.3105,15.0667,51.8913,603.0693,29.7,857.9675,254.8982,0.0526,3.2231,19.9907,119.1558,139.374,2121.552,24.1526,119.3302,1954.7975,0.0,0.0,0,29,62,496,399,34,26,1.032258064516129,64.0,0.1935483870967742,12.0,0.16129032258064516,10.0,0.3064516129032258,19.0,0.04838709677419355,3.0,,,,,,,,,,29
|
||||||
|
job_653586,0.0,0,64,512,4,1.5938,0,0.8594,3329.0859,3448.2411,3212.6911,31752,1,3171.8807,1207059.0,16.0,0.9547,0.0,0.0,0.0453,0.4941,0.0633,0.1659,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0435,-0.0529,0.0,1.0,0.0,0.0237,166590859444224.0,1.8302711074127872e+17,0.0,0.7955,0.0163,0.7031,0.4941,15.2241,52.0995,601.8652,29.8,857.9675,256.1023,0.0917,5.5872,23.2574,143.9244,167.4704,1800.873,28.7964,144.1208,1599.0148,0.0,0.0,0,30,61,488,408,28,51,0.4426229508196721,27.0,0.01639344262295082,1.0,0.01639344262295082,1.0,0.7049180327868853,43.0,0.11475409836065574,7.0,0.01639344262295082,1.0,0.0073,8.1466,,,6.1638,,,30
|
||||||
|
job_653586,0.0,0,64,512,4,1.7188,0,0.875,4176.25,4287.3301,4101.4706,22905,1,3442.4973,1477397.0,13.0,0.9695,0.0,0.0,0.0305,0.4023,0.0198,0.0959,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0487,-0.0138,0.0,1.0,0.0,0.0177,17930.9121,21793098.0,0.0,0.8509,0.0185,0.5312,0.4023,14.9604,51.8221,601.6973,29.9,857.9675,256.2702,0.0519,4.5315,30.1351,156.4327,186.8623,1635.6999,31.1503,156.6733,1413.1506,0.0,0.0,0,31,62,496,427,25,63,0.0967741935483871,6.0,0.016129032258064516,1.0,,,1.0483870967741935,65.0,0.016129032258064516,1.0,,,,,,,,0.04838709677419355,3.0,31
|
||||||
|
job_653586,0.0,0,64,512,5,1.7031,0,0.875,4112.8223,4007.204,4181.0836,31560,1,3957.1982,1471823.0,15.0,0.9631,0.0,0.0,0.0369,0.3926,0.0092,0.1152,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0462,-0.0013,0.0,1.0,0.0,0.0137,113758040.0,137789751296.0,0.0,0.8584,0.0187,0.5156,0.3926,15.251,52.1536,599.0416,30.2,857.9675,258.9259,0.0694,5.7857,28.8593,179.0917,208.2315,1896.1165,35.5573,179.3023,1646.5384,0.0,0.0,0,32,62,496,454,17,33,0.14516129032258066,9.0,0.08064516129032258,5.0,,,0.3870967741935484,24.0,0.016129032258064516,1.0,,,0.0086,11.4834,,,,0.06451612903225806,4.0,32
|
||||||
|
job_653586,0.0,0,64,512,3,1.6719,0,0.8906,3989.2363,3479.5046,4367.2007,23645,1,3195.8215,1377193.0,19.0,0.9806,0.0,0.0,0.0194,0.4258,0.0032,0.0398,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0433,-0.0003,0.0,1.0,0.0,0.0155,214550183936.0,251782945046528.0,0.0,0.9224,0.0178,0.4844,0.4258,15.0996,52.0021,598.8062,30.2,857.9675,259.1614,0.0535,4.9425,28.9784,147.4112,176.6704,1164.9656,31.3944,147.638,951.9529,0.0,0.0,0,33,64,512,443,16,69,,,0.015625,1.0,,,1.046875,67.0,,,,,,,,,,0.015625,1.0,33
|
||||||
|
job_653586,0.0,0,64,512,5,1.7812,0,0.875,3690.3398,3317.2692,3896.0939,20066,1,3147.0993,1351091.0,13.0,0.9855,0.0,0.0,0.0145,0.3555,-0.0158,0.1224,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0489,0.0,0.0,1.0,0.0,0.0231,4346809.0,4922826240.0,0.0,0.8724,0.0195,0.4844,0.3555,15.2644,52.2489,596.6576,30.5,857.9675,261.3099,0.0489,4.2274,24.6053,147.5165,172.3903,1643.8279,30.4621,147.7346,1436.743,0.0,0.0,0,34,18,144,123,6,21,,,,,,,1.0555555555555556,19.0,0.05555555555555555,1.0,,,0.0074,8.954,,,,0.05555555555555555,1.0,34
|
||||||
|
job_653587,0.0,0,64,512,3,0.1719,0,0.0938,5270.0117,4869.6905,5658.0154,24181,1,4003.1029,1780122.0,10.0,0.987,0.0,0.0,0.013,0.4922,0.0054,0.0974,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0471,-0.0,0.0,1.0,0.0,0.0164,2104.1606,2702316.25,0.0,0.9513,0.0147,0.6066,0.4922,11.2632,50.6464,591.5163,31.1,857.9675,266.4512,0.0717,5.2462,36.047,201.8628,238.2231,4940.941,41.3945,202.1037,4656.0768,0.0,0.0,0,35,58,464,389,22,75,0.017241379310344827,1.0,0.1206896551724138,7.0,,,1.0172413793103448,59.0,0.08620689655172414,5.0,,,,,,,8.0617,0.1206896551724138,7.0,35
|
||||||
|
job_653587,0.0,0,64,512,1,1.0,1,1.0,4019.6211,3506.7433,4314.7231,31761,1,4137.767,1417266.0,17.0,0.9591,0.0,0.0,0.0409,0.3652,0.0001,0.0845,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0442,0.0,0.0,1.0,0.0,0.0192,973045248.0,849779032064.0,0.0,0.8509,0.0153,0.4844,0.3652,14.0512,51.1256,585.3741,31.8,857.9675,272.5934,0.0674,6.7554,29.7924,182.7126,212.8136,608.8706,36.372,182.9532,352.9221,0.0,0.0,0,36,64,512,464,17,41,0.109375,7.0,0.03125,2.0,,,0.5,32.0,0.078125,5.0,,,11.8491,21.5894,,,,0.03125,2.0,36
|
||||||
|
job_653587,0.0,0,64,512,2,1.5781,0,0.8906,4050.4727,3962.2194,4155.3205,26248,1,3351.8207,1378599.0,12.0,0.9874,0.0,0.0,0.0126,0.543,0.0002,0.1003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0429,0.0001,0.0,1.0,0.0,0.0164,451543132274688.0,5.2744631064356454e+17,0.0,0.9114,0.0146,0.625,0.543,14.0529,51.2486,585.133,31.8,857.9675,272.8345,0.0763,5.0232,27.0398,157.8818,185.2402,1221.9848,32.8949,158.1234,998.821,0.0,0.0,0,37,60,480,426,16,51,0.05,3.0,0.03333333333333333,2.0,,,0.8,48.0,0.06666666666666667,4.0,,,,,,,,0.06666666666666667,4.0,37
|
||||||
|
job_653587,0.0,0,64,512,3,1.7812,0,0.8906,4166.3809,3516.1595,4705.1357,31759,1,3839.5371,1482005.0,13.0,0.9562,0.0,0.0,0.0438,0.4531,-0.0024,0.1352,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0404,-0.0007,0.0,1.0,0.0,0.019,5167771222540288.0,6.12676136445621e+18,0.0,0.8858,0.0136,0.5781,0.4531,14.5875,51.5708,583.2682,32.0,857.9675,274.6993,0.0874,6.1662,33.6069,178.474,212.3933,1845.865,36.6567,178.6985,1590.6434,0.0,0.0,0,38,62,496,429,21,67,,,0.016129032258064516,1.0,,,0.8387096774193549,52.0,0.11290322580645161,7.0,,,0.0087,9.8733,,,,0.14516129032258066,9.0,38
|
||||||
|
job_653587,0.0,0,64,512,4,1.9531,0,0.8906,4455.7715,4539.2418,4379.7761,31568,1,4244.5861,1548541.0,19.0,0.9715,0.0,0.0,0.0285,0.4766,0.0051,0.0819,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0407,-0.0,0.0,1.0,0.0,0.017,4.4555619643529626e+19,5.5445188610444524e+22,0.0,0.8705,0.0132,0.5781,0.4766,14.9482,52.0431,582.4987,32.1,857.9675,275.4688,0.0874,6.7551,37.2039,192.8015,230.35,1216.2695,35.0045,193.0581,944.1541,0.0,0.0,0,39,62,496,424,23,72,,,0.06451612903225806,4.0,,,0.7580645161290323,47.0,0.11290322580645161,7.0,,,,,,,,0.22580645161290322,14.0,39
|
||||||
|
job_653587,0.0,0,64,512,5,1.8906,0,0.8906,3978.8047,4426.8494,3586.5604,20630,1,3356.5378,1380244.0,18.0,0.9762,0.0,0.0,0.0238,0.4668,0.0008,0.0972,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0396,0.0,0.0,1.0,0.0,0.0151,8027375480602624.0,9.356849449931899e+18,0.0,0.8635,0.0152,0.5312,0.4668,14.7535,51.8218,581.7134,32.2,857.9675,276.2541,0.0558,4.3532,24.4683,152.7384,177.4882,1338.8646,39.4864,152.9636,1117.5303,0.0,0.0,0,40,61,488,425,18,55,0.13114754098360656,8.0,0.03278688524590164,2.0,,,0.7049180327868853,43.0,0.01639344262295082,1.0,,,0.0073,8.861,,,6.7398,0.19672131147540983,12.0,40
|
||||||
|
job_653587,0.0,0,64,512,4,1.6719,0,0.8281,3751.0664,3454.8347,4004.3659,28890,1,3641.0091,1296617.0,12.0,0.9877,0.0,0.0,0.0123,0.4609,0.0022,0.08,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0406,-0.0,0.0,1.0,0.0,0.0169,1345461420032.0,1527438681047040.0,0.0,0.8763,0.0139,0.5781,0.4609,14.9705,51.9844,581.6381,32.2,857.9675,276.3294,0.0797,5.3066,30.2736,162.8792,193.4836,1768.759,31.3917,163.1298,1538.5717,0.0,0.0,0,41,64,512,449,21,62,0.015625,1.0,0.0625,4.0,,,0.703125,45.0,0.15625,10.0,,,,,,,,0.046875,3.0,41
|
||||||
|
job_653587,0.0,0,64,512,4,1.9375,0,0.9062,4489.3984,4501.6872,4481.8391,31683,1,4385.293,1584898.0,26.0,0.939,0.0,0.0,0.061,0.3809,0.0119,0.0673,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0434,0.0,0.0,1.0,0.0,0.0124,602914881536.0,717373355065344.0,0.0,0.8819,0.0152,0.4375,0.3809,15.4638,52.4785,580.4658,32.3,857.9675,277.5018,0.0671,6.1951,32.6832,192.1724,225.2152,1528.4712,36.6995,192.4644,1260.3568,0.0,0.0,0,42,62,496,445,20,45,0.0967741935483871,6.0,0.016129032258064516,1.0,,,0.5645161290322581,35.0,0.14516129032258066,9.0,,,0.0091,8.6068,,,,,,42
|
||||||
|
job_653587,0.0,0,64,512,3,1.7344,0,0.9062,4579.6543,4076.3868,4935.2967,31795,1,4209.4791,1582217.0,11.0,0.9742,0.0,0.0,0.0258,0.4141,0.003,0.0933,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0423,0.0,0.0,1.0,0.0,0.0148,1.2523732566881721e+22,1.575312744563639e+25,0.0,0.9056,0.0142,0.5,0.4141,15.3868,52.4961,580.9841,32.3,857.9675,276.9834,0.0969,5.8831,35.5495,203.7713,239.6873,1429.6029,46.2261,204.0403,1137.7979,0.0,0.0,0,43,58,464,404,16,52,0.13793103448275862,8.0,0.034482758620689655,2.0,,,0.7068965517241379,41.0,0.08620689655172414,5.0,,,,,,,,0.20689655172413793,12.0,43
|
||||||
|
job_653587,0.0,0,64,512,4,1.7031,0,0.875,3711.0449,3450.7154,3979.6389,31858,1,3450.5205,1281843.0,21.0,0.9584,0.0,0.0,0.0416,0.5078,0.0026,0.0904,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0389,0.0001,0.0,1.0,0.0,0.0186,10689648640.0,12101714182144.0,0.0,0.8716,0.0135,0.6094,0.5078,15.393,52.5665,580.2031,32.4,857.9675,277.7644,0.0674,6.9397,27.3191,159.9392,187.542,1259.6237,27.2463,160.1551,1037.8905,0.0,0.0,0,44,64,512,477,14,35,,,0.03125,2.0,,,0.25,16.0,0.09375,6.0,,,0.0088,9.1974,,,,0.125,8.0,44
|
||||||
|
job_653587,0.0,0,64,512,4,1.7969,0,0.9219,4316.8047,4018.2939,4590.7191,31802,1,3563.2553,1489843.0,18.0,0.9623,0.0,0.0,0.0377,0.4785,0.0011,0.0785,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0413,0.0,0.0,1.0,0.0,0.0143,1.399725969946154e+24,1.7084933114951544e+27,0.0,0.9425,0.0143,0.5312,0.4785,15.4081,52.5731,579.7631,32.4,857.9675,278.2045,0.0681,6.996,33.0651,177.8596,211.2338,1402.8534,32.9762,178.1002,1151.6417,0.0,0.0,0,45,63,504,456,15,48,,,0.015873015873015872,1.0,,,0.5396825396825397,34.0,0.06349206349206349,4.0,,,,,,,6.2605,0.14285714285714285,9.0,45
|
||||||
|
job_653587,0.0,0,64,512,4,2.125,0,0.9688,4054.1738,3689.888,4427.0988,31767,1,3753.6733,1394041.0,18.0,0.9628,0.0,0.0,0.0372,0.5059,0.0001,0.1048,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0372,-0.0001,0.0,1.0,0.0,0.019,667826126848.0,788490430185472.0,0.0,0.9108,0.0129,0.5781,0.5059,15.491,52.6367,579.8985,32.4,857.9675,278.069,0.0677,6.2085,29.2944,171.0908,200.675,1235.1016,35.891,171.3125,992.3206,0.0,0.0,0,46,63,504,465,16,39,,,0.047619047619047616,3.0,,,0.49206349206349204,31.0,0.06349206349206349,4.0,,,0.0085,8.8618,,,,0.1111111111111111,7.0,46
|
||||||
|
job_653587,0.0,0,64,512,4,2.0781,0,0.8906,4380.8105,4373.7847,4388.8992,31787,1,4245.6796,1564842.0,28.0,0.9236,0.0,0.0,0.0764,0.5352,0.0002,0.1101,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.04,0.0,0.0,1.0,0.0,0.0178,6668615680000.0,6459941124571136.0,0.0,0.917,0.0138,0.625,0.5352,15.551,52.6967,579.8947,32.4,857.9675,278.0728,0.0683,5.9577,30.8067,192.315,223.44,1375.8745,34.2379,192.5645,1112.233,0.0,0.0,0,47,60,480,429,17,51,,,0.03333333333333333,2.0,,,0.8166666666666667,49.0,0.05,3.0,,,,,,,,0.06666666666666667,4.0,47
|
||||||
|
job_653587,0.0,0,64,512,4,1.7344,0,0.7812,3799.5566,3462.6914,4136.4219,31660,1,3327.2914,1342599.0,18.0,0.9759,0.0,0.0,0.0241,0.5,-0.0104,0.0643,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0401,0.0,0.0,1.0,0.0,0.0186,6386052608.0,7399555989504.0,0.0,0.8917,0.0134,0.5938,0.5,16.2215,53.1401,577.9128,32.6,857.9675,280.0547,0.0875,5.6471,29.4067,156.0174,185.712,1598.9196,29.9299,156.2173,1377.6224,0.0,0.0,0,48,63,504,429,29,67,0.12698412698412698,8.0,0.06349206349206349,4.0,,,0.6190476190476191,39.0,0.15873015873015872,10.0,,,0.0081,8.3558,,,,0.2698412698412698,17.0,48
|
||||||
|
job_653587,0.0,0,64,512,4,1.8125,0,0.8594,3419.8555,3529.5067,3333.892,31764,1,3464.6841,1225041.0,9.0,0.9602,0.0,0.0,0.0398,0.4395,0.0036,0.0933,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0381,0.0001,0.0,1.0,0.0,0.0193,22766618624.0,22522718322688.0,0.0,0.8525,0.0126,0.5781,0.4395,16.5361,53.4547,577.4091,32.7,857.9675,280.5584,0.063,5.9879,28.4774,153.5356,182.3468,1442.6214,29.7255,153.806,1224.5562,0.0,0.0,0,49,63,504,448,24,49,0.1111111111111111,7.0,,,,,0.5079365079365079,32.0,0.1111111111111111,7.0,,,,,,,,0.1111111111111111,7.0,49
|
||||||
|
job_653587,0.0,0,64,512,4,1.9375,0,0.8906,4705.582,3464.1697,5295.879,31742,1,4707.6528,1701349.0,45.0,0.9029,0.0,0.0,0.0971,0.3223,0.0002,0.0516,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0422,-0.0001,0.0,1.0,0.0,0.0187,464861233152.0,492378775355392.0,0.0,0.9011,0.0146,0.3906,0.3223,16.5455,53.464,576.9939,32.7,857.9675,280.9736,0.0903,7.0541,33.012,211.9908,245.3293,1863.3175,39.9846,212.2265,1570.9436,0.0,0.0,0,50,62,496,411,24,78,0.11290322580645161,7.0,0.11290322580645161,7.0,,,0.7903225806451613,49.0,0.11290322580645161,7.0,,,0.0085,8.5286,,,5.7884,0.24193548387096775,15.0,50
|
||||||
|
job_653587,0.0,0,64,512,3,1.625,0,0.8906,3573.1074,3325.9396,3709.4242,21691,1,3523.2654,1255628.0,17.0,0.9817,0.0,0.0,0.0183,0.3555,-0.0051,0.0953,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0379,-0.0,0.0,1.0,0.0,0.0153,165468454912.0,185397279719424.0,0.0,0.8323,0.0132,0.4844,0.3555,16.3303,53.2484,577.5415,32.7,857.9675,280.426,0.0574,5.3966,29.4475,162.4351,192.1689,1459.9757,33.9624,162.6636,1228.4414,0.0,0.0,0,51,64,512,420,29,92,,,0.140625,9.0,,,0.875,56.0,0.171875,11.0,,,,,,,,0.265625,17.0,51
|
||||||
|
job_653587,0.0,0,64,512,4,1.8438,0,0.9688,3976.1758,4500.2031,3661.7594,29854,1,3987.1647,1358843.0,25.0,0.9848,0.0,0.0,0.0152,0.375,0.0005,0.099,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0394,-0.0,0.0,1.0,0.0,0.0151,7481026.5,7046713344.0,0.0,0.8285,0.0148,0.4688,0.375,16.6743,53.6125,576.8345,32.8,857.9675,281.133,0.084,5.6133,29.8419,175.3867,205.5518,1339.7774,38.5075,175.6254,1090.0995,0.0,0.0,0,52,63,504,433,23,61,0.06349206349206349,4.0,0.09523809523809523,6.0,,,0.49206349206349204,31.0,0.015873015873015872,1.0,0.09523809523809523,6.0,0.0072,8.6118,,,,0.38095238095238093,24.0,52
|
||||||
|
job_653587,0.0,0,64,512,3,1.7812,0,0.9062,3691.3457,3547.4033,3821.3755,22436,1,3372.4346,1290968.0,11.0,0.9861,0.0,0.0,0.0139,0.4746,-0.0042,0.0709,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0381,0.0,0.0,1.0,0.0,0.0124,1.2458670639324969e+23,1.4152924048125494e+26,0.0,0.8636,0.0132,0.5625,0.4746,16.4417,53.4303,577.7429,32.7,857.9675,280.2247,0.0685,5.4589,25.5377,153.0849,178.9488,1705.9455,29.4538,153.3421,1492.0762,0.0,0.0,0,53,64,512,463,19,48,0.015625,1.0,,,,,0.5625,36.0,0.0625,4.0,,,,,,,,0.125,8.0,53
|
||||||
|
job_653587,0.0,0,64,512,4,1.7188,0,0.9062,3767.5215,3727.4593,3795.1551,18986,1,2995.1567,1342725.0,37.0,0.968,0.0,0.0,0.032,0.4082,0.0064,0.0877,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0444,-0.0,0.0,1.0,0.0,0.0185,327695663104.0,379507068370944.0,0.0,0.9092,0.0151,0.4844,0.4082,16.3465,53.3348,577.0278,32.7,857.9675,280.9397,0.0444,3.8435,25.2314,140.1255,165.6541,1372.4083,33.4557,140.3778,1169.4509,0.0,0.0,0,54,62,496,426,22,70,,,0.08064516129032258,5.0,,,0.9354838709677419,58.0,0.04838709677419355,3.0,,,0.0087,9.1348,,,,0.06451612903225806,4.0,54
|
||||||
|
job_653587,0.0,0,64,512,4,2.0469,0,0.9219,4275.6836,3941.0486,4465.0031,31662,1,4375.7459,1463491.0,11.0,0.9714,0.0,0.0,0.0286,0.3613,-0.0046,0.1022,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0408,0.0,0.0,1.0,0.0,0.0216,4036787148161024.0,4.865830434606416e+18,0.0,0.8604,0.0139,0.4688,0.3613,16.6034,53.5885,576.6667,32.8,857.9675,281.3008,0.157,6.1578,33.4267,203.4849,237.3352,1613.5996,28.3574,203.7511,1341.7459,0.0,0.0,0,55,61,488,424,21,57,0.11475409836065574,7.0,0.14754098360655737,9.0,,,0.819672131147541,50.0,0.04918032786885246,3.0,,,,,,,5.8658,0.01639344262295082,1.0,55
|
||||||
|
job_653587,0.0,0,64,512,4,1.7344,0,0.8906,4390.9277,3929.1513,4585.9,31767,1,4238.3757,1548471.0,22.0,0.9401,0.0,0.0,0.0599,0.2969,-0.001,0.0628,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,-0.0042,0.0,1.0,0.0,0.015,8617810003492864.0,1.0647384730359562e+19,0.0,0.8761,0.015,0.375,0.2969,16.7065,53.8438,576.3802,32.8,857.9675,281.5873,0.0907,6.1411,30.9645,191.9417,223.2241,1633.3771,35.1631,192.1684,1368.8439,0.0,0.0,0,56,64,512,478,15,34,,,0.046875,3.0,,,0.390625,25.0,0.09375,6.0,,,0.0081,8.2677,,,,,,56
|
||||||
|
job_653587,0.0,0,64,512,4,1.5469,0,0.8281,4453.5156,4019.9141,4887.1172,31740,1,4404.608,1552609.0,34.0,0.9144,0.0,0.0,0.0856,0.5,0.0024,0.1244,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0436,-0.0001,0.0,1.0,0.0,0.0147,3.5539692025682697e+21,3.740553229295638e+24,0.0,0.9382,0.0158,0.6094,0.5,16.787,53.9058,575.7407,32.9,857.9675,282.2268,0.0908,6.1332,38.8151,208.6843,247.8062,1501.615,35.5265,208.8997,1212.1417,0.0,0.0,0,57,61,488,455,11,29,0.06557377049180328,4.0,0.08196721311475409,5.0,,,0.5081967213114754,31.0,0.03278688524590164,2.0,,,,,,,,0.01639344262295082,1.0,57
|
||||||
|
job_653587,0.0,0,64,512,3,1.7031,0,0.875,3884.9863,4063.3161,3777.094,31844,1,3730.8638,1405568.0,16.0,0.9487,0.0,0.0,0.0513,0.377,0.0085,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0444,-0.0,0.0,1.0,0.0,0.0189,5.697833777259119e+21,6.752966200220271e+24,0.0,0.921,0.0151,0.5,0.377,17.2032,54.3492,574.8936,33.0,857.9675,283.0739,0.1129,6.2873,28.6923,178.963,207.988,1448.7366,30.4954,179.1824,1203.9568,0.0,0.0,0,58,64,512,453,16,53,0.078125,5.0,0.125,8.0,,,0.671875,43.0,0.03125,2.0,0.015625,1.0,0.0078,8.5897,,,,,,58
|
||||||
|
job_653587,0.0,0,64,512,4,1.5938,0,0.875,3719.3281,3822.8142,3577.5139,31652,1,3793.566,1302863.0,9.0,0.9494,0.0,0.0,0.0506,0.5781,0.0061,0.1067,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0407,-0.0008,0.0,1.0,0.0,0.0169,2.916946366431232e+16,3.229335359815313e+19,0.0,0.8894,0.0141,0.6875,0.5781,17.3769,54.5306,573.5134,33.2,857.9675,284.4541,0.09,5.9757,31.3476,170.0586,201.7858,1663.1725,33.9156,170.3478,1421.4882,0.0,0.0,0,59,62,496,460,15,35,,,0.03225806451612903,2.0,0.016129032258064516,1.0,0.5,31.0,0.04838709677419355,3.0,,,,,,,,,,59
|
||||||
|
job_653587,0.0,0,64,512,3,1.7188,0,0.875,4468.502,2738.5871,5586.5498,31799,1,4489.8129,1573926.0,34.0,0.9358,0.0,0.0,0.0642,0.3926,-0.0178,0.0646,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0453,-0.0,0.0,1.0,0.0,0.0139,5716752465920.0,7041144689000448.0,0.0,0.9321,0.0155,0.4688,0.3926,17.3024,54.4753,572.2839,33.3,857.9675,285.6836,0.108,6.3356,35.3958,205.7623,241.5432,1507.1501,25.3052,206.039,1233.9571,0.0,0.0,0,60,62,496,439,20,57,,,0.08064516129032258,5.0,,,0.6935483870967742,43.0,0.11290322580645161,7.0,,,0.0079,8.517,,,6.2107,,,60
|
||||||
|
job_653588,0.0,0,64,512,3,0.4219,0,0.2188,3685.4219,3527.8649,4019.75,16769,1,2307.0638,1257847.0,2.0,0.9993,0.0,0.0,0.0007,0.6797,-0.0006,0.0628,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0381,0.0,0.0,1.0,0.0,0.0162,1.0002,1.312,0.0,0.9678,0.0118,0.7544,0.6797,8.3562,50.2278,634.6806,26.0,857.9676,223.287,0.0717,3.4039,26.6535,126.6473,153.6151,4895.4874,32.7345,126.8502,4705.7335,0.0,0.0,0,61,54,432,360,23,69,0.05555555555555555,3.0,0.1111111111111111,6.0,,,1.2592592592592593,68.0,0.2222222222222222,12.0,,,,,,,,,,61
|
||||||
|
job_653588,0.0,0,64,512,1,0.9219,0,0.9219,3966.3945,3777.5464,4071.4377,31765,1,4125.9274,1408431.0,14.0,0.9555,0.0,0.0,0.0445,0.3574,0.0054,0.0859,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0428,0.0,0.0,1.0,0.0,0.0312,4798878720.0,5626906804224.0,0.0,0.8313,0.0134,0.4688,0.3574,13.8742,50.7912,624.6771,27.2,857.9676,233.2905,0.0654,5.9309,30.7554,183.7737,214.8289,884.5667,32.4223,184.0075,631.377,0.0,0.0,0,62,56,448,392,17,55,0.017857142857142856,1.0,0.14285714285714285,8.0,,,0.7857142857142857,44.0,0.08928571428571429,5.0,,,8.4412,17.7924,,,,,,62
|
||||||
|
job_653588,0.0,0,64,512,2,1.6094,0,0.9688,4484.8535,4127.1455,4739.6756,31775,1,4685.7602,1592891.0,27.0,0.914,0.0,0.0,0.086,0.416,0.0051,0.0481,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0438,0.0,0.0,1.0,0.0,0.014,1.49631712790315e+18,1.8779827487400106e+21,0.0,0.8914,0.0146,0.4531,0.416,14.0978,50.9457,622.3339,27.5,857.9676,235.6337,0.0902,5.9053,35.778,202.1171,238.2046,1021.0078,34.2374,202.3359,742.6567,0.0,0.0,0,63,65,520,475,16,39,0.09230769230769231,6.0,0.07692307692307693,5.0,,,0.4461538461538462,29.0,0.07692307692307693,5.0,,,,,,,,,,63
|
||||||
|
job_653588,0.0,0,64,512,3,1.9219,0,0.8438,4086.5547,3298.7422,4704.1777,31791,1,3910.7476,1452474.0,19.0,0.9538,0.0,0.0,0.0462,0.4395,0.0004,0.0928,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0441,-0.0,0.0,1.0,0.0,0.0169,680327517306880.0,8.199202519226778e+17,0.0,0.9173,0.0141,0.5938,0.4395,14.9725,51.1755,619.0883,27.8,857.9676,238.8793,0.0605,5.8121,35.1041,187.1695,222.5884,1993.5726,35.8302,187.4233,1729.3387,0.0,0.0,0,64,60,480,427,15,53,,,0.05,3.0,,,0.75,45.0,0.15,9.0,,,0.0083,8.9351,,,,,,64
|
||||||
|
job_653588,0.0,0,64,512,4,1.9531,0,0.8594,4482.5859,4188.5991,4726.175,31715,1,4433.6859,1614330.0,23.0,0.916,0.0,0.0,0.084,0.4531,0.0098,0.1061,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,0.0,0.0,1.0,0.0,0.0159,8.676132800661553e+23,8.132848488751659e+26,0.0,0.8934,0.0148,0.5781,0.4531,15.4079,51.6347,616.659,28.1,857.9676,241.3086,0.0896,5.6735,34.9766,204.9392,240.2083,1228.4685,32.6219,205.1416,949.9589,0.0,0.0,0,65,63,504,428,24,63,0.20634920634920634,13.0,0.047619047619047616,3.0,,,0.8253968253968254,52.0,0.12698412698412698,8.0,,,,,,,6.8083,,,65
|
||||||
|
job_653588,0.0,0,64,512,5,1.9219,0,0.9062,4357.2324,4060.3786,4557.0752,31769,1,4668.8661,1580861.0,37.0,0.8957,0.0,0.0,0.1043,0.4023,0.0064,0.0877,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,-0.0002,0.0,1.0,0.0,0.0172,1.054274388152946e+23,1.2798084482887822e+26,0.0,0.8563,0.015,0.5312,0.4023,15.6097,51.7144,614.0408,28.4,857.9676,243.9268,0.0622,6.0018,33.4297,213.4844,247.2341,1499.3639,32.0364,213.7417,1214.0885,0.0,0.0,0,66,61,488,438,15,44,0.09836065573770492,6.0,0.01639344262295082,1.0,,,0.6065573770491803,37.0,0.09836065573770492,6.0,,,0.008,9.2733,,,,,,66
|
||||||
|
job_653588,0.0,0,64,512,4,1.7344,0,0.8906,4419.9375,3553.4759,4918.4862,31735,1,4351.8887,1540086.0,31.0,0.9326,0.0,0.0,0.0674,0.3652,-0.0012,0.0879,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0436,-0.0053,0.0,1.0,0.0,0.0166,7106447872.0,7634057953280.0,0.0,0.9049,0.0147,0.4844,0.3652,15.652,51.8026,611.4824,28.7,857.9676,246.4852,0.0622,6.0487,33.6584,200.2845,234.3115,1811.2337,34.6927,200.5905,1536.1294,0.0,0.0,0,67,58,464,396,26,66,0.034482758620689655,2.0,0.05172413793103448,3.0,,,0.8275862068965517,48.0,0.25862068965517243,15.0,,,,,,,,0.05172413793103448,3.0,67
|
||||||
|
job_653588,0.0,0,64,512,4,1.5156,0,0.8906,4066.9102,4048.2335,4078.5905,31837,1,4387.0666,1427586.0,25.0,0.9098,0.0,0.0,0.0902,0.3848,-0.0114,0.0895,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0438,-0.0,0.0,1.0,0.0,0.0141,3215390490492928.0,3.841650471617954e+18,0.0,0.8661,0.015,0.4688,0.3848,16.0492,52.3116,609.3674,29.0,857.9676,248.6002,0.0897,6.1724,31.0609,203.1937,234.6137,1248.5725,32.0514,203.4626,975.7293,0.0,0.0,0,68,63,504,455,13,46,0.047619047619047616,3.0,0.031746031746031744,2.0,,,0.6190476190476191,39.0,0.07936507936507936,5.0,,,0.0086,8.9362,,,,,,68
|
||||||
|
job_653588,0.0,0,64,512,8,2.0156,0,0.9375,4388.7012,3827.7276,5121.5045,31658,1,3792.0489,1553676.0,18.0,0.9743,0.0,0.0,0.0257,0.5664,-0.0064,0.0802,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0449,0.0,0.0,1.0,0.0,0.0138,942792901656576.0,1.1535304966096814e+18,0.0,0.9097,0.0145,0.6406,0.5664,16.1733,52.4356,607.0387,29.2,857.9676,250.9289,0.1701,5.7865,33.4522,177.4228,211.3543,1399.7571,37.3866,177.7316,1145.224,0.0,0.0,0,69,62,496,433,24,58,0.08064516129032258,5.0,0.03225806451612903,2.0,,,0.5967741935483871,37.0,0.2903225806451613,18.0,,,,,,,,0.04838709677419355,3.0,69
|
||||||
|
job_653588,0.0,0,64,512,3,1.5938,0,0.9219,4478.5273,4706.2548,4322.7138,31693,1,4388.6691,1567154.0,14.0,0.9596,0.0,0.0,0.0404,0.4062,0.0146,0.0753,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.045,-0.0017,0.0,1.0,0.0,0.0203,37698882502656.0,3.898363573436416e+16,0.0,0.8784,0.0142,0.5,0.4062,16.1904,52.4454,604.8707,29.5,857.9676,253.0969,0.0912,5.8848,33.2322,191.6949,225.2911,1712.3059,39.3967,191.9673,1441.7301,0.0,0.0,0,70,63,504,441,22,60,0.047619047619047616,3.0,0.19047619047619047,12.0,,,0.6349206349206349,40.0,0.14285714285714285,9.0,,,0.009,9.5302,,,6.3806,,,70
|
||||||
|
job_653588,0.0,0,64,512,4,1.6094,0,0.8594,3870.6582,3993.4078,3804.6757,31660,1,3994.7037,1414538.0,14.0,0.939,0.0,0.0,0.061,0.3496,-0.014,0.1576,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0445,-0.0001,0.0,1.0,0.0,0.0232,6.827100124823203e+21,8.119763290842973e+24,0.0,0.8805,0.014,0.5,0.3496,16.2904,52.5068,603.2638,29.7,857.9676,254.7038,0.064,5.8584,27.7403,178.662,206.7592,1436.6431,32.111,178.9542,1191.9108,0.0,0.0,0,71,63,504,457,19,47,,,0.031746031746031744,2.0,,,0.6507936507936508,41.0,0.06349206349206349,4.0,,,,,,,,,,71
|
||||||
|
job_653588,0.0,0,64,512,3,1.6094,0,0.875,4486.5332,3847.2383,5028.8953,31774,1,4668.9327,1632410.0,30.0,0.8736,0.0,0.0,0.1264,0.459,0.004,0.0411,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0437,0.0,0.0,1.0,0.0,0.0117,2.0079791588560077e+17,2.5626006126882246e+20,0.0,0.9135,0.0143,0.5,0.459,16.2309,52.5107,603.6477,29.6,857.9676,254.3199,0.063,5.8585,33.7104,209.3972,243.4676,1574.2016,33.0528,209.6937,1291.8188,0.0,0.0,0,72,60,480,436,12,43,0.016666666666666666,1.0,0.08333333333333333,5.0,,,0.55,33.0,0.11666666666666667,7.0,,,0.0102,9.398,,,,0.016666666666666666,1.0,72
|
||||||
|
job_653588,0.0,0,64,512,3,1.625,0,0.9219,4030.5508,3747.5877,4257.7183,31764,1,3919.6801,1430904.0,31.0,0.9122,0.0,0.0,0.0878,0.4453,-0.0008,0.0867,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0396,-0.0001,0.0,1.0,0.0,0.0184,436822212608.0,400335680045056.0,0.0,0.9088,0.0133,0.5469,0.4453,16.4671,52.6346,602.5194,29.8,857.9676,255.4482,0.0953,5.9026,29.5781,177.2825,207.2711,1411.7807,48.8291,177.5971,1149.7701,0.0,0.0,0,73,64,512,477,18,30,0.078125,5.0,0.046875,3.0,,,0.375,24.0,0.046875,3.0,,,,,,,,,,73
|
||||||
|
job_653588,0.0,0,64,512,6,1.9062,0,0.9375,4928.0742,5089.388,4838.3465,31829,1,4458.2535,1709041.0,22.0,0.9236,0.0,0.0,0.0764,0.3574,0.028,0.1375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0459,0.0,0.0,1.0,0.0,0.0152,30543561359360.0,3.860194628573594e+16,0.0,0.941,0.0148,0.4844,0.3574,16.4362,52.6385,601.1992,29.9,857.9676,256.7684,0.0942,6.2333,36.9222,215.6981,253.0054,1584.8429,35.8218,215.9884,1289.7748,0.0,0.0,0,74,60,480,413,18,57,0.16666666666666666,10.0,0.016666666666666666,1.0,,,0.9833333333333333,59.0,0.08333333333333333,5.0,,,0.0087,9.1809,,,,,,74
|
||||||
|
job_653588,0.0,0,64,512,4,2.0469,1,1.0,4001.4727,3601.346,4281.9601,31741,1,3888.6649,1397275.0,23.0,0.9487,0.0,0.0,0.0513,0.4121,0.0048,0.063,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0436,-0.0026,0.0,1.0,0.0,0.0135,8.207850865374003e+16,9.702147778100068e+19,0.0,0.8586,0.0144,0.5312,0.4121,16.4568,52.7637,599.1791,30.2,857.9676,258.7885,0.0656,5.9013,30.0436,169.248,199.6035,1374.3084,36.1758,169.4935,1132.5895,0.0,0.0,0,75,63,504,422,22,64,0.2857142857142857,18.0,0.1111111111111111,7.0,,,0.8253968253968254,52.0,0.09523809523809523,6.0,,,,,,,6.5286,,,75
|
||||||
|
job_653588,0.0,0,64,512,4,2.4219,2,1.0,4137.8809,3989.7778,4218.1777,31802,1,4719.6405,1531507.0,24.0,0.8609,0.0,0.0,0.1391,0.3516,0.0177,0.1025,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0403,-0.0055,0.0,1.0,0.0,0.0169,3.817858620540846e+17,4.6935484414353107e+20,0.0,0.8424,0.0134,0.4531,0.3516,16.7703,53.0803,597.8522,30.3,857.9676,260.1154,0.0901,7.0109,36.2508,203.9976,240.5622,1688.1237,39.0785,204.2208,1401.4078,0.0,0.0,0,76,62,496,434,20,62,,,0.016129032258064516,1.0,,,0.8548387096774194,53.0,0.16129032258064516,10.0,,,0.0106,9.297,,,,,,76
|
||||||
|
job_653588,0.0,0,64,512,4,3.1875,3,1.0,4233.2773,3857.1,4420.2661,31765,1,4184.3433,1522291.0,17.0,0.9239,0.0,0.0,0.0761,0.332,0.0005,0.0567,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0415,-0.0,0.0,1.0,0.0,0.0102,2.0191042252585673e+22,2.49119404899192e+25,0.0,0.8799,0.0145,0.4219,0.332,16.9553,53.3296,595.7158,30.6,857.9676,262.2518,0.0855,5.9291,31.0768,185.4559,216.9091,1561.9809,30.9184,185.7463,1308.1743,0.0,0.0,0,77,62,496,395,33,95,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.9193548387096774,57.0,0.5806451612903226,36.0,,,,,,,,,,77
|
||||||
|
job_653588,0.0,0,64,512,5,4.4219,4,1.0,4333.3574,4615.6172,4239.2708,31662,1,4448.1268,1530711.0,16.0,0.9692,0.0,0.0,0.0308,0.25,0.0094,0.0823,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0453,0.0,0.0,1.0,0.0,0.0122,1.5951538773913764e+21,1.973554746569748e+24,0.0,0.8063,0.0155,0.3594,0.25,17.0168,53.3903,595.507,30.6,857.9676,262.4606,0.0912,5.6163,32.4987,196.5511,229.4039,1882.9354,24.8707,196.8133,1622.983,0.0,0.0,0,78,96,768,690,25,71,0.07291666666666667,7.0,0.020833333333333332,2.0,,,0.53125,51.0,0.1875,18.0,,,0.0105,9.6184,,,,,,78
|
||||||
|
job_653588,0.0,0,64,512,0,0.0,0,0.0,3256.0234,2826.6427,4613.9837,13362,1,1838.6673,1104019.0,5.0,0.996,0.0,0.0,0.004,0.7598,-0.0018,0.0763,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0334,-0.0011,0.0,1.0,0.0,0.0149,8603.9307,9039291.0,0.0,1.0023,0.0111,0.8125,0.7598,16.4131,52.7717,596.1255,30.5,857.9676,261.842,0.0432,2.9452,20.7324,109.5379,130.5083,2969.965,32.266,109.7322,2804.1924,0.0,0.0,1,79,61,488,428,22,60,,,0.03278688524590164,2.0,,,0.819672131147541,50.0,0.19672131147540983,12.0,,,,,,,,0.03278688524590164,2.0,79
|
||||||
|
job_653588,0.0,0,64,512,1,0.7188,0,0.7188,4464.1445,4825.3981,4220.9477,31774,1,4860.2721,1603625.0,28.0,0.9096,0.0,0.0,0.0904,0.4023,0.0085,0.1165,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0434,-0.0001,0.0,1.0,0.0,0.0207,2.0403219462220677e+19,2.583743478900106e+22,0.0,0.8802,0.0142,0.5312,0.4023,17.2875,53.606,593.67,30.8,857.9676,264.2975,0.092,5.9528,32.0153,208.4403,240.7613,2243.0022,39.1203,208.6536,1957.1655,0.0,0.0,1,80,57,456,397,21,54,0.08771929824561403,5.0,0.05263157894736842,3.0,,,0.8070175438596491,46.0,0.08771929824561403,5.0,,,0.0085,10.2197,,,6.298,,,80
|
||||||
|
job_653589,0.0,0,64,512,0,0.0,0,0.0,3458.0977,3170.4289,4497.3333,17781,1,2176.1485,1160818.0,8.0,0.9962,0.0,0.0,0.0038,0.7832,0.0044,0.0294,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.033,-0.002,0.0,1.0,0.0,0.0092,12.0706,11267.6396,0.0,1.0022,0.0104,0.7969,0.7832,8.6151,50.2076,591.2495,31.1,857.9676,266.7181,0.0741,3.6039,30.4349,124.3234,155.0454,3316.0311,31.9403,124.5359,3125.441,0.0,0.0,1,82,3,24,24,0,0,,,,,,,,,,,,,13.1363,21.5484,,,,,,82
|
||||||
|
3
training_logs/20260609_214045_reward_vs_steps.png
Normal file
3
training_logs/20260609_214045_reward_vs_steps.png
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:612865ae3e1c26bbb62acd3b75a903c3737903b78337e3f078bfae1aef54a63e
|
||||||
|
size 279436
|
||||||
44756
training_logs/20260609_214045_trial_results.csv
Normal file
44756
training_logs/20260609_214045_trial_results.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:8ecd219d48c7d160a3992c1b141f6ffcf6ad7d6e00a5da914040d313d22249ca
|
||||||
|
size 124671
|
||||||
7107
training_logs/20260609_214045_vllm_metrics_job_653585.csv
Normal file
7107
training_logs/20260609_214045_vllm_metrics_job_653585.csv
Normal file
File diff suppressed because it is too large
Load Diff
8495
training_logs/20260609_214045_vllm_metrics_job_653586.csv
Normal file
8495
training_logs/20260609_214045_vllm_metrics_job_653586.csv
Normal file
File diff suppressed because it is too large
Load Diff
8509
training_logs/20260609_214045_vllm_metrics_job_653587.csv
Normal file
8509
training_logs/20260609_214045_vllm_metrics_job_653587.csv
Normal file
File diff suppressed because it is too large
Load Diff
7128
training_logs/20260609_214045_vllm_metrics_job_653588.csv
Normal file
7128
training_logs/20260609_214045_vllm_metrics_job_653588.csv
Normal file
File diff suppressed because it is too large
Load Diff
661
training_logs/20260609_214045_vllm_metrics_job_653589.csv
Normal file
661
training_logs/20260609_214045_vllm_metrics_job_653589.csv
Normal file
@@ -0,0 +1,661 @@
|
|||||||
|
datetime_str,n_engines_reporting,unique_ips,total_prompt_throughput_tokens_per_sec,total_generation_throughput_tokens_per_sec,total_running_requests,total_waiting_requests,avg_prompt_throughput_per_engine,avg_generation_throughput_per_engine,avg_running_requests_per_engine,avg_waiting_requests_per_engine,avg_gpu_kv_cache_usage_pct,avg_prefix_cache_hit_rate_pct,min_running_requests,max_running_requests,min_generation_throughput,max_generation_throughput
|
||||||
|
06-09 20:15:11,1,1,8.0,0.0,1,0,8.0,0.0,1.0,0.0,0.3,0.0,1,1,0.0,0.0
|
||||||
|
06-09 20:15:15,1,1,8.4,0.0,1,0,8.4,0.0,1.0,0.0,0.3,0.0,1,1,0.0,0.0
|
||||||
|
06-09 20:15:21,1,1,0.0,89.0,1,0,0.0,89.0,1.0,0.0,0.4,32.1,1,1,89.0,89.0
|
||||||
|
06-09 20:15:26,1,1,38.6,85.6,2,0,38.6,85.6,2.0,0.0,0.8,55.8,2,2,85.6,85.6
|
||||||
|
06-09 20:15:31,1,1,0.0,69.5,1,0,0.0,69.5,1.0,0.0,0.5,55.8,1,1,69.5,69.5
|
||||||
|
06-09 20:15:36,1,1,65.8,83.9,2,0,65.8,83.9,2.0,0.0,1.1,66.6,2,2,83.9,83.9
|
||||||
|
06-09 20:15:41,1,1,0.0,94.7,2,0,0.0,94.7,2.0,0.0,1.2,66.6,2,2,94.7,94.7
|
||||||
|
06-09 20:15:46,1,1,0.0,94.5,2,0,0.0,94.5,2.0,0.0,1.3,66.6,2,2,94.5,94.5
|
||||||
|
06-09 20:15:51,1,1,0.0,94.5,2,0,0.0,94.5,2.0,0.0,1.5,66.6,2,2,94.5,94.5
|
||||||
|
06-09 20:15:56,1,1,0.0,35.5,1,0,0.0,35.5,1.0,0.0,0.7,0.0,1,1,35.5,35.5
|
||||||
|
06-09 20:16:01,1,1,0.0,35.7,1,0,0.0,35.7,1.0,0.0,0.8,0.0,1,1,35.7,35.7
|
||||||
|
06-09 20:16:06,1,1,0.0,36.9,1,0,0.0,36.9,1.0,0.0,0.8,0.0,1,1,36.9,36.9
|
||||||
|
06-09 20:16:11,1,1,0.0,50.6,1,0,0.0,50.6,1.0,0.0,0.3,0.0,1,1,50.6,50.6
|
||||||
|
06-09 20:16:16,1,1,0.0,50.9,1,0,0.0,50.9,1.0,0.0,0.4,0.0,1,1,50.9,50.9
|
||||||
|
06-09 20:16:21,1,1,0.0,36.0,1,0,0.0,36.0,1.0,0.0,1.0,0.0,1,1,36.0,36.0
|
||||||
|
06-09 20:16:26,1,1,0.0,36.2,1,0,0.0,36.2,1.0,0.0,1.0,0.0,1,1,36.2,36.2
|
||||||
|
06-09 20:16:31,1,1,34.3,63.0,2,0,34.3,63.0,2.0,0.0,1.0,56.2,2,2,63.0,63.0
|
||||||
|
06-09 20:16:36,1,1,50.1,97.0,3,0,50.1,97.0,3.0,0.0,1.4,64.4,3,3,97.0,97.0
|
||||||
|
06-09 20:16:41,1,1,0.0,137.4,3,0,0.0,137.4,3.0,0.0,1.6,64.4,3,3,137.4,137.4
|
||||||
|
06-09 20:16:46,1,1,142.4,117.9,3,0,142.4,117.9,3.0,0.0,1.8,69.6,3,3,117.9,117.9
|
||||||
|
06-09 20:16:51,1,1,0.0,138.3,3,0,0.0,138.3,3.0,0.0,2.0,69.6,3,3,138.3,138.3
|
||||||
|
06-09 20:16:56,1,1,0.0,139.7,3,0,0.0,139.7,3.0,0.0,2.2,69.6,3,3,139.7,139.7
|
||||||
|
06-09 20:17:01,1,1,0.0,98.1,1,0,0.0,98.1,1.0,0.0,0.8,69.6,1,1,98.1,98.1
|
||||||
|
06-09 20:17:06,1,1,0.0,46.4,1,0,0.0,46.4,1.0,0.0,0.9,69.6,1,1,46.4,46.4
|
||||||
|
06-09 20:17:11,1,1,0.0,47.7,1,0,0.0,47.7,1.0,0.0,0.8,53.5,1,1,47.7,47.7
|
||||||
|
06-09 20:17:16,1,1,431.3,220.3,6,0,431.3,220.3,6.0,0.0,6.0,81.2,6,6,220.3,220.3
|
||||||
|
06-09 20:17:21,1,1,0.0,270.0,6,0,0.0,270.0,6.0,0.0,6.4,81.2,6,6,270.0,270.0
|
||||||
|
06-09 20:17:27,1,1,0.0,195.1,3,0,0.0,195.1,3.0,0.0,5.4,84.7,3,3,195.1,195.1
|
||||||
|
06-09 20:17:32,1,1,0.0,146.5,3,0,0.0,146.5,3.0,0.0,4.9,84.3,3,3,146.5,146.5
|
||||||
|
06-09 20:17:37,1,1,21.3,147.2,4,0,21.3,147.2,4.0,0.0,5.9,85.7,4,4,147.2,147.2
|
||||||
|
06-09 20:17:42,1,1,0.0,195.6,4,0,0.0,195.6,4.0,0.0,6.2,85.7,4,4,195.6,195.6
|
||||||
|
06-09 20:17:47,1,1,0.0,242.3,5,0,0.0,242.3,5.0,0.0,8.2,79.9,5,5,242.3,242.3
|
||||||
|
06-09 20:17:52,1,1,259.8,215.9,5,0,259.8,215.9,5.0,0.0,8.8,80.3,5,5,215.9,215.9
|
||||||
|
06-09 20:17:57,1,1,0.0,241.3,5,0,0.0,241.3,5.0,0.0,9.2,80.3,5,5,241.3,241.3
|
||||||
|
06-09 20:18:02,1,1,0.0,241.7,5,0,0.0,241.7,5.0,0.0,9.5,80.3,5,5,241.7,241.7
|
||||||
|
06-09 20:18:07,1,1,0.0,247.3,5,0,0.0,247.3,5.0,0.0,9.8,80.3,5,5,247.3,247.3
|
||||||
|
06-09 20:18:12,1,1,0.0,247.6,5,0,0.0,247.6,5.0,0.0,10.2,80.3,5,5,247.6,247.6
|
||||||
|
06-09 20:18:17,1,1,0.0,248.0,5,0,0.0,248.0,5.0,0.0,10.5,80.3,5,5,248.0,248.0
|
||||||
|
06-09 20:18:22,1,1,0.0,245.9,5,0,0.0,245.9,5.0,0.0,10.9,80.3,5,5,245.9,245.9
|
||||||
|
06-09 20:18:27,1,1,0.0,247.8,5,0,0.0,247.8,5.0,0.0,11.2,80.3,5,5,247.8,247.8
|
||||||
|
06-09 20:18:32,1,1,0.0,248.3,5,0,0.0,248.3,5.0,0.0,11.6,80.3,5,5,248.3,248.3
|
||||||
|
06-09 20:18:37,1,1,77.5,252.4,6,0,77.5,252.4,6.0,0.0,9.7,90.5,6,6,252.4,252.4
|
||||||
|
06-09 20:18:42,1,1,0.0,247.8,5,0,0.0,247.8,5.0,0.0,8.1,90.5,5,5,247.8,247.8
|
||||||
|
06-09 20:18:47,1,1,112.3,257.6,6,0,112.3,257.6,6.0,0.0,10.5,91.2,6,6,257.6,257.6
|
||||||
|
06-09 20:18:52,1,1,0.0,275.5,6,0,0.0,275.5,6.0,0.0,10.9,91.2,6,6,275.5,275.5
|
||||||
|
06-09 20:18:57,1,1,0.0,275.8,6,0,0.0,275.8,6.0,0.0,11.3,91.2,6,6,275.8,275.8
|
||||||
|
06-09 20:19:02,1,1,0.0,127.9,3,0,0.0,127.9,3.0,0.0,5.8,83.7,3,3,127.9,127.9
|
||||||
|
06-09 20:19:07,1,1,0.0,103.6,2,0,0.0,103.6,2.0,0.0,3.8,83.7,2,2,103.6,103.6
|
||||||
|
06-09 20:19:12,1,1,0.0,75.6,2,0,0.0,75.6,2.0,0.0,3.9,83.7,2,2,75.6,75.6
|
||||||
|
06-09 20:19:17,1,1,0.0,166.2,4,0,0.0,166.2,4.0,0.0,10.8,90.7,4,4,166.2,166.2
|
||||||
|
06-09 20:19:22,1,1,0.0,166.7,4,0,0.0,166.7,4.0,0.0,11.1,90.7,4,4,166.7,166.7
|
||||||
|
06-09 20:19:27,1,1,128.6,231.8,7,0,128.6,231.8,7.0,0.0,11.8,86.0,7,7,231.8,231.8
|
||||||
|
06-09 20:19:32,1,1,0.0,144.4,3,0,0.0,144.4,3.0,0.0,6.9,86.8,3,3,144.4,144.4
|
||||||
|
06-09 20:19:37,1,1,20.1,109.9,4,0,20.1,109.9,4.0,0.0,7.9,87.3,4,4,109.9,109.9
|
||||||
|
06-09 20:19:42,1,1,0.0,149.4,4,0,0.0,149.4,4.0,0.0,8.2,87.3,4,4,149.4,149.4
|
||||||
|
06-09 20:19:47,1,1,120.9,162.8,4,0,120.9,162.8,4.0,0.0,7.6,86.8,4,4,162.8,162.8
|
||||||
|
06-09 20:19:52,1,1,171.0,139.3,5,0,171.0,139.3,5.0,0.0,8.0,87.5,5,5,139.3,139.3
|
||||||
|
06-09 20:19:57,1,1,0.0,183.9,5,0,0.0,183.9,5.0,0.0,8.3,87.5,5,5,183.9,183.9
|
||||||
|
06-09 20:20:02,1,1,0.0,167.2,4,0,0.0,167.2,4.0,0.0,5.5,87.5,4,4,167.2,167.2
|
||||||
|
06-09 20:20:07,1,1,83.4,154.9,5,0,83.4,154.9,5.0,0.0,8.9,88.5,5,5,154.9,154.9
|
||||||
|
06-09 20:20:12,1,1,0.0,172.9,4,0,0.0,172.9,4.0,0.0,8.7,88.5,4,4,172.9,172.9
|
||||||
|
06-09 20:20:17,1,1,102.0,158.8,5,0,102.0,158.8,5.0,0.0,9.5,88.4,5,5,158.8,158.8
|
||||||
|
06-09 20:20:22,1,1,91.3,213.1,6,0,91.3,213.1,6.0,0.0,9.9,88.1,6,6,213.1,213.1
|
||||||
|
06-09 20:20:28,1,1,0.0,224.0,6,0,0.0,224.0,6.0,0.0,10.2,88.1,6,6,224.0,224.0
|
||||||
|
06-09 20:20:33,1,1,139.2,205.4,6,0,139.2,205.4,6.0,0.0,14.7,87.6,6,6,205.4,205.4
|
||||||
|
06-09 20:20:38,1,1,42.1,219.1,4,0,42.1,219.1,4.0,0.0,10.8,86.9,4,4,219.1,219.1
|
||||||
|
06-09 20:20:43,1,1,0.0,191.2,4,0,0.0,191.2,4.0,0.0,11.1,86.9,4,4,191.2,191.2
|
||||||
|
06-09 20:20:48,1,1,0.0,49.3,1,0,0.0,49.3,1.0,0.0,2.6,85.5,1,1,49.3,49.3
|
||||||
|
06-09 20:20:53,1,1,0.0,49.3,1,0,0.0,49.3,1.0,0.0,2.7,85.5,1,1,49.3,49.3
|
||||||
|
06-09 20:20:58,1,1,64.3,209.9,5,0,64.3,209.9,5.0,0.0,14.0,89.0,5,5,209.9,209.9
|
||||||
|
06-09 20:21:03,1,1,0.0,238.4,5,0,0.0,238.4,5.0,0.0,14.3,89.0,5,5,238.4,238.4
|
||||||
|
06-09 20:21:08,1,1,22.3,227.0,5,0,22.3,227.0,5.0,0.0,14.7,90.0,5,5,227.0,227.0
|
||||||
|
06-09 20:21:13,1,1,41.2,124.2,3,0,41.2,124.2,3.0,0.0,7.3,83.8,3,3,124.2,124.2
|
||||||
|
06-09 20:21:18,1,1,62.1,154.7,3,0,62.1,154.7,3.0,0.0,7.3,83.7,3,3,154.7,154.7
|
||||||
|
06-09 20:21:23,1,1,0.0,49.0,1,0,0.0,49.0,1.0,0.0,3.1,85.5,1,1,49.0,49.0
|
||||||
|
06-09 20:21:28,1,1,0.0,49.1,1,0,0.0,49.1,1.0,0.0,3.2,85.5,1,1,49.1,49.1
|
||||||
|
06-09 20:21:33,1,1,153.3,192.0,5,0,153.3,192.0,5.0,0.0,10.0,86.2,5,5,192.0,192.0
|
||||||
|
06-09 20:21:38,1,1,0.0,300.6,6,0,0.0,300.6,6.0,0.0,13.5,84.1,6,6,300.6,300.6
|
||||||
|
06-09 20:21:43,1,1,123.8,241.0,6,0,123.8,241.0,6.0,0.0,14.0,84.7,6,6,241.0,241.0
|
||||||
|
06-09 20:21:48,1,1,89.8,251.7,7,0,89.8,251.7,7.0,0.0,14.8,85.9,7,7,251.7,251.7
|
||||||
|
06-09 20:21:53,1,1,0.0,302.1,7,0,0.0,302.1,7.0,0.0,15.2,85.9,7,7,302.1,302.1
|
||||||
|
06-09 20:21:58,1,1,0.0,328.9,7,0,0.0,328.9,7.0,0.0,16.5,85.1,7,7,328.9,328.9
|
||||||
|
06-09 20:22:03,1,1,0.0,328.4,7,0,0.0,328.4,7.0,0.0,17.0,85.1,7,7,328.4,328.4
|
||||||
|
06-09 20:22:08,1,1,171.5,235.6,6,0,171.5,235.6,6.0,0.0,12.9,87.1,6,6,235.6,235.6
|
||||||
|
06-09 20:22:13,1,1,0.0,237.0,6,0,0.0,237.0,6.0,0.0,13.2,87.1,6,6,237.0,237.0
|
||||||
|
06-09 20:22:18,1,1,20.9,231.1,6,0,20.9,231.1,6.0,0.0,13.6,88.3,6,6,231.1,231.1
|
||||||
|
06-09 20:22:23,1,1,0.0,240.4,6,0,0.0,240.4,6.0,0.0,13.9,88.3,6,6,240.4,240.4
|
||||||
|
06-09 20:22:28,1,1,0.0,209.8,5,0,0.0,209.8,5.0,0.0,13.3,88.3,5,5,209.8,209.8
|
||||||
|
06-09 20:22:33,1,1,185.9,164.8,5,0,185.9,164.8,5.0,0.0,12.9,89.2,5,5,164.8,164.8
|
||||||
|
06-09 20:22:38,1,1,169.1,206.4,6,0,169.1,206.4,6.0,0.0,15.2,89.2,6,6,206.4,206.4
|
||||||
|
06-09 20:22:43,1,1,0.0,234.8,6,0,0.0,234.8,6.0,0.0,15.6,89.2,6,6,234.8,234.8
|
||||||
|
06-09 20:22:48,1,1,0.0,198.5,4,0,0.0,198.5,4.0,0.0,9.7,89.2,4,4,198.5,198.5
|
||||||
|
06-09 20:22:54,1,1,115.3,180.6,5,0,115.3,180.6,5.0,0.0,14.0,89.8,5,5,180.6,180.6
|
||||||
|
06-09 20:22:59,1,1,61.0,186.2,5,0,61.0,186.2,5.0,0.0,15.3,90.2,5,5,186.2,186.2
|
||||||
|
06-09 20:23:04,1,1,0.0,202.0,5,0,0.0,202.0,5.0,0.0,15.6,90.2,5,5,202.0,202.0
|
||||||
|
06-09 20:23:09,1,1,106.2,224.9,6,0,106.2,224.9,6.0,0.0,16.0,90.4,6,6,224.9,224.9
|
||||||
|
06-09 20:23:14,1,1,96.5,244.8,7,0,96.5,244.8,7.0,0.0,17.8,90.5,7,7,244.8,244.8
|
||||||
|
06-09 20:23:19,1,1,28.7,256.4,6,0,28.7,256.4,6.0,0.0,15.7,90.5,6,6,256.4,256.4
|
||||||
|
06-09 20:23:24,1,1,0.0,215.5,4,0,0.0,215.5,4.0,0.0,14.1,90.5,4,4,215.5,215.5
|
||||||
|
06-09 20:23:29,1,1,115.0,193.3,6,0,115.0,193.3,6.0,0.0,16.4,90.6,6,6,193.3,193.3
|
||||||
|
06-09 20:23:34,1,1,0.0,239.5,6,0,0.0,239.5,6.0,0.0,16.8,90.6,6,6,239.5,239.5
|
||||||
|
06-09 20:23:39,1,1,94.1,221.8,4,0,94.1,221.8,4.0,0.0,16.3,86.2,4,4,221.8,221.8
|
||||||
|
06-09 20:23:44,1,1,120.4,197.6,5,0,120.4,197.6,5.0,0.0,15.7,90.4,5,5,197.6,197.6
|
||||||
|
06-09 20:23:49,1,1,0.0,234.2,5,0,0.0,234.2,5.0,0.0,18.1,87.0,5,5,234.2,234.2
|
||||||
|
06-09 20:23:54,1,1,0.0,234.2,5,0,0.0,234.2,5.0,0.0,18.4,87.0,5,5,234.2,234.2
|
||||||
|
06-09 20:23:59,1,1,0.0,233.5,5,0,0.0,233.5,5.0,0.0,18.7,87.0,5,5,233.5,233.5
|
||||||
|
06-09 20:24:04,1,1,216.6,252.4,6,0,216.6,252.4,6.0,0.0,19.4,86.8,6,6,252.4,252.4
|
||||||
|
06-09 20:24:09,1,1,0.0,280.8,6,0,0.0,280.8,6.0,0.0,19.8,86.8,6,6,280.8,280.8
|
||||||
|
06-09 20:24:14,1,1,0.0,269.5,5,0,0.0,269.5,5.0,0.0,19.9,86.8,5,5,269.5,269.5
|
||||||
|
06-09 20:24:19,1,1,162.8,255.2,6,0,162.8,255.2,6.0,0.0,20.7,87.4,6,6,255.2,255.2
|
||||||
|
06-09 20:24:24,1,1,0.0,257.7,5,0,0.0,257.7,5.0,0.0,20.7,87.4,5,5,257.7,257.7
|
||||||
|
06-09 20:24:29,1,1,0.0,235.8,5,0,0.0,235.8,5.0,0.0,21.0,87.4,5,5,235.8,235.8
|
||||||
|
06-09 20:24:34,1,1,68.2,273.3,6,0,68.2,273.3,6.0,0.0,21.9,87.3,6,6,273.3,273.3
|
||||||
|
06-09 20:24:39,1,1,0.0,259.1,5,0,0.0,259.1,5.0,0.0,21.6,87.3,5,5,259.1,259.1
|
||||||
|
06-09 20:24:44,1,1,121.1,265.7,7,0,121.1,265.7,7.0,0.0,22.8,87.3,7,7,265.7,265.7
|
||||||
|
06-09 20:24:49,1,1,0.0,326.6,7,0,0.0,326.6,7.0,0.0,23.3,87.3,7,7,326.6,326.6
|
||||||
|
06-09 20:24:54,1,1,0.0,416.8,9,0,0.0,416.8,9.0,0.0,33.9,86.3,9,9,416.8,416.8
|
||||||
|
06-09 20:24:59,1,1,137.3,449.0,10,0,137.3,449.0,10.0,0.0,35.5,86.2,10,10,449.0,449.0
|
||||||
|
06-09 20:25:04,1,1,9.6,477.2,11,0,9.6,477.2,11.0,0.0,36.2,86.3,11,11,477.2,477.2
|
||||||
|
06-09 20:25:09,1,1,72.3,496.9,12,0,72.3,496.9,12.0,0.0,37.0,86.1,12,12,496.9,496.9
|
||||||
|
06-09 20:25:14,1,1,0.0,339.9,6,0,0.0,339.9,6.0,0.0,20.0,86.7,6,6,339.9,339.9
|
||||||
|
06-09 20:25:19,1,1,0.0,539.6,12,0,0.0,539.6,12.0,0.0,38.5,86.1,12,12,539.6,539.6
|
||||||
|
06-09 20:25:24,1,1,0.0,539.8,12,0,0.0,539.8,12.0,0.0,39.3,86.1,12,12,539.8,539.8
|
||||||
|
06-09 20:25:29,1,1,0.0,540.6,12,0,0.0,540.6,12.0,0.0,40.0,86.1,12,12,540.6,540.6
|
||||||
|
06-09 20:25:34,1,1,0.0,539.5,11,0,0.0,539.5,11.0,0.0,39.3,86.1,11,11,539.5,539.5
|
||||||
|
06-09 20:25:39,1,1,18.7,537.1,12,0,18.7,537.1,12.0,0.0,41.6,86.6,12,12,537.1,537.1
|
||||||
|
06-09 20:25:44,1,1,0.0,541.0,12,0,0.0,541.0,12.0,0.0,42.3,86.6,12,12,541.0,541.0
|
||||||
|
06-09 20:25:49,1,1,0.0,542.1,12,0,0.0,542.1,12.0,0.0,43.1,86.6,12,12,542.1,542.1
|
||||||
|
06-09 20:25:55,1,1,0.0,240.9,5,0,0.0,240.9,5.0,0.0,18.9,86.8,5,5,240.9,240.9
|
||||||
|
06-09 20:26:00,1,1,80.9,263.2,6,0,80.9,263.2,6.0,0.0,12.4,86.9,6,6,263.2,263.2
|
||||||
|
06-09 20:26:05,1,1,0.0,278.5,6,0,0.0,278.5,6.0,0.0,12.7,86.9,6,6,278.5,278.5
|
||||||
|
06-09 20:26:10,1,1,0.0,278.0,6,0,0.0,278.0,6.0,0.0,13.1,86.9,6,6,278.0,278.0
|
||||||
|
06-09 20:26:15,1,1,0.0,192.4,4,0,0.0,192.4,4.0,0.0,12.9,91.9,4,4,192.4,192.4
|
||||||
|
06-09 20:26:20,1,1,102.3,267.9,6,0,102.3,267.9,6.0,0.0,14.3,91.9,6,6,267.9,267.9
|
||||||
|
06-09 20:26:25,1,1,101.4,278.9,6,0,101.4,278.9,6.0,0.0,14.1,91.8,6,6,278.9,278.9
|
||||||
|
06-09 20:26:30,1,1,59.5,294.1,7,0,59.5,294.1,7.0,0.0,15.4,91.8,7,7,294.1,294.1
|
||||||
|
06-09 20:26:35,1,1,143.2,346.1,8,0,143.2,346.1,8.0,0.0,18.6,91.9,8,8,346.1,346.1
|
||||||
|
06-09 20:26:40,1,1,0.0,335.5,7,0,0.0,335.5,7.0,0.0,16.1,91.9,7,7,335.5,335.5
|
||||||
|
06-09 20:26:45,1,1,65.0,320.8,7,0,65.0,320.8,7.0,0.0,15.9,91.9,7,7,320.8,320.8
|
||||||
|
06-09 20:26:50,1,1,21.2,340.3,8,0,21.2,340.3,8.0,0.0,17.4,91.9,8,8,340.3,340.3
|
||||||
|
06-09 20:26:55,1,1,9.6,372.0,9,0,9.6,372.0,9.0,0.0,20.9,92.2,9,9,372.0,372.0
|
||||||
|
06-09 20:27:00,1,1,0.0,380.7,7,0,0.0,380.7,7.0,0.0,17.9,92.2,7,7,380.7,380.7
|
||||||
|
06-09 20:27:05,1,1,0.0,307.2,6,0,0.0,307.2,6.0,0.0,17.0,92.2,6,6,307.2,307.2
|
||||||
|
06-09 20:27:10,1,1,0.0,284.8,6,0,0.0,284.8,6.0,0.0,37.6,89.9,6,6,284.8,284.8
|
||||||
|
06-09 20:27:15,1,1,0.0,276.4,6,0,0.0,276.4,6.0,0.0,38.0,89.9,6,6,276.4,276.4
|
||||||
|
06-09 20:27:20,1,1,113.2,284.0,7,0,113.2,284.0,7.0,0.0,40.1,90.0,7,7,284.0,284.0
|
||||||
|
06-09 20:27:25,1,1,0.0,310.5,6,0,0.0,310.5,6.0,0.0,31.6,90.0,6,6,310.5,310.5
|
||||||
|
06-09 20:27:30,1,1,0.0,275.9,6,0,0.0,275.9,6.0,0.0,32.0,90.0,6,6,275.9,275.9
|
||||||
|
06-09 20:27:35,1,1,0.0,174.7,4,0,0.0,174.7,4.0,0.0,21.9,88.5,4,4,174.7,174.7
|
||||||
|
06-09 20:27:40,1,1,77.8,195.7,5,0,77.8,195.7,5.0,0.0,22.3,88.3,5,5,195.7,195.7
|
||||||
|
06-09 20:27:45,1,1,0.0,202.8,5,0,0.0,202.8,5.0,0.0,22.6,88.3,5,5,202.8,202.8
|
||||||
|
06-09 20:27:50,1,1,0.0,204.6,5,0,0.0,204.6,5.0,0.0,22.9,88.3,5,5,204.6,204.6
|
||||||
|
06-09 20:27:55,1,1,0.0,190.6,4,0,0.0,190.6,4.0,0.0,22.8,88.3,4,4,190.6,190.6
|
||||||
|
06-09 20:28:00,1,1,0.0,158.0,4,0,0.0,158.0,4.0,0.0,23.0,88.3,4,4,158.0,158.0
|
||||||
|
06-09 20:28:05,1,1,0.0,158.9,4,0,0.0,158.9,4.0,0.0,23.2,88.3,4,4,158.9,158.9
|
||||||
|
06-09 20:28:10,1,1,89.4,167.8,5,0,89.4,167.8,5.0,0.0,23.9,88.2,5,5,167.8,167.8
|
||||||
|
06-09 20:28:15,1,1,0.0,202.6,5,0,0.0,202.6,5.0,0.0,24.2,88.2,5,5,202.6,202.6
|
||||||
|
06-09 20:28:20,1,1,0.0,198.6,5,0,0.0,198.6,5.0,0.0,24.5,88.2,5,5,198.6,198.6
|
||||||
|
06-09 20:28:25,1,1,0.0,199.5,5,0,0.0,199.5,5.0,0.0,24.8,88.2,5,5,199.5,199.5
|
||||||
|
06-09 20:28:31,1,1,0.0,203.2,5,0,0.0,203.2,5.0,0.0,25.0,88.2,5,5,203.2,203.2
|
||||||
|
06-09 20:28:36,1,1,0.0,162.3,4,0,0.0,162.3,4.0,0.0,24.6,88.2,4,4,162.3,162.3
|
||||||
|
06-09 20:28:41,1,1,0.0,160.4,4,0,0.0,160.4,4.0,0.0,24.8,88.2,4,4,160.4,160.4
|
||||||
|
06-09 20:28:46,1,1,0.0,249.2,4,0,0.0,249.2,4.0,0.0,24.0,86.3,4,4,249.2,249.2
|
||||||
|
06-09 20:28:51,1,1,132.1,266.5,6,0,132.1,266.5,6.0,0.0,26.3,86.6,6,6,266.5,266.5
|
||||||
|
06-09 20:28:56,1,1,0.0,291.6,5,0,0.0,291.6,5.0,0.0,24.8,86.6,5,5,291.6,291.6
|
||||||
|
06-09 20:29:01,1,1,0.0,245.4,4,0,0.0,245.4,4.0,0.0,24.8,86.6,4,4,245.4,245.4
|
||||||
|
06-09 20:29:06,1,1,104.8,221.4,5,0,104.8,221.4,5.0,0.0,27.2,86.9,5,5,221.4,221.4
|
||||||
|
06-09 20:29:11,1,1,39.9,270.3,6,0,39.9,270.3,6.0,0.0,27.9,86.9,6,6,270.3,270.3
|
||||||
|
06-09 20:29:16,1,1,0.0,266.5,5,0,0.0,266.5,5.0,0.0,27.9,86.9,5,5,266.5,266.5
|
||||||
|
06-09 20:29:21,1,1,0.0,222.4,4,0,0.0,222.4,4.0,0.0,26.0,86.9,4,4,222.4,222.4
|
||||||
|
06-09 20:29:26,1,1,44.2,242.3,5,0,44.2,242.3,5.0,0.0,26.5,86.9,5,5,242.3,242.3
|
||||||
|
06-09 20:29:31,1,1,120.4,261.4,5,0,120.4,261.4,5.0,0.0,29.0,87.2,5,5,261.4,261.4
|
||||||
|
06-09 20:29:36,1,1,39.5,334.5,7,0,39.5,334.5,7.0,0.0,30.0,87.3,7,7,334.5,334.5
|
||||||
|
06-09 20:29:41,1,1,0.0,339.3,6,0,0.0,339.3,6.0,0.0,27.8,87.3,6,6,339.3,339.3
|
||||||
|
06-09 20:29:46,1,1,41.1,268.7,6,0,41.1,268.7,6.0,0.0,28.3,87.4,6,6,268.7,268.7
|
||||||
|
06-09 20:29:51,1,1,107.5,238.3,4,0,107.5,238.3,4.0,0.0,19.2,87.3,4,4,238.3,238.3
|
||||||
|
06-09 20:29:56,1,1,0.0,201.4,4,0,0.0,201.4,4.0,0.0,19.4,87.3,4,4,201.4,201.4
|
||||||
|
06-09 20:30:01,1,1,133.2,251.0,6,0,133.2,251.0,6.0,0.0,23.2,87.8,6,6,251.0,251.0
|
||||||
|
06-09 20:30:06,1,1,0.0,212.6,4,0,0.0,212.6,4.0,0.0,12.1,87.5,4,4,212.6,212.6
|
||||||
|
06-09 20:30:11,1,1,0.0,195.5,4,0,0.0,195.5,4.0,0.0,21.7,87.4,4,4,195.5,195.5
|
||||||
|
06-09 20:30:16,1,1,0.0,195.1,4,0,0.0,195.1,4.0,0.0,22.0,87.4,4,4,195.1,195.1
|
||||||
|
06-09 20:30:21,1,1,0.0,194.6,4,0,0.0,194.6,4.0,0.0,22.3,87.4,4,4,194.6,194.6
|
||||||
|
06-09 20:30:26,1,1,0.0,194.0,4,0,0.0,194.0,4.0,0.0,22.5,87.4,4,4,194.0,194.0
|
||||||
|
06-09 20:30:31,1,1,0.0,194.2,4,0,0.0,194.2,4.0,0.0,22.8,87.4,4,4,194.2,194.2
|
||||||
|
06-09 20:30:36,1,1,0.0,194.5,4,0,0.0,194.5,4.0,0.0,23.1,87.4,4,4,194.5,194.5
|
||||||
|
06-09 20:30:41,1,1,78.4,279.3,6,0,78.4,279.3,6.0,0.0,19.4,86.5,6,6,279.3,279.3
|
||||||
|
06-09 20:30:47,1,1,182.5,362.7,8,0,182.5,362.7,8.0,0.0,29.5,92.6,8,8,362.7,362.7
|
||||||
|
06-09 20:30:52,1,1,0.0,380.6,8,0,0.0,380.6,8.0,0.0,30.0,92.6,8,8,380.6,380.6
|
||||||
|
06-09 20:30:57,1,1,0.0,379.7,8,0,0.0,379.7,8.0,0.0,30.5,92.6,8,8,379.7,379.7
|
||||||
|
06-09 20:31:02,1,1,0.0,380.5,8,0,0.0,380.5,8.0,0.0,31.0,92.6,8,8,380.5,380.5
|
||||||
|
06-09 20:31:07,1,1,0.0,380.0,8,0,0.0,380.0,8.0,0.0,31.6,92.6,8,8,380.0,380.0
|
||||||
|
06-09 20:31:12,1,1,0.0,380.8,8,0,0.0,380.8,8.0,0.0,32.1,92.6,8,8,380.8,380.8
|
||||||
|
06-09 20:31:17,1,1,86.8,423.1,9,0,86.8,423.1,9.0,0.0,32.8,92.6,9,9,423.1,423.1
|
||||||
|
06-09 20:31:22,1,1,0.0,92.4,2,0,0.0,92.4,2.0,0.0,5.1,87.9,2,2,92.4,92.4
|
||||||
|
06-09 20:31:27,1,1,269.5,119.3,3,0,269.5,119.3,3.0,0.0,6.5,87.6,3,3,119.3,119.3
|
||||||
|
06-09 20:31:32,1,1,0.0,138.7,3,0,0.0,138.7,3.0,0.0,6.7,87.6,3,3,138.7,138.7
|
||||||
|
06-09 20:31:37,1,1,0.0,117.5,2,0,0.0,117.5,2.0,0.0,5.4,87.6,2,2,117.5,117.5
|
||||||
|
06-09 20:31:42,1,1,0.0,92.5,2,0,0.0,92.5,2.0,0.0,5.6,87.6,2,2,92.5,92.5
|
||||||
|
06-09 20:31:47,1,1,0.0,92.5,2,0,0.0,92.5,2.0,0.0,5.7,87.6,2,2,92.5,92.5
|
||||||
|
06-09 20:31:52,1,1,0.0,84.2,1,0,0.0,84.2,1.0,0.0,4.7,87.6,1,1,84.2,84.2
|
||||||
|
06-09 20:31:57,1,1,0.0,138.3,3,0,0.0,138.3,3.0,0.0,6.7,86.7,3,3,138.3,138.3
|
||||||
|
06-09 20:32:02,1,1,0.0,138.1,3,0,0.0,138.1,3.0,0.0,6.9,86.7,3,3,138.1,138.1
|
||||||
|
06-09 20:32:07,1,1,0.0,138.1,3,0,0.0,138.1,3.0,0.0,7.1,86.7,3,3,138.1,138.1
|
||||||
|
06-09 20:32:12,1,1,0.0,138.2,3,0,0.0,138.2,3.0,0.0,7.3,86.7,3,3,138.2,138.2
|
||||||
|
06-09 20:32:17,1,1,0.0,138.2,3,0,0.0,138.2,3.0,0.0,7.5,86.7,3,3,138.2,138.2
|
||||||
|
06-09 20:32:22,1,1,0.0,138.1,3,0,0.0,138.1,3.0,0.0,7.7,86.7,3,3,138.1,138.1
|
||||||
|
06-09 20:32:27,1,1,0.0,138.3,3,0,0.0,138.3,3.0,0.0,7.9,86.7,3,3,138.3,138.3
|
||||||
|
06-09 20:32:32,1,1,0.0,137.8,3,0,0.0,137.8,3.0,0.0,8.1,86.7,3,3,137.8,137.8
|
||||||
|
06-09 20:32:37,1,1,0.0,137.7,3,0,0.0,137.7,3.0,0.0,8.3,86.7,3,3,137.7,137.7
|
||||||
|
06-09 20:32:42,1,1,53.4,179.9,4,0,53.4,179.9,4.0,0.0,8.8,86.6,4,4,179.9,179.9
|
||||||
|
06-09 20:32:47,1,1,0.0,84.9,2,0,0.0,84.9,2.0,0.0,9.5,91.0,2,2,84.9,84.9
|
||||||
|
06-09 20:32:52,1,1,0.0,83.3,2,0,0.0,83.3,2.0,0.0,9.6,91.0,2,2,83.3,83.3
|
||||||
|
06-09 20:32:57,1,1,0.0,82.0,2,0,0.0,82.0,2.0,0.0,9.7,91.0,2,2,82.0,82.0
|
||||||
|
06-09 20:33:02,1,1,0.0,83.1,2,0,0.0,83.1,2.0,0.0,9.8,91.0,2,2,83.1,83.1
|
||||||
|
06-09 20:33:07,1,1,0.0,82.5,2,0,0.0,82.5,2.0,0.0,9.9,91.0,2,2,82.5,82.5
|
||||||
|
06-09 20:33:12,1,1,0.0,82.0,2,0,0.0,82.0,2.0,0.0,10.0,91.0,2,2,82.0,82.0
|
||||||
|
06-09 20:33:17,1,1,0.0,71.7,1,0,0.0,71.7,1.0,0.0,2.2,91.0,1,1,71.7,71.7
|
||||||
|
06-09 20:33:22,1,1,0.0,41.8,1,0,0.0,41.8,1.0,0.0,2.2,91.0,1,1,41.8,41.8
|
||||||
|
06-09 20:33:27,1,1,0.0,41.6,1,0,0.0,41.6,1.0,0.0,2.3,91.0,1,1,41.6,41.6
|
||||||
|
06-09 20:33:32,1,1,0.0,41.4,1,0,0.0,41.4,1.0,0.0,2.3,91.0,1,1,41.4,41.4
|
||||||
|
06-09 20:33:37,1,1,0.0,42.0,1,0,0.0,42.0,1.0,0.0,2.4,91.0,1,1,42.0,42.0
|
||||||
|
06-09 20:33:42,1,1,0.0,40.9,1,0,0.0,40.9,1.0,0.0,2.5,91.0,1,1,40.9,40.9
|
||||||
|
06-09 20:33:47,1,1,0.0,41.1,1,0,0.0,41.1,1.0,0.0,2.5,91.0,1,1,41.1,41.1
|
||||||
|
06-09 20:33:52,1,1,185.7,75.8,2,0,185.7,75.8,2.0,0.0,4.6,90.9,2,2,75.8,75.8
|
||||||
|
06-09 20:33:58,1,1,111.1,563.7,13,0,111.1,563.7,13.0,0.0,45.5,86.9,13,13,563.7,563.7
|
||||||
|
06-09 20:34:03,1,1,0.0,602.8,13,0,0.0,602.8,13.0,0.0,46.4,86.9,13,13,602.8,602.8
|
||||||
|
06-09 20:34:08,1,1,0.0,602.5,13,0,0.0,602.5,13.0,0.0,47.2,86.9,13,13,602.5,602.5
|
||||||
|
06-09 20:34:13,1,1,0.0,569.5,11,0,0.0,569.5,11.0,0.0,45.8,86.9,11,11,569.5,569.5
|
||||||
|
06-09 20:34:18,1,1,111.0,526.0,12,0,111.0,526.0,12.0,0.0,47.7,86.9,12,12,526.0,526.0
|
||||||
|
06-09 20:34:23,1,1,174.7,580.4,13,0,174.7,580.4,13.0,0.0,49.9,86.9,13,13,580.4,580.4
|
||||||
|
06-09 20:34:28,1,1,2.0,610.8,13,0,2.0,610.8,13.0,0.0,50.3,87.0,13,13,610.8,610.8
|
||||||
|
06-09 20:34:33,1,1,45.3,617.9,14,0,45.3,617.9,14.0,0.0,52.5,87.5,14,14,617.9,617.9
|
||||||
|
06-09 20:34:38,1,1,0.0,384.7,8,0,0.0,384.7,8.0,0.0,31.6,89.7,8,8,384.7,384.7
|
||||||
|
06-09 20:34:43,1,1,0.0,385.7,8,0,0.0,385.7,8.0,0.0,32.2,89.7,8,8,385.7,385.7
|
||||||
|
06-09 20:34:48,1,1,0.0,383.5,8,0,0.0,383.5,8.0,0.0,32.7,89.7,8,8,383.5,383.5
|
||||||
|
06-09 20:34:53,1,1,0.0,385.1,8,0,0.0,385.1,8.0,0.0,33.2,89.7,8,8,385.1,385.1
|
||||||
|
06-09 20:34:58,1,1,0.0,384.7,8,0,0.0,384.7,8.0,0.0,33.8,89.7,8,8,384.7,384.7
|
||||||
|
06-09 20:35:03,1,1,0.0,384.5,8,0,0.0,384.5,8.0,0.0,34.3,89.7,8,8,384.5,384.5
|
||||||
|
06-09 20:35:08,1,1,50.0,327.0,10,0,50.0,327.0,10.0,0.0,36.2,86.0,10,10,327.0,327.0
|
||||||
|
06-09 20:35:13,1,1,0.0,360.8,10,0,0.0,360.8,10.0,0.0,36.8,86.0,10,10,360.8,360.8
|
||||||
|
06-09 20:35:18,1,1,0.0,347.5,10,0,0.0,347.5,10.0,0.0,37.2,86.0,10,10,347.5,347.5
|
||||||
|
06-09 20:35:23,1,1,189.2,337.9,10,0,189.2,337.9,10.0,0.0,38.9,86.0,10,10,337.9,337.9
|
||||||
|
06-09 20:35:28,1,1,8.7,368.5,11,0,8.7,368.5,11.0,0.0,39.4,86.0,11,11,368.5,368.5
|
||||||
|
06-09 20:35:33,1,1,0.0,384.1,11,0,0.0,384.1,11.0,0.0,39.9,86.0,11,11,384.1,384.1
|
||||||
|
06-09 20:35:39,1,1,0.0,382.4,10,0,0.0,382.4,10.0,0.0,40.4,86.0,10,10,382.4,382.4
|
||||||
|
06-09 20:35:44,1,1,0.0,363.3,10,0,0.0,363.3,10.0,0.0,40.9,86.0,10,10,363.3,363.3
|
||||||
|
06-09 20:35:49,1,1,119.7,352.6,10,0,119.7,352.6,10.0,0.0,41.1,86.2,10,10,352.6,352.6
|
||||||
|
06-09 20:35:54,1,1,0.0,351.5,10,0,0.0,351.5,10.0,0.0,41.6,86.2,10,10,351.5,351.5
|
||||||
|
06-09 20:35:59,1,1,0.0,345.5,9,0,0.0,345.5,9.0,0.0,40.3,86.2,9,9,345.5,345.5
|
||||||
|
06-09 20:36:04,1,1,0.0,237.3,5,0,0.0,237.3,5.0,0.0,20.6,86.0,5,5,237.3,237.3
|
||||||
|
06-09 20:36:09,1,1,0.0,237.1,5,0,0.0,237.1,5.0,0.0,20.9,86.0,5,5,237.1,237.1
|
||||||
|
06-09 20:36:14,1,1,122.4,257.3,7,0,122.4,257.3,7.0,0.0,21.8,85.8,7,7,257.3,257.3
|
||||||
|
06-09 20:36:19,1,1,0.0,329.8,7,0,0.0,329.8,7.0,0.0,22.3,85.8,7,7,329.8,329.8
|
||||||
|
06-09 20:36:24,1,1,0.0,632.9,17,0,0.0,632.9,17.0,0.0,63.4,89.0,17,17,632.9,632.9
|
||||||
|
06-09 20:36:29,1,1,40.9,334.4,7,0,40.9,334.4,7.0,0.0,22.6,86.0,7,7,334.4,334.4
|
||||||
|
06-09 20:36:34,1,1,0.0,314.2,6,0,0.0,314.2,6.0,0.0,22.7,86.0,6,6,314.2,314.2
|
||||||
|
06-09 20:36:39,1,1,69.7,287.3,7,0,69.7,287.3,7.0,0.0,23.5,86.0,7,7,287.3,287.3
|
||||||
|
06-09 20:36:44,1,1,43.9,347.2,8,0,43.9,347.2,8.0,0.0,24.7,86.2,8,8,347.2,347.2
|
||||||
|
06-09 20:36:49,1,1,0.0,370.6,7,0,0.0,370.6,7.0,0.0,24.5,86.2,7,7,370.6,370.6
|
||||||
|
06-09 20:36:54,1,1,0.0,331.3,7,0,0.0,331.3,7.0,0.0,24.9,86.2,7,7,331.3,331.3
|
||||||
|
06-09 20:36:59,1,1,81.9,249.7,4,0,81.9,249.7,4.0,0.0,14.2,92.4,4,4,249.7,249.7
|
||||||
|
06-09 20:37:04,1,1,29.3,187.3,5,0,29.3,187.3,5.0,0.0,15.3,92.4,5,5,187.3,187.3
|
||||||
|
06-09 20:37:09,1,1,0.0,220.6,4,0,0.0,220.6,4.0,0.0,14.7,92.4,4,4,220.6,220.6
|
||||||
|
06-09 20:37:14,1,1,0.0,183.5,4,0,0.0,183.5,4.0,0.0,15.0,92.4,4,4,183.5,183.5
|
||||||
|
06-09 20:37:19,1,1,0.0,184.7,4,0,0.0,184.7,4.0,0.0,15.2,92.4,4,4,184.7,184.7
|
||||||
|
06-09 20:37:24,1,1,0.0,184.1,4,0,0.0,184.1,4.0,0.0,15.5,92.4,4,4,184.1,184.1
|
||||||
|
06-09 20:37:29,1,1,0.0,183.3,4,0,0.0,183.3,4.0,0.0,15.8,92.4,4,4,183.3,183.3
|
||||||
|
06-09 20:37:34,1,1,0.0,181.0,4,0,0.0,181.0,4.0,0.0,16.0,92.4,4,4,181.0,181.0
|
||||||
|
06-09 20:37:39,1,1,20.9,193.4,5,0,20.9,193.4,5.0,0.0,17.1,92.4,5,5,193.4,193.4
|
||||||
|
06-09 20:37:44,1,1,0.0,229.0,5,0,0.0,229.0,5.0,0.0,17.5,92.4,5,5,229.0,229.0
|
||||||
|
06-09 20:37:49,1,1,0.0,199.7,4,0,0.0,199.7,4.0,0.0,16.8,92.4,4,4,199.7,199.7
|
||||||
|
06-09 20:37:54,1,1,52.5,549.2,12,0,52.5,549.2,12.0,0.0,43.2,90.1,12,12,549.2,549.2
|
||||||
|
06-09 20:37:59,1,1,0.0,232.7,5,0,0.0,232.7,5.0,0.0,22.7,90.4,5,5,232.7,232.7
|
||||||
|
06-09 20:38:04,1,1,2.8,235.8,6,0,2.8,235.8,6.0,0.0,23.1,90.4,6,6,235.8,235.8
|
||||||
|
06-09 20:38:09,1,1,1.2,306.4,7,0,1.2,306.4,7.0,0.0,24.4,90.6,7,7,306.4,306.4
|
||||||
|
06-09 20:38:14,1,1,0.0,319.0,7,0,0.0,319.0,7.0,0.0,24.8,90.6,7,7,319.0,319.0
|
||||||
|
06-09 20:38:19,1,1,0.0,318.8,7,0,0.0,318.8,7.0,0.0,25.3,90.6,7,7,318.8,318.8
|
||||||
|
06-09 20:38:24,1,1,0.0,319.3,7,0,0.0,319.3,7.0,0.0,25.7,90.6,7,7,319.3,319.3
|
||||||
|
06-09 20:38:29,1,1,0.0,319.7,7,0,0.0,319.7,7.0,0.0,26.2,90.6,7,7,319.7,319.7
|
||||||
|
06-09 20:38:34,1,1,52.9,355.8,7,0,52.9,355.8,7.0,0.0,26.2,90.6,7,7,355.8,355.8
|
||||||
|
06-09 20:38:39,1,1,0.0,319.2,7,0,0.0,319.2,7.0,0.0,26.6,90.6,7,7,319.2,319.2
|
||||||
|
06-09 20:38:44,1,1,0.0,319.3,7,0,0.0,319.3,7.0,0.0,27.1,90.6,7,7,319.3,319.3
|
||||||
|
06-09 20:38:49,1,1,0.0,319.1,7,0,0.0,319.1,7.0,0.0,27.5,90.6,7,7,319.1,319.1
|
||||||
|
06-09 20:38:54,1,1,275.5,349.7,9,0,275.5,349.7,9.0,0.0,29.7,90.3,9,9,349.7,349.7
|
||||||
|
06-09 20:38:59,1,1,0.0,407.9,9,0,0.0,407.9,9.0,0.0,30.3,90.3,9,9,407.9,407.9
|
||||||
|
06-09 20:39:04,1,1,72.1,430.6,9,0,72.1,430.6,9.0,0.0,30.7,90.2,9,9,430.6,430.6
|
||||||
|
06-09 20:39:09,1,1,0.0,408.4,9,0,0.0,408.4,9.0,0.0,31.3,90.2,9,9,408.4,408.4
|
||||||
|
06-09 20:39:15,1,1,0.0,409.2,9,0,0.0,409.2,9.0,0.0,31.9,90.2,9,9,409.2,409.2
|
||||||
|
06-09 20:39:20,1,1,72.4,407.5,10,0,72.4,407.5,10.0,0.0,32.8,90.2,10,10,407.5,407.5
|
||||||
|
06-09 20:39:25,1,1,0.0,454.4,10,0,0.0,454.4,10.0,0.0,33.5,90.2,10,10,454.4,454.4
|
||||||
|
06-09 20:39:30,1,1,0.0,642.6,16,0,0.0,642.6,16.0,0.0,82.5,88.4,16,16,642.6,642.6
|
||||||
|
06-09 20:39:35,1,1,0.0,456.3,13,0,0.0,456.3,13.0,0.0,60.2,86.9,13,13,456.3,456.3
|
||||||
|
06-09 20:39:40,1,1,0.0,454.8,12,0,0.0,454.8,12.0,0.0,60.6,86.9,12,12,454.8,454.8
|
||||||
|
06-09 20:39:45,1,1,1.2,437.8,13,0,1.2,437.8,13.0,0.0,61.2,87.1,13,13,437.8,437.8
|
||||||
|
06-09 20:39:50,1,1,0.0,461.5,13,0,0.0,461.5,13.0,0.0,61.9,87.1,13,13,461.5,461.5
|
||||||
|
06-09 20:39:55,1,1,193.6,347.7,8,0,193.6,347.7,8.0,0.0,25.8,89.8,8,8,347.7,347.7
|
||||||
|
06-09 20:40:00,1,1,0.0,350.4,7,0,0.0,350.4,7.0,0.0,25.4,89.8,7,7,350.4,350.4
|
||||||
|
06-09 20:40:05,1,1,83.8,341.7,8,0,83.8,341.7,8.0,0.0,27.3,89.8,8,8,341.7,341.7
|
||||||
|
06-09 20:40:10,1,1,0.0,355.0,7,0,0.0,355.0,7.0,0.0,26.1,89.8,7,7,355.0,355.0
|
||||||
|
06-09 20:40:15,1,1,98.8,323.9,8,0,98.8,323.9,8.0,0.0,28.6,89.9,8,8,323.9,323.9
|
||||||
|
06-09 20:40:20,1,1,0.0,332.5,7,0,0.0,332.5,7.0,0.0,27.5,89.9,7,7,332.5,332.5
|
||||||
|
06-09 20:40:25,1,1,52.9,485.1,15,0,52.9,485.1,15.0,0.0,67.3,87.1,15,15,485.1,485.1
|
||||||
|
06-09 20:40:30,1,1,0.0,501.1,14,0,0.0,501.1,14.0,0.0,67.6,87.1,14,14,501.1,501.1
|
||||||
|
06-09 20:40:35,1,1,23.4,525.9,14,0,23.4,525.9,14.0,0.0,60.7,87.3,14,14,525.9,525.9
|
||||||
|
06-09 20:40:40,1,1,0.0,467.8,13,0,0.0,467.8,13.0,0.0,60.8,87.3,13,13,467.8,467.8
|
||||||
|
06-09 20:40:45,1,1,0.0,441.4,12,0,0.0,441.4,12.0,0.0,60.1,87.3,12,12,441.4,441.4
|
||||||
|
06-09 20:40:50,1,1,192.5,516.7,11,0,192.5,516.7,11.0,0.0,46.6,90.7,11,11,516.7,516.7
|
||||||
|
06-09 20:40:55,1,1,0.0,573.8,11,0,0.0,573.8,11.0,0.0,47.4,90.7,11,11,573.8,573.8
|
||||||
|
06-09 20:41:00,1,1,0.0,255.0,5,0,0.0,255.0,5.0,0.0,25.0,96.8,5,5,255.0,255.0
|
||||||
|
06-09 20:41:05,1,1,0.0,524.8,15,0,0.0,524.8,15.0,0.0,60.1,88.8,15,15,524.8,524.8
|
||||||
|
06-09 20:41:10,1,1,0.0,590.4,12,0,0.0,590.4,12.0,0.0,57.1,90.1,12,12,590.4,590.4
|
||||||
|
06-09 20:41:15,1,1,0.0,558.3,12,0,0.0,558.3,12.0,0.0,57.8,90.1,12,12,558.3,558.3
|
||||||
|
06-09 20:41:20,1,1,225.6,607.3,15,0,225.6,607.3,15.0,0.0,62.6,90.1,15,15,607.3,607.3
|
||||||
|
06-09 20:41:25,1,1,119.4,641.2,13,0,119.4,641.2,13.0,0.0,52.9,90.1,13,13,641.2,641.2
|
||||||
|
06-09 20:41:30,1,1,0.0,598.4,13,0,0.0,598.4,13.0,0.0,53.8,90.1,13,13,598.4,598.4
|
||||||
|
06-09 20:41:35,1,1,0.0,599.6,13,0,0.0,599.6,13.0,0.0,54.6,90.1,13,13,599.6,599.6
|
||||||
|
06-09 20:41:40,1,1,0.0,594.3,12,0,0.0,594.3,12.0,0.0,53.6,90.1,12,12,594.3,594.3
|
||||||
|
06-09 20:41:45,1,1,0.0,556.3,12,0,0.0,556.3,12.0,0.0,54.4,90.1,12,12,556.3,556.3
|
||||||
|
06-09 20:41:50,1,1,0.0,555.0,12,0,0.0,555.0,12.0,0.0,55.2,90.1,12,12,555.0,555.0
|
||||||
|
06-09 20:41:55,1,1,0.0,619.5,13,0,0.0,619.5,13.0,0.0,71.2,89.2,13,13,619.5,619.5
|
||||||
|
06-09 20:42:00,1,1,0.0,593.5,13,0,0.0,593.5,13.0,0.0,72.0,89.2,13,13,593.5,593.5
|
||||||
|
06-09 20:42:05,1,1,0.0,588.1,13,0,0.0,588.1,13.0,0.0,72.8,89.2,13,13,588.1,588.1
|
||||||
|
06-09 20:42:10,1,1,0.0,582.7,13,0,0.0,582.7,13.0,0.0,73.6,89.2,13,13,582.7,582.7
|
||||||
|
06-09 20:42:16,1,1,0.0,577.4,13,0,0.0,577.4,13.0,0.0,74.5,89.2,13,13,577.4,577.4
|
||||||
|
06-09 20:42:21,1,1,0.0,572.4,13,0,0.0,572.4,13.0,0.0,75.3,89.2,13,13,572.4,572.4
|
||||||
|
06-09 20:42:26,1,1,3.2,605.8,14,0,3.2,605.8,14.0,0.0,76.1,89.3,14,14,605.8,605.8
|
||||||
|
06-09 20:42:30,1,1,0.0,558.7,12,0,0.0,558.7,12.0,0.0,61.4,90.2,12,12,558.7,558.7
|
||||||
|
06-09 20:42:36,1,1,20.8,390.7,9,0,20.8,390.7,9.0,0.0,28.5,87.4,9,9,390.7,390.7
|
||||||
|
06-09 20:42:41,1,1,0.0,432.2,9,0,0.0,432.2,9.0,0.0,29.1,87.4,9,9,432.2,432.2
|
||||||
|
06-09 20:42:46,1,1,0.0,402.0,8,0,0.0,402.0,8.0,0.0,28.4,87.4,8,8,402.0,402.0
|
||||||
|
06-09 20:42:51,1,1,0.0,381.5,8,0,0.0,381.5,8.0,0.0,29.0,87.4,8,8,381.5,381.5
|
||||||
|
06-09 20:42:56,1,1,0.0,383.1,8,0,0.0,383.1,8.0,0.0,29.5,87.4,8,8,383.1,383.1
|
||||||
|
06-09 20:43:01,1,1,0.0,383.0,8,0,0.0,383.0,8.0,0.0,30.0,87.4,8,8,383.0,383.0
|
||||||
|
06-09 20:43:06,1,1,7.8,556.8,12,0,7.8,556.8,12.0,0.0,55.4,89.5,12,12,556.8,556.8
|
||||||
|
06-09 20:43:11,1,1,0.0,565.9,12,0,0.0,565.9,12.0,0.0,56.2,89.5,12,12,565.9,565.9
|
||||||
|
06-09 20:43:16,1,1,0.0,567.1,12,0,0.0,567.1,12.0,0.0,57.0,89.5,12,12,567.1,567.1
|
||||||
|
06-09 20:43:21,1,1,0.0,328.7,7,0,0.0,328.7,7.0,0.0,33.9,90.0,7,7,328.7,328.7
|
||||||
|
06-09 20:43:26,1,1,0.0,483.2,10,0,0.0,483.2,10.0,0.0,48.8,89.5,10,10,483.2,483.2
|
||||||
|
06-09 20:43:31,1,1,63.1,498.4,11,0,63.1,498.4,11.0,0.0,52.2,89.7,11,11,498.4,498.4
|
||||||
|
06-09 20:43:36,1,1,0.0,518.8,11,0,0.0,518.8,11.0,0.0,52.9,89.7,11,11,518.8,518.8
|
||||||
|
06-09 20:43:41,1,1,0.0,384.1,11,0,0.0,384.1,11.0,0.0,45.6,87.2,11,11,384.1,384.1
|
||||||
|
06-09 20:43:46,1,1,47.2,434.8,13,0,47.2,434.8,13.0,0.0,46.5,87.2,13,13,434.8,434.8
|
||||||
|
06-09 20:43:51,1,1,0.0,440.2,12,0,0.0,440.2,12.0,0.0,46.7,87.2,12,12,440.2,440.2
|
||||||
|
06-09 20:43:56,1,1,77.7,438.5,13,0,77.7,438.5,13.0,0.0,47.5,87.2,13,13,438.5,438.5
|
||||||
|
06-09 20:44:01,1,1,0.0,636.3,14,0,0.0,636.3,14.0,0.0,62.6,93.2,14,14,636.3,636.3
|
||||||
|
06-09 20:44:06,1,1,0.0,635.0,14,0,0.0,635.0,14.0,0.0,63.5,93.2,14,14,635.0,635.0
|
||||||
|
06-09 20:44:11,1,1,0.0,633.6,14,0,0.0,633.6,14.0,0.0,64.3,93.2,14,14,633.6,633.6
|
||||||
|
06-09 20:44:16,1,1,0.0,634.5,14,0,0.0,634.5,14.0,0.0,65.2,93.2,14,14,634.5,634.5
|
||||||
|
06-09 20:44:21,1,1,0.0,604.1,13,0,0.0,604.1,13.0,0.0,64.7,93.2,13,13,604.1,604.1
|
||||||
|
06-09 20:44:26,1,1,0.0,544.1,12,0,0.0,544.1,12.0,0.0,65.1,93.2,12,12,544.1,544.1
|
||||||
|
06-09 20:44:31,1,1,0.0,544.4,12,0,0.0,544.4,12.0,0.0,65.9,93.2,12,12,544.4,544.4
|
||||||
|
06-09 20:44:36,1,1,0.0,545.9,12,0,0.0,545.9,12.0,0.0,66.7,93.2,12,12,545.9,545.9
|
||||||
|
06-09 20:44:41,1,1,0.0,545.1,12,0,0.0,545.1,12.0,0.0,67.4,93.2,12,12,545.1,545.1
|
||||||
|
06-09 20:44:46,1,1,11.0,563.6,13,0,11.0,563.6,13.0,0.0,68.2,93.2,13,13,563.6,563.6
|
||||||
|
06-09 20:44:51,1,1,0.0,552.4,12,0,0.0,552.4,12.0,0.0,60.1,93.2,12,12,552.4,552.4
|
||||||
|
06-09 20:44:56,1,1,0.0,546.7,12,0,0.0,546.7,12.0,0.0,60.8,93.2,12,12,546.7,546.7
|
||||||
|
06-09 20:45:01,1,1,0.0,545.3,12,0,0.0,545.3,12.0,0.0,61.6,93.2,12,12,545.3,545.3
|
||||||
|
06-09 20:45:06,1,1,0.0,545.2,12,0,0.0,545.2,12.0,0.0,62.3,93.2,12,12,545.2,545.2
|
||||||
|
06-09 20:45:11,1,1,0.0,545.4,12,0,0.0,545.4,12.0,0.0,63.1,93.2,12,12,545.4,545.4
|
||||||
|
06-09 20:45:16,1,1,0.0,544.2,12,0,0.0,544.2,12.0,0.0,63.9,93.2,12,12,544.2,544.2
|
||||||
|
06-09 20:45:21,1,1,46.1,578.2,13,0,46.1,578.2,13.0,0.0,65.3,93.2,13,13,578.2,578.2
|
||||||
|
06-09 20:45:26,1,1,77.9,596.3,14,0,77.9,596.3,14.0,0.0,66.6,93.2,14,14,596.3,596.3
|
||||||
|
06-09 20:45:31,1,1,0.0,595.6,13,0,0.0,595.6,13.0,0.0,66.9,93.2,13,13,595.6,595.6
|
||||||
|
06-09 20:45:36,1,1,0.0,557.7,12,0,0.0,557.7,12.0,0.0,67.1,93.2,12,12,557.7,557.7
|
||||||
|
06-09 20:45:41,1,1,0.0,548.0,12,0,0.0,548.0,12.0,0.0,67.8,93.2,12,12,548.0,548.0
|
||||||
|
06-09 20:45:46,1,1,0.0,548.2,12,0,0.0,548.2,12.0,0.0,68.6,93.2,12,12,548.2,548.2
|
||||||
|
06-09 20:45:51,1,1,0.0,545.7,12,0,0.0,545.7,12.0,0.0,69.4,93.2,12,12,545.7,545.7
|
||||||
|
06-09 20:45:56,1,1,0.0,544.7,12,0,0.0,544.7,12.0,0.0,70.1,93.2,12,12,544.7,544.7
|
||||||
|
06-09 20:46:02,1,1,0.0,511.6,17,0,0.0,511.6,17.0,0.0,64.8,87.1,17,17,511.6,511.6
|
||||||
|
06-09 20:46:07,1,1,0.0,514.8,17,0,0.0,514.8,17.0,0.0,65.5,87.1,17,17,514.8,514.8
|
||||||
|
06-09 20:46:12,1,1,93.5,548.4,18,0,93.5,548.4,18.0,0.0,66.5,87.1,18,18,548.4,548.4
|
||||||
|
06-09 20:46:17,1,1,0.0,536.3,18,0,0.0,536.3,18.0,0.0,67.3,87.1,18,18,536.3,536.3
|
||||||
|
06-09 20:46:22,1,1,79.6,530.0,17,0,79.6,530.0,17.0,0.0,67.1,87.1,17,17,530.0,530.0
|
||||||
|
06-09 20:46:27,1,1,9.2,498.5,18,0,9.2,498.5,18.0,0.0,67.8,87.1,18,18,498.5,498.5
|
||||||
|
06-09 20:46:32,1,1,0.0,538.0,18,0,0.0,538.0,18.0,0.0,68.6,87.1,18,18,538.0,538.0
|
||||||
|
06-09 20:46:37,1,1,0.0,564.7,16,0,0.0,564.7,16.0,0.0,67.4,87.1,16,16,564.7,564.7
|
||||||
|
06-09 20:46:42,1,1,10.2,554.8,16,0,10.2,554.8,16.0,0.0,68.0,87.1,16,16,554.8,554.8
|
||||||
|
06-09 20:46:47,1,1,0.0,535.5,15,0,0.0,535.5,15.0,0.0,68.6,87.1,15,15,535.5,535.5
|
||||||
|
06-09 20:46:52,1,1,0.0,665.0,19,0,0.0,665.0,19.0,0.0,83.2,88.4,19,19,665.0,665.0
|
||||||
|
06-09 20:46:57,1,1,91.6,519.0,16,0,91.6,519.0,16.0,0.0,70.6,87.1,16,16,519.0,519.0
|
||||||
|
06-09 20:47:02,1,1,0.0,568.9,16,0,0.0,568.9,16.0,0.0,71.4,87.1,16,16,568.9,568.9
|
||||||
|
06-09 20:47:07,1,1,53.0,546.1,16,0,53.0,546.1,16.0,0.0,72.8,87.2,16,16,546.1,546.1
|
||||||
|
06-09 20:47:12,1,1,0.0,378.7,8,0,0.0,378.7,8.0,0.0,44.0,90.3,8,8,378.7,378.7
|
||||||
|
06-09 20:47:17,1,1,0.0,379.4,8,0,0.0,379.4,8.0,0.0,44.6,90.3,8,8,379.4,379.4
|
||||||
|
06-09 20:47:22,1,1,0.0,379.9,8,0,0.0,379.9,8.0,0.0,45.1,90.3,8,8,379.9,379.9
|
||||||
|
06-09 20:47:27,1,1,57.3,410.1,9,0,57.3,410.1,9.0,0.0,46.7,90.4,9,9,410.1,410.1
|
||||||
|
06-09 20:47:32,1,1,7.6,427.6,9,0,7.6,427.6,9.0,0.0,39.5,90.4,9,9,427.6,427.6
|
||||||
|
06-09 20:47:37,1,1,0.0,387.8,8,0,0.0,387.8,8.0,0.0,38.9,90.4,8,8,387.8,387.8
|
||||||
|
06-09 20:47:42,1,1,0.0,363.2,10,0,0.0,363.2,10.0,0.0,47.9,87.2,10,10,363.2,363.2
|
||||||
|
06-09 20:47:47,1,1,0.0,369.6,10,0,0.0,369.6,10.0,0.0,48.4,87.2,10,10,369.6,369.6
|
||||||
|
06-09 20:47:52,1,1,0.0,356.5,10,0,0.0,356.5,10.0,0.0,48.9,87.2,10,10,356.5,356.5
|
||||||
|
06-09 20:47:57,1,1,0.0,356.0,10,0,0.0,356.0,10.0,0.0,49.4,87.2,10,10,356.0,356.0
|
||||||
|
06-09 20:48:02,1,1,0.0,365.1,10,0,0.0,365.1,10.0,0.0,49.9,87.2,10,10,365.1,365.1
|
||||||
|
06-09 20:48:07,1,1,0.0,355.8,10,0,0.0,355.8,10.0,0.0,50.4,87.2,10,10,355.8,355.8
|
||||||
|
06-09 20:48:12,1,1,0.0,357.9,10,0,0.0,357.9,10.0,0.0,50.9,87.2,10,10,357.9,357.9
|
||||||
|
06-09 20:48:17,1,1,0.0,357.4,10,0,0.0,357.4,10.0,0.0,51.4,87.2,10,10,357.4,357.4
|
||||||
|
06-09 20:48:22,1,1,0.0,350.6,10,0,0.0,350.6,10.0,0.0,51.9,87.2,10,10,350.6,350.6
|
||||||
|
06-09 20:48:27,1,1,0.0,359.2,10,0,0.0,359.2,10.0,0.0,52.4,87.2,10,10,359.2,359.2
|
||||||
|
06-09 20:48:32,1,1,0.0,365.6,10,0,0.0,365.6,10.0,0.0,52.9,87.2,10,10,365.6,365.6
|
||||||
|
06-09 20:48:37,1,1,0.0,350.9,10,0,0.0,350.9,10.0,0.0,53.4,87.2,10,10,350.9,350.9
|
||||||
|
06-09 20:48:42,1,1,0.0,355.6,10,0,0.0,355.6,10.0,0.0,53.9,87.2,10,10,355.6,355.6
|
||||||
|
06-09 20:48:47,1,1,90.6,365.9,11,0,90.6,365.9,11.0,0.0,54.5,87.1,11,11,365.9,365.9
|
||||||
|
06-09 20:48:52,1,1,0.0,392.1,11,0,0.0,392.1,11.0,0.0,55.1,87.1,11,11,392.1,392.1
|
||||||
|
06-09 20:48:57,1,1,0.0,389.2,11,0,0.0,389.2,11.0,0.0,55.6,87.1,11,11,389.2,389.2
|
||||||
|
06-09 20:49:02,1,1,0.0,396.7,11,0,0.0,396.7,11.0,0.0,56.2,87.1,11,11,396.7,396.7
|
||||||
|
06-09 20:49:07,1,1,0.0,389.7,11,0,0.0,389.7,11.0,0.0,56.7,87.1,11,11,389.7,389.7
|
||||||
|
06-09 20:49:12,1,1,0.0,392.0,10,0,0.0,392.0,10.0,0.0,56.9,87.1,10,10,392.0,392.0
|
||||||
|
06-09 20:49:17,1,1,0.0,403.7,10,0,0.0,403.7,10.0,0.0,57.4,87.1,10,10,403.7,403.7
|
||||||
|
06-09 20:49:22,1,1,0.0,328.5,9,0,0.0,328.5,9.0,0.0,49.1,87.1,9,9,328.5,328.5
|
||||||
|
06-09 20:49:27,1,1,0.0,316.7,9,0,0.0,316.7,9.0,0.0,49.5,87.1,9,9,316.7,316.7
|
||||||
|
06-09 20:49:32,1,1,2.2,341.0,10,0,2.2,341.0,10.0,0.0,50.0,87.2,10,10,341.0,341.0
|
||||||
|
06-09 20:49:37,1,1,0.0,353.0,10,0,0.0,353.0,10.0,0.0,50.5,87.2,10,10,353.0,353.0
|
||||||
|
06-09 20:49:42,1,1,0.0,321.5,8,0,0.0,321.5,8.0,0.0,42.2,87.2,8,8,321.5,321.5
|
||||||
|
06-09 20:49:47,1,1,0.0,293.3,8,0,0.0,293.3,8.0,0.0,42.6,87.2,8,8,293.3,293.3
|
||||||
|
06-09 20:49:52,1,1,0.0,282.6,8,0,0.0,282.6,8.0,0.0,43.0,87.2,8,8,282.6,282.6
|
||||||
|
06-09 20:49:57,1,1,0.0,279.1,8,0,0.0,279.1,8.0,0.0,43.4,87.2,8,8,279.1,279.1
|
||||||
|
06-09 20:50:02,1,1,0.0,280.2,8,0,0.0,280.2,8.0,0.0,43.8,87.2,8,8,280.2,280.2
|
||||||
|
06-09 20:50:07,1,1,0.0,282.8,8,0,0.0,282.8,8.0,0.0,44.2,87.2,8,8,282.8,282.8
|
||||||
|
06-09 20:50:12,1,1,74.3,268.2,8,0,74.3,268.2,8.0,0.0,36.2,87.1,8,8,268.2,268.2
|
||||||
|
06-09 20:50:17,1,1,0.0,285.6,8,0,0.0,285.6,8.0,0.0,36.6,87.1,8,8,285.6,285.6
|
||||||
|
06-09 20:50:22,1,1,0.0,295.3,8,0,0.0,295.3,8.0,0.0,37.0,87.1,8,8,295.3,295.3
|
||||||
|
06-09 20:50:27,1,1,0.0,264.0,7,0,0.0,264.0,7.0,0.0,36.8,87.1,7,7,264.0,264.0
|
||||||
|
06-09 20:50:32,1,1,0.0,251.4,7,0,0.0,251.4,7.0,0.0,37.1,87.1,7,7,251.4,251.4
|
||||||
|
06-09 20:50:37,1,1,0.0,255.7,7,0,0.0,255.7,7.0,0.0,37.5,87.1,7,7,255.7,255.7
|
||||||
|
06-09 20:50:43,1,1,247.4,910.5,24,2,247.4,910.5,24.0,2.0,85.0,76.6,24,24,910.5,910.5
|
||||||
|
06-09 20:50:48,1,1,12.6,930.1,24,0,12.6,930.1,24.0,0.0,76.6,76.7,24,24,930.1,930.1
|
||||||
|
06-09 20:50:53,1,1,0.0,424.3,10,0,0.0,424.3,10.0,0.0,42.6,90.0,10,10,424.3,424.3
|
||||||
|
06-09 20:50:58,1,1,0.0,423.2,10,0,0.0,423.2,10.0,0.0,43.2,90.0,10,10,423.2,423.2
|
||||||
|
06-09 20:51:03,1,1,0.0,419.2,9,0,0.0,419.2,9.0,0.0,43.3,90.0,9,9,419.2,419.2
|
||||||
|
06-09 20:51:08,1,1,96.7,434.6,12,0,96.7,434.6,12.0,0.0,45.1,90.0,12,12,434.6,434.6
|
||||||
|
06-09 20:51:13,1,1,0.0,507.5,12,0,0.0,507.5,12.0,0.0,45.8,90.0,12,12,507.5,507.5
|
||||||
|
06-09 20:51:18,1,1,0.0,526.4,15,0,0.0,526.4,15.0,0.0,62.5,87.2,15,15,526.4,526.4
|
||||||
|
06-09 20:51:23,1,1,169.0,510.7,16,0,169.0,510.7,16.0,0.0,63.9,87.1,16,16,510.7,510.7
|
||||||
|
06-09 20:51:28,1,1,0.0,541.0,15,0,0.0,541.0,15.0,0.0,64.0,87.1,15,15,541.0,541.0
|
||||||
|
06-09 20:51:33,1,1,65.5,498.3,14,0,65.5,498.3,14.0,0.0,63.9,87.1,14,14,498.3,498.3
|
||||||
|
06-09 20:51:38,1,1,0.0,380.4,9,0,0.0,380.4,9.0,0.0,39.5,90.1,9,9,380.4,380.4
|
||||||
|
06-09 20:51:43,1,1,54.5,519.0,15,0,54.5,519.0,15.0,0.0,66.0,87.2,15,15,519.0,519.0
|
||||||
|
06-09 20:51:48,1,1,0.0,502.0,13,0,0.0,502.0,13.0,0.0,65.0,87.2,13,13,502.0,502.0
|
||||||
|
06-09 20:51:53,1,1,105.3,471.1,14,0,105.3,471.1,14.0,0.0,66.5,87.2,14,14,471.1,471.1
|
||||||
|
06-09 20:51:58,1,1,78.4,499.9,15,0,78.4,499.9,15.0,0.0,68.2,87.3,15,15,499.9,499.9
|
||||||
|
06-09 20:52:03,1,1,758.6,528.9,16,0,758.6,528.9,16.0,0.0,72.4,86.8,16,16,528.9,528.9
|
||||||
|
06-09 20:52:08,1,1,35.4,518.5,18,0,35.4,518.5,18.0,0.0,74.1,87.0,18,18,518.5,518.5
|
||||||
|
06-09 20:52:13,1,1,0.0,539.6,18,0,0.0,539.6,18.0,0.0,74.8,87.0,18,18,539.6,539.6
|
||||||
|
06-09 20:52:18,1,1,0.0,460.8,10,0,0.0,460.8,10.0,0.0,45.1,89.9,10,10,460.8,460.8
|
||||||
|
06-09 20:52:23,1,1,0.0,511.0,17,0,0.0,511.0,17.0,0.0,67.4,87.0,17,17,511.0,511.0
|
||||||
|
06-09 20:52:28,1,1,0.0,616.0,13,0,0.0,616.0,13.0,0.0,44.6,89.5,13,13,616.0,616.0
|
||||||
|
06-09 20:52:33,1,1,0.0,603.1,12,0,0.0,603.1,12.0,0.0,43.6,89.5,12,12,603.1,603.1
|
||||||
|
06-09 20:52:38,1,1,0.0,506.4,11,0,0.0,506.4,11.0,0.0,33.3,92.6,11,11,506.4,506.4
|
||||||
|
06-09 20:52:43,1,1,0.0,615.4,13,0,0.0,615.4,13.0,0.0,46.6,89.4,13,13,615.4,615.4
|
||||||
|
06-09 20:52:48,1,1,0.0,609.1,13,0,0.0,609.1,13.0,0.0,47.5,89.4,13,13,609.1,609.1
|
||||||
|
06-09 20:52:53,1,1,0.0,609.1,13,0,0.0,609.1,13.0,0.0,48.3,89.4,13,13,609.1,609.1
|
||||||
|
06-09 20:52:58,1,1,0.0,609.3,13,0,0.0,609.3,13.0,0.0,49.2,89.4,13,13,609.3,609.3
|
||||||
|
06-09 20:53:03,1,1,22.2,617.5,14,0,22.2,617.5,14.0,0.0,51.7,89.5,14,14,617.5,617.5
|
||||||
|
06-09 20:53:08,1,1,0.0,662.8,14,0,0.0,662.8,14.0,0.0,52.6,89.5,14,14,662.8,662.8
|
||||||
|
06-09 20:53:13,1,1,202.7,650.7,14,0,202.7,650.7,14.0,0.0,53.9,89.5,14,14,650.7,650.7
|
||||||
|
06-09 20:53:18,1,1,0.0,429.3,10,0,0.0,429.3,10.0,0.0,50.9,89.9,10,10,429.3,429.3
|
||||||
|
06-09 20:53:23,1,1,0.0,664.1,14,0,0.0,664.1,14.0,0.0,55.7,89.5,14,14,664.1,664.1
|
||||||
|
06-09 20:53:28,1,1,0.0,660.9,14,0,0.0,660.9,14.0,0.0,56.7,89.5,14,14,660.9,660.9
|
||||||
|
06-09 20:53:33,1,1,0.0,660.4,14,0,0.0,660.4,14.0,0.0,57.6,89.5,14,14,660.4,660.4
|
||||||
|
06-09 20:53:38,1,1,0.0,661.7,14,0,0.0,661.7,14.0,0.0,58.5,89.5,14,14,661.7,661.7
|
||||||
|
06-09 20:53:43,1,1,0.0,657.6,14,0,0.0,657.6,14.0,0.0,59.4,89.5,14,14,657.6,657.6
|
||||||
|
06-09 20:53:48,1,1,0.0,661.0,14,0,0.0,661.0,14.0,0.0,60.3,89.5,14,14,661.0,661.0
|
||||||
|
06-09 20:53:53,1,1,31.9,517.3,15,0,31.9,517.3,15.0,0.0,69.5,87.4,15,15,517.3,517.3
|
||||||
|
06-09 20:53:58,1,1,0.0,512.8,15,0,0.0,512.8,15.0,0.0,70.2,87.4,15,15,512.8,512.8
|
||||||
|
06-09 20:54:03,1,1,0.0,942.5,24,1,0.0,942.5,24.0,1.0,55.4,75.8,24,24,942.5,942.5
|
||||||
|
06-09 20:54:08,1,1,0.0,942.1,24,2,0.0,942.1,24.0,2.0,56.7,75.8,24,24,942.1,942.1
|
||||||
|
06-09 20:54:13,1,1,0.0,942.9,24,2,0.0,942.9,24.0,2.0,58.0,75.8,24,24,942.9,942.9
|
||||||
|
06-09 20:54:18,1,1,0.0,943.2,24,2,0.0,943.2,24.0,2.0,59.3,75.8,24,24,943.2,943.2
|
||||||
|
06-09 20:54:23,1,1,20.9,941.1,24,3,20.9,941.1,24.0,3.0,61.4,76.0,24,24,941.1,941.1
|
||||||
|
06-09 20:54:28,1,1,0.0,516.9,17,0,0.0,516.9,17.0,0.0,76.4,87.4,17,17,516.9,516.9
|
||||||
|
06-09 20:54:33,1,1,0.0,941.6,24,4,0.0,941.6,24.0,4.0,63.6,76.1,24,24,941.6,941.6
|
||||||
|
06-09 20:54:38,1,1,49.2,942.0,24,5,49.2,942.0,24.0,5.0,64.3,76.2,24,24,942.0,942.0
|
||||||
|
06-09 20:54:43,1,1,0.0,940.3,24,5,0.0,940.3,24.0,5.0,65.7,76.2,24,24,940.3,940.3
|
||||||
|
06-09 20:54:48,1,1,0.0,942.9,24,5,0.0,942.9,24.0,5.0,67.0,76.2,24,24,942.9,942.9
|
||||||
|
06-09 20:54:53,1,1,0.0,943.6,24,5,0.0,943.6,24.0,5.0,68.3,76.2,24,24,943.6,943.6
|
||||||
|
06-09 20:54:58,1,1,0.0,941.9,24,5,0.0,941.9,24.0,5.0,69.6,76.2,24,24,941.9,941.9
|
||||||
|
06-09 20:55:03,1,1,1.0,940.4,24,4,1.0,940.4,24.0,4.0,69.4,76.3,24,24,940.4,940.4
|
||||||
|
06-09 20:55:09,1,1,0.0,942.7,24,5,0.0,942.7,24.0,5.0,70.7,76.3,24,24,942.7,942.7
|
||||||
|
06-09 20:55:14,1,1,0.0,942.0,24,5,0.0,942.0,24.0,5.0,72.0,76.3,24,24,942.0,942.0
|
||||||
|
06-09 20:55:19,1,1,0.0,941.2,24,5,0.0,941.2,24.0,5.0,73.4,76.3,24,24,941.2,941.2
|
||||||
|
06-09 20:55:24,1,1,0.0,943.1,24,5,0.0,943.1,24.0,5.0,74.7,76.3,24,24,943.1,943.1
|
||||||
|
06-09 20:55:29,1,1,0.0,940.5,24,5,0.0,940.5,24.0,5.0,76.0,76.3,24,24,940.5,940.5
|
||||||
|
06-09 20:55:34,1,1,0.0,939.3,24,5,0.0,939.3,24.0,5.0,77.3,76.3,24,24,939.3,939.3
|
||||||
|
06-09 20:55:39,1,1,83.2,938.6,24,4,83.2,938.6,24.0,4.0,78.4,76.2,24,24,938.6,938.6
|
||||||
|
06-09 20:55:44,1,1,0.0,943.7,24,4,0.0,943.7,24.0,4.0,79.7,76.2,24,24,943.7,943.7
|
||||||
|
06-09 20:55:49,1,1,0.0,943.5,24,4,0.0,943.5,24.0,4.0,81.0,76.2,24,24,943.5,943.5
|
||||||
|
06-09 20:55:54,1,1,0.0,942.8,24,5,0.0,942.8,24.0,5.0,82.3,76.2,24,24,942.8,942.8
|
||||||
|
06-09 20:55:59,1,1,0.0,942.4,24,5,0.0,942.4,24.0,5.0,83.6,76.2,24,24,942.4,942.4
|
||||||
|
06-09 20:56:04,1,1,100.1,938.0,24,5,100.1,938.0,24.0,5.0,85.2,76.3,24,24,938.0,938.0
|
||||||
|
06-09 20:56:09,1,1,9.4,938.3,24,5,9.4,938.3,24.0,5.0,85.8,76.3,24,24,938.3,938.3
|
||||||
|
06-09 20:56:14,1,1,0.0,942.5,24,5,0.0,942.5,24.0,5.0,87.2,76.3,24,24,942.5,942.5
|
||||||
|
06-09 20:56:19,1,1,0.0,943.2,24,5,0.0,943.2,24.0,5.0,88.5,76.3,24,24,943.2,943.2
|
||||||
|
06-09 20:56:24,1,1,0.0,929.5,24,5,0.0,929.5,24.0,5.0,89.8,76.3,24,24,929.5,929.5
|
||||||
|
06-09 20:56:29,1,1,0.0,908.8,24,5,0.0,908.8,24.0,5.0,91.0,76.3,24,24,908.8,908.8
|
||||||
|
06-09 20:56:34,1,1,1.6,902.7,24,4,1.6,902.7,24.0,4.0,92.0,76.3,24,24,902.7,902.7
|
||||||
|
06-09 20:56:39,1,1,0.0,895.5,24,4,0.0,895.5,24.0,4.0,93.3,76.3,24,24,895.5,895.5
|
||||||
|
06-09 20:56:44,1,1,0.0,886.8,24,4,0.0,886.8,24.0,4.0,94.5,76.3,24,24,886.8,886.8
|
||||||
|
06-09 20:56:49,1,1,0.0,879.9,24,4,0.0,879.9,24.0,4.0,95.7,76.3,24,24,879.9,879.9
|
||||||
|
06-09 20:56:54,1,1,0.0,739.4,20,8,0.0,739.4,20.0,8.0,97.5,14.6,20,20,739.4,739.4
|
||||||
|
06-09 20:56:59,1,1,0.0,677.4,16,0,0.0,677.4,16.0,0.0,80.9,91.9,16,16,677.4,677.4
|
||||||
|
06-09 20:57:04,1,1,0.0,636.2,16,0,0.0,636.2,16.0,0.0,81.8,91.9,16,16,636.2,636.2
|
||||||
|
06-09 20:57:09,1,1,0.0,628.9,15,0,0.0,628.9,15.0,0.0,74.2,91.9,15,15,628.9,628.9
|
||||||
|
06-09 20:57:14,1,1,0.0,616.0,15,0,0.0,616.0,15.0,0.0,75.1,91.9,15,15,616.0,616.0
|
||||||
|
06-09 20:57:19,1,1,0.0,644.9,15,0,0.0,644.9,15.0,0.0,76.0,91.9,15,15,644.9,644.9
|
||||||
|
06-09 20:57:24,1,1,2.4,666.7,16,0,2.4,666.7,16.0,0.0,76.9,92.0,16,16,666.7,666.7
|
||||||
|
06-09 20:57:29,1,1,0.0,674.8,16,0,0.0,674.8,16.0,0.0,77.9,92.0,16,16,674.8,674.8
|
||||||
|
06-09 20:57:34,1,1,0.0,702.2,16,0,0.0,702.2,16.0,0.0,64.7,88.9,16,16,702.2,702.2
|
||||||
|
06-09 20:57:39,1,1,10.5,585.4,17,0,10.5,585.4,17.0,0.0,70.8,92.0,17,17,585.4,585.4
|
||||||
|
06-09 20:57:44,1,1,0.0,576.3,17,0,0.0,576.3,17.0,0.0,71.6,92.0,17,17,576.3,576.3
|
||||||
|
06-09 20:57:49,1,1,0.0,613.3,16,0,0.0,613.3,16.0,0.0,71.6,92.0,16,16,613.3,613.3
|
||||||
|
06-09 20:57:54,1,1,528.9,609.2,16,0,528.9,609.2,16.0,0.0,72.9,91.7,16,16,609.2,609.2
|
||||||
|
06-09 20:57:59,1,1,0.0,622.7,16,0,0.0,622.7,16.0,0.0,73.8,91.7,16,16,622.7,622.7
|
||||||
|
06-09 20:58:04,1,1,145.6,586.8,15,0,145.6,586.8,15.0,0.0,73.7,91.7,15,15,586.8,586.8
|
||||||
|
06-09 20:58:09,1,1,65.2,591.1,16,0,65.2,591.1,16.0,0.0,74.9,91.7,16,16,591.1,591.1
|
||||||
|
06-09 20:58:14,1,1,0.0,587.8,15,0,0.0,587.8,15.0,0.0,66.8,91.7,15,15,587.8,587.8
|
||||||
|
06-09 20:58:19,1,1,0.0,593.5,15,0,0.0,593.5,15.0,0.0,67.6,91.7,15,15,593.5,593.5
|
||||||
|
06-09 20:58:24,1,1,0.0,569.3,14,0,0.0,569.3,14.0,0.0,67.9,91.7,14,14,569.3,569.3
|
||||||
|
06-09 20:58:29,1,1,0.0,551.3,14,0,0.0,551.3,14.0,0.0,68.7,91.7,14,14,551.3,551.3
|
||||||
|
06-09 20:58:34,1,1,104.5,549.4,14,0,104.5,549.4,14.0,0.0,69.4,91.6,14,14,549.4,549.4
|
||||||
|
06-09 20:58:39,1,1,0.0,536.0,14,0,0.0,536.0,14.0,0.0,70.2,91.6,14,14,536.0,536.0
|
||||||
|
06-09 20:58:44,1,1,0.0,551.3,14,0,0.0,551.3,14.0,0.0,71.0,91.6,14,14,551.3,551.3
|
||||||
|
06-09 20:58:49,1,1,0.0,537.4,14,0,0.0,537.4,14.0,0.0,71.7,91.6,14,14,537.4,537.4
|
||||||
|
06-09 20:58:54,1,1,32.5,571.3,15,0,32.5,571.3,15.0,0.0,72.5,91.7,15,15,571.3,571.3
|
||||||
|
06-09 20:58:59,1,1,152.2,593.3,16,0,152.2,593.3,16.0,0.0,74.4,91.6,16,16,593.3,593.3
|
||||||
|
06-09 20:59:04,1,1,89.5,595.4,17,0,89.5,595.4,17.0,0.0,77.8,91.7,17,17,595.4,595.4
|
||||||
|
06-09 20:59:09,1,1,0.0,603.9,16,0,0.0,603.9,16.0,0.0,77.7,91.7,16,16,603.9,603.9
|
||||||
|
06-09 20:59:14,1,1,267.7,692.6,18,0,267.7,692.6,18.0,0.0,82.1,88.8,18,18,692.6,692.6
|
||||||
|
06-09 20:59:19,1,1,0.0,694.9,16,0,0.0,694.9,16.0,0.0,81.9,88.8,16,16,694.9,694.9
|
||||||
|
06-09 20:59:24,1,1,0.0,683.3,16,0,0.0,683.3,16.0,0.0,82.9,88.8,16,16,683.3,683.3
|
||||||
|
06-09 20:59:29,1,1,8.6,660.9,18,0,8.6,660.9,18.0,0.0,58.6,37.0,18,18,660.9,660.9
|
||||||
|
06-09 20:59:34,1,1,0.0,590.1,16,17,0.0,590.1,16.0,17.0,97.8,75.4,16,16,590.1,590.1
|
||||||
|
06-09 20:59:39,1,1,0.0,681.8,17,0,0.0,681.8,17.0,0.0,60.2,37.0,17,17,681.8,681.8
|
||||||
|
06-09 20:59:44,1,1,0.0,656.2,17,0,0.0,656.2,17.0,0.0,61.1,37.0,17,17,656.2,656.2
|
||||||
|
06-09 20:59:49,1,1,124.0,674.1,18,0,124.0,674.1,18.0,0.0,63.2,37.1,18,18,674.1,674.1
|
||||||
|
06-09 20:59:54,1,1,0.0,693.7,18,0,0.0,693.7,18.0,0.0,64.2,37.1,18,18,693.7,693.7
|
||||||
|
06-09 20:59:59,1,1,0.0,692.3,18,0,0.0,692.3,18.0,0.0,65.2,37.1,18,18,692.3,692.3
|
||||||
|
06-09 21:00:04,1,1,0.0,652.8,17,0,0.0,652.8,17.0,0.0,64.8,37.1,17,17,652.8,652.8
|
||||||
|
06-09 21:00:10,1,1,0.0,453.4,11,0,0.0,453.4,11.0,0.0,52.7,91.0,11,11,453.4,453.4
|
||||||
|
06-09 21:00:15,1,1,8.4,615.5,12,0,8.4,615.5,12.0,0.0,53.6,91.0,12,12,615.5,615.5
|
||||||
|
06-09 21:00:20,1,1,0.0,556.4,12,0,0.0,556.4,12.0,0.0,54.4,91.0,12,12,556.4,556.4
|
||||||
|
06-09 21:00:25,1,1,171.2,764.5,20,0,171.2,764.5,20.0,0.0,69.8,37.3,20,20,764.5,764.5
|
||||||
|
06-09 21:00:30,1,1,102.0,763.7,20,0,102.0,763.7,20.0,0.0,70.9,37.3,20,20,763.7,763.7
|
||||||
|
06-09 21:00:35,1,1,0.0,599.2,13,0,0.0,599.2,13.0,0.0,67.9,88.8,13,13,599.2,599.2
|
||||||
|
06-09 21:00:40,1,1,0.0,759.7,19,0,0.0,759.7,19.0,0.0,72.3,37.3,19,19,759.7,759.7
|
||||||
|
06-09 21:00:45,1,1,0.0,717.7,17,0,0.0,717.7,17.0,0.0,71.5,37.3,17,17,717.7,717.7
|
||||||
|
06-09 21:00:50,1,1,0.0,660.5,17,0,0.0,660.5,17.0,0.0,72.4,37.3,17,17,660.5,660.5
|
||||||
|
06-09 21:00:55,1,1,166.2,656.8,18,0,166.2,656.8,18.0,0.0,74.3,37.4,18,18,656.8,656.8
|
||||||
|
06-09 21:01:00,1,1,0.0,697.3,18,0,0.0,697.3,18.0,0.0,75.3,37.4,18,18,697.3,697.3
|
||||||
|
06-09 21:01:05,1,1,0.0,684.9,17,0,0.0,684.9,17.0,0.0,75.6,88.7,17,17,684.9,684.9
|
||||||
|
06-09 21:01:10,1,1,0.0,700.9,16,0,0.0,700.9,16.0,0.0,75.5,88.7,16,16,700.9,700.9
|
||||||
|
06-09 21:01:15,1,1,0.0,685.5,15,0,0.0,685.5,15.0,0.0,75.2,88.7,15,15,685.5,685.5
|
||||||
|
06-09 21:01:20,1,1,0.0,774.6,20,0,0.0,774.6,20.0,0.0,79.6,37.5,20,20,774.6,774.6
|
||||||
|
06-09 21:01:25,1,1,54.4,768.9,19,0,54.4,768.9,19.0,0.0,80.2,37.6,19,19,768.9,768.9
|
||||||
|
06-09 21:01:30,1,1,0.0,738.5,19,0,0.0,738.5,19.0,0.0,81.2,37.6,19,19,738.5,738.5
|
||||||
|
06-09 21:01:35,1,1,0.0,736.8,19,0,0.0,736.8,19.0,0.0,82.2,37.6,19,19,736.8,736.8
|
||||||
|
06-09 21:01:40,1,1,101.3,761.8,20,0,101.3,761.8,20.0,0.0,83.4,37.6,20,20,761.8,761.8
|
||||||
|
06-09 21:01:45,1,1,0.0,756.2,19,0,0.0,756.2,19.0,0.0,84.3,37.6,19,19,756.2,756.2
|
||||||
|
06-09 21:01:50,1,1,0.0,733.4,19,0,0.0,733.4,19.0,0.0,85.3,37.6,19,19,733.4,733.4
|
||||||
|
06-09 21:01:55,1,1,11.8,743.7,20,0,11.8,743.7,20.0,0.0,86.5,37.6,20,20,743.7,743.7
|
||||||
|
06-09 21:02:00,1,1,241.5,788.0,23,0,241.5,788.0,23.0,0.0,90.9,37.8,23,23,788.0,788.0
|
||||||
|
06-09 21:02:05,1,1,0.0,878.9,23,0,0.0,878.9,23.0,0.0,92.1,37.8,23,23,878.9,878.9
|
||||||
|
06-09 21:02:10,1,1,0.0,853.2,22,0,0.0,853.2,22.0,0.0,93.0,37.8,22,22,853.2,853.2
|
||||||
|
06-09 21:02:15,1,1,0.0,817.9,21,0,0.0,817.9,21.0,0.0,93.8,37.8,21,21,817.9,817.9
|
||||||
|
06-09 21:02:20,1,1,0.0,779.0,20,0,0.0,779.0,20.0,0.0,92.1,37.8,20,20,779.0,779.0
|
||||||
|
06-09 21:02:25,1,1,0.0,734.6,19,0,0.0,734.6,19.0,0.0,84.3,37.8,19,19,734.6,734.6
|
||||||
|
06-09 21:02:30,1,1,122.9,794.0,21,0,122.9,794.0,21.0,0.0,86.4,37.9,21,21,794.0,794.0
|
||||||
|
06-09 21:02:35,1,1,0.0,807.0,21,0,0.0,807.0,21.0,0.0,87.5,37.9,21,21,807.0,807.0
|
||||||
|
06-09 21:02:40,1,1,0.0,808.4,21,0,0.0,808.4,21.0,0.0,88.6,37.9,21,21,808.4,808.4
|
||||||
|
06-09 21:02:45,1,1,0.0,807.6,21,0,0.0,807.6,21.0,0.0,89.8,37.9,21,21,807.6,807.6
|
||||||
|
06-09 21:02:50,1,1,0.0,805.7,21,0,0.0,805.7,21.0,0.0,90.9,37.9,21,21,805.7,805.7
|
||||||
|
06-09 21:02:55,1,1,1157.2,789.2,22,0,1157.2,789.2,22.0,0.0,93.6,37.8,22,22,789.2,789.2
|
||||||
|
06-09 21:03:00,1,1,0.0,848.1,22,0,0.0,848.1,22.0,0.0,94.8,37.8,22,22,848.1,848.1
|
||||||
|
06-09 21:03:05,1,1,0.0,844.0,22,0,0.0,844.0,22.0,0.0,96.0,37.8,22,22,844.0,844.0
|
||||||
|
06-09 21:03:10,1,1,0.0,506.1,11,0,0.0,506.1,11.0,0.0,53.8,90.8,11,11,506.1,506.1
|
||||||
|
06-09 21:03:15,1,1,0.0,487.1,10,0,0.0,487.1,10.0,0.0,45.7,90.8,10,10,487.1,487.1
|
||||||
|
06-09 21:03:20,1,1,0.0,461.4,10,0,0.0,461.4,10.0,0.0,46.3,90.8,10,10,461.4,461.4
|
||||||
|
06-09 21:03:25,1,1,9.6,420.5,10,0,9.6,420.5,10.0,0.0,42.4,90.8,10,10,420.5,420.5
|
||||||
|
06-09 21:03:30,1,1,0.0,460.5,10,0,0.0,460.5,10.0,0.0,43.0,90.8,10,10,460.5,460.5
|
||||||
|
06-09 21:03:35,1,1,58.6,429.4,10,0,58.6,429.4,10.0,0.0,44.3,90.8,10,10,429.4,429.4
|
||||||
|
06-09 21:03:40,1,1,0.0,464.6,10,0,0.0,464.6,10.0,0.0,44.9,90.8,10,10,464.6,464.6
|
||||||
|
06-09 21:03:45,1,1,0.0,688.2,15,0,0.0,688.2,15.0,0.0,71.0,13.7,15,15,688.2,688.2
|
||||||
|
06-09 21:03:50,1,1,43.1,726.3,16,0,43.1,726.3,16.0,0.0,74.3,13.8,16,16,726.3,726.3
|
||||||
|
06-09 21:03:55,1,1,0.0,732.5,16,0,0.0,732.5,16.0,0.0,75.3,13.8,16,16,732.5,732.5
|
||||||
|
06-09 21:04:00,1,1,0.0,731.7,16,0,0.0,731.7,16.0,0.0,76.4,13.8,16,16,731.7,731.7
|
||||||
|
06-09 21:04:05,1,1,0.0,731.8,16,0,0.0,731.8,16.0,0.0,77.4,13.8,16,16,731.8,731.8
|
||||||
|
06-09 21:04:10,1,1,0.0,731.6,16,0,0.0,731.6,16.0,0.0,78.4,13.8,16,16,731.6,731.6
|
||||||
|
06-09 21:04:15,1,1,60.3,719.8,17,0,60.3,719.8,17.0,0.0,79.5,13.8,17,17,719.8,719.8
|
||||||
|
06-09 21:04:20,1,1,69.3,676.6,17,0,69.3,676.6,17.0,0.0,78.1,13.8,17,17,676.6,676.6
|
||||||
|
06-09 21:04:25,1,1,11.2,690.9,18,0,11.2,690.9,18.0,0.0,79.2,13.8,18,18,690.9,690.9
|
||||||
|
06-09 21:04:30,1,1,0.0,714.7,18,0,0.0,714.7,18.0,0.0,80.2,13.8,18,18,714.7,714.7
|
||||||
|
06-09 21:04:35,1,1,0.0,700.6,16,0,0.0,700.6,16.0,0.0,72.1,13.8,16,16,700.6,700.6
|
||||||
|
06-09 21:04:40,1,1,0.0,731.0,16,0,0.0,731.0,16.0,0.0,73.1,13.8,16,16,731.0,731.0
|
||||||
|
06-09 21:04:45,1,1,0.0,729.3,16,0,0.0,729.3,16.0,0.0,74.1,13.8,16,16,729.3,729.3
|
||||||
|
06-09 21:04:50,1,1,21.8,691.3,17,0,21.8,691.3,17.0,0.0,76.4,13.8,17,17,691.3,691.3
|
||||||
|
06-09 21:04:55,1,1,0.0,674.6,17,0,0.0,674.6,17.0,0.0,77.4,13.8,17,17,674.6,674.6
|
||||||
|
06-09 21:05:00,1,1,73.2,688.9,18,0,73.2,688.9,18.0,0.0,78.4,13.8,18,18,688.9,688.9
|
||||||
|
06-09 21:05:05,1,1,373.3,679.9,18,0,373.3,679.9,18.0,0.0,78.8,13.8,18,18,679.9,679.9
|
||||||
|
06-09 21:05:10,1,1,2.0,732.0,19,0,2.0,732.0,19.0,0.0,79.8,13.7,19,19,732.0,732.0
|
||||||
|
06-09 21:05:15,1,1,91.8,748.4,20,0,91.8,748.4,20.0,0.0,80.7,13.7,20,20,748.4,748.4
|
||||||
|
06-09 21:05:20,1,1,82.6,787.6,19,0,82.6,787.6,19.0,0.0,80.8,13.8,19,19,787.6,787.6
|
||||||
|
06-09 21:05:25,1,1,0.0,721.1,18,0,0.0,721.1,18.0,0.0,81.6,13.8,18,18,721.1,721.1
|
||||||
|
06-09 21:05:30,1,1,0.0,706.1,17,0,0.0,706.1,17.0,0.0,82.4,13.8,17,17,706.1,706.1
|
||||||
|
06-09 21:05:35,1,1,2.4,693.6,18,0,2.4,693.6,18.0,0.0,83.4,13.8,18,18,693.6,693.6
|
||||||
|
06-09 21:05:41,1,1,0.0,549.8,14,0,0.0,549.8,14.0,0.0,66.0,86.6,14,14,549.8,549.8
|
||||||
|
06-09 21:05:46,1,1,184.0,563.2,15,0,184.0,563.2,15.0,0.0,69.7,86.7,15,15,563.2,563.2
|
||||||
|
06-09 21:05:51,1,1,0.0,599.5,15,0,0.0,599.5,15.0,0.0,70.5,86.7,15,15,599.5,599.5
|
||||||
|
06-09 21:05:56,1,1,0.0,587.7,15,0,0.0,587.7,15.0,0.0,71.4,86.7,15,15,587.7,587.7
|
||||||
|
06-09 21:06:01,1,1,0.0,592.6,15,0,0.0,592.6,15.0,0.0,72.2,86.7,15,15,592.6,592.6
|
||||||
|
06-09 21:06:06,1,1,0.0,559.9,14,0,0.0,559.9,14.0,0.0,71.1,86.7,14,14,559.9,559.9
|
||||||
|
06-09 21:06:11,1,1,0.0,535.2,13,0,0.0,535.2,13.0,0.0,68.7,86.7,13,13,535.2,535.2
|
||||||
|
06-09 21:06:16,1,1,0.0,528.4,13,0,0.0,528.4,13.0,0.0,69.4,86.7,13,13,528.4,528.4
|
||||||
|
06-09 21:06:21,1,1,0.0,500.3,12,0,0.0,500.3,12.0,0.0,61.2,86.7,12,12,500.3,500.3
|
||||||
|
06-09 21:06:26,1,1,0.0,497.5,12,0,0.0,497.5,12.0,0.0,61.9,86.7,12,12,497.5,497.5
|
||||||
|
06-09 21:06:31,1,1,0.0,461.0,11,0,0.0,461.0,11.0,0.0,53.7,86.7,11,11,461.0,461.0
|
||||||
|
06-09 21:06:36,1,1,0.0,446.4,11,0,0.0,446.4,11.0,0.0,54.3,86.7,11,11,446.4,446.4
|
||||||
|
06-09 21:06:41,1,1,0.0,443.7,11,0,0.0,443.7,11.0,0.0,54.9,86.7,11,11,443.7,443.7
|
||||||
|
06-09 21:06:46,1,1,82.9,450.3,12,0,82.9,450.3,12.0,0.0,58.8,86.9,12,12,450.3,450.3
|
||||||
|
06-09 21:06:51,1,1,223.4,472.7,13,0,223.4,472.7,13.0,0.0,57.2,86.9,13,13,472.7,472.7
|
||||||
|
06-09 21:06:56,1,1,1226.9,537.8,14,0,1226.9,537.8,14.0,0.0,59.9,86.1,14,14,537.8,537.8
|
||||||
|
06-09 21:07:01,1,1,0.0,548.3,12,0,0.0,548.3,12.0,0.0,58.5,86.1,12,12,548.3,548.3
|
||||||
|
06-09 21:07:06,1,1,0.0,498.9,12,0,0.0,498.9,12.0,0.0,59.2,86.1,12,12,498.9,498.9
|
||||||
|
06-09 21:07:11,1,1,0.0,493.4,12,0,0.0,493.4,12.0,0.0,59.9,86.1,12,12,493.4,493.4
|
||||||
|
06-09 21:07:16,1,1,0.0,487.2,12,0,0.0,487.2,12.0,0.0,60.5,86.1,12,12,487.2,487.2
|
||||||
|
06-09 21:07:21,1,1,0.0,494.6,12,0,0.0,494.6,12.0,0.0,61.2,86.1,12,12,494.6,494.6
|
||||||
|
06-09 21:07:26,1,1,1.8,504.1,13,0,1.8,504.1,13.0,0.0,61.9,86.1,13,13,504.1,504.1
|
||||||
|
06-09 21:07:31,1,1,0.0,519.3,13,0,0.0,519.3,13.0,0.0,62.7,86.1,13,13,519.3,519.3
|
||||||
|
06-09 21:07:36,1,1,0.0,515.7,13,0,0.0,515.7,13.0,0.0,63.4,86.1,13,13,515.7,515.7
|
||||||
|
06-09 21:07:41,1,1,86.4,556.6,14,0,86.4,556.6,14.0,0.0,66.4,86.2,14,14,556.6,556.6
|
||||||
|
06-09 21:07:46,1,1,10.3,614.9,15,0,10.3,614.9,15.0,0.0,67.2,86.6,15,15,614.9,614.9
|
||||||
|
06-09 21:07:51,1,1,0.0,602.0,15,0,0.0,602.0,15.0,0.0,68.0,86.6,15,15,602.0,602.0
|
||||||
|
06-09 21:07:56,1,1,0.0,562.6,14,0,0.0,562.6,14.0,0.0,66.5,86.6,14,14,562.6,562.6
|
||||||
|
06-09 21:08:01,1,1,0.0,555.2,14,0,0.0,555.2,14.0,0.0,67.2,86.6,14,14,555.2,555.2
|
||||||
|
06-09 21:08:06,1,1,0.0,516.7,13,0,0.0,516.7,13.0,0.0,59.0,86.6,13,13,516.7,516.7
|
||||||
|
06-09 21:08:11,1,1,0.0,529.3,13,0,0.0,529.3,13.0,0.0,59.7,86.6,13,13,529.3,529.3
|
||||||
|
06-09 21:08:16,1,1,54.9,559.9,14,0,54.9,559.9,14.0,0.0,60.8,86.6,14,14,559.9,559.9
|
||||||
|
06-09 21:08:21,1,1,0.0,537.3,13,0,0.0,537.3,13.0,0.0,61.2,86.6,13,13,537.3,537.3
|
||||||
|
06-09 21:08:26,1,1,8.0,528.1,13,0,8.0,528.1,13.0,0.0,61.9,86.8,13,13,528.1,528.1
|
||||||
|
06-09 21:08:31,1,1,41.0,800.1,20,0,41.0,800.1,20.0,0.0,76.3,13.6,20,20,800.1,800.1
|
||||||
|
06-09 21:08:36,1,1,0.0,522.8,13,0,0.0,522.8,13.0,0.0,65.3,86.9,13,13,522.8,522.8
|
||||||
|
06-09 21:08:41,1,1,1.8,554.0,14,0,1.8,554.0,14.0,0.0,66.1,87.0,14,14,554.0,554.0
|
||||||
|
06-09 21:08:46,1,1,0.0,538.5,13,0,0.0,538.5,13.0,0.0,64.2,87.0,13,13,538.5,538.5
|
||||||
|
06-09 21:08:51,1,1,0.0,525.8,13,0,0.0,525.8,13.0,0.0,64.9,87.0,13,13,525.8,525.8
|
||||||
|
06-09 21:08:56,1,1,0.0,323.4,9,0,0.0,323.4,9.0,0.0,39.1,88.0,9,9,323.4,323.4
|
||||||
|
06-09 21:09:01,1,1,0.0,332.0,9,0,0.0,332.0,9.0,0.0,39.6,88.0,9,9,332.0,332.0
|
||||||
|
06-09 21:09:06,1,1,0.0,320.0,9,0,0.0,320.0,9.0,0.0,40.0,88.0,9,9,320.0,320.0
|
||||||
|
06-09 21:09:11,1,1,0.0,378.6,11,0,0.0,378.6,11.0,0.0,59.2,87.0,11,11,378.6,378.6
|
||||||
|
06-09 21:09:16,1,1,0.0,329.2,8,0,0.0,329.2,8.0,0.0,39.9,88.0,8,8,329.2,329.2
|
||||||
|
06-09 21:09:21,1,1,0.0,304.5,8,0,0.0,304.5,8.0,0.0,40.3,88.0,8,8,304.5,304.5
|
||||||
|
06-09 21:09:26,1,1,0.0,314.4,8,0,0.0,314.4,8.0,0.0,40.8,88.0,8,8,314.4,314.4
|
||||||
|
06-09 21:09:31,1,1,0.0,315.5,8,0,0.0,315.5,8.0,0.0,41.2,88.0,8,8,315.5,315.5
|
||||||
|
06-09 21:09:36,1,1,0.0,303.6,8,0,0.0,303.6,8.0,0.0,41.6,88.0,8,8,303.6,303.6
|
||||||
|
06-09 21:09:41,1,1,0.0,286.8,8,0,0.0,286.8,8.0,0.0,42.0,88.0,8,8,286.8,286.8
|
||||||
|
06-09 21:09:46,1,1,10.6,309.6,9,0,10.6,309.6,9.0,0.0,42.5,88.1,9,9,309.6,309.6
|
||||||
|
06-09 21:09:51,1,1,0.0,326.1,9,0,0.0,326.1,9.0,0.0,42.9,88.1,9,9,326.1,326.1
|
||||||
|
06-09 21:09:56,1,1,184.3,323.6,9,0,184.3,323.6,9.0,0.0,44.4,88.0,9,9,323.6,323.6
|
||||||
|
06-09 21:10:01,1,1,0.0,318.7,9,0,0.0,318.7,9.0,0.0,44.9,88.0,9,9,318.7,318.7
|
||||||
|
06-09 21:10:06,1,1,0.0,318.4,9,0,0.0,318.4,9.0,0.0,45.3,88.0,9,9,318.4,318.4
|
||||||
|
06-09 21:10:11,1,1,0.0,323.3,9,0,0.0,323.3,9.0,0.0,45.8,88.0,9,9,323.3,323.3
|
||||||
|
06-09 21:10:16,1,1,0.0,628.4,14,0,0.0,628.4,14.0,0.0,63.9,90.3,14,14,628.4,628.4
|
||||||
|
06-09 21:10:19,1,1,0.0,511.9,11,0,0.0,511.9,11.0,0.0,48.6,90.9,11,11,511.9,511.9
|
||||||
|
31896
training_logs/20260609_214045_vllm_metrics_table.csv
Normal file
31896
training_logs/20260609_214045_vllm_metrics_table.csv
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,358 @@
|
|||||||
|
|
||||||
|
The following have been reloaded with a version change:
|
||||||
|
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||||
|
|
||||||
|
|
||||||
|
Lmod is automatically replacing "GCC/14.3.0" with
|
||||||
|
"nvidia-compilers/25.9-CUDA-13".
|
||||||
|
|
||||||
|
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||||
|
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||||||
|
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||||
|
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||||
|
[ray] RAY_TMPDIR=/tmp/ray/ray_653583
|
||||||
|
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_653583
|
||||||
|
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_653583
|
||||||
|
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||||||
|
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||||||
|
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||||||
|
[proxy] Tunnel port: 7003
|
||||||
|
[proxy] Node IP: 10.128.16.35 (workers will connect here)
|
||||||
|
[proxy] ✓ SSH tunnel started successfully
|
||||||
|
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653583.conf
|
||||||
|
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||||||
|
[proxy] - External traffic (internet) → PROXY via tunnel
|
||||||
|
[proxy] ✓ Daytona timeout settings configured
|
||||||
|
[proxy] Testing proxy connectivity...
|
||||||
|
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653583.conf
|
||||||
|
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||||||
|
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||||
|
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||||||
|
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
|
||||||
|
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||||||
|
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||||||
|
=== Universal RL Training Runner ===
|
||||||
|
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/configs/ablation-pymethods2test-seqmean-arm0-tis_rl_config.json
|
||||||
|
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||||||
|
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||||
|
Python version: Python 3.12.12
|
||||||
|
UV_USE_IO_URING: 0
|
||||||
|
Proxy: DISABLED (direct internet or not configured)
|
||||||
|
========================================
|
||||||
|
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0-tis ===
|
||||||
|
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
|
||||||
|
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
|
||||||
|
HF_TOKEN=****pDbg
|
||||||
|
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||||||
|
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||||||
|
Environment configured:
|
||||||
|
TENSOR_PARALLEL_SIZE=1
|
||||||
|
NUM_INFERENCE_ENGINES=56
|
||||||
|
POLICY_NUM_NODES=14
|
||||||
|
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
|
||||||
|
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||||||
|
Cleaning up existing Ray instances...
|
||||||
|
=== Starting Ray Cluster ===
|
||||||
|
Nodes: 14
|
||||||
|
GPUs per node: 4
|
||||||
|
CPUs per node: 288
|
||||||
|
Head node: jpbo-001-35 (10.128.16.35)
|
||||||
|
Ray port: 6379
|
||||||
|
============================
|
||||||
|
Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray head on jpbo-001-35
|
||||||
|
Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 1 on jpbo-001-36
|
||||||
|
Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 2 on jpbo-001-37
|
||||||
|
Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 3 on jpbo-001-38
|
||||||
|
Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 4 on jpbo-001-41
|
||||||
|
Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 5 on jpbo-001-42
|
||||||
|
Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 6 on jpbo-001-43
|
||||||
|
Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 7 on jpbo-001-44
|
||||||
|
Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 8 on jpbo-001-45
|
||||||
|
Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 9 on jpbo-001-46
|
||||||
|
Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 10 on jpbo-001-47
|
||||||
|
Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 11 on jpbo-001-48
|
||||||
|
Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 12 on jpbo-003-37
|
||||||
|
Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
|
||||||
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||||||
|
Started Ray worker 13 on jpbo-003-39
|
||||||
|
Waiting for cluster (56 GPUs, 14 nodes)...
|
||||||
|
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
|
||||||
|
[Ray wait] Connection attempt failed:
|
||||||
|
[Ray wait] Retrying in 10s...
|
||||||
|
Ray connection established, polling for resources...
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||||||
|
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
|
||||||
|
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
|
||||||
|
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
|
||||||
|
2026-06-07 14:37:25,227 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
|
||||||
|
[2026-06-07 14:37:30,234 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||||||
|
[2026-06-07 14:38:00,235 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||||
|
[2026-06-07 14:38:06,240 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||||||
|
[2026-06-07 14:38:36,240 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||||
|
[2026-06-07 14:38:42,244 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||||||
|
[2026-06-07 14:39:12,244 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||||||
|
[2026-06-07 14:39:18,247 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||||||
|
[2026-06-07 14:39:46,160 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:47,183 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:48,184 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:49,184 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:50,185 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:51,186 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:52,186 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:53,187 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:54,187 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:55,188 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:56,189 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:57,189 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:58,190 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:39:59,190 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:00,191 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:01,192 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:02,192 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:03,193 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:04,193 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:05,194 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:06,195 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:07,195 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:08,196 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:09,196 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:10,197 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:11,198 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:12,198 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:13,199 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:14,199 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:15,200 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
2026-06-07 14:40:26,227 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
|
||||||
|
[2026-06-07 14:40:26,230 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:27,231 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:28,231 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:29,232 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:30,233 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||||||
|
[2026-06-07 14:40:31,233 I 1579458 1579458] global_state_accessor.cc:487: This node has an IP address of 10.128.16.35, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
|
||||||
|
2026-06-07 14:40:31,235 INFO worker.py:2007 -- Connected to Ray cluster.
|
||||||
|
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||||||
|
warnings.warn(
|
||||||
|
[33m(raylet)[0m [2026-06-07 14:41:01,184 E 1580205 1580205] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||||
|
[36m(pid=1580366)[0m [2026-06-07 14:41:17,685 E 1580366 1581078] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||||
|
[36m(pid=1580422)[0m [2026-06-07 14:41:23,475 E 1580422 1582060] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
||||||
|
[2026-06-07 14:41:27,125 E 1579458 1580363] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||||||
|
[36m(pid=1580544)[0m [2026-06-07 14:41:27,119 E 1580544 1596600] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
||||||
|
main()
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
||||||
|
wait_for_cluster(
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
||||||
|
raise TimeoutError(
|
||||||
|
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=8, gpus=32.0)
|
||||||
|
srun: error: jpbo-001-35: task 0: Exited with exit code 1
|
||||||
|
|
||||||
|
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
|
||||||
|
result = subprocess.run(
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
||||||
|
raise CalledProcessError(retcode, process.args,
|
||||||
|
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
||||||
|
|
||||||
|
The above exception was the direct cause of the following exception:
|
||||||
|
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1285, in run
|
||||||
|
training_exit_code = self._run_with_ray()
|
||||||
|
^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1532, in _run_with_ray
|
||||||
|
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
|
||||||
|
self.start()
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
|
||||||
|
self._wait_for_cluster()
|
||||||
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
|
||||||
|
raise RuntimeError(
|
||||||
|
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||||||
|
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
||||||
|
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
|
||||||
|
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs, skipping upload.
|
||||||
|
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ray_logs/
|
||||||
|
slurmstepd: error: *** JOB 653583 ON jpbo-001-35 CANCELLED AT 2026-06-07T15:15:50 ***
|
||||||
1892
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653584.out
Normal file
1892
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653584.out
Normal file
File diff suppressed because one or more lines are too long
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:a383fcb87f44eb8202d493c3a075d21b029c8a48345094d3b87e1b5b26bc7e48
|
||||||
|
size 23207424
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:6eb312e5d802af500569a6b68dbce64468e96a285edd5cf1acd375e81b907b99
|
||||||
|
size 29452449
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:f3b467320252ddc62ad9ee956a782634573d6751244537854604e946364df5b9
|
||||||
|
size 39383504
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:6306300b94a9bf493d8856124479d39ace101f1de2d402289704da14505fbdb9
|
||||||
|
size 37839447
|
||||||
34646
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653589.out
Normal file
34646
training_logs/ablation-pymethods2test-seqmean-arm0-tis_653589.out
Normal file
File diff suppressed because one or more lines are too long
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user