初始化项目，由ModelHub XC社区提供模型

Model: laion/a3-rl-laion_exp_rpt_stack-bash-v3-70-8B
Source: Original Platform

.gitattributes

@@ -0,0 +1,41 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+training_logs/20260606_151436_reward_vs_steps.png filter=lfs diff=lfs merge=lfs -text
+training_logs/20260606_151436_turn_count_distribution.png filter=lfs diff=lfs merge=lfs -text
+training_logs/a3-rl-laion_exp_rpt_stack-bash-v3_589413.out filter=lfs diff=lfs merge=lfs -text
+training_logs/a3-rl-laion_exp_rpt_stack-bash-v3_589414.out filter=lfs diff=lfs merge=lfs -text
+training_logs/a3-rl-laion_exp_rpt_stack-bash-v3_589415.out filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,29 @@
+---
+base_model: laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink
+tags:
+- rl
+- skyrl
+- qwen3
+---
+
+# a3-rl-laion_exp_rpt_stack-bash-v3 (global_step_70)
+
+RL (RLOO_n / SkyRL FSDP2) fine-tune of
+`laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink`
+(Qwen3-8B) on the `laion/exp_rpt_stack-bash-v3` agentic task set, agent `terminus-2`.
+
+**PARTIAL / CANCELLED MID-RUN.** This run was part of the a3 RL series, which was
+concluded as uninformative; SLURM job 589415 was cancelled at training step ~73
+(2026-06-06). This checkpoint is the **latest available HF-ready export
+(global_step_70)** — preserved as the run artifact, not a converged model. The
+reward signal was low and flat throughout (avg_raw_reward ~0.08-0.21, EMA peak
+~0.19 around step 60); grad-norm stayed healthy (<0.012) with no collapse.
+
+## Training Traces
+
+Training-time Daytona/Harbor rollouts for this run are uploaded as a companion
+dataset:
+**[DCAgent/a3-rl-laion_exp_rpt_stack-bash-v3](https://huggingface.co/datasets/DCAgent/a3-rl-laion_exp_rpt_stack-bash-v3)**
+
+The dataset contains the last episode of each of the ~41,263 trials
+(per make_and_upload_trace_dataset --episodes last).

a3-rl-laion_exp_rpt_stack-bash-v3_rl_config.json

@@ -0,0 +1,156 @@
+{
+  "job_name": "a3-rl-laion_exp_rpt_stack-bash-v3",
+  "experiments_dir": "/e/data1/datasets/playground/ot-baf/a3-rl-laion_exp_rpt_stack-bash-v3",
+  "cluster_name": "jupiter",
+  "skyrl_entrypoint": "examples.terminal_bench.entrypoints.main_tbench",
+  "skyrl_hydra_args": [
+    "+terminal_bench_config=terminal_bench",
+    "trainer.strategy=fsdp2",
+    "trainer.algorithm.advantage_estimator=rloo_n",
+    "trainer.algorithm.use_kl_loss=false",
+    "trainer.algorithm.kl_loss_coef=0.0",
+    "trainer.algorithm.eps_clip_low=0.2",
+    "trainer.algorithm.eps_clip_high=0.05",
+    "trainer.algorithm.loss_reduction=token_mean",
+    "trainer.epochs=2",
+    "trainer.max_steps=80",
+    "trainer.update_epochs_per_batch=1",
+    "trainer.train_batch_size=64",
+    "trainer.policy_mini_batch_size=64",
+    "trainer.eval_batch_size=64",
+    "trainer.micro_forward_batch_size_per_gpu=4",
+    "trainer.micro_train_batch_size_per_gpu=1",
+    "trainer.max_prompt_length=999999",
+    "trainer.eval_interval=999999",
+    "trainer.eval_before_train=false",
+    "trainer.ckpt_interval=2",
+    "trainer.resume_mode=latest",
+    "trainer.hf_save_interval=5",
+    "++trainer.hf_hub_repo_id=laion/a3-rl-laion_exp_rpt_stack-bash-v3",
+    "++trainer.hf_hub_private=false",
+    "++trainer.hf_hub_revision=main",
+    "++trainer.enable_db_registration=false",
+    "trainer.project_name=OpenThoughts-Agent",
+    "trainer.log_level=INFO",
+    "trainer.tracker_commit_each_step=true",
+    "trainer.logger=console",
+    "trainer.run_name=a3-rl-laion_exp_rpt_stack-bash-v3",
+    "trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-laion_exp_rpt_stack-bash-v3/a3-rl-laion_exp_rpt_stack-bash-v3/checkpoints",
+    "trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-laion_exp_rpt_stack-bash-v3/a3-rl-laion_exp_rpt_stack-bash-v3/exports",
+    "trainer.policy.optimizer_config.lr=8e-6",
+    "trainer.policy.optimizer_config.weight_decay=0.0",
+    "trainer.policy.optimizer_config.adam_betas=[0.9,0.999]",
+    "trainer.policy.optimizer_config.max_grad_norm=0.9",
+    "trainer.policy.fsdp_config.cpu_offload=false",
+    "trainer.policy.fsdp_config.reshard_after_forward=true",
+    "trainer.policy.fsdp_config.fsdp_size=4",
+    "trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
+    "trainer.ref.fsdp_config.cpu_offload=false",
+    "trainer.ref.fsdp_config.reshard_after_forward=true",
+    "trainer.ref.fsdp_config.fsdp_size=4",
+    "trainer.placement.colocate_all=false",
+    "trainer.placement.policy_num_nodes=2",
+    "trainer.placement.ref_num_nodes=2",
+    "trainer.placement.policy_num_gpus_per_node=4",
+    "trainer.placement.ref_num_gpus_per_node=4",
+    "trainer.fully_async.max_staleness_steps=16",
+    "trainer.fully_async.num_parallel_generation_workers=338",
+    "generator.backend=vllm",
+    "generator.timeout_multiplier=1.0",
+    "generator.model_dtype=bfloat16",
+    "generator.inference_engine_tensor_parallel_size=1",
+    "generator.num_inference_engines=48",
+    "generator.n_samples_per_prompt=8",
+    "generator.eval_n_samples_per_prompt=8",
+    "generator.gpu_memory_utilization=0.75",
+    "generator.max_num_seqs=24",
+    "generator.max_num_batched_tokens=65536",
+    "generator.enable_prefix_caching=true",
+    "generator.enable_chunked_prefill=true",
+    "generator.run_engines_locally=true",
+    "generator.weight_sync_backend=nccl",
+    "generator.async_engine=true",
+    "generator.batched=false",
+    "generator.enable_http_endpoint=true",
+    "generator.enable_ray_prometheus_stats=false",
+    "generator.vllm_stats_interval=1",
+    "generator.append_eos_token_after_stop_str_in_multi_turn=true",
+    "generator.max_turns=999999",
+    "generator.sampling_params.max_generate_length=4096",
+    "generator.sampling_params.temperature=0.7",
+    "generator.sampling_params.top_p=0.95",
+    "generator.sampling_params.top_k=20",
+    "++generator.engine_init_kwargs.max_model_len=32768",
+    "++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2",
+    "++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
+    "data.train_data=[\"/e/scratch/jureap59/feuer1/tasks/exp_rpt_stack-bash-v3\"]",
+    "data.val_data=[]",
+    "+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-laion_exp_rpt_stack-bash-v3/a3-rl-laion_exp_rpt_stack-bash-v3/trace_jobs",
+    "+terminal_bench_config.harbor.name=terminus-2",
+    "+terminal_bench_config.harbor.max_episodes=999999",
+    "+terminal_bench_config.harbor.enable_summarize=false",
+    "+terminal_bench_config.harbor.store_all_messages=true",
+    "+terminal_bench_config.harbor.trajectory_config.raw_content=true",
+    "+terminal_bench_config.harbor.enable_episode_logging=false",
+    "+terminal_bench_config.harbor.record_terminal_session=false",
+    "+terminal_bench_config.harbor.enable_pane_logging=false",
+    "+terminal_bench_config.harbor.strict_json_parser=true",
+    "+terminal_bench_config.harbor.interleaved_thinking=true",
+    "+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true",
+    "+terminal_bench_config.harbor.override_timeout_sec=900",
+    "+terminal_bench_config.harbor.override_cpus=1",
+    "+terminal_bench_config.harbor.override_memory_mb=2048",
+    "+terminal_bench_config.harbor.override_storage_mb=2048",
+    "+terminal_bench_config.harbor.auto_snapshot=true",
+    "+terminal_bench_config.harbor.verifier_override_timeout_sec=120",
+    "+terminal_bench_config.harbor.max_retries=3",
+    "+terminal_bench_config.harbor.min_wait_sec=60.0",
+    "+terminal_bench_config.harbor.max_wait_sec=600.0",
+    "+terminal_bench_config.harbor.wait_multiplier=2.0",
+    "+terminal_bench_config.harbor.exclude_exceptions=[\"VerifierTimeoutError\",\"VerifierRuntimeError\",\"RewardFileNotFoundError\",\"RewardFileEmptyError\",\"VerifierOutputParseError\"]",
+    "+terminal_bench_config.harbor.n_concurrent_trials=675",
+    "+terminal_bench_config.harbor.log_level=INFO",
+    "+terminal_bench_config.harbor.enable_reward_shaping=false",
+    "+terminal_bench_config.harbor.enable_error_classification=true",
+    "+terminal_bench_config.harbor.mask_exceptions=[\"DaytonaError\",\"EnvironmentStartTimeoutError\",\"NetworkError\",\"ConnectionError\",\"RewardFileNotFoundError\",\"RewardFileEmptyError\",\"AgentEnvironmentTimeoutError\",\"ContextLengthExceededError\"]",
+    "+terminal_bench_config.harbor.default_error_treatment=zero",
+    "+terminal_bench_config.harbor.passthrough_exceptions=[\"AgentTimeoutError\"]",
+    "+terminal_bench_config.harbor.zero_exceptions=[]",
+    "+terminal_bench_config.model_info.max_input_tokens=32000",
+    "+terminal_bench_config.model_info.max_output_tokens=4096",
+    "+terminal_bench_config.archiving.enabled=false",
+    "+terminal_bench_config.trace_upload.enabled=true",
+    "+terminal_bench_config.trace_upload.repo_org=DCAgent",
+    "+terminal_bench_config.trace_upload.episodes=last",
+    "+terminal_bench_config.trace_upload.dataset_type=SFT",
+    "+terminal_bench_config.trace_upload.cleanup=true"
+  ],
+  "model_path": "/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6",
+  "train_data": [
+    "/e/scratch/jureap59/feuer1/tasks/exp_rpt_stack-bash-v3"
+  ],
+  "val_data": [],
+  "num_nodes": 14,
+  "gpus_per_node": 4,
+  "cpus_per_node": 288,
+  "tensor_parallel_size": 1,
+  "ray_port": 6379,
+  "master_port": 12345,
+  "checkpoints_dir": null,
+  "export_path": "/e/data1/datasets/playground/ot-baf/a3-rl-laion_exp_rpt_stack-bash-v3/a3-rl-laion_exp_rpt_stack-bash-v3/exports",
+  "needs_ssh_tunnel": true,
+  "needs_cuda_detection": false,
+  "pinggy_persistent_url": null,
+  "pinggy_token": null,
+  "agent_name": "terminus-2",
+  "harbor_env": "daytona",
+  "proxychains_binary": null,
+  "container_sif": null,
+  "container_binds": [],
+  "ray_object_store_gb": 40.0,
+  "trace_upload_enabled": true,
+  "trace_upload_repo_org": "DCAgent",
+  "trace_upload_episodes": "last",
+  "trace_upload_dataset_type": "SFT",
+  "trace_upload_cleanup": true
+}

added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,89 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,68 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 40960,
+  "max_window_layers": 36,
+  "model_type": "qwen3",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.6",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "4.57.6"
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c04da15e099ce970e397dc9a05129410ecba1ee05a13ea7186f7abadcfa7615
+size 4902257696

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6f401fd17461edc831d55abf5fede241f9bd1e5fbf67ed585ef727848c62673
+size 4915960368

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8236d91cf5b6c3cec3ba8f438b4a7321c23f353f207488f2d4767ad948b90ea
+size 4983068496

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6948d25d00bf8d4a5eb5f4d0aee265dbe1278eae95208663482cbd6eef373edb
+size 1580230264

model.safetensors.index.json

@@ -0,0 +1,407 @@
+{
+  "metadata": {
+    "total_parameters": 8190735360,
+    "total_size": 16381470720
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.norm.weight": "model-00004-of-00004.safetensors"
+  }
+}

rl_config.yaml

@@ -0,0 +1,316 @@
+entrypoint: examples.terminal_bench.entrypoints.main_tbench
+
+# Hydra config groups (+ prefix in CLI)
+config_groups:
+  terminal_bench_config: terminal_bench
+
+# Terminal bench / agentic environment settings
+terminal_bench:
+  # trials_dir: Directory for Harbor trial artifacts (derived from experiments_dir if null)
+  trials_dir: null
+
+  # Harbor configuration - schema-driven mapping to TrialConfig
+  harbor:
+    # Agent settings
+    name: terminus-2
+    max_episodes: 999999
+    enable_summarize: false
+    store_all_messages: true
+    trajectory_config:
+      raw_content: true
+    enable_episode_logging: false
+    record_terminal_session: false
+    enable_pane_logging: false
+
+    # Strict JSON parser
+    strict_json_parser: true
+
+    # Interleaved Thinking Settings
+    interleaved_thinking: true
+    extra_body:
+      chat_template_kwargs:
+        enable_thinking: true
+    # 2026-05-27: 1800 → 900. The boundary-hugging a3 datasets (stack-junit,
+    # nemotron, methods2test) hit the 1800s agent-timeout wall on ~70-95% of
+    # trials (GLM-4_7-swesmith can't finish these in 1800s), so every group
+    # waited ~1800s for its slowest sample → generation buffer crawled (5/64
+    # in 4h19) and chains never reached training step 1. Halving to 900s
+    # ~2× the valid-group yield. Tradeoff: lower solve-rate on the hardest
+    # tasks (they fail faster) — acceptable vs. TIMEOUT-at-step-0. Fast-task
+    # chains (e2egit, median 7 turns / <900s) are unaffected. See
+    # agent_logs/2026-05-27_rl_chains_stuck_step0.md. AgentTimeoutError stays
+    # PASSTHROUGH (standing pref — partial trajectory+reward is meaningful).
+    override_timeout_sec: 900
+
+    # Environment settings
+    override_cpus: 1
+    override_memory_mb: 2048
+    override_storage_mb: 2048
+
+    # ==========================================================================
+    # AUTO SNAPSHOT: Reduce Daytona rate limits with hash-based snapshot caching
+    # ==========================================================================
+    # When true, automatically creates a snapshot from the Dockerfile on first use,
+    # then reuses it for all subsequent sandboxes with the same Dockerfile content.
+    # Snapshots are named: harbor__<sha256[:12]>__snapshot
+    auto_snapshot: true
+
+    # Verifier settings
+    verifier_override_timeout_sec: 120
+
+    # Retry settings
+    max_retries: 3
+    min_wait_sec: 60.0
+    max_wait_sec: 600.0
+    wait_multiplier: 2.0
+
+    exclude_exceptions:
+      - VerifierTimeoutError
+      - VerifierRuntimeError
+      - RewardFileNotFoundError
+      - RewardFileEmptyError
+      - VerifierOutputParseError
+
+    # 3x base (300 → 900): triple Daytona concurrency for higher throughput.
+    # Mirrors ALCC/56GPU_base.yaml — per-job sandbox count; doesn't affect
+    # the cross-cluster RUNNING-RL cap (≤ 6) which is about job count, not
+    # sandboxes-per-job.
+    # NOTE (2026-05-27): a temporary 900→500 walk-back was tried to mitigate
+    # the FD-exhaustion / uv__epoll_ctl_prep SIGABRT but REVERTED — the real
+    # fix is porting harbor's terminus-2 FD-taming features from
+    # penfever/temp-override into penfever/otagent-latest (that branch had the
+    # issue tamed at full concurrency). See agent_logs/2026-05-27_rl_chains_stuck_step0.md.
+    # 2026-05-28: 900 → 675 (-25%). Fresh a3 chains (#12/#13) abort with a
+    # driver SIGABRT that is NOT FD exhaustion (fd-monitor showed 1.6% of the
+    # 131072 limit) — suspected host-memory OOM from too many concurrent
+    # never-completing trials piling up resident state in skyrl_entrypoint on
+    # hard datasets. Cutting concurrent trial count reduces that resident
+    # memory. Paired with num_parallel_generation_workers -25%. See
+    # agent_logs/2026-05-28_fresh_a3_chain_crash_not_fd.md.
+    n_concurrent_trials: 675
+
+    # Logging settings
+    log_level: INFO
+
+    # Reward shaping (disabled - binary rewards)
+    enable_reward_shaping: false
+
+    # RLOO-N error classification
+    enable_error_classification: true
+    mask_exceptions:
+      - DaytonaError
+      - EnvironmentStartTimeoutError
+      - NetworkError
+      - ConnectionError
+      - RewardFileNotFoundError
+      - RewardFileEmptyError
+      - AgentEnvironmentTimeoutError
+      - ContextLengthExceededError
+    default_error_treatment: zero
+    # NOTE (2026-05-27): AgentTimeoutError / ContextLengthExceededError moved
+    # OUT of passthrough_exceptions into zero_exceptions. Passthrough routed
+    # these soft-limit trials through the *normal* (live-trajectory) path in
+    # terminal_bench_generator._process_trial_result, keeping the trial's
+    # dangling vLLM ObjectRefs + orphaned litellm async-callback closures
+    # alive in the training batch. That fed Ray's distributed-refcount race
+    # (reference_count.cc:1619 → SIGABRT / WorkerCrashedError) that killed the
+    # nemotron-junit chain (litellm +1 32769 VLLMValidationError +
+    # ContextLengthExceeded on near-budget prompts). Both mask + zero take the
+    # EARLY-RETURN path (response_ids=[0]) which never consumes rollout_details,
+    # so no dangling refs enter training. 2026-05-27: ContextLengthExceededError
+    # moved to mask_exceptions (above) — excluded from the RLOO-N baseline, since
+    # GLM-4_7-swesmith ~32k prompts overflow routinely and we don't want hard
+    # zeros dragging the baseline. AgentTimeoutError stays PASSTHROUGH (partial trajectory+reward is
+    # meaningful signal; infrequent vs ContextLengthExceeded so low refcount risk). Belt-and-suspenders w/ the _harbor_compat rollback_hook that
+    # already truncates these trials' dangling prompt-without-response.
+    passthrough_exceptions:
+      - AgentTimeoutError
+    zero_exceptions: []
+
+  # Model info for Harbor's hosted_vllm validation
+  model_info:
+    # Lowered 32767 -> 32000 (2026-05-27). Harbor's litellm token counter is
+    # +2 below vLLM's actual BPE tokenization on near-budget prompts (litellm
+    # +1 plus chat-template/special-token drift), so a 32767-capped prompt
+    # tokenizes to 32769 at the vLLM serving layer and is rejected with
+    # `VLLMValidationError: 32769 input tokens ... context length is only
+    # 32768`. vLLM returns this as an ErrorResponse (handled, not an engine
+    # crash), but harbor's litellm then retries the *deterministically*
+    # over-budget request (max_retries=3, exp backoff) across n_concurrent
+    # trials. On a boundary-heavy task distribution (e.g. nemotron-junit: 111+
+    # such overflows per chain link) the accumulating open sockets exhaust the
+    # skyrl_entrypoint actor's file descriptors -> libuv uv__epoll_ctl_prep
+    # aborts -> SIGABRT in the uvloop event loop -> ray.WorkerCrashedError kills
+    # the chain before training step 1 (job a3-rl ...nemotron-junit #11,
+    # chain 521442-448). A 768-token buffer below max_model_len=32768 ensures
+    # harbor truncates before vLLM ever rejects, eliminating the retry storm.
+    # Cost is ~2.3% usable context for all a3 chains; chains that don't hug the
+    # boundary (e2egit reached step 68 fine at 32767) are unaffected.
+    max_input_tokens: 32000
+    max_output_tokens: 4096
+
+  archiving:
+    # Enable trial archiving callback
+    enabled: false
+
+  # Post-training trace upload to HuggingFace
+  trace_upload:
+    enabled: true
+    repo_org: DCAgent
+    episodes: last
+    dataset_type: SFT
+    cleanup: true
+
+# Trainer configuration
+trainer:
+  strategy: fsdp2
+  algorithm:
+    advantage_estimator: rloo_n
+    use_kl_loss: false
+    kl_loss_coef: 0.0
+    eps_clip_low: 0.2
+    # eps_clip_high=0.05 mirrors 24GPU_base — midpoint between 0.2 default and
+    # 0.01 tight. Engages on collapse-onset ratios without over-clamping
+    # healthy updates. Asymmetric — only tightening upper bound.
+    eps_clip_high: 0.05
+    loss_reduction: token_mean
+
+  # Training loop settings
+  epochs: 2
+  max_steps: 80
+  update_epochs_per_batch: 1
+
+  # Batch sizes
+  train_batch_size: 64
+  policy_mini_batch_size: 64
+  eval_batch_size: 64
+
+  # Micro batch sizes (micro1x4 variant)
+  micro_forward_batch_size_per_gpu: 4
+  micro_train_batch_size_per_gpu: 1
+
+  max_prompt_length: 999999
+
+  # Evaluation and checkpointing
+  eval_interval: 999999
+  eval_before_train: false
+  # Resumable checkpointing
+  ckpt_interval: 2
+  resume_mode: latest
+  # HF upload-ready checkpoints
+  hf_save_interval: 5
+  # HuggingFace Hub upload (set via CLI: trainer.hf_hub_repo_id=org/repo)
+  hf_hub_repo_id: null
+  hf_hub_private: false
+  hf_hub_revision: main
+
+  # Database registration (auto-registers trained model to Supabase)
+  # Requires KEYS env var pointing to Supabase credentials file
+  enable_db_registration: false
+
+  # Logging
+  project_name: OpenThoughts-Agent
+  log_level: INFO
+  tracker_commit_each_step: true
+  logger: console
+
+  # Paths
+  run_name: null
+  ckpt_path: null
+  export_path: null
+
+  # Policy optimizer
+  # max_grad_norm=0.9 mirrors 24GPU_base — guardrail against grad-norm spikes
+  # entering correlation-mode-collapse territory (>1.0). 0.9 is just above the
+  # natural healthy peak observed on this dataset+base (‖g‖ peak ~0.81).
+  policy:
+    optimizer_config:
+      lr: 8e-6
+      weight_decay: 0.0
+      adam_betas: [0.9, 0.999]
+      max_grad_norm: 0.9
+    fsdp_config:
+      cpu_offload: false
+      reshard_after_forward: true
+      fsdp_size: 4
+  # Reference model
+  ref:
+    fsdp_config:
+      cpu_offload: false
+      reshard_after_forward: true
+      fsdp_size: 4
+
+  # Model placement (async training) - 8 shared GPUs for policy/ref
+  placement:
+    colocate_all: false
+    policy_num_nodes: 2
+    ref_num_nodes: 2
+    policy_num_gpus_per_node: 4
+    ref_num_gpus_per_node: 4
+
+  # Fully async generation (settings from v2_maxconcurrent)
+  fully_async:
+    max_staleness_steps: 16
+    # Setting conservatively to 1 / 2 of total concurrency
+    # 2026-05-28: 450 → 338 (-25%), paired with n_concurrent_trials 900→675,
+    # to cut driver memory pressure (suspected OOM on fresh hard-dataset a3
+    # chains). See agent_logs/2026-05-28_fresh_a3_chain_crash_not_fd.md.
+    num_parallel_generation_workers: 338
+
+# Generator configuration
+generator:
+  backend: vllm
+  timeout_multiplier: 1.0
+  model_dtype: bfloat16
+
+  inference_engine_tensor_parallel_size: 1
+  # 3x base (16 → 48): triple vLLM engines to lift gen throughput above training rate.
+  # Goal: build a real surplus of completed groups so the trainer never waits on gen.
+  # Layout: 48 engines × TP=1 + 8 GPUs for policy/ref = 56 GPUs total = 14 nodes.
+  num_inference_engines: 48
+
+  n_samples_per_prompt: 8
+  eval_n_samples_per_prompt: 8
+
+  # Jupiter-specific gpu_memory_utilization: 0.75 (vs ALCC's 0.85).
+  # GH200's 96 GB HBM has more headroom than ALCC's A100-80, but Jupiter
+  # also runs hosted_vllm in the same Ray cluster as the engines and we
+  # keep 0.75 across all jupiter yamls for consistency / fragmentation
+  # safety. Bump to 0.85 only if KV-cache pressure justifies it.
+  gpu_memory_utilization: 0.75
+
+  max_num_seqs: 24
+  # Jupiter-specific 65536 (vs ALCC's 16384) — matches the existing
+  # 24GPU_base.yaml on Jupiter; larger batched-token budget keeps the
+  # engine fed when many concurrent requests arrive in bursts.
+  max_num_batched_tokens: 65536
+
+  enable_prefix_caching: true
+  enable_chunked_prefill: true
+
+  run_engines_locally: true
+  weight_sync_backend: nccl
+  async_engine: true
+  batched: false
+  enable_http_endpoint: true
+  enable_ray_prometheus_stats: false
+  vllm_stats_interval: 1
+  append_eos_token_after_stop_str_in_multi_turn: true
+  max_turns: 999999
+
+  sampling_params:
+    max_generate_length: 4096
+    temperature: 0.7
+    top_p: 0.95
+    top_k: 20
+
+  engine_init_kwargs:
+    max_model_len: 32768
+    # Interleaved thinking chat template: preserves <think> blocks on ALL
+    # historical assistant turns (stock Qwen3 template strips them).
+    custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
+
+# Data paths
+data:
+  train_data: []
+  val_data: []

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,240 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

training_logs/20260606_151436_metrics_job_589413.csv

@@ -0,0 +1,35 @@
+async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_EnvironmentStartTimeoutError,batch_errors/total_EnvironmentStartTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_KeyError,batch_errors/total_KeyError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,timing/save_hf_model,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_RewardFileEmptyError,batch_errors/total_RewardFileEmptyError,batch_errors/avg_RewardFileNotFoundError,batch_errors/total_RewardFileNotFoundError
+0.0,0,64,512,0,0.0,0,0.0,8008.4238,6454.3125,8296.2222,30557,1,6145.4578,0.1562,0.0042,0.0042,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1503,-0.002,0.0,1.0,0.0,0.002,0.1719,0.1562,12.5361,50.6815,570.2606,33.5,857.9676,287.707,0.0777,4.9764,65.8731,326.3489,392.5719,3542.5041,18.4328,326.6207,3126.5227,0,1,128,1024,932,28,82,0.078125,10.0,0.6328125,81,0.0078125,1.0,,,,,,,,,,,,,,,,,,,
+0.0,0,64,512,1,0.9531,0,0.9531,9989.8789,11171.0226,9575.3879,31013,1,7902.9149,0.2598,0.0034,0.0616,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1219,0.0,0.0,1.0,0.0,0.0108,0.3125,0.2598,13.2463,50.9355,567.341,33.9,857.9676,290.6266,0.0959,5.465,67.1229,396.8681,464.4001,1080.6092,19.4978,397.1805,591.2388,0,2,61,488,347,36,139,,,2.2950819672131146,140,,,14.8142,23.9576,0.03278688524590164,2.0,,,,,,,,,,,,,,,
+0.0,0,64,512,2,1.8906,1,1.0,7605.9121,12283.3469,7110.892,29953,1,7792.5022,0.0957,0.0049,0.0482,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1016,0.0,0.0,1.0,0.0,0.0071,0.1875,0.0957,13.525,51.199,566.0598,34.0,857.9676,291.9078,0.0622,4.8118,58.9185,372.0508,431.3149,1056.2852,17.8522,372.3337,602.3031,0,3,61,488,389,28,88,0.13114754098360656,8.0,1.4098360655737705,86,,,,,0.03278688524590164,2.0,0.01639344262295082,1.0,0.01639344262295082,1.0,0.01639344262295082,1.0,,,,,,,,,
+0.0,0,64,512,3,2.3281,1,1.0,9737.7754,11271.8636,9318.0,31498,1,8568.9139,0.2148,0.0024,0.0199,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1175,-0.0,0.0,1.0,0.0,0.0078,0.2656,0.2148,14.0266,51.7184,564.809,34.2,857.9676,293.1586,0.077,5.6117,77.6427,431.842,509.8442,1290.1709,19.2028,432.1239,755.5076,0,4,63,504,389,30,101,0.2222222222222222,14.0,1.6349206349206349,103,,,0.0083,11.984,,,0.031746031746031744,2.0,,,,,0.015873015873015872,1.0,,,,,,,
+0.0,0,64,512,4,2.6406,1,1.0,9354.8848,11739.8,8846.2536,29922,1,8139.2713,0.1758,0.009,0.0379,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1108,0.0,0.0,1.0,0.0,0.0137,0.2656,0.1758,14.1067,51.8126,564.4687,34.2,857.9676,293.4989,0.0932,5.784,65.5331,368.6572,434.5888,1003.4574,19.5787,368.9621,543.503,0,5,63,504,388,29,99,0.25396825396825395,16.0,1.5873015873015872,100,,,,,0.015873015873015872,1.0,,,,,0.031746031746031744,2.0,0.031746031746031744,2.0,6.7546,,,,,,
+0.0,0,64,512,4,2.3594,0,0.9688,8796.3184,9492.7564,8671.1521,31266,1,8002.8664,0.1523,0.0003,0.0408,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.107,0.0,0.0,1.0,0.0,0.0057,0.2031,0.1523,14.3447,52.0127,564.0888,34.3,857.9676,293.8788,0.0991,5.3188,61.6207,383.0313,445.0269,1142.9704,18.7342,383.3065,673.8844,0,6,62,496,397,32,97,0.03225806451612903,2.0,1.6451612903225807,102,,,0.0086,11.9245,,,,,,,,,,,,,,,,,
+0.0,0,64,512,4,2.625,0,0.9844,8902.6406,10492.863,8638.2073,31311,1,7655.1331,0.1426,0.0167,0.0409,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1192,0.0,0.0,1.0,0.0,0.0116,0.2188,0.1426,14.496,52.2004,562.8091,34.4,857.9676,295.1584,0.0971,5.1807,60.7299,376.1769,437.326,874.1424,19.3489,376.4986,412.2793,0,7,64,512,431,21,70,0.125,8.0,1.078125,69,,,,,0.03125,2.0,,,,,0.015625,1.0,,,,0.015625,1.0,,,,
+0.0,0,64,512,4,2.5312,0,0.9688,9234.748,9460.3158,9216.6646,30860,1,7267.3486,0.0742,0.0082,0.0159,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1202,0.0,0.0,1.0,0.0,0.0126,0.1094,0.0742,14.6129,52.1974,561.8904,34.5,857.9676,296.0771,0.0953,5.2113,68.9415,372.7546,442.0454,1105.2377,19.7903,373.0081,638.1832,0,8,61,488,365,27,105,0.26229508196721313,16.0,1.819672131147541,111,,,0.0084,13.4096,0.03278688524590164,2.0,,,,,0.01639344262295082,1.0,,,,,,,,,
+0.0,0,64,512,4,2.5156,0,0.9688,8680.2656,9195.8696,8629.3691,31066,1,7737.2296,0.0898,0.0011,0.0098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1075,-0.0016,0.0,1.0,0.0,0.01,0.1406,0.0898,14.6152,52.1984,560.4534,34.7,857.9676,297.5142,0.0957,5.2168,62.3325,369.4867,432.1807,1089.1024,21.8229,369.7521,629.8738,0,9,63,504,423,26,80,,,1.2698412698412698,80,,,,,0.015873015873015872,1.0,0.015873015873015872,1.0,,,0.047619047619047616,3.0,,,,,,,,,
+0.0,0,64,512,4,2.5156,0,0.9688,10459.5703,11588.1058,10171.9044,30481,1,7871.4302,0.2031,0.0048,0.0376,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1215,0.0,0.0,1.0,0.0,0.0079,0.2812,0.2031,14.7728,52.3819,559.487,34.8,857.9676,298.4806,0.0749,5.5545,74.3819,404.7054,479.4846,1142.0953,22.9006,405.0274,634.1471,0,10,60,480,361,31,100,0.26666666666666666,16.0,1.6666666666666667,100,,,0.0085,13.7449,0.05,3.0,,,,,,,0.016666666666666666,1.0,6.0456,,,,,,
+0.0,0,64,512,5,2.6562,0,0.9844,9537.0527,10914.949,9210.8841,31403,1,8564.2418,0.1914,-0.0013,0.0527,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1103,0.0,0.0,1.0,0.0,0.0085,0.3125,0.1914,14.9898,52.5877,558.656,34.9,857.9676,299.3116,0.0997,5.368,71.5852,424.0463,496.0457,1178.5975,19.4113,424.3602,657.7694,0,11,61,488,408,25,78,,,1.360655737704918,83,,,,,0.01639344262295082,1.0,,,,,0.03278688524590164,2.0,,,,0.01639344262295082,1.0,,,,
+0.0,0,64,512,5,2.4219,0,0.9688,10249.832,12585.6413,9738.1786,29817,1,8232.7329,0.1797,0.0106,0.0884,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1233,-0.002,0.0,1.0,0.0,0.01,0.2969,0.1797,15.3224,52.9479,557.6371,35.0,857.9676,300.3305,0.0957,5.2695,71.4084,417.8217,489.605,1047.7479,18.5522,418.1004,534.3138,0,12,63,504,377,34,123,0.047619047619047616,3.0,1.8888888888888888,119,,,0.008,13.0838,0.015873015873015872,1.0,0.06349206349206349,4.0,,,,,,,,,,,,,
+0.0,0,64,512,7,2.5312,1,1.0,9863.5742,12363.7143,9680.1258,31273,1,8388.053,0.0684,-0.0033,0.0352,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1073,0.0,0.0,1.0,0.0,0.0086,0.0938,0.0684,15.172,52.7369,556.8109,35.1,857.9676,301.1567,0.0961,5.3201,73.1788,402.4707,476.0375,1228.0298,19.6119,402.7622,727.0533,0,13,63,504,385,34,103,0.25396825396825395,16.0,1.619047619047619,102,,,,,,,,,,,0.015873015873015872,1.0,,,,,,,,,
+0.0,0,64,512,4,2.4375,1,1.0,10328.373,12811.4426,9992.5255,30437,1,8842.8434,0.1191,0.0083,0.0233,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1067,0.0,0.0,1.0,0.0,0.011,0.1875,0.1191,15.2037,52.733,631.45,26.4,857.9676,226.5176,0.2101,5.3545,75.6881,431.7699,507.9602,861.5604,19.7888,432.0614,328.4525,0,14,60,480,385,25,93,0.016666666666666666,1.0,1.7333333333333334,104,,,0.008,12.6653,0.016666666666666666,1.0,,,,,0.016666666666666666,1.0,,,,,,,,,
+0.0,0,64,512,5,2.5156,0,0.9219,9168.6328,10240.6912,9004.4437,29749,1,7654.3379,0.1328,0.0025,0.0467,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1149,-0.0,0.0,1.0,0.0,0.0082,0.25,0.1328,15.8657,52.7222,626.7126,27.0,857.9676,231.2549,0.0624,4.9571,67.7609,367.1356,435.2305,1080.6331,18.1439,367.4068,622.2968,0,15,62,496,351,40,135,0.2903225806451613,18.0,2.2096774193548385,137,,,,,,,0.016129032258064516,1.0,,,0.04838709677419355,3.0,,,6.0878,,,,,,
+0.0,0,64,512,5,2.7812,0,0.9844,9860.5078,13834.0633,9135.5404,30958,1,9126.4354,0.1543,0.0025,0.0364,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.095,0.0,0.0,1.0,0.0,0.0082,0.2812,0.1543,15.8339,52.6867,622.6899,27.4,857.9676,235.2776,0.0997,5.2914,70.2154,425.0592,495.6344,1291.7286,18.9487,425.3188,771.8469,0,16,63,504,391,37,103,0.15873015873015872,10.0,1.6349206349206349,103,,,0.0086,12.4342,,,,,,,,,,,,,,,,,
+0.0,0,64,512,4,2.6875,0,0.9844,10984.0449,13339.6324,10623.2793,30946,1,8860.174,0.1328,0.006,0.0495,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1103,-0.0,0.0,1.0,0.0,0.0075,0.1875,0.1328,15.8974,52.7485,619.0869,27.8,857.9676,238.8807,0.0701,5.3616,71.2226,437.1859,508.7899,1012.7019,18.2569,437.4967,480.289,0,17,62,496,365,28,111,0.3064516129032258,19.0,1.7096774193548387,106,,,,,0.016129032258064516,1.0,,,,,0.0967741935483871,6.0,,,,,,,,,
+0.0,0,64,512,5,2.5156,0,0.9688,9343.375,11583.6966,8872.0071,30532,1,8242.8542,0.1738,0.0043,0.0404,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1106,0.0,0.0,1.0,0.0,0.012,0.2656,0.1738,15.8437,52.7366,616.8957,28.1,857.9676,241.0719,0.0968,5.1671,71.7476,398.261,470.3905,1019.3083,19.5815,398.5457,524.1618,0,18,63,504,335,40,158,0.15873015873015872,10.0,2.3968253968253967,151,,,0.0081,12.7041,0.015873015873015872,1.0,,,,,0.07936507936507936,5.0,0.031746031746031744,2.0,,,,,,,
+0.0,0,64,512,4,2.6719,1,1.0,8471.2129,13121.2,7479.5095,31520,1,8710.7976,0.1758,0.0039,0.009,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0945,0.0,0.0,1.0,0.0,0.0032,0.2344,0.1758,16.4859,52.7443,633.4617,26.2,857.9676,224.5059,0.0872,5.2245,66.0855,407.5046,473.9292,1109.522,19.6532,407.7559,610.7109,0,19,65,520,405,31,104,0.15384615384615385,10.0,1.5692307692307692,102,,,,,0.015384615384615385,1.0,,,,,0.03076923076923077,2.0,,,,,,,,,
+0.0,0,64,512,4,2.4688,0,0.9688,9397.5605,10769.1373,9245.8243,29721,1,8090.9631,0.0996,-0.0018,0.0196,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1076,0.0,0.0,1.0,0.0,0.0054,0.1406,0.0996,16.5093,52.7675,627.9171,26.8,857.9676,230.0505,0.0761,5.269,71.6405,396.6428,468.6066,1307.4116,19.3738,396.8895,814.1585,0,20,63,504,367,36,119,0.2222222222222222,14.0,1.6031746031746033,101,,,0.0077,12.3336,0.06349206349206349,4.0,0.06349206349206349,4.0,,,0.2222222222222222,14.0,,,6.113,,,,,,
+0.0,0,64,512,4,2.3594,0,0.9531,8876.3809,11866.2886,7649.1185,31020,1,8144.0439,0.291,0.0083,0.0317,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1037,-0.002,0.0,1.0,0.0,0.0063,0.3594,0.291,16.8527,53.1653,624.288,27.2,857.9676,233.6796,0.0644,5.1935,65.7614,389.564,455.6589,1135.649,18.4184,389.8327,656.37,0,21,63,504,370,34,121,0.14285714285714285,9.0,1.873015873015873,118,,,,,0.031746031746031744,2.0,0.015873015873015872,1.0,,,0.09523809523809523,6.0,,,,0.031746031746031744,2.0,0.015873015873015872,1.0,,
+0.0,0,64,512,4,2.4062,0,0.9844,9612.6992,12727.119,9001.4579,30518,1,8610.0533,0.1641,0.0001,0.0381,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1071,-0.0,0.0,1.0,0.0,0.007,0.25,0.1641,16.8425,53.2195,620.9487,27.6,857.9676,237.0189,0.0666,5.2176,71.3936,408.7832,480.5136,1077.8042,19.3161,409.053,572.7531,0,22,63,504,371,32,122,0.15873015873015872,10.0,1.8253968253968254,115,,,0.0083,14.3305,0.015873015873015872,1.0,0.047619047619047616,3.0,,,0.07936507936507936,5.0,,,,,,,,0.07936507936507936,5.0
+0.0,0,64,512,4,2.3125,1,1.0,9644.4805,12499.4219,9236.6317,30247,1,8449.5651,0.125,0.004,0.0329,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0975,-0.0,0.0,1.0,0.0,0.0066,0.1719,0.125,16.898,53.3349,629.8778,26.6,857.9676,228.0898,0.0955,5.124,68.4055,407.5341,476.3226,1267.6702,18.3902,407.8206,767.8278,0,23,65,520,326,45,175,0.26153846153846155,17.0,2.5846153846153848,168,,,,,0.03076923076923077,2.0,0.015384615384615385,1.0,,,0.09230769230769231,6.0,,,,,,,,,
+0.0,0,64,512,4,2.6094,0,0.9844,8941.7324,12333.4851,8108.236,30654,1,9059.652,0.1973,-0.0011,0.0307,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0891,0.0,0.0,1.0,0.0,0.0064,0.2812,0.1973,17.0198,53.3979,625.2485,27.1,857.9676,232.7191,0.1002,5.0625,64.3428,413.9089,478.6224,1422.179,18.4004,414.1787,920.0877,0,24,64,512,348,39,150,0.15625,10.0,2.25,144,,,0.0084,12.0063,0.0625,4.0,,,,,0.09375,6.0,,,,,,,,,
+0.0,0,64,512,4,2.0781,0,0.9688,8478.0488,9041.8118,8365.8244,31197,1,8623.2484,0.166,0.0034,0.0454,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.099,0.0,0.0,1.0,0.0,0.0084,0.2344,0.166,17.0027,53.4105,621.9629,27.5,857.9676,236.0047,0.1051,5.0928,65.4935,394.6093,460.5277,1256.6936,18.4525,394.9286,772.6144,0,25,63,504,371,30,129,0.047619047619047616,3.0,1.8253968253968254,115,,,,,0.015873015873015872,1.0,,,,,0.2222222222222222,14.0,,,6.1077,,,,,,
+0.0,0,64,512,4,2.1719,0,0.9844,9143.0312,10793.6909,8944.3742,30837,1,7933.9127,0.1074,-0.0014,0.0226,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1039,0.0,0.0,1.0,0.0,0.0049,0.1406,0.1074,16.9659,53.344,619.0737,27.8,857.9676,238.8939,0.0999,5.2223,65.8763,368.7657,435.0494,1056.9012,17.4192,369.0727,599.2038,0,26,64,512,349,36,156,0.078125,5.0,2.25,144,,,0.0083,12.3343,0.03125,2.0,,,,,0.1875,12.0,,,,,,,,,
+0.0,0,64,512,4,2.2969,1,1.0,9318.0684,13478.1053,8370.3381,30076,1,8927.2675,0.1855,-0.0011,0.0127,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0987,0.0,0.0,1.0,0.0,0.0025,0.2344,0.1855,17.2687,53.7057,615.7143,28.2,857.9676,242.2533,0.0941,5.2193,75.3222,424.8138,500.4749,1506.0501,17.7312,425.0582,982.618,0,27,63,504,365,43,130,0.12698412698412698,8.0,1.7936507936507937,113,,,,,0.015873015873015872,1.0,,,,,0.2698412698412698,17.0,,,,,,,,,
+0.0,0,64,512,4,2.3125,0,0.9844,9706.3535,12505.9143,9262.9842,31295,1,8703.7321,0.1367,0.0164,0.0632,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1061,-0.0,0.0,1.0,0.0,0.0111,0.2344,0.1367,17.2266,53.6643,613.2701,28.5,857.9676,244.6974,0.0669,5.773,65.2001,413.5376,479.086,1433.1909,17.2395,413.8186,931.0875,0,28,64,512,338,40,162,0.1875,12.0,2.265625,145,,,0.0092,12.5456,,,0.046875,3.0,,,0.3125,20.0,,,,,,,,,
+0.0,0,64,512,4,2.3906,0,0.9844,9190.4043,14896.2414,8461.4648,31487,1,9135.4476,0.1133,0.002,0.0487,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0966,0.0,0.0,1.0,0.0,0.0068,0.1719,0.1133,17.3366,53.7334,611.0821,28.8,857.9676,246.8855,0.1016,5.197,71.0217,433.8967,505.3155,1360.4476,18.3518,434.1918,831.5773,0,29,60,480,381,30,88,0.16666666666666666,10.0,1.3666666666666667,82,,,,,0.016666666666666666,1.0,,,,,0.26666666666666666,16.0,,,,,,,,,
+0.0,0,64,512,4,2.1562,0,0.9688,9495.8145,9631.0294,9475.1059,31385,1,7976.2685,0.1328,0.0033,0.0175,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1173,-0.0033,0.0,1.0,0.0,0.0061,0.1875,0.1328,17.3372,53.7932,608.0651,29.1,857.9676,249.9025,0.0913,5.3592,68.7826,391.6734,460.8187,1262.0092,18.3274,391.9444,777.4995,0,30,64,512,338,40,157,0.25,16.0,2.359375,151,,,0.0093,12.804,0.015625,1.0,0.015625,1.0,,,0.078125,5.0,,,5.7747,,,,,0.015625,1.0
+0.0,0,64,512,4,2.4062,1,1.0,9377.334,13592.5495,8466.2067,31245,1,8932.7859,0.1777,0.0029,0.0383,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0993,-0.0,0.0,1.0,0.0,0.0061,0.25,0.1777,17.642,54.1039,605.7375,29.4,857.9676,252.23,0.0721,5.2867,72.9809,411.0495,484.4737,1259.668,18.722,411.4202,751.1815,0,31,63,504,377,36,116,0.14285714285714285,9.0,1.7301587301587302,109,,,,,0.031746031746031744,2.0,0.047619047619047616,3.0,,,0.07936507936507936,5.0,,,,,,,,0.015873015873015872,1.0
+0.0,0,64,512,4,2.2656,0,0.9531,10975.1738,11508.35,10904.3982,31286,1,9253.1616,0.1172,0.0006,0.0289,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1081,-0.0,0.0,1.0,0.0,0.0089,0.1719,0.1172,17.6508,54.1074,603.5103,29.7,857.9676,254.4573,0.0782,5.6393,76.5317,453.5022,530.3774,1462.4104,18.7241,453.7671,907.6655,0,32,62,496,344,37,132,0.2903225806451613,18.0,1.9193548387096775,119,,,0.008,13.2254,0.03225806451612903,2.0,0.03225806451612903,2.0,,,0.25806451612903225,16.0,,,,,,,,,
+0.0,0,64,512,4,2.2656,0,0.9844,10141.4238,13411.0159,9682.6615,31058,1,9259.9469,0.123,0.0048,0.0311,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0991,0.0,0.0,1.0,0.0,0.0056,0.1719,0.123,17.9439,54.3633,601.2052,29.9,857.9676,256.7624,0.0899,5.5213,77.6245,440.932,519.1285,1156.1656,18.025,441.4135,613.4848,0,33,64,512,375,36,128,0.0625,4.0,1.8125,116,,,,,0.0625,4.0,,,,,0.1875,12.0,,,,0.015625,1.0,,,,
+0.0,0,64,512,4,2.25,0,0.9375,9173.3125,10323.3913,9059.7854,30789,1,8248.3461,0.0898,0.0034,0.0353,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1055,0.0,0.0,1.0,0.0,0.0064,0.1406,0.0898,18.9124,54.4227,597.6184,30.3,857.9676,260.3492,0.0665,5.2916,67.6079,383.5392,451.4687,1258.212,17.6079,383.7938,783.8386,0,34,29,232,161,19,68,,,2.0689655172413794,60,,,0.0073,12.0016,0.10344827586206896,3.0,,,,,0.27586206896551724,8.0,,,,,,,,,

training_logs/20260606_151436_metrics_job_589414.csv

@@ -0,0 +1,33 @@
+async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/save_hf_model,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_DaytonaNotFoundError,batch_errors/total_DaytonaNotFoundError
+0.0,0,64,512,4,0.8438,0,0.3125,10437.5781,12216.3191,9761.5606,30906,1,8425.2048,0.2754,0.0045,0.0183,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,-0.002,0.0,1.0,0.0,0.007,0.3636,0.2754,12.9912,50.7759,629.5773,26.6,857.9675,228.3903,0.2767,5.2124,74.8968,412.6478,488.0814,7.2362,4779.1047,18.2716,412.907,4267.539,0,35,63,504,287,47,208,3.0952380952380953,195,0.23809523809523808,15.0,0.20634920634920634,13.0,0.031746031746031744,2.0,,,,,,,,,,
+0.0,0,64,512,1,1.0,1,1.0,7617.0176,12086.0,7217.6617,29796,1,8549.5628,0.082,0.0123,0.0609,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0835,-0.0015,0.0,1.0,0.0,0.0109,0.1719,0.082,13.0166,50.8255,628.3799,26.8,857.9675,229.5876,0.0642,4.757,62.0209,360.6818,423.056,,863.6853,19.4267,360.9703,416.4341,0,36,62,496,334,34,128,1.967741935483871,122,0.532258064516129,33.0,0.1935483870967742,12.0,0.016129032258064516,1.0,14.7598,22.9295,0.016129032258064516,1.0,,,,,,
+0.0,0,64,512,2,1.6719,0,0.9531,8760.6465,10886.4028,8412.7955,30724,1,8278.5732,0.1406,0.0003,0.0457,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0928,-0.0,0.0,1.0,0.0,0.0056,0.1875,0.1406,13.2816,51.1987,628.3003,26.8,857.9675,229.6672,0.0686,4.9887,70.2967,382.4549,453.1113,,1683.2463,18.1475,382.7453,1206.9952,0,37,63,504,340,38,150,2.3174603174603177,146,0.19047619047619047,12.0,0.031746031746031744,2.0,0.031746031746031744,2.0,,,0.015873015873015872,1.0,,,,,,
+0.0,0,64,512,3,2.125,1,1.0,9711.0859,13647.7812,8802.6178,30819,1,9164.2999,0.1875,0.0068,0.0556,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0973,-0.0,0.0,1.0,0.0,0.0079,0.2969,0.1875,13.7075,51.4554,624.343,27.2,857.9675,233.6246,0.0742,5.2921,70.8615,430.3457,501.5565,,1089.0308,19.4588,430.6204,562.719,0,38,63,504,345,41,140,2.142857142857143,135,0.1746031746031746,11.0,0.047619047619047616,3.0,0.1111111111111111,7.0,0.0101,8.3872,0.031746031746031744,2.0,0.015873015873015872,1.0,,,,
+0.0,0,64,512,4,2.4688,0,0.9688,9830.0312,14998.9714,9450.7589,31096,1,9133.2781,0.0684,0.0078,0.0391,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1011,-0.002,0.0,1.0,0.0,0.0063,0.1094,0.0684,13.9773,51.7114,620.9083,27.6,857.9675,237.0592,0.0719,5.1953,74.5293,443.5724,518.4862,,1413.4816,21.7271,443.8847,868.0682,0,39,64,512,339,43,168,2.40625,154,0.046875,3.0,0.140625,9.0,0.015625,1.0,,,0.078125,5.0,0.015625,1.0,,,,
+0.0,0,64,512,4,2.1562,0,0.9844,9801.043,16505.9767,9186.3049,30907,1,9197.5424,0.084,0.0054,0.0597,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0935,-0.0005,0.0,1.0,0.0,0.0058,0.1562,0.084,14.1064,51.8289,625.7814,27.1,857.9675,232.1861,0.0741,5.1608,69.67,427.6333,497.6731,5.8701,1297.6163,20.2719,427.9286,774.5065,0,40,63,504,309,42,147,2.2857142857142856,144,0.7301587301587301,46.0,0.047619047619047616,3.0,0.031746031746031744,2.0,0.0096,8.4634,,,,,,,,
+0.0,0,64,512,4,2.125,0,0.9688,8527.4961,11893.8046,7838.3929,30476,1,9153.2018,0.1699,0.0071,0.0525,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0915,0.0,0.0,1.0,0.0,0.0051,0.2656,0.1699,14.1085,52.0214,621.533,27.6,857.9675,236.4345,0.0745,5.0385,69.091,420.8784,490.3688,,1109.7026,18.6338,421.2029,595.6564,0,41,62,496,370,32,100,1.6129032258064515,100,0.3870967741935484,24.0,,,0.03225806451612903,2.0,,,,,,,,,,
+0.0,0,64,512,4,2.0781,0,0.9062,8602.8281,13130.1719,7956.0647,30220,1,8072.8774,0.125,0.0056,0.0104,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,-0.0,0.0,1.0,0.0,0.0056,0.1719,0.125,14.3199,52.3275,618.3289,27.9,857.9675,239.6387,0.0678,4.8823,60.7727,365.3284,427.1973,,1310.0776,19.9813,366.3565,858.0116,0,42,64,512,335,42,177,2.765625,177,,,,,,,0.0091,13.1783,,,,,,,,
+0.0,0,64,512,4,2.4844,0,0.9844,9943.8965,14522.3788,9266.3632,31302,1,9568.7495,0.1289,0.0004,0.0148,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0922,0.0,0.0,1.0,0.0,0.0061,0.25,0.1289,14.4575,52.3593,615.0375,28.3,857.9675,242.9301,0.0709,5.3536,73.4756,439.6526,513.5252,,1370.015,19.0401,439.9785,832.0917,0,43,62,496,309,39,175,2.935483870967742,182,0.12903225806451613,8.0,0.016129032258064516,1.0,0.03225806451612903,2.0,,,0.016129032258064516,1.0,0.016129032258064516,1.0,0.016129032258064516,1.0,,
+0.0,0,64,512,4,2.2188,0,0.9844,9086.3555,11450.5517,8784.3216,31262,1,9393.9818,0.1133,0.0007,0.0481,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0907,-0.0007,0.0,1.0,0.0,0.0072,0.2188,0.1133,15.2404,52.5447,611.1252,28.8,857.9675,246.8423,0.0753,5.1796,81.3851,430.771,512.6133,,1326.4544,19.2724,431.1526,789.3843,0,44,64,512,328,41,170,2.640625,169,0.1875,12.0,,,0.03125,2.0,0.0088,8.375,0.015625,1.0,,,,,,
+0.0,0,64,512,4,2.0469,0,0.9844,9583.3691,13273.5522,9027.7685,30598,1,9337.8756,0.1309,0.0045,0.0705,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0902,-0.0036,0.0,1.0,0.0,0.0074,0.2031,0.1309,15.2057,52.4645,608.826,29.0,857.9675,249.1415,0.0722,6.294,66.9948,423.8895,491.2079,5.9741,1420.4363,20.0032,424.1404,902.927,0,45,63,504,324,46,175,2.7777777777777777,175,0.015873015873015872,1.0,,,0.031746031746031744,2.0,,,,,0.015873015873015872,1.0,0.015873015873015872,1.0,,
+0.0,0,64,512,4,2.2969,0,0.9844,9691.5176,14166.0521,8658.9327,31244,1,9305.0755,0.1875,0.0014,0.0454,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0908,0.0,0.0,1.0,0.0,0.008,0.3281,0.1875,15.2065,52.5456,606.6011,29.3,857.9675,251.3664,0.0752,5.4224,76.2767,430.1979,506.8676,,1193.1841,19.3454,430.5153,661.5444,0,46,64,512,345,42,158,2.46875,158,0.140625,9.0,,,,,0.0091,8.4932,,,,,,,,
+0.0,0,64,512,4,2.1094,0,0.9844,9212.3242,11344.0481,8668.9436,31190,1,9416.1534,0.2031,0.0114,0.052,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0936,-0.002,0.0,1.0,0.0,0.0081,0.3125,0.2031,14.9799,52.4084,622.5466,27.4,857.9675,235.421,0.0721,5.1673,74.6655,426.7696,501.7978,,1222.5144,20.3347,427.0598,695.2104,0,47,65,520,365,39,150,2.3076923076923075,150,,,,,0.07692307692307693,5.0,,,,,,,,,,
+0.0,0,64,512,4,2.25,0,0.9688,9360.1367,11736.4615,8754.4069,31205,1,8692.4675,0.2031,0.0014,0.0459,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0967,0.0,0.0,1.0,0.0,0.006,0.2969,0.2031,15.0214,52.425,618.7219,27.9,857.9675,239.2457,0.0711,5.2233,75.0136,431.6495,507.052,,1314.5149,19.6438,431.967,782.5914,0,48,64,512,367,36,128,2.0,128,0.25,16.0,,,0.015625,1.0,0.0085,8.0854,,,,,,,,
+0.0,0,64,512,4,2.25,0,0.9844,9074.0957,10469.2639,8845.7955,30308,1,8363.6284,0.1406,0.0047,0.0279,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1028,0.0,0.0,1.0,0.0,0.0051,0.1875,0.1406,15.0687,52.4651,615.8871,28.2,857.9675,242.0804,0.0895,5.114,66.3779,388.6285,455.3327,,1176.5771,19.9722,388.8649,696.1539,0,49,63,504,329,39,153,2.380952380952381,150,0.2857142857142857,18.0,0.015873015873015872,1.0,0.047619047619047616,3.0,,,0.031746031746031744,2.0,0.015873015873015872,1.0,,,,
+0.0,0,64,512,4,2.1562,0,0.9688,8850.0449,11473.1616,8221.2591,31212,1,8601.1534,0.1934,0.0041,0.0483,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0926,-0.0,0.0,1.0,0.0,0.0075,0.2812,0.1934,15.1635,52.538,613.2601,28.5,857.9675,244.7074,0.0716,6.2678,65.1271,384.0105,449.5106,6.1327,1156.8611,19.9659,384.3115,681.1123,0,50,63,504,368,39,119,1.8571428571428572,117,0.2698412698412698,17.0,,,,,0.0083,8.5298,0.031746031746031744,2.0,,,,,,
+0.0,0,64,512,4,2.375,0,0.9688,9589.3242,10869.4848,9399.8834,31288,1,8733.0224,0.1289,-0.0003,0.0192,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,0.0,0.0,1.0,0.0,0.0027,0.2031,0.1289,15.4259,52.851,610.0452,28.9,857.9675,247.9223,0.0715,5.3738,72.1756,418.9982,491.5295,,1212.2518,20.0825,419.282,695.2615,0,51,61,488,328,38,137,2.2295081967213113,136,0.3770491803278688,23.0,,,,,,,0.01639344262295082,1.0,,,,,,
+0.0,0,64,512,4,2.2188,0,0.9844,8594.4082,12730.0615,7993.0268,30309,1,8273.7344,0.127,0.0042,0.0227,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0919,-0.0003,0.0,1.0,0.0,0.0049,0.1719,0.127,15.3691,52.7689,607.3633,29.2,857.9675,250.6042,0.0692,6.0694,65.242,372.8057,438.4157,,1065.0034,19.6838,373.1041,600.8298,0,52,64,512,350,37,154,2.359375,151,0.03125,2.0,,,0.078125,5.0,0.0082,9.0665,0.046875,3.0,0.015625,1.0,,,,
+0.0,0,64,512,4,2.2969,0,0.9219,9746.8672,12776.9178,9243.0091,30786,1,8989.9087,0.1426,0.0061,0.044,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0999,-0.0034,0.0,1.0,0.0,0.0105,0.2344,0.1426,16.1131,53.1016,603.8304,29.6,857.9675,254.1371,0.074,5.2264,73.7603,426.7516,500.919,,1352.3345,22.008,427.0843,824.1767,0,53,64,512,347,41,150,2.3125,148,0.21875,14.0,,,0.015625,1.0,,,0.03125,2.0,,,,,,
+0.0,0,64,512,4,2.375,0,0.9844,8793.3281,10799.5357,8231.59,30032,1,8724.8588,0.2188,0.0003,0.0569,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0953,-0.0039,0.0,1.0,0.0,0.0075,0.3281,0.2188,16.0381,53.0909,601.3336,29.9,857.9675,256.6339,0.0685,5.1466,65.5847,414.5861,480.548,,1173.7722,19.8623,414.8945,668.2111,0,54,64,512,375,37,116,1.78125,114,0.265625,17.0,0.015625,1.0,0.046875,3.0,0.0092,9.4884,0.015625,1.0,,,,,0.015625,1.0
+0.0,0,64,512,4,2.3125,0,0.9531,10566.2969,15290.8932,9376.4841,31410,1,8879.9389,0.2012,0.0092,0.062,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1024,-0.0011,0.0,1.0,0.0,0.0116,0.3125,0.2012,16.1116,53.1949,598.9396,30.2,857.9675,259.028,0.073,5.354,71.9567,429.284,501.609,6.0853,1261.7633,20.7867,429.5788,734.0093,0,55,63,504,323,40,161,2.619047619047619,165,0.14285714285714285,9.0,0.015873015873015872,1.0,0.09523809523809523,6.0,,,,,0.015873015873015872,1.0,0.047619047619047616,3.0,0.015873015873015872,1.0
+0.0,0,64,512,4,2.2188,1,1.0,9105.8008,16860.3696,8340.3283,30634,1,8929.209,0.0898,0.0184,0.0576,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0877,-0.0022,0.0,1.0,0.0,0.0098,0.2188,0.0898,16.1151,53.3091,597.5827,30.3,857.9675,260.3848,0.0717,5.1031,70.8664,394.2463,465.4289,,1091.053,20.0046,394.4905,600.5119,0,56,60,480,364,31,107,1.7833333333333334,107,0.15,9.0,,,,,0.009,8.794,,,,,,,,
+0.0,0,64,512,4,2.2188,0,0.9688,9641.6895,12360.0233,9092.9178,31090,1,8609.0109,0.168,0.0033,0.045,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1088,-0.0,0.0,1.0,0.0,0.0066,0.25,0.168,16.5616,53.6893,595.4823,30.6,857.9675,262.4852,0.072,5.3676,73.5019,417.3243,491.1753,,994.6671,20.0253,417.601,478.0945,0,57,60,480,367,32,94,1.7,102,0.26666666666666666,16.0,,,0.05,3.0,,,,,,,,,,
+0.0,0,64,512,4,2.4219,0,0.9844,10050.9961,12024.2162,9504.793,30541,1,8807.1723,0.2168,-0.0042,0.0241,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1046,0.0,0.0,1.0,0.0,0.005,0.2969,0.2168,16.7413,53.8259,593.603,30.8,857.9675,264.3646,0.0688,5.27,75.3362,434.0958,509.781,,1039.3098,20.109,434.3756,504.1454,0,58,62,496,369,33,118,1.8870967741935485,117,0.12903225806451613,8.0,0.016129032258064516,1.0,0.016129032258064516,1.0,0.0088,8.1323,,,,,,,,
+0.0,0,64,512,4,2.5156,0,0.9688,10033.8789,11916.9434,9542.2414,30895,1,8675.2561,0.207,-0.0027,0.0237,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1062,0.0,0.0,1.0,0.0,0.004,0.2656,0.207,16.718,53.7883,591.6187,31.0,857.9675,266.3488,0.0732,5.4347,72.3408,424.4609,497.1824,,1099.6533,20.2339,424.768,576.7981,0,59,64,512,356,40,143,2.1875,140,0.1875,12.0,0.015625,1.0,0.03125,2.0,,,0.0625,4.0,0.015625,1.0,,,,
+0.0,0,64,512,5,2.625,0,0.9688,10255.4453,11782.7547,9856.6897,31411,1,9352.3936,0.207,-0.0005,0.0143,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0999,0.0,0.0,1.0,0.0,0.003,0.2656,0.207,16.6629,53.7289,589.5977,31.3,857.9675,268.3698,0.0713,5.3855,76.6821,456.4509,533.52,5.936,1404.7918,19.2674,456.7662,846.6146,0,60,59,472,317,39,140,2.542372881355932,150,0.2542372881355932,15.0,,,0.06779661016949153,4.0,0.009,8.6208,0.01694915254237288,1.0,,,,,,
+0.0,0,64,512,4,2.3281,0,0.9844,8909.3555,14929.6739,8315.0751,30786,1,8970.4052,0.0898,0.0005,0.0049,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0877,0.0,0.0,1.0,0.0,0.0025,0.1406,0.0898,17.0513,54.0333,587.6489,31.5,857.9675,270.3187,0.0684,5.2603,72.0759,407.8817,480.3137,,1017.8425,19.3396,408.1689,512.924,0,61,64,512,367,31,119,1.828125,117,0.40625,26.0,,,,,,,0.03125,2.0,,,,,,
+0.0,0,64,512,4,2.3438,1,1.0,8969.4238,8863.163,8992.7,30797,1,8054.4728,0.1797,0.0005,0.003,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1029,-0.0,0.0,1.0,0.0,0.0027,0.1875,0.1797,17.4703,54.3962,585.8682,31.7,857.9675,272.0993,0.0708,5.3578,65.7888,362.4732,428.5896,,1111.1134,18.7856,362.7297,658.3761,0,62,61,488,349,35,120,1.9672131147540983,120,0.29508196721311475,18.0,,,0.01639344262295082,1.0,0.009,8.2242,,,,,,,,
+0.0,0,64,512,4,2.3594,0,0.9531,9888.6504,13162.0123,9273.471,31070,1,9029.6223,0.1582,0.0057,0.0317,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1012,-0.0,0.0,1.0,0.0,0.0078,0.2188,0.1582,17.4786,54.3968,584.5446,31.9,857.9675,273.4229,0.0747,5.2953,79.8089,427.4805,507.6453,,1382.5263,19.5701,427.7613,850.0112,0,63,58,464,331,35,120,2.1724137931034484,126,0.1896551724137931,11.0,0.017241379310344827,1.0,0.034482758620689655,2.0,,,0.06896551724137931,4.0,,,,,,
+0.0,0,64,512,4,2.4844,0,0.9688,9433.4355,12847.9314,8583.9756,31078,1,8889.5713,0.1992,0.0044,0.0373,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0999,-0.0012,0.0,1.0,0.0,0.0082,0.3125,0.1992,17.2842,54.336,583.1888,32.0,857.9675,274.7787,0.0706,5.3302,70.4561,431.1564,501.9916,,1036.7697,21.6661,431.4646,507.7772,0,64,63,504,374,30,127,2.0,126,0.031746031746031744,2.0,,,0.015873015873015872,1.0,0.0082,8.205,0.015873015873015872,1.0,,,,,,
+0.0,0,64,512,4,2.3594,0,0.9844,10180.8359,9118.8947,10365.945,31446,1,9044.9768,0.1484,0.0006,0.0191,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1007,0.0,0.0,1.0,0.0,0.0047,0.2031,0.1484,17.3231,54.4015,621.9341,27.5,857.9675,236.0334,0.0728,5.5421,78.2487,435.2774,513.9476,5.8354,1157.1932,19.0014,435.6258,618.6976,0,65,62,496,350,33,137,2.2903225806451615,142,0.016129032258064516,1.0,0.016129032258064516,1.0,0.03225806451612903,2.0,,,,,0.06451612903225806,4.0,0.016129032258064516,1.0,0.03225806451612903,2.0
+0.0,0,64,512,4,2.4688,1,1.0,9031.8203,11650.358,8539.7053,30031,1,8235.59,0.1582,0.0053,0.0558,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0975,0.0,0.0,1.0,0.0,0.0097,0.2344,0.1582,17.4368,54.4995,616.9651,28.1,857.9675,241.0024,0.0701,5.1933,66.5767,384.447,451.3545,,1015.6966,19.7349,384.7073,539.4094,0,66,42,336,228,26,103,2.4761904761904763,104,0.11904761904761904,5.0,,,,,0.0085,8.349,,,,,,,,

training_logs/20260606_151436_metrics_job_589415.csv

@@ -0,0 +1,8 @@
+async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,timing/save_hf_model,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_DaytonaNotFoundError,batch_errors/total_DaytonaNotFoundError
+0.0,0,64,512,4,1.1406,0,0.4062,9320.8516,14550.6567,8533.4427,31363,1,9169.3835,0.1309,0.0044,0.0201,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0918,0.0,0.0,1.0,0.0,0.004,0.1852,0.1309,13.4686,50.5256,603.4965,29.7,857.9675,254.4711,0.1717,5.1623,84.3031,442.9513,527.6906,4644.6281,17.6466,443.2151,4094.1282,0,67,60,480,312,43,158,2.9166666666666665,175,0.1,6,0.06666666666666667,4,0.05,3.0,,,,,,,,,,,
+0.0,0,64,512,1,0.9688,0,0.9688,8838.2246,14955.6667,7885.395,31069,1,9454.7696,0.1348,0.0049,0.0588,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,0.0,0.0,1.0,0.0,0.0076,0.2188,0.1348,13.6993,50.7828,600.2096,30.0,857.9675,257.7579,0.0646,5.1712,76.8797,447.6366,524.8183,1058.6616,18.5204,447.8732,510.1366,0,68,61,488,348,34,126,2.0655737704918034,126,0.03278688524590164,2,0.19672131147540983,12,,,11.1932,19.3268,,,,,,,,,
+0.0,0,64,512,2,1.7344,1,1.0,8664.0488,11762.5714,8283.5285,29909,1,7605.2237,0.1094,0.0014,0.0566,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1024,-0.002,0.0,1.0,0.0,0.01,0.2031,0.1094,13.8658,50.9406,598.2016,30.3,857.9675,259.7659,0.0651,5.0053,63.8033,372.1322,436.2895,819.5224,18.7903,372.4205,359.4342,0,69,61,488,378,28,93,1.5081967213114753,92,0.08196721311475409,5,0.14754098360655737,9,0.03278688524590164,2.0,,,0.01639344262295082,1.0,0.03278688524590164,2.0,,,,,
+0.0,0,64,512,3,2.2344,0,0.9531,10562.7539,12507.5258,10108.1928,30830,1,8215.603,0.1895,0.0043,0.0703,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1069,0.0,0.0,1.0,0.0,0.0113,0.2656,0.1895,14.0821,51.1283,623.5098,27.3,857.9675,234.4578,0.0712,5.2083,76.5816,414.9592,491.8453,1037.2437,17.6609,415.1921,522.5257,0,70,62,496,359,33,122,1.935483870967742,120,0.04838709677419355,3,0.1935483870967742,12,0.03225806451612903,2.0,0.0082,7.9993,,,,,6.2945,,,,
+0.0,0,64,512,4,2.6406,0,0.9375,10671.9688,11820.5049,10382.7286,31046,1,9383.9069,0.2012,0.0104,0.0247,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1016,-0.002,0.0,1.0,0.0,0.0074,0.2656,0.2012,14.3759,51.4185,619.6084,27.8,857.9675,238.3591,0.0693,5.2885,83.9511,461.8063,546.119,1228.6645,18.3273,462.0983,658.925,0,71,60,480,328,36,136,2.216666666666667,133,0.08333333333333333,5,0.2,12,0.03333333333333333,2.0,,,0.016666666666666666,1.0,,,,0.016666666666666666,1.0,,
+0.0,0,64,512,5,2.6719,0,0.9844,9609.7773,13170.275,8950.4259,30619,1,9001.5217,0.1562,0.0077,0.0335,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0938,-0.002,0.0,1.0,0.0,0.0054,0.2188,0.1562,14.5012,51.5967,616.5872,28.1,857.9675,241.3804,0.0684,5.2467,76.3358,416.1831,492.8603,890.1009,18.801,416.4557,373.1888,0,72,59,472,334,31,125,2.169491525423729,128,0.05084745762711865,3,0.1694915254237288,10,,,0.0114,8.2721,,,,,,0.03389830508474576,2.0,,
+0.0,0,64,512,5,2.4375,0,0.9844,9353.7402,11149.3151,9055.1595,31227,1,8604.2319,0.1426,0.0035,0.0336,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1013,-0.002,0.0,1.0,0.0,0.0079,0.2031,0.1426,14.8654,51.7343,622.3488,27.5,857.9675,235.6187,0.104,5.238,69.902,401.1988,471.4352,970.1871,19.4643,401.4288,474.0452,0,73,61,488,338,35,139,2.2459016393442623,137,0.01639344262295082,1,0.13114754098360656,8,0.01639344262295082,1.0,,,0.01639344262295082,1.0,,,,0.01639344262295082,1.0,0.01639344262295082,1.0

training_logs/20260606_151436_metrics_report.md

@@ -0,0 +1,380 @@
+# SkyRL Training Metrics Analysis
+
+Generated from 3 log files
+
+## Overview
+
+| Log File | Total Steps | Metric Blocks | Final Reward (mean) | Final Reward (max) | Total Time (s) |
+|----------|-------------|---------------|---------------------|-------------------|----------------|
+| job_589413 | 34 | 34 | 0.1506 | 0.2910 | 42378.9 |
+| job_589414 | 66 | 32 | 0.1586 | 0.2754 | 42142.2 |
+| job_589415 | 73 | 7 | 0.1521 | 0.2012 | 10649.0 |
+
+## Async Metrics
+
+|                               |       Mean |      Std |   Min |      Max |   Count |
+|:------------------------------|-----------:|---------:|------:|---------:|--------:|
+| async/discard_rate            |   0        | 0        |     0 |   0      |      73 |
+| async/discarded_count         |   0        | 0        |     0 |   0      |      73 |
+| async/effective_batch_groups  |  64        | 0        |    64 |  64      |      73 |
+| async/effective_batch_samples | 512        | 0        |   512 | 512      |      73 |
+| async/staleness_max           |   3.84932  | 1.00928  |     0 |   7      |      73 |
+| async/staleness_mean          |   2.2241   | 0.483279 |     0 |   2.7812 |      73 |
+| async/staleness_min           |   0.205479 | 0.406848 |     0 |   1      |      73 |
+| async/staleness_ratio         |   0.945866 | 0.152747 |     0 |   1      |      73 |
+
+## Generate Metrics
+
+|                                      |     Mean |      Std |      Min |      Max |   Count |
+|:-------------------------------------|---------:|---------:|---------:|---------:|--------:|
+| generate/avg_num_tokens              |  9423.99 |  697.672 |  7605.91 | 10984    |      73 |
+| generate/avg_tokens_non_zero_rewards | 12123.6  | 1819.92  |  6454.31 | 16860.4  |      73 |
+| generate/avg_tokens_zero_rewards     |  8947.26 |  755.482 |  7110.89 | 10904.4  |      73 |
+| generate/max_num_tokens              | 30826.2  |  494.298 | 29721    | 31520    |      73 |
+| generate/min_num_tokens              |     1    |    0     |     1    |     1    |      73 |
+| generate/std_num_tokens              |  8593.72 |  606.039 |  6145.46 |  9568.75 |      73 |
+
+## Loss Metrics
+
+|                             |       Mean |        Std |     Min |    Max |   Count |
+|:----------------------------|-----------:|-----------:|--------:|-------:|--------:|
+| loss/avg_final_rewards      | 0.154214   | 0.0473859  |  0.0684 | 0.291  |      73 |
+| loss/avg_raw_advantages     | 0.00408356 | 0.00444472 | -0.0042 | 0.0184 |      73 |
+| loss/avg_raw_advantages_abs | 0.037111   | 0.0180256  |  0.003  | 0.0884 |      73 |
+
+## Policy Metrics
+
+|                             |         Mean |         Std |     Min |    Max |   Count |
+|:----------------------------|-------------:|------------:|--------:|-------:|--------:|
+| policy/final_loss           | -5.47945e-06 | 2.29154e-05 | -0.0001 | 0      |      73 |
+| policy/log_ratio_abs_max    |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_mean   |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_p99    |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos00  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos10  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos20  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos30  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos40  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos50  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos60  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos70  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos80  |  0           | 0           |  0      | 0      |      73 |
+| policy/log_ratio_abs_pos90  |  0           | 0           |  0      | 0      |      73 |
+| policy/n_tokens_dp_gt_10pct |  0           | 0           |  0      | 0      |      73 |
+| policy/n_tokens_dp_gt_1pct  |  0           | 0           |  0      | 0      |      73 |
+| policy/n_tokens_dp_gt_50pct |  0           | 0           |  0      | 0      |      73 |
+| policy/policy_entropy       |  0.102596    | 0.0108117   |  0.0829 | 0.1503 |      73 |
+| policy/policy_loss          | -0.000593151 | 0.00104128  | -0.0039 | 0      |      73 |
+| policy/policy_lr            |  0           | 0           |  0      | 0      |      73 |
+| policy/policy_update_steps  |  1           | 0           |  1      | 1      |      73 |
+| policy/ppo_clip_ratio       |  0           | 0           |  0      | 0      |      73 |
+| policy/raw_grad_norm        |  0.00725068  | 0.00261481  |  0.002  | 0.0137 |      73 |
+
+## Reward Metrics
+
+|                       |     Mean |       Std |    Min |    Max |   Count |
+|:----------------------|---------:|----------:|-------:|-------:|--------:|
+| reward/avg_pass_at_8  | 0.227553 | 0.0609958 | 0.0938 | 0.3636 |      73 |
+| reward/avg_raw_reward | 0.154214 | 0.0473859 | 0.0684 | 0.291  |      73 |
+
+## System Metrics
+
+|                         |     Mean |          Std |      Min |      Max |   Count |
+|:------------------------|---------:|-------------:|---------:|---------:|--------:|
+| system/process_rss_gb   |  15.5468 |  1.4605      |  12.5361 |  18.9124 |      73 |
+| system/process_vms_gb   |  52.6842 |  1.06812     |  50.5256 |  54.4995 |      73 |
+| system/ram_available_gb | 603.469  | 22.6021      | 556.811  | 633.462  |      73 |
+| system/ram_percent      |  29.6589 |  2.63686     |  26.2    |  35.1    |      73 |
+| system/ram_total_gb     | 857.968  |  5.02278e-05 | 857.967  | 857.968  |      73 |
+| system/ram_used_gb      | 254.498  | 22.6021      | 224.506  | 301.157  |      73 |
+
+## Timing Metrics
+
+|                                       |         Mean |         Std |      Min |       Max |   Count |
+|:--------------------------------------|-------------:|------------:|---------:|----------:|--------:|
+| timing/compute_advantages_and_returns |    0.0846452 |   0.0319802 |   0.0622 |    0.2767 |      73 |
+| timing/convert_to_training_input      |    5.29443   |   0.266451  |   4.757  |    6.294  |      73 |
+| timing/fwd_logprobs_values_reward     |   70.7341    |   5.46465   |  58.9185 |   84.3031 |      73 |
+| timing/policy_train                   |  408.762     |  27.3881    | 326.349  |  461.806  |      73 |
+| timing/run_training                   |  479.881     |  31.7823    | 392.572  |  546.119  |      73 |
+| timing/step                           | 1303.7       | 660.326     | 819.522  | 4779.1    |      73 |
+| timing/sync_weights                   |   19.2668    |   1.09635   |  17.2395 |   22.9006 |      73 |
+| timing/train_critic_and_policy        |  409.062     |  27.3812    | 326.621  |  462.098  |      73 |
+| timing/wait_for_generation_buffer     |  799.253     | 659.115     | 328.452  | 4267.54   |      73 |
+| timing/cleanup_old_checkpoints        |    1.14037   |   3.83885   |   0.0073 |   14.8142 |      36 |
+| timing/save_checkpoints               |   11.6225    |   3.89007   |   7.9993 |   23.9576 |      36 |
+| timing/save_hf_model                  |    6.16055   |   0.390464  |   5.7747 |    7.2362 |      14 |
+
+## Trainer Metrics
+
+|                     |   Mean |     Std |   Min |   Max |   Count |
+|:--------------------|-------:|--------:|------:|------:|--------:|
+| trainer/epoch       |      0 |  0      |     0 |     0 |      73 |
+| trainer/global_step |     37 | 21.2171 |     1 |    73 |      73 |
+
+## Batch_Errors Metrics
+
+|                                                 |        Mean |          Std |         Min |          Max |   Count |
+|:------------------------------------------------|------------:|-------------:|------------:|-------------:|--------:|
+| batch_errors/total_batches                      |  62.589     |   9.1315     |  29         |  128         |      73 |
+| batch_errors/total_instances                    | 500.712     |  73.052      | 232         | 1024         |      73 |
+| batch_errors/total_successful                   | 360.151     |  77.8734     | 161         |  932         |      73 |
+| batch_errors/total_failed                       |  34.9726    |   5.82374    |  19         |   47         |      73 |
+| batch_errors/total_masked                       | 127.575     |  28.7266     |  68         |  208         |      73 |
+| batch_errors/avg_VerifierTimeoutError           |   0.191289  |   0.121376   |   0.015873  |    0.730159  |      67 |
+| batch_errors/total_VerifierTimeoutError         |  11.9701    |   7.57182    |   1         |   46         |      67 |
+| batch_errors/avg_ContextLengthExceededError     |   2.02529   |   0.446941   |   0.632812  |    3.09524   |      73 |
+| batch_errors/total_ContextLengthExceededError   | 125.438     |  28.102      |  60         |  195         |      73 |
+| batch_errors/avg_EnvironmentStartTimeoutError   |   0.0078125 | nan          |   0.0078125 |    0.0078125 |       1 |
+| batch_errors/total_EnvironmentStartTimeoutError |   1         | nan          |   1         |    1         |       1 |
+| batch_errors/avg_RuntimeError                   |   0.0382515 |   0.0256729  |   0.0153846 |    0.111111  |      59 |
+| batch_errors/total_RuntimeError                 |   2.32203   |   1.50219    |   1         |    7         |      59 |
+| batch_errors/avg_AgentTimeoutError              |   0.0314587 |   0.0188765  |   0.0153846 |    0.078125  |      33 |
+| batch_errors/total_AgentTimeoutError            |   1.9697    |   1.18545    |   1         |    5         |      33 |
+| batch_errors/avg_KeyError                       |   0.0163934 | nan          |   0.0163934 |    0.0163934 |       1 |
+| batch_errors/total_KeyError                     |   1         | nan          |   1         |    1         |       1 |
+| batch_errors/avg_DaytonaError                   |   0.089779  |   0.0901486  |   0.015625  |    0.3125    |      46 |
+| batch_errors/total_DaytonaError                 |   5.43478   |   5.39816    |   1         |   20         |      46 |
+| batch_errors/avg_InvalidChatHistory             |   0.0240079 |   0.00894106 |   0.015873  |    0.031746  |       4 |
+| batch_errors/total_InvalidChatHistory           |   1.5       |   0.57735    |   1         |    2         |       4 |
+| batch_errors/avg_DaytonaValidationError         |   0.019991  |   0.0125129  |   0.015625  |    0.0645161 |      16 |
+| batch_errors/total_DaytonaValidationError       |   1.25      |   0.774597   |   1         |    4         |      16 |
+| batch_errors/avg_RewardFileEmptyError           |   0.015873  | nan          |   0.015873  |    0.015873  |       1 |
+| batch_errors/total_RewardFileEmptyError         |   1         | nan          |   1         |    1         |       1 |
+| batch_errors/avg_RewardFileNotFoundError        |   0.0369544 |   0.036729   |   0.015625  |    0.0793651 |       3 |
+| batch_errors/total_RewardFileNotFoundError      |   2.33333   |   2.3094     |   1         |    5         |       3 |
+| batch_errors/avg_DaytonaAuthenticationError     |   0.0257074 |   0.0142342  |   0.015873  |    0.047619  |       5 |
+| batch_errors/total_DaytonaAuthenticationError   |   1.6       |   0.894427   |   1         |    3         |       5 |
+| batch_errors/avg_DaytonaNotFoundError           |   0.0200374 |   0.00815341 |   0.015625  |    0.0322581 |       4 |
+| batch_errors/total_DaytonaNotFoundError         |   1.25      |   0.5        |   1         |    2         |       4 |
+
+## Training Progression by Log
+
+### job_589413
+
+| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
+|------|--------|--------|-----|------|---------------|-------------|
+| 1 | 0.1562 | 0.1719 | 0.000000 | -0.0000 | 3542.5 | 3126.5 |
+| 2 | 0.2598 | 0.3125 | 0.000000 | 0.0000 | 1080.6 | 591.2 |
+| 3 | 0.0957 | 0.1875 | 0.000000 | 0.0000 | 1056.3 | 602.3 |
+| 4 | 0.2148 | 0.2656 | 0.000000 | -0.0000 | 1290.2 | 755.5 |
+| 5 | 0.1758 | 0.2656 | 0.000000 | 0.0000 | 1003.5 | 543.5 |
+| 6 | 0.1523 | 0.2031 | 0.000000 | 0.0000 | 1143.0 | 673.9 |
+| 7 | 0.1426 | 0.2188 | 0.000000 | 0.0000 | 874.1 | 412.3 |
+| 8 | 0.0742 | 0.1094 | 0.000000 | 0.0000 | 1105.2 | 638.2 |
+| 9 | 0.0898 | 0.1406 | 0.000000 | -0.0000 | 1089.1 | 629.9 |
+| 10 | 0.2031 | 0.2812 | 0.000000 | 0.0000 | 1142.1 | 634.1 |
+| 11 | 0.1914 | 0.3125 | 0.000000 | 0.0000 | 1178.6 | 657.8 |
+| 12 | 0.1797 | 0.2969 | 0.000000 | -0.0000 | 1047.7 | 534.3 |
+| 13 | 0.0684 | 0.0938 | 0.000000 | 0.0000 | 1228.0 | 727.1 |
+| 14 | 0.1191 | 0.1875 | 0.000000 | 0.0000 | 861.6 | 328.5 |
+| 15 | 0.1328 | 0.2500 | 0.000000 | -0.0000 | 1080.6 | 622.3 |
+| 16 | 0.1543 | 0.2812 | 0.000000 | 0.0000 | 1291.7 | 771.8 |
+| 17 | 0.1328 | 0.1875 | 0.000000 | -0.0000 | 1012.7 | 480.3 |
+| 18 | 0.1738 | 0.2656 | 0.000000 | 0.0000 | 1019.3 | 524.2 |
+| 19 | 0.1758 | 0.2344 | 0.000000 | 0.0000 | 1109.5 | 610.7 |
+| 20 | 0.0996 | 0.1406 | 0.000000 | 0.0000 | 1307.4 | 814.2 |
+| 21 | 0.2910 | 0.3594 | 0.000000 | -0.0000 | 1135.6 | 656.4 |
+| 22 | 0.1641 | 0.2500 | 0.000000 | -0.0000 | 1077.8 | 572.8 |
+| 23 | 0.1250 | 0.1719 | 0.000000 | -0.0000 | 1267.7 | 767.8 |
+| 24 | 0.1973 | 0.2812 | 0.000000 | 0.0000 | 1422.2 | 920.1 |
+| 25 | 0.1660 | 0.2344 | 0.000000 | 0.0000 | 1256.7 | 772.6 |
+| 26 | 0.1074 | 0.1406 | 0.000000 | 0.0000 | 1056.9 | 599.2 |
+| 27 | 0.1855 | 0.2344 | 0.000000 | 0.0000 | 1506.1 | 982.6 |
+| 28 | 0.1367 | 0.2344 | 0.000000 | -0.0000 | 1433.2 | 931.1 |
+| 29 | 0.1133 | 0.1719 | 0.000000 | 0.0000 | 1360.4 | 831.6 |
+| 30 | 0.1328 | 0.1875 | 0.000000 | -0.0001 | 1262.0 | 777.5 |
+| 31 | 0.1777 | 0.2500 | 0.000000 | -0.0000 | 1259.7 | 751.2 |
+| 32 | 0.1172 | 0.1719 | 0.000000 | -0.0000 | 1462.4 | 907.7 |
+| 33 | 0.1230 | 0.1719 | 0.000000 | 0.0000 | 1156.2 | 613.5 |
+| 34 | 0.0898 | 0.1406 | 0.000000 | 0.0000 | 1258.2 | 783.8 |
+
+### job_589414
+
+| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
+|------|--------|--------|-----|------|---------------|-------------|
+| 35 | 0.2754 | 0.3636 | 0.000000 | -0.0000 | 4779.1 | 4267.5 |
+| 36 | 0.0820 | 0.1719 | 0.000000 | -0.0000 | 863.7 | 416.4 |
+| 37 | 0.1406 | 0.1875 | 0.000000 | -0.0000 | 1683.2 | 1207.0 |
+| 38 | 0.1875 | 0.2969 | 0.000000 | -0.0000 | 1089.0 | 562.7 |
+| 39 | 0.0684 | 0.1094 | 0.000000 | -0.0000 | 1413.5 | 868.1 |
+| 40 | 0.0840 | 0.1562 | 0.000000 | -0.0000 | 1297.6 | 774.5 |
+| 41 | 0.1699 | 0.2656 | 0.000000 | 0.0000 | 1109.7 | 595.7 |
+| 42 | 0.1250 | 0.1719 | 0.000000 | -0.0000 | 1310.1 | 858.0 |
+| 43 | 0.1289 | 0.2500 | 0.000000 | 0.0000 | 1370.0 | 832.1 |
+| 44 | 0.1133 | 0.2188 | 0.000000 | -0.0000 | 1326.5 | 789.4 |
+| 45 | 0.1309 | 0.2031 | 0.000000 | -0.0001 | 1420.4 | 902.9 |
+| 46 | 0.1875 | 0.3281 | 0.000000 | 0.0000 | 1193.2 | 661.5 |
+| 47 | 0.2031 | 0.3125 | 0.000000 | -0.0000 | 1222.5 | 695.2 |
+| 48 | 0.2031 | 0.2969 | 0.000000 | 0.0000 | 1314.5 | 782.6 |
+| 49 | 0.1406 | 0.1875 | 0.000000 | 0.0000 | 1176.6 | 696.2 |
+| 50 | 0.1934 | 0.2812 | 0.000000 | -0.0000 | 1156.9 | 681.1 |
+| 51 | 0.1289 | 0.2031 | 0.000000 | 0.0000 | 1212.3 | 695.3 |
+| 52 | 0.1270 | 0.1719 | 0.000000 | -0.0000 | 1065.0 | 600.8 |
+| 53 | 0.1426 | 0.2344 | 0.000000 | -0.0001 | 1352.3 | 824.2 |
+| 54 | 0.2188 | 0.3281 | 0.000000 | -0.0001 | 1173.8 | 668.2 |
+| 55 | 0.2012 | 0.3125 | 0.000000 | -0.0000 | 1261.8 | 734.0 |
+| 56 | 0.0898 | 0.2188 | 0.000000 | -0.0000 | 1091.1 | 600.5 |
+| 57 | 0.1680 | 0.2500 | 0.000000 | -0.0000 | 994.7 | 478.1 |
+| 58 | 0.2168 | 0.2969 | 0.000000 | 0.0000 | 1039.3 | 504.1 |
+| 59 | 0.2070 | 0.2656 | 0.000000 | 0.0000 | 1099.7 | 576.8 |
+| 60 | 0.2070 | 0.2656 | 0.000000 | 0.0000 | 1404.8 | 846.6 |
+| 61 | 0.0898 | 0.1406 | 0.000000 | 0.0000 | 1017.8 | 512.9 |
+| 62 | 0.1797 | 0.1875 | 0.000000 | -0.0000 | 1111.1 | 658.4 |
+| 63 | 0.1582 | 0.2188 | 0.000000 | -0.0000 | 1382.5 | 850.0 |
+| 64 | 0.1992 | 0.3125 | 0.000000 | -0.0000 | 1036.8 | 507.8 |
+| 65 | 0.1484 | 0.2031 | 0.000000 | 0.0000 | 1157.2 | 618.7 |
+| 66 | 0.1582 | 0.2344 | 0.000000 | 0.0000 | 1015.7 | 539.4 |
+
+### job_589415
+
+| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
+|------|--------|--------|-----|------|---------------|-------------|
+| 67 | 0.1309 | 0.1852 | 0.000000 | 0.0000 | 4644.6 | 4094.1 |
+| 68 | 0.1348 | 0.2188 | 0.000000 | 0.0000 | 1058.7 | 510.1 |
+| 69 | 0.1094 | 0.2031 | 0.000000 | -0.0000 | 819.5 | 359.4 |
+| 70 | 0.1895 | 0.2656 | 0.000000 | 0.0000 | 1037.2 | 522.5 |
+| 71 | 0.2012 | 0.2656 | 0.000000 | -0.0000 | 1228.7 | 658.9 |
+| 72 | 0.1562 | 0.2188 | 0.000000 | -0.0000 | 890.1 | 373.2 |
+| 73 | 0.1426 | 0.2031 | 0.000000 | -0.0000 | 970.2 | 474.0 |
+
+## Timing Analysis
+
+### Average Time Breakdown (% of step time)
+
+| Component | Avg % of Step Time |
+|-----------|-------------------|
+| wait_for_generation_buffer | 57.6% |
+| run_training | 40.4% |
+| train_critic_and_policy | 34.4% |
+| policy_train | 34.4% |
+| fwd_logprobs_values_reward | 5.9% |
+| sync_weights | 1.6% |
+| save_checkpoints | 1.0% |
+| save_hf_model | 0.5% |
+| convert_to_training_input | 0.4% |
+| cleanup_old_checkpoints | 0.1% |
+| compute_advantages_and_returns | 0.0% |
+
+## Cross-Log Comparison
+
+| Log | Avg Reward | Pass@8 | Step Time (s) | Gen Wait Time (s) | Avg Tokens | Staleness |
+|-----|------|------|------|------|------|------|
+| job_589413 | 0.1506 | 0.2178 | 1246.4373 | 751.3619 | 9413.2012 | 2.3038 |
+| job_589414 | 0.1586 | 0.2389 | 1316.9451 | 806.4623 | 9402.5320 | 2.1939 |
+| job_589415 | 0.1521 | 0.2229 | 1521.2869 | 998.9120 | 9574.4807 | 1.9755 |
+
+## vLLM Inference Engine Analysis
+
+Metrics from vLLM stat loggers (V1LoggingStatLoggerFixed).
+
+> **Note**: Ray deduplicates similar log messages with `[repeated Nx across cluster]`,
+> so we typically capture stats from one engine per timestamp. The stats shown are
+> **per-engine** values. Multiply by num_inference_engines for cluster-wide estimates.
+
+### Summary by Log (Per-Engine Stats)
+
+| Log | Avg Running/Engine | Avg Waiting/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % |
+|-----|-------------------|-------------------|--------------------------|----------------|------------------|
+| job_589413 | 8.0 | 0.0 | 362.6 tok/s | 36.4% | 91.9% |
+| job_589414 | 8.0 | 0.0 | 358.2 tok/s | 36.8% | 92.9% |
+| job_589415 | 7.2 | 0.0 | 323.6 tok/s | 32.2% | 92.1% |
+
+### Utilization Analysis (Per-Engine)
+
+Key indicators of inference engine utilization:
+
+- **Running requests/engine**: Concurrent requests being processed by each engine
+- **Waiting requests**: Requests queued (0 = engine not saturated, has spare capacity)
+- **Generation throughput**: Decode tokens/sec per engine
+  - 8B model on H100 can do **1000+ tok/s** when saturated
+  - If seeing <300 tok/s with 0 waiting, engine is **starved for requests**
+
+#### job_589413
+
+- **Running requests/engine**: avg=8.0, max=23
+- **Waiting requests**: avg=0.0, max=5
+- **Generation throughput/engine**: avg=362.6 tok/s, max=841.4 tok/s
+- **KV cache usage**: avg=36.4%
+- **Prefix cache hit rate**: avg=91.9%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+#### job_589414
+
+- **Running requests/engine**: avg=8.0, max=24
+- **Waiting requests**: avg=0.0, max=12
+- **Generation throughput/engine**: avg=358.2 tok/s, max=819.3 tok/s
+- **KV cache usage**: avg=36.8%
+- **Prefix cache hit rate**: avg=92.9%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+#### job_589415
+
+- **Running requests/engine**: avg=7.2, max=21
+- **Waiting requests**: avg=0.0, max=2
+- **Generation throughput/engine**: avg=323.6 tok/s, max=705.6 tok/s
+- **KV cache usage**: avg=32.2%
+- **Prefix cache hit rate**: avg=92.1%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+## Trial-Level Analysis (from result.json)
+
+Total trials parsed: 39462
+
+### Turn Count Statistics
+
+| Metric | Value |
+|--------|-------|
+| Mean | 17.2 |
+| Median | 15.0 |
+| Std | 10.6 |
+| Min | 1 |
+| Max | 171 |
+| Count | 39437 |
+
+### Exception Distribution
+
+| Exception Type | Count | % |
+|---------------|-------|---|
+| No exception | 26679 | 67.6% |
+| ContextLengthExceededError | 9524 | 24.1% |
+| AgentTimeoutError | 1945 | 4.9% |
+| VerifierTimeoutError | 858 | 2.2% |
+| DaytonaError | 268 | 0.7% |
+| RuntimeError | 144 | 0.4% |
+| DaytonaValidationError | 20 | 0.1% |
+| RewardFileNotFoundError | 9 | 0.0% |
+| DaytonaAuthenticationError | 8 | 0.0% |
+| DaytonaNotFoundError | 5 | 0.0% |
+| EnvironmentStartTimeoutError | 1 | 0.0% |
+| RewardFileEmptyError | 1 | 0.0% |
+
+### Turn Count by Exception Type
+
+| Exception Type | Mean Turns | Median Turns | Count |
+|---------------|-----------|-------------|-------|
+| RewardFileEmptyError | 39.0 | 39.0 | 1 |
+| AgentTimeoutError | 21.7 | 13.0 | 1944 |
+| ContextLengthExceededError | 20.9 | 19.0 | 9524 |
+| DaytonaAuthenticationError | 17.4 | 19.0 | 8 |
+| DaytonaValidationError | 16.8 | 15.5 | 20 |
+| VerifierTimeoutError | 16.5 | 15.0 | 858 |
+| No exception | 15.6 | 14.0 | 26679 |
+| RewardFileNotFoundError | 14.6 | 14.0 | 9 |
+| DaytonaError | 14.2 | 13.0 | 245 |
+| RuntimeError | 12.1 | 10.0 | 144 |
+| DaytonaNotFoundError | 11.8 | 8.0 | 5 |
+
+### Turn Count by Outcome
+
+| Outcome | Mean Turns | Median Turns | Count |
+|---------|-----------|-------------|-------|
+| Success | 14.9 | 13.0 | 6140 |
+| Failure | 16.3 | 14.0 | 22416 |
+
+### Reward Summary
+
+- Mean reward: 0.2150
+- Success rate: 21.5%
+- Trials with reward data: 28557
+

training_logs/20260606_151436_metrics_table.csv

@@ -0,0 +1,74 @@
+log_file,async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_EnvironmentStartTimeoutError,batch_errors/total_EnvironmentStartTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_KeyError,batch_errors/total_KeyError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,timing/save_hf_model,batch_errors/avg_DaytonaValidationError,batch_errors/total_DaytonaValidationError,batch_errors/avg_RewardFileEmptyError,batch_errors/total_RewardFileEmptyError,batch_errors/avg_RewardFileNotFoundError,batch_errors/total_RewardFileNotFoundError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,batch_errors/avg_DaytonaNotFoundError,batch_errors/total_DaytonaNotFoundError,global_step
+job_589413,0.0,0,64,512,0,0.0,0,0.0,8008.4238,6454.3125,8296.2222,30557,1,6145.4578,0.1562,0.0042,0.0042,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1503,-0.002,0.0,1.0,0.0,0.002,0.1719,0.1562,12.5361,50.6815,570.2606,33.5,857.9676,287.707,0.0777,4.9764,65.8731,326.3489,392.5719,3542.5041,18.4328,326.6207,3126.5227,0,1,128,1024,932,28,82,0.078125,10.0,0.6328125,81,0.0078125,1.0,,,,,,,,,,,,,,,,,,,,,,,,1
+job_589413,0.0,0,64,512,1,0.9531,0,0.9531,9989.8789,11171.0226,9575.3879,31013,1,7902.9149,0.2598,0.0034,0.0616,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1219,0.0,0.0,1.0,0.0,0.0108,0.3125,0.2598,13.2463,50.9355,567.341,33.9,857.9676,290.6266,0.0959,5.465,67.1229,396.8681,464.4001,1080.6092,19.4978,397.1805,591.2388,0,2,61,488,347,36,139,,,2.2950819672131146,140,,,14.8142,23.9576,0.03278688524590164,2.0,,,,,,,,,,,,,,,,,,,,2
+job_589413,0.0,0,64,512,2,1.8906,1,1.0,7605.9121,12283.3469,7110.892,29953,1,7792.5022,0.0957,0.0049,0.0482,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1016,0.0,0.0,1.0,0.0,0.0071,0.1875,0.0957,13.525,51.199,566.0598,34.0,857.9676,291.9078,0.0622,4.8118,58.9185,372.0508,431.3149,1056.2852,17.8522,372.3337,602.3031,0,3,61,488,389,28,88,0.13114754098360656,8.0,1.4098360655737705,86,,,,,0.03278688524590164,2.0,0.01639344262295082,1.0,0.01639344262295082,1.0,0.01639344262295082,1.0,,,,,,,,,,,,,,3
+job_589413,0.0,0,64,512,3,2.3281,1,1.0,9737.7754,11271.8636,9318.0,31498,1,8568.9139,0.2148,0.0024,0.0199,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1175,-0.0,0.0,1.0,0.0,0.0078,0.2656,0.2148,14.0266,51.7184,564.809,34.2,857.9676,293.1586,0.077,5.6117,77.6427,431.842,509.8442,1290.1709,19.2028,432.1239,755.5076,0,4,63,504,389,30,101,0.2222222222222222,14.0,1.6349206349206349,103,,,0.0083,11.984,,,0.031746031746031744,2.0,,,,,0.015873015873015872,1.0,,,,,,,,,,,,4
+job_589413,0.0,0,64,512,4,2.6406,1,1.0,9354.8848,11739.8,8846.2536,29922,1,8139.2713,0.1758,0.009,0.0379,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1108,0.0,0.0,1.0,0.0,0.0137,0.2656,0.1758,14.1067,51.8126,564.4687,34.2,857.9676,293.4989,0.0932,5.784,65.5331,368.6572,434.5888,1003.4574,19.5787,368.9621,543.503,0,5,63,504,388,29,99,0.25396825396825395,16.0,1.5873015873015872,100,,,,,0.015873015873015872,1.0,,,,,0.031746031746031744,2.0,0.031746031746031744,2.0,6.7546,,,,,,,,,,,5
+job_589413,0.0,0,64,512,4,2.3594,0,0.9688,8796.3184,9492.7564,8671.1521,31266,1,8002.8664,0.1523,0.0003,0.0408,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.107,0.0,0.0,1.0,0.0,0.0057,0.2031,0.1523,14.3447,52.0127,564.0888,34.3,857.9676,293.8788,0.0991,5.3188,61.6207,383.0313,445.0269,1142.9704,18.7342,383.3065,673.8844,0,6,62,496,397,32,97,0.03225806451612903,2.0,1.6451612903225807,102,,,0.0086,11.9245,,,,,,,,,,,,,,,,,,,,,,6
+job_589413,0.0,0,64,512,4,2.625,0,0.9844,8902.6406,10492.863,8638.2073,31311,1,7655.1331,0.1426,0.0167,0.0409,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1192,0.0,0.0,1.0,0.0,0.0116,0.2188,0.1426,14.496,52.2004,562.8091,34.4,857.9676,295.1584,0.0971,5.1807,60.7299,376.1769,437.326,874.1424,19.3489,376.4986,412.2793,0,7,64,512,431,21,70,0.125,8.0,1.078125,69,,,,,0.03125,2.0,,,,,0.015625,1.0,,,,0.015625,1.0,,,,,,,,,7
+job_589413,0.0,0,64,512,4,2.5312,0,0.9688,9234.748,9460.3158,9216.6646,30860,1,7267.3486,0.0742,0.0082,0.0159,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1202,0.0,0.0,1.0,0.0,0.0126,0.1094,0.0742,14.6129,52.1974,561.8904,34.5,857.9676,296.0771,0.0953,5.2113,68.9415,372.7546,442.0454,1105.2377,19.7903,373.0081,638.1832,0,8,61,488,365,27,105,0.26229508196721313,16.0,1.819672131147541,111,,,0.0084,13.4096,0.03278688524590164,2.0,,,,,0.01639344262295082,1.0,,,,,,,,,,,,,,8
+job_589413,0.0,0,64,512,4,2.5156,0,0.9688,8680.2656,9195.8696,8629.3691,31066,1,7737.2296,0.0898,0.0011,0.0098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1075,-0.0016,0.0,1.0,0.0,0.01,0.1406,0.0898,14.6152,52.1984,560.4534,34.7,857.9676,297.5142,0.0957,5.2168,62.3325,369.4867,432.1807,1089.1024,21.8229,369.7521,629.8738,0,9,63,504,423,26,80,,,1.2698412698412698,80,,,,,0.015873015873015872,1.0,0.015873015873015872,1.0,,,0.047619047619047616,3.0,,,,,,,,,,,,,,9
+job_589413,0.0,0,64,512,4,2.5156,0,0.9688,10459.5703,11588.1058,10171.9044,30481,1,7871.4302,0.2031,0.0048,0.0376,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1215,0.0,0.0,1.0,0.0,0.0079,0.2812,0.2031,14.7728,52.3819,559.487,34.8,857.9676,298.4806,0.0749,5.5545,74.3819,404.7054,479.4846,1142.0953,22.9006,405.0274,634.1471,0,10,60,480,361,31,100,0.26666666666666666,16.0,1.6666666666666667,100,,,0.0085,13.7449,0.05,3.0,,,,,,,0.016666666666666666,1.0,6.0456,,,,,,,,,,,10
+job_589413,0.0,0,64,512,5,2.6562,0,0.9844,9537.0527,10914.949,9210.8841,31403,1,8564.2418,0.1914,-0.0013,0.0527,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1103,0.0,0.0,1.0,0.0,0.0085,0.3125,0.1914,14.9898,52.5877,558.656,34.9,857.9676,299.3116,0.0997,5.368,71.5852,424.0463,496.0457,1178.5975,19.4113,424.3602,657.7694,0,11,61,488,408,25,78,,,1.360655737704918,83,,,,,0.01639344262295082,1.0,,,,,0.03278688524590164,2.0,,,,0.01639344262295082,1.0,,,,,,,,,11
+job_589413,0.0,0,64,512,5,2.4219,0,0.9688,10249.832,12585.6413,9738.1786,29817,1,8232.7329,0.1797,0.0106,0.0884,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1233,-0.002,0.0,1.0,0.0,0.01,0.2969,0.1797,15.3224,52.9479,557.6371,35.0,857.9676,300.3305,0.0957,5.2695,71.4084,417.8217,489.605,1047.7479,18.5522,418.1004,534.3138,0,12,63,504,377,34,123,0.047619047619047616,3.0,1.8888888888888888,119,,,0.008,13.0838,0.015873015873015872,1.0,0.06349206349206349,4.0,,,,,,,,,,,,,,,,,,12
+job_589413,0.0,0,64,512,7,2.5312,1,1.0,9863.5742,12363.7143,9680.1258,31273,1,8388.053,0.0684,-0.0033,0.0352,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1073,0.0,0.0,1.0,0.0,0.0086,0.0938,0.0684,15.172,52.7369,556.8109,35.1,857.9676,301.1567,0.0961,5.3201,73.1788,402.4707,476.0375,1228.0298,19.6119,402.7622,727.0533,0,13,63,504,385,34,103,0.25396825396825395,16.0,1.619047619047619,102,,,,,,,,,,,0.015873015873015872,1.0,,,,,,,,,,,,,,13
+job_589413,0.0,0,64,512,4,2.4375,1,1.0,10328.373,12811.4426,9992.5255,30437,1,8842.8434,0.1191,0.0083,0.0233,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1067,0.0,0.0,1.0,0.0,0.011,0.1875,0.1191,15.2037,52.733,631.45,26.4,857.9676,226.5176,0.2101,5.3545,75.6881,431.7699,507.9602,861.5604,19.7888,432.0614,328.4525,0,14,60,480,385,25,93,0.016666666666666666,1.0,1.7333333333333334,104,,,0.008,12.6653,0.016666666666666666,1.0,,,,,0.016666666666666666,1.0,,,,,,,,,,,,,,14
+job_589413,0.0,0,64,512,5,2.5156,0,0.9219,9168.6328,10240.6912,9004.4437,29749,1,7654.3379,0.1328,0.0025,0.0467,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1149,-0.0,0.0,1.0,0.0,0.0082,0.25,0.1328,15.8657,52.7222,626.7126,27.0,857.9676,231.2549,0.0624,4.9571,67.7609,367.1356,435.2305,1080.6331,18.1439,367.4068,622.2968,0,15,62,496,351,40,135,0.2903225806451613,18.0,2.2096774193548385,137,,,,,,,0.016129032258064516,1.0,,,0.04838709677419355,3.0,,,6.0878,,,,,,,,,,,15
+job_589413,0.0,0,64,512,5,2.7812,0,0.9844,9860.5078,13834.0633,9135.5404,30958,1,9126.4354,0.1543,0.0025,0.0364,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.095,0.0,0.0,1.0,0.0,0.0082,0.2812,0.1543,15.8339,52.6867,622.6899,27.4,857.9676,235.2776,0.0997,5.2914,70.2154,425.0592,495.6344,1291.7286,18.9487,425.3188,771.8469,0,16,63,504,391,37,103,0.15873015873015872,10.0,1.6349206349206349,103,,,0.0086,12.4342,,,,,,,,,,,,,,,,,,,,,,16
+job_589413,0.0,0,64,512,4,2.6875,0,0.9844,10984.0449,13339.6324,10623.2793,30946,1,8860.174,0.1328,0.006,0.0495,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1103,-0.0,0.0,1.0,0.0,0.0075,0.1875,0.1328,15.8974,52.7485,619.0869,27.8,857.9676,238.8807,0.0701,5.3616,71.2226,437.1859,508.7899,1012.7019,18.2569,437.4967,480.289,0,17,62,496,365,28,111,0.3064516129032258,19.0,1.7096774193548387,106,,,,,0.016129032258064516,1.0,,,,,0.0967741935483871,6.0,,,,,,,,,,,,,,17
+job_589413,0.0,0,64,512,5,2.5156,0,0.9688,9343.375,11583.6966,8872.0071,30532,1,8242.8542,0.1738,0.0043,0.0404,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1106,0.0,0.0,1.0,0.0,0.012,0.2656,0.1738,15.8437,52.7366,616.8957,28.1,857.9676,241.0719,0.0968,5.1671,71.7476,398.261,470.3905,1019.3083,19.5815,398.5457,524.1618,0,18,63,504,335,40,158,0.15873015873015872,10.0,2.3968253968253967,151,,,0.0081,12.7041,0.015873015873015872,1.0,,,,,0.07936507936507936,5.0,0.031746031746031744,2.0,,,,,,,,,,,,18
+job_589413,0.0,0,64,512,4,2.6719,1,1.0,8471.2129,13121.2,7479.5095,31520,1,8710.7976,0.1758,0.0039,0.009,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0945,0.0,0.0,1.0,0.0,0.0032,0.2344,0.1758,16.4859,52.7443,633.4617,26.2,857.9676,224.5059,0.0872,5.2245,66.0855,407.5046,473.9292,1109.522,19.6532,407.7559,610.7109,0,19,65,520,405,31,104,0.15384615384615385,10.0,1.5692307692307692,102,,,,,0.015384615384615385,1.0,,,,,0.03076923076923077,2.0,,,,,,,,,,,,,,19
+job_589413,0.0,0,64,512,4,2.4688,0,0.9688,9397.5605,10769.1373,9245.8243,29721,1,8090.9631,0.0996,-0.0018,0.0196,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1076,0.0,0.0,1.0,0.0,0.0054,0.1406,0.0996,16.5093,52.7675,627.9171,26.8,857.9676,230.0505,0.0761,5.269,71.6405,396.6428,468.6066,1307.4116,19.3738,396.8895,814.1585,0,20,63,504,367,36,119,0.2222222222222222,14.0,1.6031746031746033,101,,,0.0077,12.3336,0.06349206349206349,4.0,0.06349206349206349,4.0,,,0.2222222222222222,14.0,,,6.113,,,,,,,,,,,20
+job_589413,0.0,0,64,512,4,2.3594,0,0.9531,8876.3809,11866.2886,7649.1185,31020,1,8144.0439,0.291,0.0083,0.0317,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1037,-0.002,0.0,1.0,0.0,0.0063,0.3594,0.291,16.8527,53.1653,624.288,27.2,857.9676,233.6796,0.0644,5.1935,65.7614,389.564,455.6589,1135.649,18.4184,389.8327,656.37,0,21,63,504,370,34,121,0.14285714285714285,9.0,1.873015873015873,118,,,,,0.031746031746031744,2.0,0.015873015873015872,1.0,,,0.09523809523809523,6.0,,,,0.031746031746031744,2.0,0.015873015873015872,1.0,,,,,,,21
+job_589413,0.0,0,64,512,4,2.4062,0,0.9844,9612.6992,12727.119,9001.4579,30518,1,8610.0533,0.1641,0.0001,0.0381,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1071,-0.0,0.0,1.0,0.0,0.007,0.25,0.1641,16.8425,53.2195,620.9487,27.6,857.9676,237.0189,0.0666,5.2176,71.3936,408.7832,480.5136,1077.8042,19.3161,409.053,572.7531,0,22,63,504,371,32,122,0.15873015873015872,10.0,1.8253968253968254,115,,,0.0083,14.3305,0.015873015873015872,1.0,0.047619047619047616,3.0,,,0.07936507936507936,5.0,,,,,,,,0.07936507936507936,5.0,,,,,22
+job_589413,0.0,0,64,512,4,2.3125,1,1.0,9644.4805,12499.4219,9236.6317,30247,1,8449.5651,0.125,0.004,0.0329,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0975,-0.0,0.0,1.0,0.0,0.0066,0.1719,0.125,16.898,53.3349,629.8778,26.6,857.9676,228.0898,0.0955,5.124,68.4055,407.5341,476.3226,1267.6702,18.3902,407.8206,767.8278,0,23,65,520,326,45,175,0.26153846153846155,17.0,2.5846153846153848,168,,,,,0.03076923076923077,2.0,0.015384615384615385,1.0,,,0.09230769230769231,6.0,,,,,,,,,,,,,,23
+job_589413,0.0,0,64,512,4,2.6094,0,0.9844,8941.7324,12333.4851,8108.236,30654,1,9059.652,0.1973,-0.0011,0.0307,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0891,0.0,0.0,1.0,0.0,0.0064,0.2812,0.1973,17.0198,53.3979,625.2485,27.1,857.9676,232.7191,0.1002,5.0625,64.3428,413.9089,478.6224,1422.179,18.4004,414.1787,920.0877,0,24,64,512,348,39,150,0.15625,10.0,2.25,144,,,0.0084,12.0063,0.0625,4.0,,,,,0.09375,6.0,,,,,,,,,,,,,,24
+job_589413,0.0,0,64,512,4,2.0781,0,0.9688,8478.0488,9041.8118,8365.8244,31197,1,8623.2484,0.166,0.0034,0.0454,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.099,0.0,0.0,1.0,0.0,0.0084,0.2344,0.166,17.0027,53.4105,621.9629,27.5,857.9676,236.0047,0.1051,5.0928,65.4935,394.6093,460.5277,1256.6936,18.4525,394.9286,772.6144,0,25,63,504,371,30,129,0.047619047619047616,3.0,1.8253968253968254,115,,,,,0.015873015873015872,1.0,,,,,0.2222222222222222,14.0,,,6.1077,,,,,,,,,,,25
+job_589413,0.0,0,64,512,4,2.1719,0,0.9844,9143.0312,10793.6909,8944.3742,30837,1,7933.9127,0.1074,-0.0014,0.0226,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1039,0.0,0.0,1.0,0.0,0.0049,0.1406,0.1074,16.9659,53.344,619.0737,27.8,857.9676,238.8939,0.0999,5.2223,65.8763,368.7657,435.0494,1056.9012,17.4192,369.0727,599.2038,0,26,64,512,349,36,156,0.078125,5.0,2.25,144,,,0.0083,12.3343,0.03125,2.0,,,,,0.1875,12.0,,,,,,,,,,,,,,26
+job_589413,0.0,0,64,512,4,2.2969,1,1.0,9318.0684,13478.1053,8370.3381,30076,1,8927.2675,0.1855,-0.0011,0.0127,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0987,0.0,0.0,1.0,0.0,0.0025,0.2344,0.1855,17.2687,53.7057,615.7143,28.2,857.9676,242.2533,0.0941,5.2193,75.3222,424.8138,500.4749,1506.0501,17.7312,425.0582,982.618,0,27,63,504,365,43,130,0.12698412698412698,8.0,1.7936507936507937,113,,,,,0.015873015873015872,1.0,,,,,0.2698412698412698,17.0,,,,,,,,,,,,,,27
+job_589413,0.0,0,64,512,4,2.3125,0,0.9844,9706.3535,12505.9143,9262.9842,31295,1,8703.7321,0.1367,0.0164,0.0632,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1061,-0.0,0.0,1.0,0.0,0.0111,0.2344,0.1367,17.2266,53.6643,613.2701,28.5,857.9676,244.6974,0.0669,5.773,65.2001,413.5376,479.086,1433.1909,17.2395,413.8186,931.0875,0,28,64,512,338,40,162,0.1875,12.0,2.265625,145,,,0.0092,12.5456,,,0.046875,3.0,,,0.3125,20.0,,,,,,,,,,,,,,28
+job_589413,0.0,0,64,512,4,2.3906,0,0.9844,9190.4043,14896.2414,8461.4648,31487,1,9135.4476,0.1133,0.002,0.0487,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0966,0.0,0.0,1.0,0.0,0.0068,0.1719,0.1133,17.3366,53.7334,611.0821,28.8,857.9676,246.8855,0.1016,5.197,71.0217,433.8967,505.3155,1360.4476,18.3518,434.1918,831.5773,0,29,60,480,381,30,88,0.16666666666666666,10.0,1.3666666666666667,82,,,,,0.016666666666666666,1.0,,,,,0.26666666666666666,16.0,,,,,,,,,,,,,,29
+job_589413,0.0,0,64,512,4,2.1562,0,0.9688,9495.8145,9631.0294,9475.1059,31385,1,7976.2685,0.1328,0.0033,0.0175,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1173,-0.0033,0.0,1.0,0.0,0.0061,0.1875,0.1328,17.3372,53.7932,608.0651,29.1,857.9676,249.9025,0.0913,5.3592,68.7826,391.6734,460.8187,1262.0092,18.3274,391.9444,777.4995,0,30,64,512,338,40,157,0.25,16.0,2.359375,151,,,0.0093,12.804,0.015625,1.0,0.015625,1.0,,,0.078125,5.0,,,5.7747,,,,,0.015625,1.0,,,,,30
+job_589413,0.0,0,64,512,4,2.4062,1,1.0,9377.334,13592.5495,8466.2067,31245,1,8932.7859,0.1777,0.0029,0.0383,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0993,-0.0,0.0,1.0,0.0,0.0061,0.25,0.1777,17.642,54.1039,605.7375,29.4,857.9676,252.23,0.0721,5.2867,72.9809,411.0495,484.4737,1259.668,18.722,411.4202,751.1815,0,31,63,504,377,36,116,0.14285714285714285,9.0,1.7301587301587302,109,,,,,0.031746031746031744,2.0,0.047619047619047616,3.0,,,0.07936507936507936,5.0,,,,,,,,0.015873015873015872,1.0,,,,,31
+job_589413,0.0,0,64,512,4,2.2656,0,0.9531,10975.1738,11508.35,10904.3982,31286,1,9253.1616,0.1172,0.0006,0.0289,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1081,-0.0,0.0,1.0,0.0,0.0089,0.1719,0.1172,17.6508,54.1074,603.5103,29.7,857.9676,254.4573,0.0782,5.6393,76.5317,453.5022,530.3774,1462.4104,18.7241,453.7671,907.6655,0,32,62,496,344,37,132,0.2903225806451613,18.0,1.9193548387096775,119,,,0.008,13.2254,0.03225806451612903,2.0,0.03225806451612903,2.0,,,0.25806451612903225,16.0,,,,,,,,,,,,,,32
+job_589413,0.0,0,64,512,4,2.2656,0,0.9844,10141.4238,13411.0159,9682.6615,31058,1,9259.9469,0.123,0.0048,0.0311,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0991,0.0,0.0,1.0,0.0,0.0056,0.1719,0.123,17.9439,54.3633,601.2052,29.9,857.9676,256.7624,0.0899,5.5213,77.6245,440.932,519.1285,1156.1656,18.025,441.4135,613.4848,0,33,64,512,375,36,128,0.0625,4.0,1.8125,116,,,,,0.0625,4.0,,,,,0.1875,12.0,,,,0.015625,1.0,,,,,,,,,33
+job_589413,0.0,0,64,512,4,2.25,0,0.9375,9173.3125,10323.3913,9059.7854,30789,1,8248.3461,0.0898,0.0034,0.0353,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1055,0.0,0.0,1.0,0.0,0.0064,0.1406,0.0898,18.9124,54.4227,597.6184,30.3,857.9676,260.3492,0.0665,5.2916,67.6079,383.5392,451.4687,1258.212,17.6079,383.7938,783.8386,0,34,29,232,161,19,68,,,2.0689655172413794,60,,,0.0073,12.0016,0.10344827586206896,3.0,,,,,0.27586206896551724,8.0,,,,,,,,,,,,,,34
+job_589414,0.0,0,64,512,4,0.8438,0,0.3125,10437.5781,12216.3191,9761.5606,30906,1,8425.2048,0.2754,0.0045,0.0183,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,-0.002,0.0,1.0,0.0,0.007,0.3636,0.2754,12.9912,50.7759,629.5773,26.6,857.9675,228.3903,0.2767,5.2124,74.8968,412.6478,488.0814,4779.1047,18.2716,412.907,4267.539,0,35,63,504,287,47,208,0.23809523809523808,15.0,3.0952380952380953,195,,,,,0.031746031746031744,2.0,,,,,0.20634920634920634,13.0,,,7.2362,,,,,,,,,,,35
+job_589414,0.0,0,64,512,1,1.0,1,1.0,7617.0176,12086.0,7217.6617,29796,1,8549.5628,0.082,0.0123,0.0609,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0835,-0.0015,0.0,1.0,0.0,0.0109,0.1719,0.082,13.0166,50.8255,628.3799,26.8,857.9675,229.5876,0.0642,4.757,62.0209,360.6818,423.056,863.6853,19.4267,360.9703,416.4341,0,36,62,496,334,34,128,0.532258064516129,33.0,1.967741935483871,122,,,14.7598,22.9295,0.016129032258064516,1.0,0.016129032258064516,1.0,,,0.1935483870967742,12.0,,,,,,,,,,,,,,36
+job_589414,0.0,0,64,512,2,1.6719,0,0.9531,8760.6465,10886.4028,8412.7955,30724,1,8278.5732,0.1406,0.0003,0.0457,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0928,-0.0,0.0,1.0,0.0,0.0056,0.1875,0.1406,13.2816,51.1987,628.3003,26.8,857.9675,229.6672,0.0686,4.9887,70.2967,382.4549,453.1113,1683.2463,18.1475,382.7453,1206.9952,0,37,63,504,340,38,150,0.19047619047619047,12.0,2.3174603174603177,146,,,,,0.031746031746031744,2.0,0.015873015873015872,1.0,,,0.031746031746031744,2.0,,,,,,,,,,,,,,37
+job_589414,0.0,0,64,512,3,2.125,1,1.0,9711.0859,13647.7812,8802.6178,30819,1,9164.2999,0.1875,0.0068,0.0556,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0973,-0.0,0.0,1.0,0.0,0.0079,0.2969,0.1875,13.7075,51.4554,624.343,27.2,857.9675,233.6246,0.0742,5.2921,70.8615,430.3457,501.5565,1089.0308,19.4588,430.6204,562.719,0,38,63,504,345,41,140,0.1746031746031746,11.0,2.142857142857143,135,,,0.0101,8.3872,0.1111111111111111,7.0,0.031746031746031744,2.0,,,0.047619047619047616,3.0,,,,0.015873015873015872,1.0,,,,,,,,,38
+job_589414,0.0,0,64,512,4,2.4688,0,0.9688,9830.0312,14998.9714,9450.7589,31096,1,9133.2781,0.0684,0.0078,0.0391,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1011,-0.002,0.0,1.0,0.0,0.0063,0.1094,0.0684,13.9773,51.7114,620.9083,27.6,857.9675,237.0592,0.0719,5.1953,74.5293,443.5724,518.4862,1413.4816,21.7271,443.8847,868.0682,0,39,64,512,339,43,168,0.046875,3.0,2.40625,154,,,,,0.015625,1.0,0.078125,5.0,,,0.140625,9.0,,,,0.015625,1.0,,,,,,,,,39
+job_589414,0.0,0,64,512,4,2.1562,0,0.9844,9801.043,16505.9767,9186.3049,30907,1,9197.5424,0.084,0.0054,0.0597,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0935,-0.0005,0.0,1.0,0.0,0.0058,0.1562,0.084,14.1064,51.8289,625.7814,27.1,857.9675,232.1861,0.0741,5.1608,69.67,427.6333,497.6731,1297.6163,20.2719,427.9286,774.5065,0,40,63,504,309,42,147,0.7301587301587301,46.0,2.2857142857142856,144,,,0.0096,8.4634,0.031746031746031744,2.0,,,,,0.047619047619047616,3.0,,,5.8701,,,,,,,,,,,40
+job_589414,0.0,0,64,512,4,2.125,0,0.9688,8527.4961,11893.8046,7838.3929,30476,1,9153.2018,0.1699,0.0071,0.0525,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0915,0.0,0.0,1.0,0.0,0.0051,0.2656,0.1699,14.1085,52.0214,621.533,27.6,857.9675,236.4345,0.0745,5.0385,69.091,420.8784,490.3688,1109.7026,18.6338,421.2029,595.6564,0,41,62,496,370,32,100,0.3870967741935484,24.0,1.6129032258064515,100,,,,,0.03225806451612903,2.0,,,,,,,,,,,,,,,,,,,,41
+job_589414,0.0,0,64,512,4,2.0781,0,0.9062,8602.8281,13130.1719,7956.0647,30220,1,8072.8774,0.125,0.0056,0.0104,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,-0.0,0.0,1.0,0.0,0.0056,0.1719,0.125,14.3199,52.3275,618.3289,27.9,857.9675,239.6387,0.0678,4.8823,60.7727,365.3284,427.1973,1310.0776,19.9813,366.3565,858.0116,0,42,64,512,335,42,177,,,2.765625,177,,,0.0091,13.1783,,,,,,,,,,,,,,,,,,,,,,42
+job_589414,0.0,0,64,512,4,2.4844,0,0.9844,9943.8965,14522.3788,9266.3632,31302,1,9568.7495,0.1289,0.0004,0.0148,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0922,0.0,0.0,1.0,0.0,0.0061,0.25,0.1289,14.4575,52.3593,615.0375,28.3,857.9675,242.9301,0.0709,5.3536,73.4756,439.6526,513.5252,1370.015,19.0401,439.9785,832.0917,0,43,62,496,309,39,175,0.12903225806451613,8.0,2.935483870967742,182,,,,,0.03225806451612903,2.0,0.016129032258064516,1.0,,,0.016129032258064516,1.0,,,,0.016129032258064516,1.0,,,,,0.016129032258064516,1.0,,,43
+job_589414,0.0,0,64,512,4,2.2188,0,0.9844,9086.3555,11450.5517,8784.3216,31262,1,9393.9818,0.1133,0.0007,0.0481,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0907,-0.0007,0.0,1.0,0.0,0.0072,0.2188,0.1133,15.2404,52.5447,611.1252,28.8,857.9675,246.8423,0.0753,5.1796,81.3851,430.771,512.6133,1326.4544,19.2724,431.1526,789.3843,0,44,64,512,328,41,170,0.1875,12.0,2.640625,169,,,0.0088,8.375,0.03125,2.0,0.015625,1.0,,,,,,,,,,,,,,,,,,44
+job_589414,0.0,0,64,512,4,2.0469,0,0.9844,9583.3691,13273.5522,9027.7685,30598,1,9337.8756,0.1309,0.0045,0.0705,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0902,-0.0036,0.0,1.0,0.0,0.0074,0.2031,0.1309,15.2057,52.4645,608.826,29.0,857.9675,249.1415,0.0722,6.294,66.9948,423.8895,491.2079,1420.4363,20.0032,424.1404,902.927,0,45,63,504,324,46,175,0.015873015873015872,1.0,2.7777777777777777,175,,,,,0.031746031746031744,2.0,,,,,,,,,5.9741,0.015873015873015872,1.0,,,,,0.015873015873015872,1.0,,,45
+job_589414,0.0,0,64,512,4,2.2969,0,0.9844,9691.5176,14166.0521,8658.9327,31244,1,9305.0755,0.1875,0.0014,0.0454,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0908,0.0,0.0,1.0,0.0,0.008,0.3281,0.1875,15.2065,52.5456,606.6011,29.3,857.9675,251.3664,0.0752,5.4224,76.2767,430.1979,506.8676,1193.1841,19.3454,430.5153,661.5444,0,46,64,512,345,42,158,0.140625,9.0,2.46875,158,,,0.0091,8.4932,,,,,,,,,,,,,,,,,,,,,,46
+job_589414,0.0,0,64,512,4,2.1094,0,0.9844,9212.3242,11344.0481,8668.9436,31190,1,9416.1534,0.2031,0.0114,0.052,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0936,-0.002,0.0,1.0,0.0,0.0081,0.3125,0.2031,14.9799,52.4084,622.5466,27.4,857.9675,235.421,0.0721,5.1673,74.6655,426.7696,501.7978,1222.5144,20.3347,427.0598,695.2104,0,47,65,520,365,39,150,,,2.3076923076923075,150,,,,,0.07692307692307693,5.0,,,,,,,,,,,,,,,,,,,,47
+job_589414,0.0,0,64,512,4,2.25,0,0.9688,9360.1367,11736.4615,8754.4069,31205,1,8692.4675,0.2031,0.0014,0.0459,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0967,0.0,0.0,1.0,0.0,0.006,0.2969,0.2031,15.0214,52.425,618.7219,27.9,857.9675,239.2457,0.0711,5.2233,75.0136,431.6495,507.052,1314.5149,19.6438,431.967,782.5914,0,48,64,512,367,36,128,0.25,16.0,2.0,128,,,0.0085,8.0854,0.015625,1.0,,,,,,,,,,,,,,,,,,,,48
+job_589414,0.0,0,64,512,4,2.25,0,0.9844,9074.0957,10469.2639,8845.7955,30308,1,8363.6284,0.1406,0.0047,0.0279,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1028,0.0,0.0,1.0,0.0,0.0051,0.1875,0.1406,15.0687,52.4651,615.8871,28.2,857.9675,242.0804,0.0895,5.114,66.3779,388.6285,455.3327,1176.5771,19.9722,388.8649,696.1539,0,49,63,504,329,39,153,0.2857142857142857,18.0,2.380952380952381,150,,,,,0.047619047619047616,3.0,0.031746031746031744,2.0,,,0.015873015873015872,1.0,,,,0.015873015873015872,1.0,,,,,,,,,49
+job_589414,0.0,0,64,512,4,2.1562,0,0.9688,8850.0449,11473.1616,8221.2591,31212,1,8601.1534,0.1934,0.0041,0.0483,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0926,-0.0,0.0,1.0,0.0,0.0075,0.2812,0.1934,15.1635,52.538,613.2601,28.5,857.9675,244.7074,0.0716,6.2678,65.1271,384.0105,449.5106,1156.8611,19.9659,384.3115,681.1123,0,50,63,504,368,39,119,0.2698412698412698,17.0,1.8571428571428572,117,,,0.0083,8.5298,,,0.031746031746031744,2.0,,,,,,,6.1327,,,,,,,,,,,50
+job_589414,0.0,0,64,512,4,2.375,0,0.9688,9589.3242,10869.4848,9399.8834,31288,1,8733.0224,0.1289,-0.0003,0.0192,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,0.0,0.0,1.0,0.0,0.0027,0.2031,0.1289,15.4259,52.851,610.0452,28.9,857.9675,247.9223,0.0715,5.3738,72.1756,418.9982,491.5295,1212.2518,20.0825,419.282,695.2615,0,51,61,488,328,38,137,0.3770491803278688,23.0,2.2295081967213113,136,,,,,,,0.01639344262295082,1.0,,,,,,,,,,,,,,,,,,51
+job_589414,0.0,0,64,512,4,2.2188,0,0.9844,8594.4082,12730.0615,7993.0268,30309,1,8273.7344,0.127,0.0042,0.0227,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0919,-0.0003,0.0,1.0,0.0,0.0049,0.1719,0.127,15.3691,52.7689,607.3633,29.2,857.9675,250.6042,0.0692,6.0694,65.242,372.8057,438.4157,1065.0034,19.6838,373.1041,600.8298,0,52,64,512,350,37,154,0.03125,2.0,2.359375,151,,,0.0082,9.0665,0.078125,5.0,0.046875,3.0,,,,,,,,0.015625,1.0,,,,,,,,,52
+job_589414,0.0,0,64,512,4,2.2969,0,0.9219,9746.8672,12776.9178,9243.0091,30786,1,8989.9087,0.1426,0.0061,0.044,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0999,-0.0034,0.0,1.0,0.0,0.0105,0.2344,0.1426,16.1131,53.1016,603.8304,29.6,857.9675,254.1371,0.074,5.2264,73.7603,426.7516,500.919,1352.3345,22.008,427.0843,824.1767,0,53,64,512,347,41,150,0.21875,14.0,2.3125,148,,,,,0.015625,1.0,0.03125,2.0,,,,,,,,,,,,,,,,,,53
+job_589414,0.0,0,64,512,4,2.375,0,0.9844,8793.3281,10799.5357,8231.59,30032,1,8724.8588,0.2188,0.0003,0.0569,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0953,-0.0039,0.0,1.0,0.0,0.0075,0.3281,0.2188,16.0381,53.0909,601.3336,29.9,857.9675,256.6339,0.0685,5.1466,65.5847,414.5861,480.548,1173.7722,19.8623,414.8945,668.2111,0,54,64,512,375,37,116,0.265625,17.0,1.78125,114,,,0.0092,9.4884,0.046875,3.0,0.015625,1.0,,,0.015625,1.0,,,,,,,,,,,,0.015625,1.0,54
+job_589414,0.0,0,64,512,4,2.3125,0,0.9531,10566.2969,15290.8932,9376.4841,31410,1,8879.9389,0.2012,0.0092,0.062,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1024,-0.0011,0.0,1.0,0.0,0.0116,0.3125,0.2012,16.1116,53.1949,598.9396,30.2,857.9675,259.028,0.073,5.354,71.9567,429.284,501.609,1261.7633,20.7867,429.5788,734.0093,0,55,63,504,323,40,161,0.14285714285714285,9.0,2.619047619047619,165,,,,,0.09523809523809523,6.0,,,,,0.015873015873015872,1.0,,,6.0853,0.015873015873015872,1.0,,,,,0.047619047619047616,3.0,0.015873015873015872,1.0,55
+job_589414,0.0,0,64,512,4,2.2188,1,1.0,9105.8008,16860.3696,8340.3283,30634,1,8929.209,0.0898,0.0184,0.0576,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0877,-0.0022,0.0,1.0,0.0,0.0098,0.2188,0.0898,16.1151,53.3091,597.5827,30.3,857.9675,260.3848,0.0717,5.1031,70.8664,394.2463,465.4289,1091.053,20.0046,394.4905,600.5119,0,56,60,480,364,31,107,0.15,9.0,1.7833333333333334,107,,,0.009,8.794,,,,,,,,,,,,,,,,,,,,,,56
+job_589414,0.0,0,64,512,4,2.2188,0,0.9688,9641.6895,12360.0233,9092.9178,31090,1,8609.0109,0.168,0.0033,0.045,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1088,-0.0,0.0,1.0,0.0,0.0066,0.25,0.168,16.5616,53.6893,595.4823,30.6,857.9675,262.4852,0.072,5.3676,73.5019,417.3243,491.1753,994.6671,20.0253,417.601,478.0945,0,57,60,480,367,32,94,0.26666666666666666,16.0,1.7,102,,,,,0.05,3.0,,,,,,,,,,,,,,,,,,,,57
+job_589414,0.0,0,64,512,4,2.4219,0,0.9844,10050.9961,12024.2162,9504.793,30541,1,8807.1723,0.2168,-0.0042,0.0241,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1046,0.0,0.0,1.0,0.0,0.005,0.2969,0.2168,16.7413,53.8259,593.603,30.8,857.9675,264.3646,0.0688,5.27,75.3362,434.0958,509.781,1039.3098,20.109,434.3756,504.1454,0,58,62,496,369,33,118,0.12903225806451613,8.0,1.8870967741935485,117,,,0.0088,8.1323,0.016129032258064516,1.0,,,,,0.016129032258064516,1.0,,,,,,,,,,,,,,58
+job_589414,0.0,0,64,512,4,2.5156,0,0.9688,10033.8789,11916.9434,9542.2414,30895,1,8675.2561,0.207,-0.0027,0.0237,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1062,0.0,0.0,1.0,0.0,0.004,0.2656,0.207,16.718,53.7883,591.6187,31.0,857.9675,266.3488,0.0732,5.4347,72.3408,424.4609,497.1824,1099.6533,20.2339,424.768,576.7981,0,59,64,512,356,40,143,0.1875,12.0,2.1875,140,,,,,0.03125,2.0,0.0625,4.0,,,0.015625,1.0,,,,0.015625,1.0,,,,,,,,,59
+job_589414,0.0,0,64,512,5,2.625,0,0.9688,10255.4453,11782.7547,9856.6897,31411,1,9352.3936,0.207,-0.0005,0.0143,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0999,0.0,0.0,1.0,0.0,0.003,0.2656,0.207,16.6629,53.7289,589.5977,31.3,857.9675,268.3698,0.0713,5.3855,76.6821,456.4509,533.52,1404.7918,19.2674,456.7662,846.6146,0,60,59,472,317,39,140,0.2542372881355932,15.0,2.542372881355932,150,,,0.009,8.6208,0.06779661016949153,4.0,0.01694915254237288,1.0,,,,,,,5.936,,,,,,,,,,,60
+job_589414,0.0,0,64,512,4,2.3281,0,0.9844,8909.3555,14929.6739,8315.0751,30786,1,8970.4052,0.0898,0.0005,0.0049,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0877,0.0,0.0,1.0,0.0,0.0025,0.1406,0.0898,17.0513,54.0333,587.6489,31.5,857.9675,270.3187,0.0684,5.2603,72.0759,407.8817,480.3137,1017.8425,19.3396,408.1689,512.924,0,61,64,512,367,31,119,0.40625,26.0,1.828125,117,,,,,,,0.03125,2.0,,,,,,,,,,,,,,,,,,61
+job_589414,0.0,0,64,512,4,2.3438,1,1.0,8969.4238,8863.163,8992.7,30797,1,8054.4728,0.1797,0.0005,0.003,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1029,-0.0,0.0,1.0,0.0,0.0027,0.1875,0.1797,17.4703,54.3962,585.8682,31.7,857.9675,272.0993,0.0708,5.3578,65.7888,362.4732,428.5896,1111.1134,18.7856,362.7297,658.3761,0,62,61,488,349,35,120,0.29508196721311475,18.0,1.9672131147540983,120,,,0.009,8.2242,0.01639344262295082,1.0,,,,,,,,,,,,,,,,,,,,62
+job_589414,0.0,0,64,512,4,2.3594,0,0.9531,9888.6504,13162.0123,9273.471,31070,1,9029.6223,0.1582,0.0057,0.0317,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1012,-0.0,0.0,1.0,0.0,0.0078,0.2188,0.1582,17.4786,54.3968,584.5446,31.9,857.9675,273.4229,0.0747,5.2953,79.8089,427.4805,507.6453,1382.5263,19.5701,427.7613,850.0112,0,63,58,464,331,35,120,0.1896551724137931,11.0,2.1724137931034484,126,,,,,0.034482758620689655,2.0,0.06896551724137931,4.0,,,0.017241379310344827,1.0,,,,,,,,,,,,,,63
+job_589414,0.0,0,64,512,4,2.4844,0,0.9688,9433.4355,12847.9314,8583.9756,31078,1,8889.5713,0.1992,0.0044,0.0373,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0999,-0.0012,0.0,1.0,0.0,0.0082,0.3125,0.1992,17.2842,54.336,583.1888,32.0,857.9675,274.7787,0.0706,5.3302,70.4561,431.1564,501.9916,1036.7697,21.6661,431.4646,507.7772,0,64,63,504,374,30,127,0.031746031746031744,2.0,2.0,126,,,0.0082,8.205,0.015873015873015872,1.0,0.015873015873015872,1.0,,,,,,,,,,,,,,,,,,64
+job_589414,0.0,0,64,512,4,2.3594,0,0.9844,10180.8359,9118.8947,10365.945,31446,1,9044.9768,0.1484,0.0006,0.0191,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1007,0.0,0.0,1.0,0.0,0.0047,0.2031,0.1484,17.3231,54.4015,621.9341,27.5,857.9675,236.0334,0.0728,5.5421,78.2487,435.2774,513.9476,1157.1932,19.0014,435.6258,618.6976,0,65,62,496,350,33,137,0.016129032258064516,1.0,2.2903225806451615,142,,,,,0.03225806451612903,2.0,,,,,0.016129032258064516,1.0,,,5.8354,0.06451612903225806,4.0,,,,,0.016129032258064516,1.0,0.03225806451612903,2.0,65
+job_589414,0.0,0,64,512,4,2.4688,1,1.0,9031.8203,11650.358,8539.7053,30031,1,8235.59,0.1582,0.0053,0.0558,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0975,0.0,0.0,1.0,0.0,0.0097,0.2344,0.1582,17.4368,54.4995,616.9651,28.1,857.9675,241.0024,0.0701,5.1933,66.5767,384.447,451.3545,1015.6966,19.7349,384.7073,539.4094,0,66,42,336,228,26,103,0.11904761904761904,5.0,2.4761904761904763,104,,,0.0085,8.349,,,,,,,,,,,,,,,,,,,,,,66
+job_589415,0.0,0,64,512,4,1.1406,0,0.4062,9320.8516,14550.6567,8533.4427,31363,1,9169.3835,0.1309,0.0044,0.0201,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0918,0.0,0.0,1.0,0.0,0.004,0.1852,0.1309,13.4686,50.5256,603.4965,29.7,857.9675,254.4711,0.1717,5.1623,84.3031,442.9513,527.6906,4644.6281,17.6466,443.2151,4094.1282,0,67,60,480,312,43,158,0.06666666666666667,4.0,2.9166666666666665,175,,,,,0.1,6.0,,,,,0.05,3.0,,,,,,,,,,,,,,67
+job_589415,0.0,0,64,512,1,0.9688,0,0.9688,8838.2246,14955.6667,7885.395,31069,1,9454.7696,0.1348,0.0049,0.0588,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,0.0,0.0,1.0,0.0,0.0076,0.2188,0.1348,13.6993,50.7828,600.2096,30.0,857.9675,257.7579,0.0646,5.1712,76.8797,447.6366,524.8183,1058.6616,18.5204,447.8732,510.1366,0,68,61,488,348,34,126,0.19672131147540983,12.0,2.0655737704918034,126,,,11.1932,19.3268,0.03278688524590164,2.0,,,,,,,,,,,,,,,,,,,,68
+job_589415,0.0,0,64,512,2,1.7344,1,1.0,8664.0488,11762.5714,8283.5285,29909,1,7605.2237,0.1094,0.0014,0.0566,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1024,-0.002,0.0,1.0,0.0,0.01,0.2031,0.1094,13.8658,50.9406,598.2016,30.3,857.9675,259.7659,0.0651,5.0053,63.8033,372.1322,436.2895,819.5224,18.7903,372.4205,359.4342,0,69,61,488,378,28,93,0.14754098360655737,9.0,1.5081967213114753,92,,,,,0.08196721311475409,5.0,,,,,0.03278688524590164,2.0,,,,0.01639344262295082,1.0,,,,,0.03278688524590164,2.0,,,69
+job_589415,0.0,0,64,512,3,2.2344,0,0.9531,10562.7539,12507.5258,10108.1928,30830,1,8215.603,0.1895,0.0043,0.0703,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1069,0.0,0.0,1.0,0.0,0.0113,0.2656,0.1895,14.0821,51.1283,623.5098,27.3,857.9675,234.4578,0.0712,5.2083,76.5816,414.9592,491.8453,1037.2437,17.6609,415.1921,522.5257,0,70,62,496,359,33,122,0.1935483870967742,12.0,1.935483870967742,120,,,0.0082,7.9993,0.04838709677419355,3.0,,,,,0.03225806451612903,2.0,,,6.2945,,,,,,,,,,,70
+job_589415,0.0,0,64,512,4,2.6406,0,0.9375,10671.9688,11820.5049,10382.7286,31046,1,9383.9069,0.2012,0.0104,0.0247,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1016,-0.002,0.0,1.0,0.0,0.0074,0.2656,0.2012,14.3759,51.4185,619.6084,27.8,857.9675,238.3591,0.0693,5.2885,83.9511,461.8063,546.119,1228.6645,18.3273,462.0983,658.925,0,71,60,480,328,36,136,0.2,12.0,2.216666666666667,133,,,,,0.08333333333333333,5.0,0.016666666666666666,1.0,,,0.03333333333333333,2.0,,,,0.016666666666666666,1.0,,,,,,,,,71
+job_589415,0.0,0,64,512,5,2.6719,0,0.9844,9609.7773,13170.275,8950.4259,30619,1,9001.5217,0.1562,0.0077,0.0335,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0938,-0.002,0.0,1.0,0.0,0.0054,0.2188,0.1562,14.5012,51.5967,616.5872,28.1,857.9675,241.3804,0.0684,5.2467,76.3358,416.1831,492.8603,890.1009,18.801,416.4557,373.1888,0,72,59,472,334,31,125,0.1694915254237288,10.0,2.169491525423729,128,,,0.0114,8.2721,0.05084745762711865,3.0,0.03389830508474576,2.0,,,,,,,,,,,,,,,,,,72
+job_589415,0.0,0,64,512,5,2.4375,0,0.9844,9353.7402,11149.3151,9055.1595,31227,1,8604.2319,0.1426,0.0035,0.0336,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1013,-0.002,0.0,1.0,0.0,0.0079,0.2031,0.1426,14.8654,51.7343,622.3488,27.5,857.9675,235.6187,0.104,5.238,69.902,401.1988,471.4352,970.1871,19.4643,401.4288,474.0452,0,73,61,488,338,35,139,0.13114754098360656,8.0,2.2459016393442623,137,,,,,0.01639344262295082,1.0,0.01639344262295082,1.0,,,0.01639344262295082,1.0,,,,0.01639344262295082,1.0,,,,,,,0.01639344262295082,1.0,73

training_logs/20260606_151436_reward_vs_steps.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f0077dc9e1928e6925a77725e14debbba717bc11bb24c8c474eb6711b8b0f9b
+size 329652

training_logs/20260606_151436_turn_count_distribution.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e636add6c8bfa10d6c1708e1efef34774e87edcd684ba499a785d1796696b487
+size 158806

training_logs/a3-rl-laion_exp_rpt_stack-bash-v3_589413.out

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f77a4f2151f6738a1edc6bc68084fa4ad5fcb0bc8f41e0c228e3d877a33c54c7
+size 96794246

training_logs/a3-rl-laion_exp_rpt_stack-bash-v3_589414.out

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b1899450c49e66b1f3880d3ba4cbdf7bb5ab38b3baa0505a2208fbd33d42710
+size 97478628

training_logs/a3-rl-laion_exp_rpt_stack-bash-v3_589415.out

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ff15c7bf3d2632c9cb7eb09fe1b21f2d9b50358f833d5328ce720e38ad20627
+size 24158119

upload.log

@@ -0,0 +1,699 @@
+Hint: A new version of huggingface_hub (1.18.0) is available! You are using version 1.14.0.
+To update, run: hf update
+Start hashing 26 files.
+Finished hashing 26 files.
+
+Processing Files (0 / 0)      : |          |  0.00B /  0.00B            
+
+New Data Upload               : |          |  0.00B /  0.00B            
+
+
+  ...0001-of-00004.safetensors:   4%|▍         |  184MB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:   4%|▎         |  184MB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  12%|█▏        |  192MB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:   4%|▍         |  192MB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...0001-of-00004.safetensors:   4%|▍         |  184MB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:   4%|▎         |  184MB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  12%|█▏        |  192MB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:   4%|▍         |  192MB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :   5%|▍         |  763MB / 16.4GB,   ???B/s  
+
+
+  ...0001-of-00004.safetensors:   8%|▊         |  383MB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:   8%|▊         |  384MB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  24%|██▍       |  384MB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:   8%|▊         |  384MB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :   9%|▉         | 1.55GB / 16.4GB,   ???B/s  
+
+
+  ...0001-of-00004.safetensors:  12%|█▏        |  573MB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  12%|█▏        |  576MB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  36%|███▋      |  576MB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  12%|█▏        |  576MB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  14%|█▍        | 2.31GB / 16.4GB,  222MB/s  
+
+
+  ...0001-of-00004.safetensors:  16%|█▌        |  763MB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  16%|█▌        |  768MB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  49%|████▊     |  768MB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  15%|█▌        |  768MB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  19%|█▉        | 3.08GB / 16.4GB,  291MB/s  
+
+
+  ...0001-of-00004.safetensors:  19%|█▉        |  953MB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  20%|█▉        |  960MB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  60%|█████▉    |  948MB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  19%|█▉        |  960MB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  23%|██▎       | 3.83GB / 16.4GB,  357MB/s  
+
+
+  ...0001-of-00004.safetensors:  23%|██▎       | 1.14GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  23%|██▎       | 1.15GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  72%|███████▏  | 1.14GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  23%|██▎       | 1.15GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  28%|██▊       | 4.60GB / 16.4GB,  421MB/s  
+
+
+  ...0001-of-00004.safetensors:  27%|██▋       | 1.32GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  27%|██▋       | 1.34GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  85%|████████▍ | 1.34GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  27%|██▋       | 1.34GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  33%|███▎      | 5.36GB / 16.4GB,  484MB/s  
+
+
+  ...0001-of-00004.safetensors:  31%|███       | 1.51GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  31%|███       | 1.54GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  96%|█████████▌| 1.52GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  31%|███       | 1.54GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  37%|███▋      | 6.11GB / 16.4GB,  544MB/s  
+
+
+  ...0001-of-00004.safetensors:  35%|███▍      | 1.71GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  35%|███▌      | 1.73GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  35%|███▍      | 1.73GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  41%|████      | 6.74GB / 16.4GB,  590MB/s  
+
+
+  ...0001-of-00004.safetensors:  39%|███▊      | 1.90GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  39%|███▉      | 1.92GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  39%|███▊      | 1.92GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  45%|████▍     | 7.32GB / 16.4GB,  630MB/s  
+
+
+  ...0001-of-00004.safetensors:  43%|████▎     | 2.10GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  43%|████▎     | 2.11GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  42%|████▏     | 2.11GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  48%|████▊     | 7.90GB / 16.4GB,  669MB/s  
+
+
+  ...0001-of-00004.safetensors:  47%|████▋     | 2.29GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  47%|████▋     | 2.31GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  46%|████▋     | 2.31GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  52%|█████▏    | 8.49GB / 16.4GB,  709MB/s  
+
+
+  ...0001-of-00004.safetensors:  51%|█████     | 2.48GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  51%|█████     | 2.50GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  50%|█████     | 2.50GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  55%|█████▌    | 9.07GB / 16.4GB,  745MB/s  
+
+
+  ...0001-of-00004.safetensors:  55%|█████▍    | 2.68GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  55%|█████▍    | 2.70GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  54%|█████▍    | 2.70GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  59%|█████▉    | 9.65GB / 16.4GB,  781MB/s  
+
+
+  ...0001-of-00004.safetensors:  59%|█████▊    | 2.87GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  59%|█████▉    | 2.90GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  58%|█████▊    | 2.90GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  63%|██████▎   | 10.2GB / 16.4GB,  817MB/s  
+
+
+  ...0001-of-00004.safetensors:  63%|██████▎   | 3.07GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  63%|██████▎   | 3.09GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  62%|██████▏   | 3.09GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  66%|██████▌   | 10.8GB / 16.4GB,  851MB/s  
+
+
+  ...0001-of-00004.safetensors:  67%|██████▋   | 3.27GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  67%|██████▋   | 3.29GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  66%|██████▌   | 3.29GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  70%|██████▉   | 11.4GB / 16.4GB,  885MB/s  
+
+
+  ...0001-of-00004.safetensors:  71%|███████   | 3.47GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  71%|███████   | 3.48GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  70%|██████▉   | 3.48GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  73%|███████▎  | 12.0GB / 16.4GB,  918MB/s  
+
+
+  ...0001-of-00004.safetensors:  75%|███████▍  | 3.66GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  75%|███████▍  | 3.68GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  74%|███████▍  | 3.68GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  77%|███████▋  | 12.6GB / 16.4GB,  950MB/s  
+
+
+  ...0001-of-00004.safetensors:  79%|███████▊  | 3.86GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  79%|███████▉  | 3.88GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  78%|███████▊  | 3.88GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  81%|████████  | 13.2GB / 16.4GB,  982MB/s  
+
+
+  ...0001-of-00004.safetensors:  83%|████████▎ | 4.05GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  83%|████████▎ | 4.07GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  82%|████████▏ | 4.07GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  84%|████████▍ | 13.8GB / 16.4GB, 1.01GB/s  
+
+
+  ...0001-of-00004.safetensors:  87%|████████▋ | 4.25GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  87%|████████▋ | 4.26GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  86%|████████▌ | 4.26GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  88%|████████▊ | 14.4GB / 16.4GB, 1.04GB/s  
+
+
+  ...0001-of-00004.safetensors:  91%|█████████ | 4.44GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  91%|█████████ | 4.46GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  90%|████████▉ | 4.46GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  91%|█████████ | 15.0GB / 16.4GB, 1.07GB/s  
+
+
+  ...0001-of-00004.safetensors:  95%|█████████▍| 4.64GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  95%|█████████▍| 4.66GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  93%|█████████▎| 4.66GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  95%|█████████▍| 15.5GB / 16.4GB, 1.10GB/s  
+
+
+  ...0001-of-00004.safetensors:  99%|█████████▊| 4.83GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors:  99%|█████████▉| 4.86GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+
+
+  ...0003-of-00004.safetensors:  97%|█████████▋| 4.86GB / 4.98GB            
+
+
+
+
+
+
+  ...h-v3-70-8B/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 5)      :  98%|█████████▊| 16.1GB / 16.4GB, 1.12GB/s  
+
+
+  ...0001-of-00004.safetensors: 100%|█████████▉| 4.88GB / 4.90GB            
+
+
+
+  ...0002-of-00004.safetensors: 100%|██████████| 4.92GB / 4.92GB            
+
+
+
+
+  ...0004-of-00004.safetensors:  99%|█████████▉| 1.57GB / 1.58GB            
+
+
+
+