laion/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified-70-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
3e52217125c5998e2e7177659dc62fb58bef4acf
a3-rl-DCAgent_selfinstruct-…/training_logs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_567549.out
ModelHub XC 3e52217125 初始化项目，由ModelHub XC社区提供模型 
Model: laion/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified-70-8B
Source: Original Platform
2026-08-07 10:27:18 +08:00

249 lines
53 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_567549
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_567549
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_567549
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.16.49 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_567549.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_567549.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.16.49:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/configs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-002-01 (10.128.16.49)
Ray port: 6379
============================
Starting Ray head on jpbo-002-01 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-002-01.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.49 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.49 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-002-01
Starting Ray worker on jpbo-002-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-02.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-002-02
Starting Ray worker on jpbo-002-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-03.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-002-03
Starting Ray worker on jpbo-002-04 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-04.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.52 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.52 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-002-04
Starting Ray worker on jpbo-002-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-05.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.53 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.53 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-002-05
Starting Ray worker on jpbo-002-06 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-06.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.54 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.54 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-002-06
Starting Ray worker on jpbo-002-07 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-07.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.55 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.55 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-002-07
Starting Ray worker on jpbo-002-08 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-08.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.56 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-08 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.56 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-002-08
Starting Ray worker on jpbo-002-09 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-09.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.57 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-09 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.57 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-002-09
Starting Ray worker on jpbo-002-10 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-10.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.58 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-10 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.58 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-002-10
Starting Ray worker on jpbo-002-11 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-11.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.59 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-11 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.59 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-002-11
Starting Ray worker on jpbo-002-12 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-12.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.60 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.60 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-002-12
Starting Ray worker on jpbo-002-13 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-13.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.61 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-13 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.61 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-002-13
Starting Ray worker on jpbo-002-14 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-14.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.62 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-14 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.62 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-002-14
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.16.49:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
srun stderr: 2026-06-03 16:26:49,474 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.49:6379...
2026-06-03 16:26:49,483 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.16.62) [2026-06-03 16:27:17,835 E 835178 835178] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=983379) [2026-06-03 16:27:34,323 E 983379 984025] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=983396) [2026-06-03 16:27:42,172 E 983396 985096] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
[2026-06-03 16:27:45,463 E 983335 983376] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=983680) [2026-06-03 16:27:45,417 E 983680 997125] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster]
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
wait_for_cluster(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
raise TimeoutError(
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0)
srun: error: jpbo-002-01: task 0: Exited with exit code 1
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 599, in _wait_for_cluster
result = subprocess.run(
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 930, in run
training_exit_code = self._run_with_ray()
^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1171, in _run_with_ray
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 826, in __enter__
self.start()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 366, in start
self._wait_for_cluster()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 627, in _wait_for_cluster
raise RuntimeError(
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
Collecting Ray logs from worker jpbo-002-02...
Collecting Ray logs from worker jpbo-002-03...
Collecting Ray logs from worker jpbo-002-04...
Collecting Ray logs from worker jpbo-002-05...
Collecting Ray logs from worker jpbo-002-06...
Collecting Ray logs from worker jpbo-002-07...
Collecting Ray logs from worker jpbo-002-08...
Collecting Ray logs from worker jpbo-002-09...
Collecting Ray logs from worker jpbo-002-10...
Collecting Ray logs from worker jpbo-002-11...
WARNING: Failed to collect logs from jpbo-002-11
Collecting Ray logs from worker jpbo-002-12...
Collecting Ray logs from worker jpbo-002-13...
Collecting Ray logs from worker jpbo-002-14...
[RLJobRunner] Crash-time Ray log preservation complete.
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified
episodes: last
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/logs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/ray_logs/
Collecting Ray logs from worker jpbo-002-02...
Collecting Ray logs from worker jpbo-002-03...
Collecting Ray logs from worker jpbo-002-04...
Collecting Ray logs from worker jpbo-002-05...
Collecting Ray logs from worker jpbo-002-06...
Collecting Ray logs from worker jpbo-002-07...
Collecting Ray logs from worker jpbo-002-08...
Collecting Ray logs from worker jpbo-002-09...
Collecting Ray logs from worker jpbo-002-10...
Collecting Ray logs from worker jpbo-002-11...
WARNING: Failed to collect logs from jpbo-002-11
Collecting Ray logs from worker jpbo-002-12...
Collecting Ray logs from worker jpbo-002-13...
Collecting Ray logs from worker jpbo-002-14...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink