laion/ablation-pymethods2test-seqmean-arm0-30-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
0c293fc953b2b15bcd1bacb72bb3526c7c06369e
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0_661733.out
ModelHub XC 0c293fc953 初始化项目，由ModelHub XC社区提供模型 
Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B
Source: Original Platform
2026-07-18 18:02:02 +08:00

365 lines
58 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_661733
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_661733
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_661733
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.16.35 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-001-35 (10.128.16.35)
Ray port: 6379
============================
Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-001-35
Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-001-36
Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-001-37
Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-001-38
Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-001-41
Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-001-42
Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-001-43
Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-001-44
Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-001-45
Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-001-46
Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-001-47
Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-001-48
Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-003-37
Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-003-39
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 40.0, 'memory': 7408839753728.0, 'accelerator_type:GH200': 10.0, 'object_store_memory': 429496729600.0, 'CPU': 2880.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=11/14 GPUs=44.0/56.0 resources={'GPU': 44.0, 'memory': 8175128150016.0, 'accelerator_type:GH200': 11.0, 'object_store_memory': 472446402560.0, 'node:10.128.16.135': 1.0, 'CPU': 3168.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
2026-06-08 13:34:49,641 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
2026-06-08 13:34:49,647 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
2026-06-08 13:34:49,649 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(pid=1699456) [2026-06-08 13:35:34,739 E 1699456 1700253] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=1699477) [2026-06-08 13:35:42,369 E 1699477 1701156] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
[2026-06-08 13:35:45,201 E 1699412 1699453] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(pid=1699674) [2026-06-08 13:35:45,144 E 1699674 1715593] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster]
(raylet, ip=10.128.16.133) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster]
(raylet, ip=10.128.16.47) [2026-06-08 13:37:33,645 E 1598284 1598284] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.135) [2026-06-08 13:37:44,343 E 3199851 3199851] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.133) [2026-06-08 13:37:47,415 E 2150584 2150584] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.16.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
wait_for_cluster(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
raise TimeoutError(
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0)
srun: error: jpbo-001-35: task 0: Exited with exit code 1
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
result = subprocess.run(
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1291, in run
training_exit_code = self._run_with_ray()
^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1538, in _run_with_ray
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
self.start()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
self._wait_for_cluster()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
raise RuntimeError(
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
Collecting Ray logs from worker jpbo-001-36...
Collecting Ray logs from worker jpbo-001-37...
Collecting Ray logs from worker jpbo-001-38...
Collecting Ray logs from worker jpbo-001-41...
Collecting Ray logs from worker jpbo-001-42...
Collecting Ray logs from worker jpbo-001-43...
Collecting Ray logs from worker jpbo-001-44...
Collecting Ray logs from worker jpbo-001-45...
Collecting Ray logs from worker jpbo-001-46...
Collecting Ray logs from worker jpbo-001-47...
Collecting Ray logs from worker jpbo-001-48...
Collecting Ray logs from worker jpbo-003-37...
Collecting Ray logs from worker jpbo-003-39...
[RLJobRunner] Crash-time Ray log preservation complete.
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/trace_jobs, skipping upload.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ray_logs/
Collecting Ray logs from worker jpbo-001-36...
Collecting Ray logs from worker jpbo-001-37...
Collecting Ray logs from worker jpbo-001-38...
Collecting Ray logs from worker jpbo-001-41...
Collecting Ray logs from worker jpbo-001-42...
Collecting Ray logs from worker jpbo-001-43...
Collecting Ray logs from worker jpbo-001-44...
Collecting Ray logs from worker jpbo-001-45...
Collecting Ray logs from worker jpbo-001-46...
Collecting Ray logs from worker jpbo-001-47...
Collecting Ray logs from worker jpbo-001-48...
Collecting Ray logs from worker jpbo-003-37...
Collecting Ray logs from worker jpbo-003-39...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink