laion/ablation-pymethods2test-seqmean-arm0-30-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0_630128.out
ModelHub XC 0c293fc953 初始化项目，由ModelHub XC社区提供模型 
Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B
Source: Original Platform
2026-07-18 18:02:02 +08:00

371 lines
57 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_630128
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630128
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630128
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.32.33 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.32.33:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-041-33 (10.128.32.33)
Ray port: 6379
============================
Starting Ray head on jpbo-041-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-33.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-041-33
Starting Ray worker on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-34.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-041-34
Starting Ray worker on jpbo-041-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-35.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-041-35
Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-041-36
Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-041-37
Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-041-38
Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-041-39
Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-041-40
Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-041-41
Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-041-42
Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-041-43
Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-041-44
Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-041-45
Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-041-46
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.32.33:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
[Ray wait] nodes=3/14 GPUs=12.0/56.0 resources={'GPU': 12.0, 'CPU': 864.0, 'accelerator_type:GH200': 3.0, 'memory': 2273135755264.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 128849018880.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'CPU': 1152.0, 'accelerator_type:GH200': 4.0, 'memory': 3033402310656.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'memory': 3033402310656.0, 'accelerator_type:GH200': 4.0, 'CPU': 1152.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
[Ray wait] nodes=5/14 GPUs=20.0/56.0 resources={'GPU': 20.0, 'memory': 3809847214080.0, 'accelerator_type:GH200': 5.0, 'CPU': 1440.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 214748364800.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
[Ray wait] nodes=6/14 GPUs=24.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 4562133647360.0, 'accelerator_type:GH200': 6.0, 'CPU': 1728.0, 'GPU': 24.0, 'object_store_memory': 257698037760.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 6067020955648.0, 'accelerator_type:GH200': 8.0, 'CPU': 2304.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
2026-06-07 14:35:10,656 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.33:6379...
2026-06-07 14:35:10,666 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.43) [2026-06-07 14:35:49,687 E 2691736 2691736] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=352159) [2026-06-07 14:35:54,163 E 352159 352769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(pid=352184) [2026-06-07 14:36:00,280 E 352184 353748] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[2026-06-07 14:36:04,817 E 352114 352155] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.44) [2026-06-07 14:36:16,668 E 3005389 3005389] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=352328) [2026-06-07 14:36:04,815 E 352328 368299] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster]
(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.42) [2026-06-07 14:36:31,723 E 3108436 3108436] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.46) [2026-06-07 14:36:47,990 E 2656361 2656361] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
(raylet, ip=10.128.32.34) [2026-06-07 14:37:02,131 E 2733873 2733873] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.40) [2026-06-07 14:37:06,688 E 2571074 2571074] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.36) [2026-06-07 14:37:33,965 E 2625779 2625779] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
(raylet, ip=10.128.32.38) [2026-06-07 14:38:06,213 E 2699635 2699635] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.45) [2026-06-07 14:38:14,810 E 2825651 2825651] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
wait_for_cluster(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
raise TimeoutError(
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=10, gpus=40.0)
srun: error: jpbo-041-33: task 0: Exited with exit code 1
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
result = subprocess.run(
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1285, in run
training_exit_code = self._run_with_ray()
^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1532, in _run_with_ray
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
self.start()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
self._wait_for_cluster()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
raise RuntimeError(
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
episodes: last
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
Collecting Ray logs from worker jpbo-041-34...
Collecting Ray logs from worker jpbo-041-35...
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink