The following have been reloaded with a version change: 1) GCCcore/.14.3.0 => GCCcore/14.3.0 Lmod is automatically replacing "GCC/14.3.0" with "nvidia-compilers/25.9-CUDA-13". Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python [ray] RAY_TMPDIR=/tmp/ray/ray_661733 [triton_cache] Triton cache: /tmp/triton_cache_feuer1_661733 [triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_661733 [proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 [proxy] Setting up SSH tunnel to jpbl-s01-01 [proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc [proxy] Tunnel port: 7003 [proxy] Node IP: 10.128.16.35 (workers will connect here) [proxy] ✓ SSH tunnel started successfully [proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf [proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT [proxy] - External traffic (internet) → PROXY via tunnel [proxy] ✓ Daytona timeout settings configured [proxy] Testing proxy connectivity... [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary) [proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail) [proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers) [container_runtime] Using cloud backend: daytona (no local container setup) === Universal RL Training Runner === Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python version: Python 3.12.12 UV_USE_IO_URING: 0 Proxy: DISABLED (direct internet or not configured) ======================================== === RLJobRunner: ablation-pymethods2test-seqmean-arm0 === [wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb [wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb HF_TOKEN=****pDbg HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config) Environment configured: TENSOR_PARALLEL_SIZE=1 NUM_INFERENCE_ENGINES=56 POLICY_NUM_NODES=14 WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb Starting Ray cluster with 14 nodes, 4 GPUs/node Cleaning up existing Ray instances... === Starting Ray Cluster === Nodes: 14 GPUs per node: 4 CPUs per node: 288 Head node: jpbo-001-35 (10.128.16.35) Ray port: 6379 ============================ Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray head on jpbo-001-35 Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 1 on jpbo-001-36 Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 2 on jpbo-001-37 Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 3 on jpbo-001-38 Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 4 on jpbo-001-41 Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 5 on jpbo-001-42 Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 6 on jpbo-001-43 Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 7 on jpbo-001-44 Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 8 on jpbo-001-45 Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 9 on jpbo-001-46 Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 10 on jpbo-001-47 Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 11 on jpbo-001-48 Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 12 on jpbo-003-37 Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 13 on jpbo-003-39 Waiting for cluster (56 GPUs, 14 nodes)... Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs) Ray connection established, polling for resources... [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 40.0, 'memory': 7408839753728.0, 'accelerator_type:GH200': 10.0, 'object_store_memory': 429496729600.0, 'CPU': 2880.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=11/14 GPUs=44.0/56.0 resources={'GPU': 44.0, 'memory': 8175128150016.0, 'accelerator_type:GH200': 11.0, 'object_store_memory': 472446402560.0, 'node:10.128.16.135': 1.0, 'CPU': 3168.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0} [Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0} [Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0} srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29" srun: error: _find_node_record: lookup failure for node "jpbo-123-30" srun: error: _find_node_record: lookup failure for node "jpbo-123-31" srun: error: _find_node_record: lookup failure for node "jpbo-123-32" srun: error: _find_node_record: lookup failure for node "jpbo-123-33" srun: error: _find_node_record: lookup failure for node "jpbo-123-34" srun: error: _find_node_record: lookup failure for node "jpbo-123-35" srun: error: _find_node_record: lookup failure for node "jpbo-123-36" srun: error: _find_node_record: lookup failure for node "jpbo-123-37" srun: error: _find_node_record: lookup failure for node "jpbo-123-38" srun: error: _find_node_record: lookup failure for node "jpbo-123-39" srun: error: _find_node_record: lookup failure for node "jpbo-123-40" srun: error: _find_node_record: lookup failure for node "jpbo-123-41" srun: error: _find_node_record: lookup failure for node "jpbo-123-42" srun: error: _find_node_record: lookup failure for node "jpbo-123-43" srun: error: _find_node_record: lookup failure for node "jpbo-123-44" srun: error: _find_node_record: lookup failure for node "jpbo-123-45" srun: error: _find_node_record: lookup failure for node "jpbo-123-46" srun: error: _find_node_record: lookup failure for node "jpbo-123-47" srun: error: _find_node_record: lookup failure for node "jpbo-123-48" srun: error: _find_node_record: lookup failure for node "jpbo-124-01" srun: error: _find_node_record: lookup failure for node "jpbo-124-02" srun: error: _find_node_record: lookup failure for node "jpbo-124-03" srun: error: _find_node_record: lookup failure for node "jpbo-124-04" srun: error: _find_node_record: lookup failure for node "jpbo-124-05" srun: error: _find_node_record: lookup failure for node "jpbo-124-06" srun: error: _find_node_record: lookup failure for node "jpbo-124-07" srun: error: _find_node_record: lookup failure for node "jpbo-124-08" srun: error: _find_node_record: lookup failure for node "jpbo-124-09" srun: error: _find_node_record: lookup failure for node "jpbo-124-10" srun: error: _find_node_record: lookup failure for node "jpbo-124-11" srun: error: _find_node_record: lookup failure for node "jpbo-124-12" srun: error: _find_node_record: lookup failure for node "jpbo-124-13" srun: error: _find_node_record: lookup failure for node "jpbo-124-14" srun: error: _find_node_record: lookup failure for node "jpbo-124-15" srun: error: _find_node_record: lookup failure for node "jpbo-124-16" srun: error: _find_node_record: lookup failure for node "jpbo-124-17" srun: error: _find_node_record: lookup failure for node "jpbo-124-18" srun: error: _find_node_record: lookup failure for node "jpbo-124-19" srun: error: _find_node_record: lookup failure for node "jpbo-124-20" srun: error: _find_node_record: lookup failure for node "jpbo-124-21" srun: error: _find_node_record: lookup failure for node "jpbo-124-22" srun: error: _find_node_record: lookup failure for node "jpbo-124-23" srun: error: _find_node_record: lookup failure for node "jpbo-124-24" srun: error: _find_node_record: lookup failure for node "jpbo-124-25" srun: error: _find_node_record: lookup failure for node "jpbo-124-26" srun: error: _find_node_record: lookup failure for node "jpbo-124-27" srun: error: _find_node_record: lookup failure for node "jpbo-124-28" srun: error: _find_node_record: lookup failure for node "jpbo-124-29" srun: error: _find_node_record: lookup failure for node "jpbo-124-30" srun: error: _find_node_record: lookup failure for node "jpbo-124-31" srun: error: _find_node_record: lookup failure for node "jpbo-124-32" srun: error: _find_node_record: lookup failure for node "jpbo-124-33" srun: error: _find_node_record: lookup failure for node "jpbo-124-34" srun: error: _find_node_record: lookup failure for node "jpbo-124-35" srun: error: _find_node_record: lookup failure for node "jpbo-124-36" srun: error: _find_node_record: lookup failure for node "jpbo-124-37" srun: error: _find_node_record: lookup failure for node "jpbo-124-38" srun: error: _find_node_record: lookup failure for node "jpbo-124-39" srun: error: _find_node_record: lookup failure for node "jpbo-124-40" srun: error: _find_node_record: lookup failure for node "jpbo-124-41" srun: error: _find_node_record: lookup failure for node "jpbo-124-42" srun: error: _find_node_record: lookup failure for node "jpbo-124-43" srun: error: _find_node_record: lookup failure for node "jpbo-124-44" srun: error: _find_node_record: lookup failure for node "jpbo-124-45" srun: error: _find_node_record: lookup failure for node "jpbo-124-46" srun: error: _find_node_record: lookup failure for node "jpbo-124-47" srun: error: _find_node_record: lookup failure for node "jpbo-124-48" srun: error: _find_node_record: lookup failure for node "jpbo-125-01" srun: error: _find_node_record: lookup failure for node "jpbo-125-02" srun: error: _find_node_record: lookup failure for node "jpbo-125-03" srun: error: _find_node_record: lookup failure for node "jpbo-125-04" srun: error: _find_node_record: lookup failure for node "jpbo-125-05" srun: error: _find_node_record: lookup failure for node "jpbo-125-06" srun: error: _find_node_record: lookup failure for node "jpbo-125-07" srun: error: _find_node_record: lookup failure for node "jpbo-125-08" srun: error: _find_node_record: lookup failure for node "jpbo-125-09" srun: error: _find_node_record: lookup failure for node "jpbo-125-10" srun: error: _find_node_record: lookup failure for node "jpbo-125-11" srun: error: _find_node_record: lookup failure for node "jpbo-125-12" srun: error: _find_node_record: lookup failure for node "jpbo-125-13" srun: error: _find_node_record: lookup failure for node "jpbo-125-14" srun: error: _find_node_record: lookup failure for node "jpbo-125-15" srun: error: _find_node_record: lookup failure for node "jpbo-125-16" srun: error: _find_node_record: lookup failure for node "jpbo-125-17" srun: error: _find_node_record: lookup failure for node "jpbo-125-18" srun: error: _find_node_record: lookup failure for node "jpbo-125-19" srun: error: _find_node_record: lookup failure for node "jpbo-125-20" srun: error: _find_node_record: lookup failure for node "jpbo-125-21" srun: error: _find_node_record: lookup failure for node "jpbo-125-22" srun: error: _find_node_record: lookup failure for node "jpbo-125-23" srun: error: _find_node_record: lookup failure for node "jpbo-125-24" srun: error: _find_node_record: lookup failure for node "jpbo-125-25" srun: error: _find_node_record: lookup failure for node "jpbo-125-26" srun: error: _find_node_record: lookup failure for node "jpbo-125-27" srun: error: _find_node_record: lookup failure for node "jpbo-125-28" srun: error: _find_node_record: lookup failure for node "jpbo-125-29" srun: error: _find_node_record: lookup failure for node "jpbo-125-30" srun: error: _find_node_record: lookup failure for node "jpbo-125-31" srun: error: _find_node_record: lookup failure for node "jpbo-125-32" srun: error: _find_node_record: lookup failure for node "jpbo-125-33" srun: error: _find_node_record: lookup failure for node "jpbo-125-34" srun: error: _find_node_record: lookup failure for node "jpbo-125-35" srun: error: _find_node_record: lookup failure for node "jpbo-125-36" srun: error: _find_node_record: lookup failure for node "jpbo-125-37" srun: error: _find_node_record: lookup failure for node "jpbo-125-38" srun: error: _find_node_record: lookup failure for node "jpbo-125-39" srun: error: _find_node_record: lookup failure for node "jpbo-125-40" srun: error: _find_node_record: lookup failure for node "jpbo-125-41" srun: error: _find_node_record: lookup failure for node "jpbo-125-42" srun: error: _find_node_record: lookup failure for node "jpbo-125-43" srun: error: _find_node_record: lookup failure for node "jpbo-125-44" srun: error: _find_node_record: lookup failure for node "jpbo-125-45" srun: error: _find_node_record: lookup failure for node "jpbo-125-46" srun: error: _find_node_record: lookup failure for node "jpbo-125-47" srun: error: _find_node_record: lookup failure for node "jpbo-125-48" srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48] 2026-06-08 13:34:49,641 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379... 2026-06-08 13:34:49,647 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team. 2026-06-08 13:34:49,649 INFO worker.py:2007 -- Connected to Ray cluster. /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( (pid=1699456) [2026-06-08 13:35:34,739 E 1699456 1700253] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (pid=1699477) [2026-06-08 13:35:42,369 E 1699477 1701156] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) [2026-06-08 13:35:45,201 E 1699412 1699453] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.16.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (pid=1699674) [2026-06-08 13:35:45,144 E 1699674 1715593] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster] (raylet, ip=10.128.16.133) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster] (raylet, ip=10.128.16.47) [2026-06-08 13:37:33,645 E 1598284 1598284] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.16.135) [2026-06-08 13:37:44,343 E 3199851 3199851] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.16.133) [2026-06-08 13:37:47,415 E 2150584 2150584] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (raylet, ip=10.128.16.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (raylet, ip=10.128.16.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e Traceback (most recent call last): File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in main() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main wait_for_cluster( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster raise TimeoutError( TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0) srun: error: jpbo-001-35: task 0: Exited with exit code 1 RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.) Traceback (most recent call last): File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster result = subprocess.run( ^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1291, in run training_exit_code = self._run_with_ray() ^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1538, in _run_with_ray with RayCluster.from_slurm(ray_cfg) as ray_cluster: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__ self.start() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start self._wait_for_cluster() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster raise RuntimeError( RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.) [RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)... Collecting Ray logs from worker jpbo-001-36... Collecting Ray logs from worker jpbo-001-37... Collecting Ray logs from worker jpbo-001-38... Collecting Ray logs from worker jpbo-001-41... Collecting Ray logs from worker jpbo-001-42... Collecting Ray logs from worker jpbo-001-43... Collecting Ray logs from worker jpbo-001-44... Collecting Ray logs from worker jpbo-001-45... Collecting Ray logs from worker jpbo-001-46... Collecting Ray logs from worker jpbo-001-47... Collecting Ray logs from worker jpbo-001-48... Collecting Ray logs from worker jpbo-003-37... Collecting Ray logs from worker jpbo-003-39... [RLJobRunner] Crash-time Ray log preservation complete. [RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/trace_jobs, skipping upload. Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ray_logs/ Collecting Ray logs from worker jpbo-001-36... Collecting Ray logs from worker jpbo-001-37... Collecting Ray logs from worker jpbo-001-38... Collecting Ray logs from worker jpbo-001-41... Collecting Ray logs from worker jpbo-001-42... Collecting Ray logs from worker jpbo-001-43... Collecting Ray logs from worker jpbo-001-44... Collecting Ray logs from worker jpbo-001-45... Collecting Ray logs from worker jpbo-001-46... Collecting Ray logs from worker jpbo-001-47... Collecting Ray logs from worker jpbo-001-48... Collecting Ray logs from worker jpbo-003-37... Collecting Ray logs from worker jpbo-003-39... Ray log preservation complete