laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0-tis_653583.out
ModelHub XC 9f22667c70 初始化项目，由ModelHub XC社区提供模型 
Model: laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B
Source: Original Platform
2026-08-09 05:27:18 +08:00

359 lines
52 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_653583
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_653583
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_653583
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.16.35 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653583.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653583.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/configs/ablation-pymethods2test-seqmean-arm0-tis_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0-tis ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-001-35 (10.128.16.35)
Ray port: 6379
============================
Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-001-35
Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-001-36
Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-001-37
Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-001-38
Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-001-41
Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-001-42
Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-001-43
Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-001-44
Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-001-45
Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-001-46
Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-001-47
Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-001-48
Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-003-37
Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-003-39
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
[Ray wait] Connection attempt failed:
[Ray wait] Retrying in 10s...
Ray connection established, polling for resources...
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
2026-06-07 14:37:25,227 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
[2026-06-07 14:37:30,234 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
[2026-06-07 14:38:00,235 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
[2026-06-07 14:38:06,240 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
[2026-06-07 14:38:36,240 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
[2026-06-07 14:38:42,244 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
[2026-06-07 14:39:12,244 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
[2026-06-07 14:39:18,247 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
[2026-06-07 14:39:46,160 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:47,183 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:48,184 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:49,184 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:50,185 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:51,186 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:52,186 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:53,187 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:54,187 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:55,188 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:56,189 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:57,189 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:58,190 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:39:59,190 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:00,191 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:01,192 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:02,192 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:03,193 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:04,193 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:05,194 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:06,195 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:07,195 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:08,196 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:09,196 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:10,197 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:11,198 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:12,198 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:13,199 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:14,199 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:15,200 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
2026-06-07 14:40:26,227 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
[2026-06-07 14:40:26,230 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:27,231 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:28,231 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:29,232 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:30,233 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
[2026-06-07 14:40:31,233 I 1579458 1579458] global_state_accessor.cc:487: This node has an IP address of 10.128.16.35, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
2026-06-07 14:40:31,235 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet) [2026-06-07 14:41:01,184 E 1580205 1580205] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=1580366) [2026-06-07 14:41:17,685 E 1580366 1581078] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=1580422) [2026-06-07 14:41:23,475 E 1580422 1582060] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
[2026-06-07 14:41:27,125 E 1579458 1580363] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=1580544) [2026-06-07 14:41:27,119 E 1580544 1596600] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster]
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
wait_for_cluster(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
raise TimeoutError(
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=8, gpus=32.0)
srun: error: jpbo-001-35: task 0: Exited with exit code 1
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
result = subprocess.run(
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1285, in run
training_exit_code = self._run_with_ray()
^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1532, in _run_with_ray
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
self.start()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
self._wait_for_cluster()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
raise RuntimeError(
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs, skipping upload.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ray_logs/
slurmstepd: error: *** JOB 653583 ON jpbo-001-35 CANCELLED AT 2026-06-07T15:15:50 ***
Reference in New Issue View Git Blame Copy Permalink