249 lines
53 KiB
Plaintext
249 lines
53 KiB
Plaintext
|
|
|||
|
|
The following have been reloaded with a version change:
|
|||
|
|
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
|||
|
|
|
|||
|
|
|
|||
|
|
Lmod is automatically replacing "GCC/14.3.0" with
|
|||
|
|
"nvidia-compilers/25.9-CUDA-13".
|
|||
|
|
|
|||
|
|
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
|||
|
|
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
|||
|
|
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
|||
|
|
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
|||
|
|
[ray] RAY_TMPDIR=/tmp/ray/ray_567549
|
|||
|
|
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_567549
|
|||
|
|
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_567549
|
|||
|
|
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
|||
|
|
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
|||
|
|
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
|||
|
|
[proxy] Tunnel port: 7003
|
|||
|
|
[proxy] Node IP: 10.128.16.49 (workers will connect here)
|
|||
|
|
[proxy] ✓ SSH tunnel started successfully
|
|||
|
|
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_567549.conf
|
|||
|
|
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
|||
|
|
[proxy] - External traffic (internet) → PROXY via tunnel
|
|||
|
|
[proxy] ✓ Daytona timeout settings configured
|
|||
|
|
[proxy] Testing proxy connectivity...
|
|||
|
|
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_567549.conf
|
|||
|
|
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
|||
|
|
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
|||
|
|
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
|||
|
|
[proxy] ⚠ Tunnel not accessible at 10.128.16.49:7003 (workers may fail)
|
|||
|
|
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
|||
|
|
[container_runtime] Using cloud backend: daytona (no local container setup)
|
|||
|
|
=== Universal RL Training Runner ===
|
|||
|
|
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/configs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_rl_config.json
|
|||
|
|
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
|||
|
|
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
|||
|
|
Python version: Python 3.12.12
|
|||
|
|
UV_USE_IO_URING: 0
|
|||
|
|
Proxy: DISABLED (direct internet or not configured)
|
|||
|
|
========================================
|
|||
|
|
=== RLJobRunner: a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified ===
|
|||
|
|
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
|
|||
|
|
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
|
|||
|
|
HF_TOKEN=****pDbg
|
|||
|
|
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
|||
|
|
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
|||
|
|
Environment configured:
|
|||
|
|
TENSOR_PARALLEL_SIZE=1
|
|||
|
|
NUM_INFERENCE_ENGINES=56
|
|||
|
|
POLICY_NUM_NODES=14
|
|||
|
|
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/wandb
|
|||
|
|
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
|||
|
|
Cleaning up existing Ray instances...
|
|||
|
|
=== Starting Ray Cluster ===
|
|||
|
|
Nodes: 14
|
|||
|
|
GPUs per node: 4
|
|||
|
|
CPUs per node: 288
|
|||
|
|
Head node: jpbo-002-01 (10.128.16.49)
|
|||
|
|
Ray port: 6379
|
|||
|
|
============================
|
|||
|
|
Starting Ray head on jpbo-002-01 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-002-01.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.49 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.49 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray head on jpbo-002-01
|
|||
|
|
Starting Ray worker on jpbo-002-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-02.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 1 on jpbo-002-02
|
|||
|
|
Starting Ray worker on jpbo-002-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-03.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 2 on jpbo-002-03
|
|||
|
|
Starting Ray worker on jpbo-002-04 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-04.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.52 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.52 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 3 on jpbo-002-04
|
|||
|
|
Starting Ray worker on jpbo-002-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-05.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.53 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.53 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 4 on jpbo-002-05
|
|||
|
|
Starting Ray worker on jpbo-002-06 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-06.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.54 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.54 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 5 on jpbo-002-06
|
|||
|
|
Starting Ray worker on jpbo-002-07 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-07.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.55 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.55 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 6 on jpbo-002-07
|
|||
|
|
Starting Ray worker on jpbo-002-08 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-08.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.56 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-08 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.56 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 7 on jpbo-002-08
|
|||
|
|
Starting Ray worker on jpbo-002-09 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-09.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.57 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-09 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.57 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 8 on jpbo-002-09
|
|||
|
|
Starting Ray worker on jpbo-002-10 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-10.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.58 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-10 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.58 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 9 on jpbo-002-10
|
|||
|
|
Starting Ray worker on jpbo-002-11 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-11.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.59 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-11 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.59 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 10 on jpbo-002-11
|
|||
|
|
Starting Ray worker on jpbo-002-12 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-12.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.60 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.60 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 11 on jpbo-002-12
|
|||
|
|
Starting Ray worker on jpbo-002-13 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-13.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.61 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-13 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.61 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 12 on jpbo-002-13
|
|||
|
|
Starting Ray worker on jpbo-002-14 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-002-14.log)...
|
|||
|
|
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.62 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-002-14 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.49:6379 --node-ip-address=10.128.16.62 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
|||
|
|
Started Ray worker 13 on jpbo-002-14
|
|||
|
|
Waiting for cluster (56 GPUs, 14 nodes)...
|
|||
|
|
Connecting to Ray at 10.128.16.49:6379 (expecting 14 nodes, 56.0 GPUs)
|
|||
|
|
Ray connection established, polling for resources...
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 10114558590976.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'object_store_memory': 558345748480.0, 'node:10.128.16.53': 1.0, 'accelerator_type:GH200': 13.0, 'memory': 10114558590976.0, 'CPU': 3744.0, 'GPU': 52.0, 'node:10.128.16.50': 1.0, 'node:10.128.16.52': 1.0, 'node:10.128.16.60': 1.0, 'node:10.128.16.49': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.55': 1.0, 'node:10.128.16.56': 1.0, 'node:10.128.16.57': 1.0, 'node:10.128.16.51': 1.0, 'node:10.128.16.58': 1.0, 'node:10.128.16.61': 1.0, 'node:10.128.16.62': 1.0, 'node:10.128.16.54': 1.0}
|
|||
|
|
srun stderr: 2026-06-03 16:26:49,474 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.49:6379...
|
|||
|
|
2026-06-03 16:26:49,483 INFO worker.py:2007 -- Connected to Ray cluster.
|
|||
|
|
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
|||
|
|
warnings.warn(
|
|||
|
|
[33m(raylet, ip=10.128.16.62)[0m [2026-06-03 16:27:17,835 E 835178 835178] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
|||
|
|
[36m(pid=983379)[0m [2026-06-03 16:27:34,323 E 983379 984025] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
|||
|
|
[36m(pid=983396)[0m [2026-06-03 16:27:42,172 E 983396 985096] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
|||
|
|
[2026-06-03 16:27:45,463 E 983335 983376] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
|||
|
|
[36m(pid=983680)[0m [2026-06-03 16:27:45,417 E 983680 997125] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
|||
|
|
Traceback (most recent call last):
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
|||
|
|
main()
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
|||
|
|
wait_for_cluster(
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
|||
|
|
raise TimeoutError(
|
|||
|
|
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0)
|
|||
|
|
srun: error: jpbo-002-01: task 0: Exited with exit code 1
|
|||
|
|
|
|||
|
|
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
|||
|
|
Traceback (most recent call last):
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 599, in _wait_for_cluster
|
|||
|
|
result = subprocess.run(
|
|||
|
|
^^^^^^^^^^^^^^^
|
|||
|
|
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
|||
|
|
raise CalledProcessError(retcode, process.args,
|
|||
|
|
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
|||
|
|
|
|||
|
|
The above exception was the direct cause of the following exception:
|
|||
|
|
|
|||
|
|
Traceback (most recent call last):
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 930, in run
|
|||
|
|
training_exit_code = self._run_with_ray()
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1171, in _run_with_ray
|
|||
|
|
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 826, in __enter__
|
|||
|
|
self.start()
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 366, in start
|
|||
|
|
self._wait_for_cluster()
|
|||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 627, in _wait_for_cluster
|
|||
|
|
raise RuntimeError(
|
|||
|
|
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-002-01', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.49:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
|||
|
|
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-02...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-03...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-04...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-05...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-06...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-07...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-08...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-09...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-10...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-11...
|
|||
|
|
WARNING: Failed to collect logs from jpbo-002-11
|
|||
|
|
Collecting Ray logs from worker jpbo-002-12...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-13...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-14...
|
|||
|
|
[RLJobRunner] Crash-time Ray log preservation complete.
|
|||
|
|
[RLJobRunner] Launching trace upload (training exit code: 1):
|
|||
|
|
repo_id: DCAgent/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified
|
|||
|
|
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified
|
|||
|
|
episodes: last
|
|||
|
|
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/logs/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified_trace_upload.log
|
|||
|
|
[RLJobRunner] Waiting for trace upload to complete...
|
|||
|
|
[RLJobRunner] Trace upload failed with exit code 1.
|
|||
|
|
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_selfinstruct-naive-sandboxes-2-verified/ray_logs/
|
|||
|
|
Collecting Ray logs from worker jpbo-002-02...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-03...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-04...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-05...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-06...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-07...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-08...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-09...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-10...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-11...
|
|||
|
|
WARNING: Failed to collect logs from jpbo-002-11
|
|||
|
|
Collecting Ray logs from worker jpbo-002-12...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-13...
|
|||
|
|
Collecting Ray logs from worker jpbo-002-14...
|
|||
|
|
Ray log preservation complete
|