359 lines
52 KiB
Plaintext
359 lines
52 KiB
Plaintext
|
||
The following have been reloaded with a version change:
|
||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||
|
||
|
||
Lmod is automatically replacing "GCC/14.3.0" with
|
||
"nvidia-compilers/25.9-CUDA-13".
|
||
|
||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
[ray] RAY_TMPDIR=/tmp/ray/ray_653583
|
||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_653583
|
||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_653583
|
||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||
[proxy] Tunnel port: 7003
|
||
[proxy] Node IP: 10.128.16.35 (workers will connect here)
|
||
[proxy] ✓ SSH tunnel started successfully
|
||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653583.conf
|
||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||
[proxy] ✓ Daytona timeout settings configured
|
||
[proxy] Testing proxy connectivity...
|
||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653583.conf
|
||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
|
||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||
=== Universal RL Training Runner ===
|
||
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/configs/ablation-pymethods2test-seqmean-arm0-tis_rl_config.json
|
||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
Python version: Python 3.12.12
|
||
UV_USE_IO_URING: 0
|
||
Proxy: DISABLED (direct internet or not configured)
|
||
========================================
|
||
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0-tis ===
|
||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
|
||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
|
||
HF_TOKEN=****pDbg
|
||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||
Environment configured:
|
||
TENSOR_PARALLEL_SIZE=1
|
||
NUM_INFERENCE_ENGINES=56
|
||
POLICY_NUM_NODES=14
|
||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
|
||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||
Cleaning up existing Ray instances...
|
||
=== Starting Ray Cluster ===
|
||
Nodes: 14
|
||
GPUs per node: 4
|
||
CPUs per node: 288
|
||
Head node: jpbo-001-35 (10.128.16.35)
|
||
Ray port: 6379
|
||
============================
|
||
Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray head on jpbo-001-35
|
||
Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 1 on jpbo-001-36
|
||
Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 2 on jpbo-001-37
|
||
Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 3 on jpbo-001-38
|
||
Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 4 on jpbo-001-41
|
||
Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 5 on jpbo-001-42
|
||
Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 6 on jpbo-001-43
|
||
Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 7 on jpbo-001-44
|
||
Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 8 on jpbo-001-45
|
||
Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 9 on jpbo-001-46
|
||
Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 10 on jpbo-001-47
|
||
Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 11 on jpbo-001-48
|
||
Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 12 on jpbo-003-37
|
||
Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
|
||
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
||
Started Ray worker 13 on jpbo-003-39
|
||
Waiting for cluster (56 GPUs, 14 nodes)...
|
||
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
|
||
[Ray wait] Connection attempt failed:
|
||
[Ray wait] Retrying in 10s...
|
||
Ray connection established, polling for resources...
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.36': 1.0, 'GPU': 32.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'object_store_memory': 343597383680.0, 'CPU': 2304.0, 'memory': 5900030640128.0, 'GPU': 32.0, 'node:10.128.16.36': 1.0, 'accelerator_type:GH200': 8.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.41': 1.0}
|
||
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
|
||
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
|
||
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
|
||
2026-06-07 14:37:25,227 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
|
||
[2026-06-07 14:37:30,234 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||
[2026-06-07 14:38:00,235 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||
[2026-06-07 14:38:06,240 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||
[2026-06-07 14:38:36,240 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||
[2026-06-07 14:38:42,244 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||
[2026-06-07 14:39:12,244 W 1579458 1579458] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
|
||
[2026-06-07 14:39:18,247 W 1579458 1579458] rpc_client.h:153: Failed to connect to GCS at address 10.128.16.35:6379 within 5 seconds.
|
||
[2026-06-07 14:39:46,160 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:47,183 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:48,184 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:49,184 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:50,185 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:51,186 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:52,186 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:53,187 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:54,187 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:55,188 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:56,189 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:57,189 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:58,190 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:39:59,190 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:00,191 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:01,192 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:02,192 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:03,193 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:04,193 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:05,194 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:06,195 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:07,195 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:08,196 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:09,196 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:10,197 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:11,198 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:12,198 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:13,199 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:14,199 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:15,200 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
2026-06-07 14:40:26,227 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
|
||
[2026-06-07 14:40:26,230 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:27,231 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:28,231 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:29,232 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:30,233 W 1579458 1579458] global_state_accessor.cc:505: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?
|
||
[2026-06-07 14:40:31,233 I 1579458 1579458] global_state_accessor.cc:487: This node has an IP address of 10.128.16.35, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
|
||
2026-06-07 14:40:31,235 INFO worker.py:2007 -- Connected to Ray cluster.
|
||
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||
warnings.warn(
|
||
[33m(raylet)[0m [2026-06-07 14:41:01,184 E 1580205 1580205] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(pid=1580366)[0m [2026-06-07 14:41:17,685 E 1580366 1581078] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(pid=1580422)[0m [2026-06-07 14:41:23,475 E 1580422 1582060] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
||
[2026-06-07 14:41:27,125 E 1579458 1580363] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
||
[36m(pid=1580544)[0m [2026-06-07 14:41:27,119 E 1580544 1596600] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
||
Traceback (most recent call last):
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
||
main()
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
||
wait_for_cluster(
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
||
raise TimeoutError(
|
||
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=8, gpus=32.0)
|
||
srun: error: jpbo-001-35: task 0: Exited with exit code 1
|
||
|
||
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||
Traceback (most recent call last):
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
|
||
result = subprocess.run(
|
||
^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
||
raise CalledProcessError(retcode, process.args,
|
||
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
||
|
||
The above exception was the direct cause of the following exception:
|
||
|
||
Traceback (most recent call last):
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1285, in run
|
||
training_exit_code = self._run_with_ray()
|
||
^^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1532, in _run_with_ray
|
||
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
|
||
self.start()
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
|
||
self._wait_for_cluster()
|
||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
|
||
raise RuntimeError(
|
||
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
||
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
||
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
|
||
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs, skipping upload.
|
||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ray_logs/
|
||
slurmstepd: error: *** JOB 653583 ON jpbo-001-35 CANCELLED AT 2026-06-07T15:15:50 ***
|