112 lines
6.2 KiB
Plaintext
112 lines
6.2 KiB
Plaintext
|
|
||
|
|
The following have been reloaded with a version change:
|
||
|
|
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||
|
|
|
||
|
|
|
||
|
|
Lmod is automatically replacing "GCC/14.3.0" with
|
||
|
|
"nvidia-compilers/25.9-CUDA-13".
|
||
|
|
|
||
|
|
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||
|
|
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||
|
|
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
|
|
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
|
|
[ray] RAY_TMPDIR=/tmp/ray/ray_485105
|
||
|
|
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_485105
|
||
|
|
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_485105
|
||
|
|
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||
|
|
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||
|
|
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||
|
|
[proxy] Tunnel port: 7003
|
||
|
|
[proxy] Node IP: 10.128.32.11 (workers will connect here)
|
||
|
|
[proxy] ✓ SSH tunnel started successfully
|
||
|
|
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_485105.conf
|
||
|
|
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||
|
|
[proxy] - External traffic (internet) → PROXY via tunnel
|
||
|
|
[proxy] ✓ Daytona timeout settings configured
|
||
|
|
[proxy] Testing proxy connectivity...
|
||
|
|
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_485105.conf
|
||
|
|
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||
|
|
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
|
|
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||
|
|
[proxy] ⚠ Tunnel not accessible at 10.128.32.11:7003 (workers may fail)
|
||
|
|
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||
|
|
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||
|
|
=== Universal RL Training Runner ===
|
||
|
|
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/configs/a3-rl-DCAgent_code-contests-noblock_rl_config.json
|
||
|
|
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||
|
|
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
|
|
Python version: Python 3.12.12
|
||
|
|
Proxy: DISABLED (direct internet or not configured)
|
||
|
|
========================================
|
||
|
|
=== RLJobRunner: a3-rl-DCAgent_code-contests-noblock ===
|
||
|
|
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||
|
|
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||
|
|
HF_TOKEN=****pDbg
|
||
|
|
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||
|
|
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||
|
|
Environment configured:
|
||
|
|
TENSOR_PARALLEL_SIZE=1
|
||
|
|
NUM_INFERENCE_ENGINES=56
|
||
|
|
POLICY_NUM_NODES=14
|
||
|
|
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||
|
|
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||
|
|
Cleaning up existing Ray instances...
|
||
|
|
=== Starting Ray Cluster ===
|
||
|
|
Nodes: 14
|
||
|
|
GPUs per node: 4
|
||
|
|
CPUs per node: 288
|
||
|
|
Head node: jpbo-041-11 (10.128.32.11)
|
||
|
|
Ray port: 6379
|
||
|
|
============================
|
||
|
|
RL job failed: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-041-11.log'
|
||
|
|
Traceback (most recent call last):
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 922, in run
|
||
|
|
training_exit_code = self._run_with_ray()
|
||
|
|
^^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1100, in _run_with_ray
|
||
|
|
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 818, in __enter__
|
||
|
|
self.start()
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 342, in start
|
||
|
|
self._start_node(self.node_list[0], is_head=True)
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 520, in _start_node
|
||
|
|
ray_log_file = open(ray_log_path, "w")
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
OSError: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-041-11.log'
|
||
|
|
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||
|
|
repo_id: DCAgent/a3-rl-DCAgent_code-contests-noblock
|
||
|
|
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock
|
||
|
|
episodes: last
|
||
|
|
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/logs/a3-rl-DCAgent_code-contests-noblock_trace_upload.log
|
||
|
|
[RLJobRunner] Waiting for trace upload to complete...
|
||
|
|
[RLJobRunner] Trace upload failed with exit code 1.
|
||
|
|
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/ray_logs/
|
||
|
|
Collecting Ray logs from worker jpbo-051-33...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-33
|
||
|
|
Collecting Ray logs from worker jpbo-051-34...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-34
|
||
|
|
Collecting Ray logs from worker jpbo-051-35...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-35
|
||
|
|
Collecting Ray logs from worker jpbo-051-36...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-36
|
||
|
|
Collecting Ray logs from worker jpbo-051-37...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-37
|
||
|
|
Collecting Ray logs from worker jpbo-051-38...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-38
|
||
|
|
Collecting Ray logs from worker jpbo-051-39...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-39
|
||
|
|
Collecting Ray logs from worker jpbo-051-40...
|
||
|
|
WARNING: Failed to collect logs from jpbo-051-40
|
||
|
|
Collecting Ray logs from worker jpbo-060-46...
|
||
|
|
WARNING: Failed to collect logs from jpbo-060-46
|
||
|
|
Collecting Ray logs from worker jpbo-066-09...
|
||
|
|
WARNING: Failed to collect logs from jpbo-066-09
|
||
|
|
Collecting Ray logs from worker jpbo-074-48...
|
||
|
|
WARNING: Failed to collect logs from jpbo-074-48
|
||
|
|
Collecting Ray logs from worker jpbo-089-02...
|
||
|
|
WARNING: Failed to collect logs from jpbo-089-02
|
||
|
|
Collecting Ray logs from worker jpbo-090-42...
|
||
|
|
WARNING: Failed to collect logs from jpbo-090-42
|
||
|
|
Ray log preservation complete
|