112 lines
6.2 KiB
Plaintext
112 lines
6.2 KiB
Plaintext
|
|
||
|
|
The following have been reloaded with a version change:
|
||
|
|
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||
|
|
|
||
|
|
|
||
|
|
Lmod is automatically replacing "GCC/14.3.0" with
|
||
|
|
"nvidia-compilers/25.9-CUDA-13".
|
||
|
|
|
||
|
|
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||
|
|
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||
|
|
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
|
|
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
|
|
[ray] RAY_TMPDIR=/tmp/ray/ray_496036
|
||
|
|
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_496036
|
||
|
|
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_496036
|
||
|
|
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||
|
|
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||
|
|
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||
|
|
[proxy] Tunnel port: 7003
|
||
|
|
[proxy] Node IP: 10.128.17.210 (workers will connect here)
|
||
|
|
[proxy] ✓ SSH tunnel started successfully
|
||
|
|
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496036.conf
|
||
|
|
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||
|
|
[proxy] - External traffic (internet) → PROXY via tunnel
|
||
|
|
[proxy] ✓ Daytona timeout settings configured
|
||
|
|
[proxy] Testing proxy connectivity...
|
||
|
|
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496036.conf
|
||
|
|
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||
|
|
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||
|
|
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||
|
|
[proxy] ⚠ Tunnel not accessible at 10.128.17.210:7003 (workers may fail)
|
||
|
|
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||
|
|
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||
|
|
=== Universal RL Training Runner ===
|
||
|
|
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/configs/a3-rl-DCAgent_code-contests-noblock_rl_config.json
|
||
|
|
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||
|
|
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||
|
|
Python version: Python 3.12.12
|
||
|
|
Proxy: DISABLED (direct internet or not configured)
|
||
|
|
========================================
|
||
|
|
=== RLJobRunner: a3-rl-DCAgent_code-contests-noblock ===
|
||
|
|
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||
|
|
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||
|
|
HF_TOKEN=****pDbg
|
||
|
|
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||
|
|
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||
|
|
Environment configured:
|
||
|
|
TENSOR_PARALLEL_SIZE=1
|
||
|
|
NUM_INFERENCE_ENGINES=56
|
||
|
|
POLICY_NUM_NODES=14
|
||
|
|
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||
|
|
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||
|
|
Cleaning up existing Ray instances...
|
||
|
|
=== Starting Ray Cluster ===
|
||
|
|
Nodes: 14
|
||
|
|
GPUs per node: 4
|
||
|
|
CPUs per node: 288
|
||
|
|
Head node: jpbo-010-34 (10.128.17.210)
|
||
|
|
Ray port: 6379
|
||
|
|
============================
|
||
|
|
RL job failed: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-010-34.log'
|
||
|
|
Traceback (most recent call last):
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 922, in run
|
||
|
|
training_exit_code = self._run_with_ray()
|
||
|
|
^^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1100, in _run_with_ray
|
||
|
|
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 818, in __enter__
|
||
|
|
self.start()
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 342, in start
|
||
|
|
self._start_node(self.node_list[0], is_head=True)
|
||
|
|
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 520, in _start_node
|
||
|
|
ray_log_file = open(ray_log_path, "w")
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
OSError: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-010-34.log'
|
||
|
|
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||
|
|
repo_id: DCAgent/a3-rl-DCAgent_code-contests-noblock
|
||
|
|
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock
|
||
|
|
episodes: last
|
||
|
|
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/logs/a3-rl-DCAgent_code-contests-noblock_trace_upload.log
|
||
|
|
[RLJobRunner] Waiting for trace upload to complete...
|
||
|
|
[RLJobRunner] Trace upload failed with exit code 1.
|
||
|
|
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/ray_logs/
|
||
|
|
Collecting Ray logs from worker jpbo-010-36...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-36
|
||
|
|
Collecting Ray logs from worker jpbo-010-38...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-38
|
||
|
|
Collecting Ray logs from worker jpbo-010-39...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-39
|
||
|
|
Collecting Ray logs from worker jpbo-010-40...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-40
|
||
|
|
Collecting Ray logs from worker jpbo-010-41...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-41
|
||
|
|
Collecting Ray logs from worker jpbo-010-43...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-43
|
||
|
|
Collecting Ray logs from worker jpbo-010-44...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-44
|
||
|
|
Collecting Ray logs from worker jpbo-010-45...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-45
|
||
|
|
Collecting Ray logs from worker jpbo-010-46...
|
||
|
|
WARNING: Failed to collect logs from jpbo-010-46
|
||
|
|
Collecting Ray logs from worker jpbo-022-02...
|
||
|
|
WARNING: Failed to collect logs from jpbo-022-02
|
||
|
|
Collecting Ray logs from worker jpbo-022-03...
|
||
|
|
WARNING: Failed to collect logs from jpbo-022-03
|
||
|
|
Collecting Ray logs from worker jpbo-022-04...
|
||
|
|
WARNING: Failed to collect logs from jpbo-022-04
|
||
|
|
Collecting Ray logs from worker jpbo-022-05...
|
||
|
|
WARNING: Failed to collect logs from jpbo-022-05
|
||
|
|
Ray log preservation complete
|