初始化项目，由ModelHub XC社区提供模型
Model: laion/a3-rl-DCAgent_code-contests-noblock-5-8B Source: Original Platform
This commit is contained in:
ModelHub XC
111
training_logs/a3-rl-DCAgent_code-contests-noblock_496037.out
Normal file
111
training_logs/a3-rl-DCAgent_code-contests-noblock_496037.out
Normal file
@@ -0,0 +1,111 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_496037
|
||||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_496037
|
||||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_496037
|
||||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||||
[proxy] Tunnel port: 7003
|
||||
[proxy] Node IP: 10.128.49.63 (workers will connect here)
|
||||
[proxy] ✓ SSH tunnel started successfully
|
||||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496037.conf
|
||||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||||
[proxy] ✓ Daytona timeout settings configured
|
||||
[proxy] Testing proxy connectivity...
|
||||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496037.conf
|
||||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||||
[proxy] ⚠ Tunnel not accessible at 10.128.49.63:7003 (workers may fail)
|
||||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||||
=== Universal RL Training Runner ===
|
||||
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/configs/a3-rl-DCAgent_code-contests-noblock_rl_config.json
|
||||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python version: Python 3.12.12
|
||||
Proxy: DISABLED (direct internet or not configured)
|
||||
========================================
|
||||
=== RLJobRunner: a3-rl-DCAgent_code-contests-noblock ===
|
||||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||||
HF_TOKEN=****pDbg
|
||||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||||
Environment configured:
|
||||
TENSOR_PARALLEL_SIZE=1
|
||||
NUM_INFERENCE_ENGINES=56
|
||||
POLICY_NUM_NODES=14
|
||||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||||
Cleaning up existing Ray instances...
|
||||
=== Starting Ray Cluster ===
|
||||
Nodes: 14
|
||||
GPUs per node: 4
|
||||
CPUs per node: 288
|
||||
Head node: jpbo-087-31 (10.128.49.63)
|
||||
Ray port: 6379
|
||||
============================
|
||||
RL job failed: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-087-31.log'
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 922, in run
|
||||
training_exit_code = self._run_with_ray()
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1100, in _run_with_ray
|
||||
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 818, in __enter__
|
||||
self.start()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 342, in start
|
||||
self._start_node(self.node_list[0], is_head=True)
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 520, in _start_node
|
||||
ray_log_file = open(ray_log_path, "w")
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
OSError: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-087-31.log'
|
||||
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||||
repo_id: DCAgent/a3-rl-DCAgent_code-contests-noblock
|
||||
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock
|
||||
episodes: last
|
||||
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/logs/a3-rl-DCAgent_code-contests-noblock_trace_upload.log
|
||||
[RLJobRunner] Waiting for trace upload to complete...
|
||||
slurmstepd: error: *** JOB 496037 ON jpbo-087-31 CANCELLED AT 2026-05-23T14:15:08 ***
|
||||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/ray_logs/
|
||||
Collecting Ray logs from worker jpbo-087-32...
|
||||
WARNING: Failed to collect logs from jpbo-087-32
|
||||
Collecting Ray logs from worker jpbo-090-02...
|
||||
WARNING: Failed to collect logs from jpbo-090-02
|
||||
Collecting Ray logs from worker jpbo-090-03...
|
||||
WARNING: Failed to collect logs from jpbo-090-03
|
||||
Collecting Ray logs from worker jpbo-090-05...
|
||||
WARNING: Failed to collect logs from jpbo-090-05
|
||||
Collecting Ray logs from worker jpbo-090-06...
|
||||
WARNING: Failed to collect logs from jpbo-090-06
|
||||
Collecting Ray logs from worker jpbo-090-08...
|
||||
WARNING: Failed to collect logs from jpbo-090-08
|
||||
Collecting Ray logs from worker jpbo-090-09...
|
||||
WARNING: Failed to collect logs from jpbo-090-09
|
||||
Collecting Ray logs from worker jpbo-090-10...
|
||||
WARNING: Failed to collect logs from jpbo-090-10
|
||||
Collecting Ray logs from worker jpbo-090-11...
|
||||
WARNING: Failed to collect logs from jpbo-090-11
|
||||
Collecting Ray logs from worker jpbo-090-12...
|
||||
WARNING: Failed to collect logs from jpbo-090-12
|
||||
Collecting Ray logs from worker jpbo-090-13...
|
||||
WARNING: Failed to collect logs from jpbo-090-13
|
||||
Collecting Ray logs from worker jpbo-090-15...
|
||||
WARNING: Failed to collect logs from jpbo-090-15
|
||||
Collecting Ray logs from worker jpbo-090-16...
|
||||
WARNING: Failed to collect logs from jpbo-090-16
|
||||
Ray log preservation complete
|
||||
Reference in New Issue
Block a user