初始化项目，由ModelHub XC社区提供模型
Model: laion/a3-rl-DCAgent_code-contests-noblock-5-8B Source: Original Platform
This commit is contained in:
ModelHub XC
111
training_logs/a3-rl-DCAgent_code-contests-noblock_496036.out
Normal file
111
training_logs/a3-rl-DCAgent_code-contests-noblock_496036.out
Normal file
@@ -0,0 +1,111 @@
|
||||
|
||||
The following have been reloaded with a version change:
|
||||
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
||||
|
||||
|
||||
Lmod is automatically replacing "GCC/14.3.0" with
|
||||
"nvidia-compilers/25.9-CUDA-13".
|
||||
|
||||
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
||||
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
||||
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
[ray] RAY_TMPDIR=/tmp/ray/ray_496036
|
||||
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_496036
|
||||
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_496036
|
||||
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
||||
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
||||
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
||||
[proxy] Tunnel port: 7003
|
||||
[proxy] Node IP: 10.128.17.210 (workers will connect here)
|
||||
[proxy] ✓ SSH tunnel started successfully
|
||||
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496036.conf
|
||||
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
||||
[proxy] - External traffic (internet) → PROXY via tunnel
|
||||
[proxy] ✓ Daytona timeout settings configured
|
||||
[proxy] Testing proxy connectivity...
|
||||
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496036.conf
|
||||
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
||||
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
||||
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
||||
[proxy] ⚠ Tunnel not accessible at 10.128.17.210:7003 (workers may fail)
|
||||
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
||||
[container_runtime] Using cloud backend: daytona (no local container setup)
|
||||
=== Universal RL Training Runner ===
|
||||
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/configs/a3-rl-DCAgent_code-contests-noblock_rl_config.json
|
||||
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
||||
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
||||
Python version: Python 3.12.12
|
||||
Proxy: DISABLED (direct internet or not configured)
|
||||
========================================
|
||||
=== RLJobRunner: a3-rl-DCAgent_code-contests-noblock ===
|
||||
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||||
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||||
HF_TOKEN=****pDbg
|
||||
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
||||
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
||||
Environment configured:
|
||||
TENSOR_PARALLEL_SIZE=1
|
||||
NUM_INFERENCE_ENGINES=56
|
||||
POLICY_NUM_NODES=14
|
||||
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
|
||||
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
||||
Cleaning up existing Ray instances...
|
||||
=== Starting Ray Cluster ===
|
||||
Nodes: 14
|
||||
GPUs per node: 4
|
||||
CPUs per node: 288
|
||||
Head node: jpbo-010-34 (10.128.17.210)
|
||||
Ray port: 6379
|
||||
============================
|
||||
RL job failed: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-010-34.log'
|
||||
Traceback (most recent call last):
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 922, in run
|
||||
training_exit_code = self._run_with_ray()
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1100, in _run_with_ray
|
||||
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 818, in __enter__
|
||||
self.start()
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 342, in start
|
||||
self._start_node(self.node_list[0], is_head=True)
|
||||
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 520, in _start_node
|
||||
ray_log_file = open(ray_log_path, "w")
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
OSError: [Errno 122] Disk quota exceeded: '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-010-34.log'
|
||||
[RLJobRunner] Launching trace upload (training exit code: 1):
|
||||
repo_id: DCAgent/a3-rl-DCAgent_code-contests-noblock
|
||||
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock
|
||||
episodes: last
|
||||
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/logs/a3-rl-DCAgent_code-contests-noblock_trace_upload.log
|
||||
[RLJobRunner] Waiting for trace upload to complete...
|
||||
[RLJobRunner] Trace upload failed with exit code 1.
|
||||
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/ray_logs/
|
||||
Collecting Ray logs from worker jpbo-010-36...
|
||||
WARNING: Failed to collect logs from jpbo-010-36
|
||||
Collecting Ray logs from worker jpbo-010-38...
|
||||
WARNING: Failed to collect logs from jpbo-010-38
|
||||
Collecting Ray logs from worker jpbo-010-39...
|
||||
WARNING: Failed to collect logs from jpbo-010-39
|
||||
Collecting Ray logs from worker jpbo-010-40...
|
||||
WARNING: Failed to collect logs from jpbo-010-40
|
||||
Collecting Ray logs from worker jpbo-010-41...
|
||||
WARNING: Failed to collect logs from jpbo-010-41
|
||||
Collecting Ray logs from worker jpbo-010-43...
|
||||
WARNING: Failed to collect logs from jpbo-010-43
|
||||
Collecting Ray logs from worker jpbo-010-44...
|
||||
WARNING: Failed to collect logs from jpbo-010-44
|
||||
Collecting Ray logs from worker jpbo-010-45...
|
||||
WARNING: Failed to collect logs from jpbo-010-45
|
||||
Collecting Ray logs from worker jpbo-010-46...
|
||||
WARNING: Failed to collect logs from jpbo-010-46
|
||||
Collecting Ray logs from worker jpbo-022-02...
|
||||
WARNING: Failed to collect logs from jpbo-022-02
|
||||
Collecting Ray logs from worker jpbo-022-03...
|
||||
WARNING: Failed to collect logs from jpbo-022-03
|
||||
Collecting Ray logs from worker jpbo-022-04...
|
||||
WARNING: Failed to collect logs from jpbo-022-04
|
||||
Collecting Ray logs from worker jpbo-022-05...
|
||||
WARNING: Failed to collect logs from jpbo-022-05
|
||||
Ray log preservation complete
|
||||
Reference in New Issue
Block a user