laion/a3-rl-DCAgent_code-contests-noblock-5-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
a3-rl-DCAgent_code-contests…/training_logs/a3-rl-DCAgent_code-contests-noblock_496419.out
ModelHub XC a508caa68d 初始化项目，由ModelHub XC社区提供模型 
Model: laion/a3-rl-DCAgent_code-contests-noblock-5-8B
Source: Original Platform
2026-08-07 12:29:36 +08:00

1665 lines
278 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_496419
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_496419
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_496419
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.17.210 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496419.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496419.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.17.210:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/configs/a3-rl-DCAgent_code-contests-noblock_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: a3-rl-DCAgent_code-contests-noblock ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-010-34 (10.128.17.210)
Ray port: 6379
============================
Starting Ray head on jpbo-010-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-010-34.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.210 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.17.210 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-010-34
Starting Ray worker on jpbo-010-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.212 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.212 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-010-36
Starting Ray worker on jpbo-010-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.214 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.214 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-010-38
Starting Ray worker on jpbo-010-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.215 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.215 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-010-39
Starting Ray worker on jpbo-010-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-40.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.216 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.216 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-010-40
Starting Ray worker on jpbo-010-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.217 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.217 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-010-41
Starting Ray worker on jpbo-010-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.219 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.219 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-010-43
Starting Ray worker on jpbo-010-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.220 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.220 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-010-44
Starting Ray worker on jpbo-010-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.221 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.221 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-010-45
Starting Ray worker on jpbo-010-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.222 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.222 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-010-46
Starting Ray worker on jpbo-022-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-02.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-022-02
Starting Ray worker on jpbo-022-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-03.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-022-03
Starting Ray worker on jpbo-022-04 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-04.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.52 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.52 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-022-04
Starting Ray worker on jpbo-022-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-05.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.53 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.53 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-022-05
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.17.210:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'accelerator_type:GH200': 14.0, 'CPU': 4032.0, 'GPU': 56.0, 'node:10.128.17.216': 1.0, 'object_store_memory': 601295421440.0, 'memory': 10779218477056.0, 'node:10.128.17.222': 1.0, 'node:10.128.17.221': 1.0, 'node:10.128.24.50': 1.0, 'node:10.128.24.52': 1.0, 'node:10.128.17.215': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.17.210': 1.0, 'node:10.128.17.214': 1.0, 'node:10.128.24.51': 1.0, 'node:10.128.24.53': 1.0, 'node:10.128.17.217': 1.0, 'node:10.128.17.220': 1.0, 'node:10.128.17.219': 1.0, 'node:10.128.17.212': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
Address: 10.128.17.210:6379
Total GPUs: 56
=========================
Ray cluster ready at 10.128.17.210:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
Running SkyRL:
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
Args: 119 Hydra arguments
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=token_mean trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/a3-rl-DCAgent_code-contests-noblock ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=true trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=a3-rl-DCAgent_code-contests-noblock trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=450 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/code-contests-noblock"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=1800 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=900 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.model_info.max_input_tokens=32768 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496419.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-05-23 15:16:29.498 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:590 - Exporting wandb api key to ray runtime env
2026-05-23 15:16:29.500 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:609 - Exporting RAY_ADDRESS to ray runtime env
2026-05-23 15:16:29,500 INFO worker.py:1680 -- Using address 10.128.17.210:6379 set in the environment variable RAY_ADDRESS
2026-05-23 15:16:29,536 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.17.210:6379...
2026-05-23 15:16:29,547 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.24.53) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-05-23 15:16:32.151 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
(skyrl_entrypoint pid=3659716) 2026-05-23 15:16:41.392 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:199 - SkyRL log level set to: INFO
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(skyrl_entrypoint pid=3659716) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=3659716) 2026-05-23 15:16:41.632 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/code-contests-noblock
[2026-05-23 15:16:59,779 E 3659268 3659688] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RegistryActor pid=1440387, ip=10.128.24.53) [2026-05-23 15:17:00,782 E 1440387 1440427] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(skyrl_entrypoint pid=3659716) 2026-05-23 15:18:34.306 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 8728 valid task directories out of 8728 total directories
(skyrl_entrypoint pid=3659716) 2026-05-23 15:18:34.306 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 8728 task paths
(skyrl_entrypoint pid=3659716) 2026-05-23 15:18:34.317 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:352 - data:
(skyrl_entrypoint pid=3659716) train_data:
(skyrl_entrypoint pid=3659716) - /e/scratch/jureap59/feuer1/tasks/code-contests-noblock
(skyrl_entrypoint pid=3659716) val_data: []
(skyrl_entrypoint pid=3659716) trainer:
(skyrl_entrypoint pid=3659716) placement:
(skyrl_entrypoint pid=3659716) colocate_all: false
(skyrl_entrypoint pid=3659716) colocate_policy_ref: true
(skyrl_entrypoint pid=3659716) policy_num_nodes: 2
(skyrl_entrypoint pid=3659716) policy_num_gpus_per_node: 4
(skyrl_entrypoint pid=3659716) critic_num_nodes: 1
(skyrl_entrypoint pid=3659716) critic_num_gpus_per_node: 4
(skyrl_entrypoint pid=3659716) ref_num_nodes: 2
(skyrl_entrypoint pid=3659716) ref_num_gpus_per_node: 4
(skyrl_entrypoint pid=3659716) sequence_parallel_backend: ulysses
(skyrl_entrypoint pid=3659716) strategy: fsdp2
(skyrl_entrypoint pid=3659716) policy:
(skyrl_entrypoint pid=3659716) model:
(skyrl_entrypoint pid=3659716) path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=3659716) lora:
(skyrl_entrypoint pid=3659716) rank: 0
(skyrl_entrypoint pid=3659716) alpha: 16
(skyrl_entrypoint pid=3659716) dropout: 0
(skyrl_entrypoint pid=3659716) lora_sync_path: /tmp/skyrl_lora_sync
(skyrl_entrypoint pid=3659716) target_modules: all-linear
(skyrl_entrypoint pid=3659716) exclude_modules: null
(skyrl_entrypoint pid=3659716) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=3659716) optimizer_config:
(skyrl_entrypoint pid=3659716) optimizer: AdamW
(skyrl_entrypoint pid=3659716) lr: 8.0e-06
(skyrl_entrypoint pid=3659716) adam_betas:
(skyrl_entrypoint pid=3659716) - 0.9
(skyrl_entrypoint pid=3659716) - 0.999
(skyrl_entrypoint pid=3659716) weight_decay: 0.0
(skyrl_entrypoint pid=3659716) max_grad_norm: 0.9
(skyrl_entrypoint pid=3659716) offload_after_step: true
(skyrl_entrypoint pid=3659716) num_warmup_steps: 0
(skyrl_entrypoint pid=3659716) scheduler: constant_with_warmup
(skyrl_entrypoint pid=3659716) optimizer_kwargs: {}
(skyrl_entrypoint pid=3659716) fsdp_config:
(skyrl_entrypoint pid=3659716) cpu_offload: false
(skyrl_entrypoint pid=3659716) reshard_after_forward: true
(skyrl_entrypoint pid=3659716) fsdp_size: 4
(skyrl_entrypoint pid=3659716) sequence_parallel_size: 1
(skyrl_entrypoint pid=3659716) use_torch_compile: false
(skyrl_entrypoint pid=3659716) record_memory: false
(skyrl_entrypoint pid=3659716) megatron_config:
(skyrl_entrypoint pid=3659716) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=3659716) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=3659716) context_parallel_size: 1
(skyrl_entrypoint pid=3659716) expert_model_parallel_size: 1
(skyrl_entrypoint pid=3659716) expert_tensor_parallel_size: null
(skyrl_entrypoint pid=3659716) ddp_config:
(skyrl_entrypoint pid=3659716) grad_reduce_in_fp32: true
(skyrl_entrypoint pid=3659716) overlap_grad_reduce: false
(skyrl_entrypoint pid=3659716) overlap_param_gather: false
(skyrl_entrypoint pid=3659716) average_in_collective: true
(skyrl_entrypoint pid=3659716) model_config_kwargs: {}
(skyrl_entrypoint pid=3659716) torch_profiler_config:
(skyrl_entrypoint pid=3659716) enable: false
(skyrl_entrypoint pid=3659716) ranks: []
(skyrl_entrypoint pid=3659716) save_path: null
(skyrl_entrypoint pid=3659716) optimizer_config_kwargs:
(skyrl_entrypoint pid=3659716) overlap_cpu_optimizer_d2h_h2d: false
(skyrl_entrypoint pid=3659716) use_precision_aware_optimizer: false
(skyrl_entrypoint pid=3659716) optimizer_cpu_offload: false
(skyrl_entrypoint pid=3659716) optimizer_offload_fraction: 0.0
(skyrl_entrypoint pid=3659716) transformer_config_kwargs:
(skyrl_entrypoint pid=3659716) recompute_granularity: full
(skyrl_entrypoint pid=3659716) recompute_modules:
(skyrl_entrypoint pid=3659716) - core_attn
(skyrl_entrypoint pid=3659716) recompute_method: uniform
(skyrl_entrypoint pid=3659716) recompute_num_layers: 1
(skyrl_entrypoint pid=3659716) empty_cuda_cache: true
(skyrl_entrypoint pid=3659716) ref:
(skyrl_entrypoint pid=3659716) model:
(skyrl_entrypoint pid=3659716) path: ${trainer.policy.model.path}
(skyrl_entrypoint pid=3659716) sequence_parallel_size: 1
(skyrl_entrypoint pid=3659716) deepspeed_config: ${deepspeed_config.eval}
(skyrl_entrypoint pid=3659716) fsdp_config:
(skyrl_entrypoint pid=3659716) cpu_offload: false
(skyrl_entrypoint pid=3659716) reshard_after_forward: true
(skyrl_entrypoint pid=3659716) fsdp_size: 4
(skyrl_entrypoint pid=3659716) megatron_config:
(skyrl_entrypoint pid=3659716) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=3659716) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=3659716) context_parallel_size: 1
(skyrl_entrypoint pid=3659716) expert_model_parallel_size: 1
(skyrl_entrypoint pid=3659716) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=3659716) model_config_kwargs: {}
(skyrl_entrypoint pid=3659716) transformer_config_kwargs: {}
(skyrl_entrypoint pid=3659716) critic:
(skyrl_entrypoint pid=3659716) model:
(skyrl_entrypoint pid=3659716) path: null
(skyrl_entrypoint pid=3659716) lora:
(skyrl_entrypoint pid=3659716) rank: 0
(skyrl_entrypoint pid=3659716) alpha: 16
(skyrl_entrypoint pid=3659716) dropout: 0
(skyrl_entrypoint pid=3659716) target_modules: all-linear
(skyrl_entrypoint pid=3659716) exclude_modules: null
(skyrl_entrypoint pid=3659716) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=3659716) optimizer_config:
(skyrl_entrypoint pid=3659716) optimizer: AdamW
(skyrl_entrypoint pid=3659716) lr: 5.0e-06
(skyrl_entrypoint pid=3659716) adam_betas:
(skyrl_entrypoint pid=3659716) - 0.9
(skyrl_entrypoint pid=3659716) - 0.999
(skyrl_entrypoint pid=3659716) weight_decay: 0.01
(skyrl_entrypoint pid=3659716) max_grad_norm: 1.0
(skyrl_entrypoint pid=3659716) offload_after_step: true
(skyrl_entrypoint pid=3659716) num_warmup_steps: 0
(skyrl_entrypoint pid=3659716) scheduler: constant_with_warmup
(skyrl_entrypoint pid=3659716) optimizer_kwargs: {}
(skyrl_entrypoint pid=3659716) fsdp_config:
(skyrl_entrypoint pid=3659716) cpu_offload: false
(skyrl_entrypoint pid=3659716) reshard_after_forward: true
(skyrl_entrypoint pid=3659716) fsdp_size: -1
(skyrl_entrypoint pid=3659716) sequence_parallel_size: 1
(skyrl_entrypoint pid=3659716) algorithm:
(skyrl_entrypoint pid=3659716) advantage_estimator: rloo_n
(skyrl_entrypoint pid=3659716) kl_ctrl:
(skyrl_entrypoint pid=3659716) type: fixed
(skyrl_entrypoint pid=3659716) kl_target: 0.1
(skyrl_entrypoint pid=3659716) horizon: 10000
(skyrl_entrypoint pid=3659716) kl_estimator_type: k3
(skyrl_entrypoint pid=3659716) use_kl_estimator_k3: false
(skyrl_entrypoint pid=3659716) use_abs_kl: false
(skyrl_entrypoint pid=3659716) use_kl_in_reward: false
(skyrl_entrypoint pid=3659716) use_kl_loss: false
(skyrl_entrypoint pid=3659716) kl_loss_coef: 0.0
(skyrl_entrypoint pid=3659716) use_entropy_loss: false
(skyrl_entrypoint pid=3659716) entropy_loss_coef: 0.01
(skyrl_entrypoint pid=3659716) advantage_batch_normalize: false
(skyrl_entrypoint pid=3659716) value_head_prefix: value_head
(skyrl_entrypoint pid=3659716) policy_loss_type: regular
(skyrl_entrypoint pid=3659716) loss_reduction: token_mean
(skyrl_entrypoint pid=3659716) grpo_norm_by_std: true
(skyrl_entrypoint pid=3659716) rloo_n_min_group_size: 4
(skyrl_entrypoint pid=3659716) rloo_n_filter_zero_reward_groups: true
(skyrl_entrypoint pid=3659716) lambd: 1.0
(skyrl_entrypoint pid=3659716) gamma: 1.0
(skyrl_entrypoint pid=3659716) eps_clip_low: 0.2
(skyrl_entrypoint pid=3659716) eps_clip_high: 0.05
(skyrl_entrypoint pid=3659716) clip_ratio_c: 3.0
(skyrl_entrypoint pid=3659716) tis_imp_ratio_cap: -1.0
(skyrl_entrypoint pid=3659716) use_tis: false
(skyrl_entrypoint pid=3659716) sapo:
(skyrl_entrypoint pid=3659716) tau_pos: 1.0
(skyrl_entrypoint pid=3659716) tau_neg: 1.05
(skyrl_entrypoint pid=3659716) value_clip: 0.2
(skyrl_entrypoint pid=3659716) dynamic_sampling:
(skyrl_entrypoint pid=3659716) type: null
(skyrl_entrypoint pid=3659716) max_sample_batches: 30
(skyrl_entrypoint pid=3659716) min_replace_ratio: 0.3
(skyrl_entrypoint pid=3659716) clip_cov:
(skyrl_entrypoint pid=3659716) clip_ratio: 0.0002
(skyrl_entrypoint pid=3659716) clip_cov_lb: 1.0
(skyrl_entrypoint pid=3659716) clip_cov_ub: 5.0
(skyrl_entrypoint pid=3659716) kl_cov:
(skyrl_entrypoint pid=3659716) kl_cov_frac: 0.2
(skyrl_entrypoint pid=3659716) ppo_kl_coef: 1.0
(skyrl_entrypoint pid=3659716) cispo:
(skyrl_entrypoint pid=3659716) cispo_eps_clip_low: 0
(skyrl_entrypoint pid=3659716) cispo_eps_clip_high: 5
(skyrl_entrypoint pid=3659716) z_clip:
(skyrl_entrypoint pid=3659716) enabled: false
(skyrl_entrypoint pid=3659716) alpha: 0.97
(skyrl_entrypoint pid=3659716) z_thresh: 2.5
(skyrl_entrypoint pid=3659716) warmup_steps: 25
(skyrl_entrypoint pid=3659716) mode: zscore
(skyrl_entrypoint pid=3659716) clip_option: adaptive_scaling
(skyrl_entrypoint pid=3659716) clip_factor: 1.0
(skyrl_entrypoint pid=3659716) skip_update_on_spike: false
(skyrl_entrypoint pid=3659716) stale_clip:
(skyrl_entrypoint pid=3659716) enabled: false
(skyrl_entrypoint pid=3659716) alpha: 0.3
(skyrl_entrypoint pid=3659716) entropy_threshold: 0.15
(skyrl_entrypoint pid=3659716) entropy_window: 10
(skyrl_entrypoint pid=3659716) min_lr_scale: 0.1
(skyrl_entrypoint pid=3659716) max_seq_len: 1004095
(skyrl_entrypoint pid=3659716) fully_async:
(skyrl_entrypoint pid=3659716) max_staleness_steps: 16
(skyrl_entrypoint pid=3659716) num_parallel_generation_workers: 450
(skyrl_entrypoint pid=3659716) gradient_checkpointing: true
(skyrl_entrypoint pid=3659716) gradient_checkpointing_use_reentrant: false
(skyrl_entrypoint pid=3659716) seed: 42
(skyrl_entrypoint pid=3659716) resume_mode: latest
(skyrl_entrypoint pid=3659716) resume_path: null
(skyrl_entrypoint pid=3659716) ckpt_path: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints
(skyrl_entrypoint pid=3659716) max_ckpts_to_keep: -1
(skyrl_entrypoint pid=3659716) ckpt_interval: 2
(skyrl_entrypoint pid=3659716) hf_save_interval: 5
(skyrl_entrypoint pid=3659716) export_path: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports
(skyrl_entrypoint pid=3659716) bf16: true
(skyrl_entrypoint pid=3659716) epochs: 2
(skyrl_entrypoint pid=3659716) max_steps: 80
(skyrl_entrypoint pid=3659716) update_epochs_per_batch: 1
(skyrl_entrypoint pid=3659716) train_batch_size: 64
(skyrl_entrypoint pid=3659716) policy_mini_batch_size: 64
(skyrl_entrypoint pid=3659716) critic_mini_batch_size: 256
(skyrl_entrypoint pid=3659716) micro_train_batch_size_per_gpu: 1
(skyrl_entrypoint pid=3659716) micro_forward_batch_size_per_gpu: 4
(skyrl_entrypoint pid=3659716) update_ref_every_epoch: false
(skyrl_entrypoint pid=3659716) use_sample_packing: true
(skyrl_entrypoint pid=3659716) eval_batch_size: 64
(skyrl_entrypoint pid=3659716) eval_before_train: false
(skyrl_entrypoint pid=3659716) eval_interval: 999999
(skyrl_entrypoint pid=3659716) max_prompt_length: 999999
(skyrl_entrypoint pid=3659716) flash_attn: true
(skyrl_entrypoint pid=3659716) disable_fast_tokenizer: false
(skyrl_entrypoint pid=3659716) target_modules: null
(skyrl_entrypoint pid=3659716) exclude_modules: null
(skyrl_entrypoint pid=3659716) project_name: OpenThoughts-Agent
(skyrl_entrypoint pid=3659716) run_name: a3-rl-DCAgent_code-contests-noblock
(skyrl_entrypoint pid=3659716) logger: console
(skyrl_entrypoint pid=3659716) tracker_commit_each_step: true
(skyrl_entrypoint pid=3659716) dump_data_batch: false
(skyrl_entrypoint pid=3659716) dump_eval_results: true
(skyrl_entrypoint pid=3659716) log_level: INFO
(skyrl_entrypoint pid=3659716) rope_scaling: null
(skyrl_entrypoint pid=3659716) rope_theta: null
(skyrl_entrypoint pid=3659716) step_wise_training: false
(skyrl_entrypoint pid=3659716) hf_hub_repo_id: laion/a3-rl-DCAgent_code-contests-noblock
(skyrl_entrypoint pid=3659716) hf_hub_private: false
(skyrl_entrypoint pid=3659716) hf_hub_revision: main
(skyrl_entrypoint pid=3659716) enable_db_registration: true
(skyrl_entrypoint pid=3659716) generator:
(skyrl_entrypoint pid=3659716) model_name: ${trainer.policy.model.path}
(skyrl_entrypoint pid=3659716) model_dtype: bfloat16
(skyrl_entrypoint pid=3659716) timeout_multiplier: 1.0
(skyrl_entrypoint pid=3659716) run_engines_locally: true
(skyrl_entrypoint pid=3659716) num_inference_engines: 48
(skyrl_entrypoint pid=3659716) backend: vllm
(skyrl_entrypoint pid=3659716) weight_sync_backend: nccl
(skyrl_entrypoint pid=3659716) fuse_weights: false
(skyrl_entrypoint pid=3659716) weight_transfer_threshold_cuda_ipc_GB: 1.0
(skyrl_entrypoint pid=3659716) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=3659716) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=3659716) inference_engine_expert_parallel_size: 1
(skyrl_entrypoint pid=3659716) inference_engine_data_parallel_size: 1
(skyrl_entrypoint pid=3659716) n_samples_per_prompt: 8
(skyrl_entrypoint pid=3659716) async_engine: true
(skyrl_entrypoint pid=3659716) batched: false
(skyrl_entrypoint pid=3659716) max_input_length: ${trainer.max_prompt_length}
(skyrl_entrypoint pid=3659716) vllm_v1_disable_multiproc: true
(skyrl_entrypoint pid=3659716) enable_prefix_caching: true
(skyrl_entrypoint pid=3659716) enable_chunked_prefill: true
(skyrl_entrypoint pid=3659716) max_num_batched_tokens: 65536
(skyrl_entrypoint pid=3659716) enforce_eager: true
(skyrl_entrypoint pid=3659716) fully_sharded_loras: false
(skyrl_entrypoint pid=3659716) enable_ray_prometheus_stats: false
(skyrl_entrypoint pid=3659716) vllm_stats_interval: 1
(skyrl_entrypoint pid=3659716) gpu_memory_utilization: 0.75
(skyrl_entrypoint pid=3659716) max_num_seqs: 24
(skyrl_entrypoint pid=3659716) remote_inference_engine_urls:
(skyrl_entrypoint pid=3659716) - 127.0.0.1:8001
(skyrl_entrypoint pid=3659716) enable_http_endpoint: true
(skyrl_entrypoint pid=3659716) http_endpoint_host: 127.0.0.1
(skyrl_entrypoint pid=3659716) http_endpoint_port: 8000
(skyrl_entrypoint pid=3659716) max_turns: 999999
(skyrl_entrypoint pid=3659716) chat_template:
(skyrl_entrypoint pid=3659716) source: name
(skyrl_entrypoint pid=3659716) name_or_path: null
(skyrl_entrypoint pid=3659716) chat_template_kwargs: {}
(skyrl_entrypoint pid=3659716) engine_init_kwargs:
(skyrl_entrypoint pid=3659716) max_model_len: 32768
(skyrl_entrypoint pid=3659716) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=3659716) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=3659716) override_existing_update_group: disable
(skyrl_entrypoint pid=3659716) sampling_params:
(skyrl_entrypoint pid=3659716) max_generate_length: 4096
(skyrl_entrypoint pid=3659716) repetition_penalty: 1.0
(skyrl_entrypoint pid=3659716) temperature: 0.7
(skyrl_entrypoint pid=3659716) top_p: 0.95
(skyrl_entrypoint pid=3659716) min_p: 0.0
(skyrl_entrypoint pid=3659716) top_k: 20
(skyrl_entrypoint pid=3659716) logprobs: null
(skyrl_entrypoint pid=3659716) stop: null
(skyrl_entrypoint pid=3659716) use_conversation_multi_turn: true
(skyrl_entrypoint pid=3659716) append_eos_token_after_stop_str_in_multi_turn: true
(skyrl_entrypoint pid=3659716) eval_sampling_params:
(skyrl_entrypoint pid=3659716) max_generate_length: ${generator.sampling_params.max_generate_length}
(skyrl_entrypoint pid=3659716) repetition_penalty: 1.0
(skyrl_entrypoint pid=3659716) temperature: 0.0
(skyrl_entrypoint pid=3659716) top_p: 1.0
(skyrl_entrypoint pid=3659716) min_p: 0.0
(skyrl_entrypoint pid=3659716) top_k: -1
(skyrl_entrypoint pid=3659716) logprobs: null
(skyrl_entrypoint pid=3659716) stop: null
(skyrl_entrypoint pid=3659716) eval_n_samples_per_prompt: 8
(skyrl_entrypoint pid=3659716) zero_reward_on_non_stop: false
(skyrl_entrypoint pid=3659716) apply_overlong_filtering: false
(skyrl_entrypoint pid=3659716) rope_scaling: ${trainer.rope_scaling}
(skyrl_entrypoint pid=3659716) rope_theta: ${trainer.rope_theta}
(skyrl_entrypoint pid=3659716) teacher:
(skyrl_entrypoint pid=3659716) model_path: null
(skyrl_entrypoint pid=3659716) top_k_logprobs: 256
(skyrl_entrypoint pid=3659716) num_inference_engines: 1
(skyrl_entrypoint pid=3659716) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=3659716) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=3659716) gpu_memory_utilization: 0.9
(skyrl_entrypoint pid=3659716) enforce_eager: false
(skyrl_entrypoint pid=3659716) backend: vllm
(skyrl_entrypoint pid=3659716) engine_init_kwargs: {}
(skyrl_entrypoint pid=3659716) environment:
(skyrl_entrypoint pid=3659716) env_class: gsm8k
(skyrl_entrypoint pid=3659716) skyrl_gym:
(skyrl_entrypoint pid=3659716) max_env_workers: 32
(skyrl_entrypoint pid=3659716) text2sql:
(skyrl_entrypoint pid=3659716) db_path: /home/ray/default/sql_data
(skyrl_entrypoint pid=3659716) llm_as_a_judge:
(skyrl_entrypoint pid=3659716) model: gpt-4o-mini
(skyrl_entrypoint pid=3659716) base_url: null
(skyrl_entrypoint pid=3659716) search:
(skyrl_entrypoint pid=3659716) log_requests: false
(skyrl_entrypoint pid=3659716) search_url: http://127.0.0.1:8000/retrieve
(skyrl_entrypoint pid=3659716) topk: 3
(skyrl_entrypoint pid=3659716) timeout: 30
(skyrl_entrypoint pid=3659716) deepspeed_config:
(skyrl_entrypoint pid=3659716) train:
(skyrl_entrypoint pid=3659716) zero_optimization:
(skyrl_entrypoint pid=3659716) stage: 3
(skyrl_entrypoint pid=3659716) offload_param:
(skyrl_entrypoint pid=3659716) device: none
(skyrl_entrypoint pid=3659716) offload_optimizer:
(skyrl_entrypoint pid=3659716) device: none
(skyrl_entrypoint pid=3659716) pin_memory: true
(skyrl_entrypoint pid=3659716) sub_group_size: auto
(skyrl_entrypoint pid=3659716) reduce_bucket_size: auto
(skyrl_entrypoint pid=3659716) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=3659716) stage3_prefetch_bucket_size: auto
(skyrl_entrypoint pid=3659716) stage3_max_live_parameters: auto
(skyrl_entrypoint pid=3659716) stage3_max_reuse_distance: auto
(skyrl_entrypoint pid=3659716) round_robin_gradients: true
(skyrl_entrypoint pid=3659716) zero_hpz_partition_size: 1
(skyrl_entrypoint pid=3659716) zero_quantized_weights: false
(skyrl_entrypoint pid=3659716) zero_quantized_gradients: false
(skyrl_entrypoint pid=3659716) torch_autocast:
(skyrl_entrypoint pid=3659716) enabled: true
(skyrl_entrypoint pid=3659716) dtype: bfloat16
(skyrl_entrypoint pid=3659716) disable_trace_cache: false
(skyrl_entrypoint pid=3659716) data_types:
(skyrl_entrypoint pid=3659716) grad_accum_dtype: fp32
(skyrl_entrypoint pid=3659716) gradient_clipping: 1.0
(skyrl_entrypoint pid=3659716) wall_clock_breakdown: false
(skyrl_entrypoint pid=3659716) prescale_gradient: false
(skyrl_entrypoint pid=3659716) eval:
(skyrl_entrypoint pid=3659716) zero_optimization:
(skyrl_entrypoint pid=3659716) stage: 3
(skyrl_entrypoint pid=3659716) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=3659716) offload_param:
(skyrl_entrypoint pid=3659716) device: cpu
(skyrl_entrypoint pid=3659716) pin_memory: true
(skyrl_entrypoint pid=3659716) torch_autocast:
(skyrl_entrypoint pid=3659716) enabled: true
(skyrl_entrypoint pid=3659716) dtype: bfloat16
(skyrl_entrypoint pid=3659716) gradient_clipping: 1.0
(skyrl_entrypoint pid=3659716) prescale_gradient: false
(skyrl_entrypoint pid=3659716) wall_clock_breakdown: false
(skyrl_entrypoint pid=3659716) terminal_bench_config:
(skyrl_entrypoint pid=3659716) trials_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs
(skyrl_entrypoint pid=3659716) harbor:
(skyrl_entrypoint pid=3659716) name: terminus-2
(skyrl_entrypoint pid=3659716) max_episodes: 999999
(skyrl_entrypoint pid=3659716) enable_summarize: false
(skyrl_entrypoint pid=3659716) store_all_messages: true
(skyrl_entrypoint pid=3659716) trajectory_config:
(skyrl_entrypoint pid=3659716) raw_content: true
(skyrl_entrypoint pid=3659716) enable_episode_logging: false
(skyrl_entrypoint pid=3659716) record_terminal_session: false
(skyrl_entrypoint pid=3659716) enable_pane_logging: false
(skyrl_entrypoint pid=3659716) strict_json_parser: true
(skyrl_entrypoint pid=3659716) interleaved_thinking: true
(skyrl_entrypoint pid=3659716) extra_body:
(skyrl_entrypoint pid=3659716) chat_template_kwargs:
(skyrl_entrypoint pid=3659716) enable_thinking: true
(skyrl_entrypoint pid=3659716) override_timeout_sec: 1800
(skyrl_entrypoint pid=3659716) override_cpus: 1
(skyrl_entrypoint pid=3659716) override_memory_mb: 2048
(skyrl_entrypoint pid=3659716) override_storage_mb: 2048
(skyrl_entrypoint pid=3659716) auto_snapshot: true
(skyrl_entrypoint pid=3659716) verifier_override_timeout_sec: 120
(skyrl_entrypoint pid=3659716) max_retries: 3
(skyrl_entrypoint pid=3659716) min_wait_sec: 60.0
(skyrl_entrypoint pid=3659716) max_wait_sec: 600.0
(skyrl_entrypoint pid=3659716) wait_multiplier: 2.0
(skyrl_entrypoint pid=3659716) exclude_exceptions:
(skyrl_entrypoint pid=3659716) - VerifierTimeoutError
(skyrl_entrypoint pid=3659716) - VerifierRuntimeError
(skyrl_entrypoint pid=3659716) - RewardFileNotFoundError
(skyrl_entrypoint pid=3659716) - RewardFileEmptyError
(skyrl_entrypoint pid=3659716) - VerifierOutputParseError
(skyrl_entrypoint pid=3659716) n_concurrent_trials: 900
(skyrl_entrypoint pid=3659716) log_level: INFO
(skyrl_entrypoint pid=3659716) enable_reward_shaping: false
(skyrl_entrypoint pid=3659716) enable_error_classification: true
(skyrl_entrypoint pid=3659716) mask_exceptions:
(skyrl_entrypoint pid=3659716) - DaytonaError
(skyrl_entrypoint pid=3659716) - EnvironmentStartTimeoutError
(skyrl_entrypoint pid=3659716) - NetworkError
(skyrl_entrypoint pid=3659716) - ConnectionError
(skyrl_entrypoint pid=3659716) - RewardFileNotFoundError
(skyrl_entrypoint pid=3659716) - RewardFileEmptyError
(skyrl_entrypoint pid=3659716) - AgentEnvironmentTimeoutError
(skyrl_entrypoint pid=3659716) default_error_treatment: zero
(skyrl_entrypoint pid=3659716) passthrough_exceptions:
(skyrl_entrypoint pid=3659716) - AgentTimeoutError
(skyrl_entrypoint pid=3659716) - ContextLengthExceededError
(skyrl_entrypoint pid=3659716) model_info:
(skyrl_entrypoint pid=3659716) max_input_tokens: 32768
(skyrl_entrypoint pid=3659716) max_output_tokens: 4096
(skyrl_entrypoint pid=3659716) archiving:
(skyrl_entrypoint pid=3659716) enabled: false
(skyrl_entrypoint pid=3659716) trace_upload:
(skyrl_entrypoint pid=3659716) enabled: true
(skyrl_entrypoint pid=3659716) repo_org: DCAgent
(skyrl_entrypoint pid=3659716) episodes: last
(skyrl_entrypoint pid=3659716) dataset_type: SFT
(skyrl_entrypoint pid=3659716) cleanup: true
(skyrl_entrypoint pid=3659716) 
(skyrl_entrypoint pid=3659716) [2026-05-23 15:17:03,082 E 3659716 3659759] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster]
(skyrl_entrypoint pid=3659716) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(skyrl_entrypoint pid=3659716) No module named 'vllm._version'
(skyrl_entrypoint pid=3659716) from .version import __version__, __version_tuple__ # isort:skip
(skyrl_entrypoint pid=3659716) W0523 15:18:39.734000 3659716 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.24.51) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster]
(pid=3294862, ip=10.128.17.219) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=3294862, ip=10.128.17.219) No module named 'vllm._version'
(pid=3294862, ip=10.128.17.219) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.17.216) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 57x across cluster]
(pid=2397096, ip=10.128.24.51) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster]
(pid=2397096, ip=10.128.24.51) No module named 'vllm._version' [repeated 8x across cluster]
(pid=2397096, ip=10.128.24.51) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster]
(raylet, ip=10.128.17.220) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 101x across cluster]
(pid=3123494, ip=10.128.17.215) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster]
(pid=3123494, ip=10.128.17.215) No module named 'vllm._version' [repeated 15x across cluster]
(pid=3123494, ip=10.128.17.215) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:18:59.704 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:18:59.706 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:18:59.706 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.677 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:01.027 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 1.88s (attempt 1/5) to avoid port collisions
(raylet, ip=10.128.24.52) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 64x across cluster]
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(pid=1440680, ip=10.128.24.53) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster]
(pid=1440680, ip=10.128.24.53) No module named 'vllm._version' [repeated 8x across cluster]
(pid=1440680, ip=10.128.24.53) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:04.230 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:04.288 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:04.288 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.504 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.792 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.793 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.793 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.793 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.823 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 2.65s (attempt 1/5) to avoid port collisions [repeated 11x across cluster]
(skyrl_entrypoint pid=3659716) [2026-05-23 15:19:06] INFO inference_engine_client_http_endpoint.py:350: Starting server on 127.0.0.1:8000
(skyrl_entrypoint pid=3659716) [2026-05-23 15:19:06] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
(raylet, ip=10.128.17.214) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 118x across cluster]
(skyrl_entrypoint pid=3659716) [2026-05-23 15:19:07] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.125 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:960 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.125 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines.
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.127 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:213 - Harbor logging level set to INFO
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.128 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:113 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 900. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.130 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:129 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 272
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:351 - Length of train_dataloader: 8728
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:352 - Number of steps per epoch: 136
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:353 - Total training steps: 80
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) No module named 'vllm._version' [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) from .version import __version__, __version_tuple__ # isort:skip [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) 2026-05-23 15:19:10.098 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) 2026-05-23 15:19:10.100 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) 2026-05-23 15:19:10.100 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 5x across cluster]
(pid=2753436, ip=10.128.17.217) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) 2026-05-23 15:19:08.832 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.876 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.877 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.877 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.877 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.898 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 2.48s (attempt 1/5) to avoid port collisions [repeated 5x across cluster]
(bundle_reservation_check_func pid=3659816) [2026-05-23 15:19:11,518 E 3659816 3659856] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 21x across cluster]
(AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) No module named 'vllm._version' [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) from .version import __version__, __version_tuple__ # isort:skip [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:14.386 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 19x across cluster]
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:14.387 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 19x across cluster]
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:14.387 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 19x across cluster]
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:15.611 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.309 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 1.81s (attempt 1/5) to avoid port collisions [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) [2026-05-23 15:19:16,018 E 2396835 2396937] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 34x across cluster]
(AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) [W523 15:19:17.519222234 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-022-03-interconnect-1.jupiter.internal]:40421 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) [W523 15:19:17.519601842 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) [rank0]:[W523 15:19:17.521578185 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:19:18 INFO [ipv4-debug] hostname=jpbo-010-41.jupiter.internal
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:19:18 INFO [ipv4-debug] _global_node.node_ip_address=10.128.17.217
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:19:18 INFO [ipv4-debug] get_node_ip_address()=10.128.17.217
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:18.138 | INFO  | skyrl_train.workers.worker:_initiate_actors:496 - Initializing process group for RayActorGroup
(AsyncVLLMInferenceEngine pid=1440679, ip=10.128.24.53) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 29x across cluster]
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) No module named 'vllm._version' [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) from .version import __version__, __version_tuple__ # isort:skip [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3294996, ip=10.128.17.219) (EngineCore_DP0 pid=3295247) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3294998, ip=10.128.17.219) (EngineCore_DP0 pid=3295229) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) (EngineCore_DP0 pid=3295221) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) Process EngineCore_DP0:
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) self.run()
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 108, in run
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) self._target(*self._args, **self._kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1010, in run_engine_core
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) raise e
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) super().__init__(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) self._init_executor()
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) init_distributed_environment(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) return TCPStore(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 34697, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40003896c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #1: <unknown function> + 0x5f29220 (0x400018b09220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #2: <unknown function> + 0x5f4326c (0x400018b2326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #3: <unknown function> + 0x5f49074 (0x400018b29074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #4: <unknown function> + 0x5f49138 (0x400018b29138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #5: <unknown function> + 0x5f2ccc4 (0x400018b0ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x400018b11564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #7: <unknown function> + 0x109a094 (0x4000128ba094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #8: <unknown function> + 0x113236c (0x40001295236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #9: <unknown function> + 0x5d6d60 (0x400011df6d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #10: <unknown function> + 0x1b7a38 (0xaaaaca047a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #12: <unknown function> + 0x169f50 (0xaaaac9ff9f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #13: <unknown function> + 0x1682e4 (0xaaaac9ff82e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #14: <unknown function> + 0x1e0ce8 (0xaaaaca070ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #15: <unknown function> + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #16: <unknown function> + 0x646b0c (0x400011e66b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #19: <unknown function> + 0x1808c0 (0xaaaaca0108c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #20: <unknown function> + 0x182bf8 (0xaaaaca012bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #21: <unknown function> + 0x25fd30 (0xaaaaca0efd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #22: <unknown function> + 0x1b7d20 (0xaaaaca047d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #23: PyObject_Vectorcall + 0x54 (0xaaaac9ff60e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaac9ff7f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #27: <unknown function> + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #28: <unknown function> + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaac9ff7fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #33: <unknown function> + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #34: <unknown function> + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #35: _PyObject_Call + 0x68 (0xaaaac9ff8488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaaca0fd8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #37: PyEval_EvalCode + 0xb4 (0xaaaaca102eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #38: <unknown function> + 0x2ccdcc (0xaaaaca15cdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #39: <unknown function> + 0x2ccef4 (0xaaaaca15cef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #40: PyRun_StringFlags + 0x90 (0xaaaaca161050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaaca1610f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #42: Py_RunMain + 0x2c8 (0xaaaaca189190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #43: Py_BytesMain + 0x64 (0xaaaaca189fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #44: <unknown function> + 0x27540 (0x400000e67540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #45: __libc_start_main + 0x98 (0x400000e67618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #46: <unknown function> + 0x10e0c0 (0xaaaac9f9e0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:16.152 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:16.152 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:16.152 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 4x across cluster]
(pid=2753511, ip=10.128.17.217) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) (EngineCore_DP0 pid=523846) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.465 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.796 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 2.12s (attempt 1/5) to avoid port collisions [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400bfefd3080>)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) return self.__get_result()
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) raise self._exception
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) super().__init__(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) self.llm = self._create_engine(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) return cls(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) self.engine_core = EngineCoreClient.make_async_mp_client(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) return AsyncMPClient(*client_args)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) super().__init__(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) next(self.gen)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) wait_for_engine_startup(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) raise RuntimeError(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) [2026-05-23 15:19:20,076 E 1909038 1909140] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=523482, ip=10.128.17.222) (EngineCore_DP0 pid=523842) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) (EngineCore_DP0 pid=523858) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 58x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) [W523 15:19:22.212466561 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-010-46.jupiter.internal]:58805 (errno: 97 - Address family not supported by protocol). [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) [W523 15:19:22.215869301 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) [rank0]:[W523 15:19:22.220727209 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) (EngineCore_DP0 pid=523862) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:11, 3.75s/it]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) (EngineCore_DP0 pid=3123723) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) No module named 'vllm._version' [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3294996, ip=10.128.17.219) (EngineCore_DP0 pid=3295247)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) (EngineCore_DP0 pid=200983) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3123494, ip=10.128.17.215) (EngineCore_DP0 pid=3123731) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:22.722 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:22.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:22.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) (EngineCore_DP0 pid=3222586) _C._set_float32_matmul_precision(precision)
(pid=3659917) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=200756, ip=10.128.17.221) (EngineCore_DP0 pid=200991) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3123493, ip=10.128.17.215) (EngineCore_DP0 pid=3123736) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:23.980 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) 2026-05-23 15:19:24.038 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 1.65s (attempt 1/5) to avoid port collisions [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3222352, ip=10.128.17.212) (EngineCore_DP0 pid=3222606) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3123492, ip=10.128.17.215) (EngineCore_DP0 pid=3123744) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) [2026-05-23 15:19:25,287 E 200625 200728] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3222353, ip=10.128.17.212) (EngineCore_DP0 pid=3222598) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=372387, ip=10.128.17.220) (EngineCore_DP0 pid=372619) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) (EngineCore_DP0 pid=1909400) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) [W523 15:19:27.784140679 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-010-40-interconnect-1.jupiter.internal]:45605 (errno: 97 - Address family not supported by protocol). [repeated 19x across cluster]
(AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) [W523 15:19:27.784665083 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 19x across cluster]
(AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) [rank0]:[W523 15:19:27.786653454 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=200757, ip=10.128.17.221) (EngineCore_DP0 pid=200995) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.15s/it] [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) (EngineCore_DP0 pid=372624) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) No module named 'vllm._version' [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=372258, ip=10.128.17.220) (EngineCore_DP0 pid=372628) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=1440679, ip=10.128.24.53) (EngineCore_DP0 pid=1440908) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) (EngineCore_DP0 pid=372633) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) (EngineCore_DP0 pid=1440912) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) (EngineCore_DP0 pid=1440912) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 18x across cluster]
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:30.331 | INFO  | skyrl_train.workers.worker:_initiate_actors:498 - Initialized process group for RayActorGroup
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:30.341 | INFO  | skyrl_train.workers.worker:_initiate_actors:500 - Mesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1)]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) `torch_dtype` is deprecated! Use `dtype` instead!
(FSDPPolicyWorkerBase pid=3659918)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217)
Loading checkpoint shards: 50%|█████ | 2/4 [00:00<00:00, 16.35it/s]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217)
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 18.60it/s]
(FSDPPolicyWorkerBase pid=3659915)
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 20.16it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 25.72it/s]
(FSDPPolicyWorkerBase pid=3659916)
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 15.36it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 15.01it/s]
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2172645, ip=10.128.24.52) [2026-05-23 15:19:31,089 E 2172645 2172743] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2172645, ip=10.128.24.52) (EngineCore_DP0 pid=2173002) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) [W523 15:19:32.269254945 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-022-05.jupiter.internal]:39961 (errno: 97 - Address family not supported by protocol). [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) [W523 15:19:32.269727257 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) [rank0]:[W523 15:19:32.271950002 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197)
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201)
(AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192)
(AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216)
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.38s/it] [repeated 42x across cluster]
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) (EngineCore_DP0 pid=3295221)
(AsyncVLLMInferenceEngine pid=3294998, ip=10.128.17.219) (EngineCore_DP0 pid=3295229)
(AsyncVLLMInferenceEngine pid=3294996, ip=10.128.17.219) (EngineCore_DP0 pid=3295247)
(AsyncVLLMInferenceEngine pid=2172775, ip=10.128.24.52) (EngineCore_DP0 pid=2173018) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=3659917) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=3659915)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s] [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=3659917)
Loading checkpoint shards: 50%|█████ | 2/4 [00:00<00:00, 11.90it/s] [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(FSDPPolicyWorkerBase pid=2753509, ip=10.128.17.217)
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 17.05it/s] [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=3659917)
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 14.54it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 14.07it/s] [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) 2026-05-23 15:19:36.781 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1156 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=1503353, ip=10.128.24.50) [2026-05-23 15:19:35,200 E 1503353 1503520] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) (EngineCore_DP0 pid=200983)
(AsyncVLLMInferenceEngine pid=200756, ip=10.128.17.221) (EngineCore_DP0 pid=200991)
(AsyncVLLMInferenceEngine pid=200757, ip=10.128.17.221) (EngineCore_DP0 pid=200995)
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001)
(AsyncVLLMInferenceEngine pid=1503353, ip=10.128.24.50) (EngineCore_DP0 pid=1503586) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) [W523 15:19:35.878750956 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-022-02.jupiter.internal]:55605 (errno: 97 - Address family not supported by protocol). [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) [W523 15:19:35.883184374 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) [rank0]:[W523 15:19:35.885823523 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.05s/it] [repeated 42x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1503355, ip=10.128.24.50) (EngineCore_DP0 pid=1503590) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=523482, ip=10.128.17.222) (EngineCore_DP0 pid=523842)
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) (EngineCore_DP0 pid=523846)
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) (EngineCore_DP0 pid=523858)
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) (EngineCore_DP0 pid=523862)
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) (EngineCore_DP0 pid=3123723)
(AsyncVLLMInferenceEngine pid=3123493, ip=10.128.17.215) (EngineCore_DP0 pid=3123736)
(AsyncVLLMInferenceEngine pid=3123492, ip=10.128.17.215) (EngineCore_DP0 pid=3123744)
(AsyncVLLMInferenceEngine pid=3123494, ip=10.128.17.215) (EngineCore_DP0 pid=3123731)
(AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1503355, ip=10.128.24.50) (EngineCore_DP0 pid=1503590)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) 2026-05-23 15:19:40.963 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1156 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 10x across cluster]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [2026-05-23 15:19:38,106 E 2753436 2753476] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396)
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420)
(AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) (EngineCore_DP0 pid=1909400)
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404)
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) [W523 15:19:40.328829609 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-010-38-interconnect-1.jupiter.internal]:50715 (errno: 97 - Address family not supported by protocol). [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) [W523 15:19:40.329285632 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=2753509, ip=10.128.17.217) [rank2]:[W523 15:19:42.418333463 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2172645, ip=10.128.24.52) (EngineCore_DP0 pid=2173002)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.16s/it] [repeated 57x across cluster]
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.040 | INFO  | skyrl_train.trainer:build_models:802 - init policy/ref/critic models done
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.042 | INFO  | harbor.orchestrators.queue:start:262 - [terminal_bench_generator:236] Started 900 workers (status every 120.0s, 0.75s launch grace period)
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.042 | INFO  | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:257 - QueueOrchestrator created and started with n_concurrent_trials=900, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.042 | INFO  | examples.terminal_bench.terminal_bench_generator:startup:226 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=900
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.043 | INFO  | skyrl_train.fully_async_trainer:train:428 - Generator startup complete
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.043 | INFO  | skyrl_train.fully_async_trainer:_train_loop:453 - Started: 'load_checkpoints'
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.088 | INFO  | skyrl_train.trainer:load_checkpoints:1595 - Loading checkpoint from: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.088 | INFO  | skyrl_train.trainer:load_checkpoints:1601 - Resuming from global_step: 18
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.128 | INFO  | skyrl_train.trainer:load_checkpoints:1617 - Successfully loaded trainer state
(AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 6x across cluster]
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.171 | INFO  | skyrl_train.trainer:load_checkpoints:1627 - Successfully loaded dataloader state
(skyrl_entrypoint pid=3659716) 2026-05-23 15:19:45.171 | INFO  | skyrl_train.trainer:load_checkpoints:1636 - Loading policy checkpoint from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy
(AsyncVLLMInferenceEngine pid=372258, ip=10.128.17.220) (EngineCore_DP0 pid=372628)
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) (EngineCore_DP0 pid=372633)
(AsyncVLLMInferenceEngine pid=372387, ip=10.128.17.220) (EngineCore_DP0 pid=372619)
(AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) (EngineCore_DP0 pid=372624)
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) 2026-05-23 15:19:46.919 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1156 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918)
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) (EngineCore_DP0 pid=1440912)
(AsyncVLLMInferenceEngine pid=1440679, ip=10.128.24.53) (EngineCore_DP0 pid=1440908)
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929)
(AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605)
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581)
(AsyncVLLMInferenceEngine pid=1503355, ip=10.128.24.50) (EngineCore_DP0 pid=1503590)
(AsyncVLLMInferenceEngine pid=1503353, ip=10.128.24.50) (EngineCore_DP0 pid=1503586)
(AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) (EngineCore_DP0 pid=3222586)
(AsyncVLLMInferenceEngine pid=3222352, ip=10.128.17.212) (EngineCore_DP0 pid=3222606)
(AsyncVLLMInferenceEngine pid=3222353, ip=10.128.17.212) (EngineCore_DP0 pid=3222598)
(AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582)
(FSDPPolicyWorkerBase pid=3659915) [2026-05-23 15:19:48,908 E 3659915 3660061] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:19<00:00, 4.98s/it] [repeated 59x across cluster]
(AsyncVLLMInferenceEngine pid=2172645, ip=10.128.24.52) (EngineCore_DP0 pid=2173002)
(AsyncVLLMInferenceEngine pid=2172775, ip=10.128.24.52) (EngineCore_DP0 pid=2173018)
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022)
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007)
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) 2026-05-23 15:19:51.235 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1156 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 11x across cluster]
(FSDPPolicyWorkerBase pid=3659917) [2026-05-23 15:19:49,079 E 3659917 3660145] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:13<00:04, 4.43s/it] [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003)
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996)
(AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998)
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990)
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) 2026-05-23 15:19:53.103 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1156 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:15<00:00, 3.76s/it] [repeated 11x across cluster]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) warnings.warn( # warn only once
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:03.601 | INFO  | skyrl_train.trainer:load_checkpoints:1646 - Successfully loaded policy checkpoint
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:03.601 | INFO  | skyrl_train.trainer:load_checkpoints:1662 - Successfully loaded complete checkpoint state from global_step_18
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:03.601 | INFO  | skyrl_train.fully_async_trainer:_train_loop:455 - Resumed training from global_step 18
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:03.623 | INFO  | skyrl_train.utils.data_tracker:load_state:97 - Loaded data tracker state: epoch=0, consumed_in_epoch=1144, total_consumed=1144
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:03.624 | WARNING  | skyrl_train.fully_async_trainer:_train_loop:480 - Data consumption count mismatch on resume: expected 1152, got 1144. This can happen after epoch boundary transitions or error recovery.
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:03.624 | INFO  | skyrl_train.fully_async_trainer:_train_loop:453 - Finished: 'load_checkpoints', time cost: 18.58s
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:03.624 | INFO  | skyrl_train.fully_async_trainer:_train_loop:487 - Started: 'init_weight_sync_state'
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) No module named 'vllm._version'
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) from .version import __version__, __version_tuple__ # isort:skip
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:20:09.761 | INFO  | logging:info:2216 - [weight-sync] Using master_addr=10.128.17.217, master_port=47881
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:20:00.782 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1156 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:20:09.760 | INFO  | logging:info:2216 - [weight-sync] get_node_ip_address()=10.128.17.217 [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=3659917) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=3659917) No module named 'vllm._version' [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=3659917) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:09.842 | INFO  | skyrl_train.fully_async_trainer:_train_loop:487 - Finished: 'init_weight_sync_state', time cost: 6.22s
(skyrl_entrypoint pid=3659716) --- Logging error in Loguru Handler #2 ---
(skyrl_entrypoint pid=3659716) Record was: {'elapsed': datetime.timedelta(seconds=212, microseconds=858264), 'exception': (type=<class 'ray.exceptions.RayTaskError(ActorDiedError)'>, value=RayTaskError(ActorDiedError)(ActorDiedError(RayTaskError('__init__', 'Traceback (most recent call last):\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n File "python/ray/_raylet.pyx", line 4329, in async_func\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\n return method(__ray_actor, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\n return method(self, *_args, **_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__\n super().__init__(*args, **kwargs)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__\n self.llm = self._create_engine(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\n return method(self, *_args, **_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\n return cls(\n ^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\n self.engine_core = EngineCoreClient.make_async_mp_client(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\n return AsyncMPClient(*client_args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\n super().__init__(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\n next(self.gen)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\n wait_for_engine_startup(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\n raise RuntimeError(\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\n', RuntimeError('Engine core initialization failed. See root cause above. Failed core proc(s): {}'), 'ray::AsyncVLLMInferenceEngine.__init__', None, None))), traceback=<traceback object at 0x400c3b492980>), 'extra': {}, 'file': (name='fully_async_trainer.py', path='/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py'), 'function': 'train', 'level': (name='ERROR', no=40, icon='❌'), 'line': 436, 'message': 'Train loop failed at global_step 18: \x1b[36mray::FSDPPolicyWorkerBase.init_weight_sync_state()\x1b[39m (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400df41fe420>)\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state\n results = await asyncio.gather(*tasks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator\n rank_offset_count += engine.tp_size() * engine.pp_size()\n ^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size\n return ray.get(self.inference_engine_actor.tp_size.remote())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^\nray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, \x1b[36mray::AsyncVLLMInferenceEngine.__init__()\x1b[39m (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400bfefd3080>)\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__\n super().__init__(*args, **kwargs)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__\n self.llm = self._create_engine(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\n return cls(\n ^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\n self.engine_core = EngineCoreClient.make_async_mp_client(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\n return AsyncMPClient(*client_args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\n super().__init__(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\n next(self.gen)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\n wait_for_engine_startup(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\n raise RuntimeError(\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}', 'module': 'fully_async_trainer', 'name': 'skyrl_train.fully_async_trainer', 'process': (id=3659716, name='MainProcess'), 'thread': (id=70368994352832, name='MainThread'), 'time': datetime(2026, 5, 23, 15, 20, 9, 843010, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200), 'CEST'))}
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank0]:[W523 15:20:09.372907026 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-010-41.jupiter.internal]:47881 (errno: 97 - Address family not supported by protocol).
(skyrl_entrypoint pid=3659716) Traceback (most recent call last):
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train
(skyrl_entrypoint pid=3659716) await self._train_loop()
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop
(skyrl_entrypoint pid=3659716) self.init_weight_sync_state()
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state
(skyrl_entrypoint pid=3659716) ray.get(
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(skyrl_entrypoint pid=3659716) return fn(*args, **kwargs)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
(skyrl_entrypoint pid=3659716) return func(*args, **kwargs)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
(skyrl_entrypoint pid=3659716) values, debugger_breakpoint = worker.get_objects(
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
(skyrl_entrypoint pid=3659716) raise value.as_instanceof_cause()
(skyrl_entrypoint pid=3659716) ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400df41fe420>)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(skyrl_entrypoint pid=3659716) return self.__get_result()
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(skyrl_entrypoint pid=3659716) raise self._exception
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state
(skyrl_entrypoint pid=3659716) results = await asyncio.gather(*tasks)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator
(skyrl_entrypoint pid=3659716) rank_offset_count += engine.tp_size() * engine.pp_size()
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size
(skyrl_entrypoint pid=3659716) return ray.get(self.inference_engine_actor.tp_size.remote())
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400bfefd3080>)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(skyrl_entrypoint pid=3659716) return self.__get_result()
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(skyrl_entrypoint pid=3659716) raise self._exception
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
(skyrl_entrypoint pid=3659716) super().__init__(*args, **kwargs)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
(skyrl_entrypoint pid=3659716) self.llm = self._create_engine(*args, **kwargs)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine
(skyrl_entrypoint pid=3659716) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(skyrl_entrypoint pid=3659716) return cls(
(skyrl_entrypoint pid=3659716) ^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(skyrl_entrypoint pid=3659716) self.engine_core = EngineCoreClient.make_async_mp_client(
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(skyrl_entrypoint pid=3659716) return AsyncMPClient(*client_args)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(skyrl_entrypoint pid=3659716) super().__init__(
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(skyrl_entrypoint pid=3659716) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(skyrl_entrypoint pid=3659716) next(self.gen)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(skyrl_entrypoint pid=3659716) wait_for_engine_startup(
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(skyrl_entrypoint pid=3659716) raise RuntimeError(
(skyrl_entrypoint pid=3659716) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(skyrl_entrypoint pid=3659716)
(skyrl_entrypoint pid=3659716) During handling of the above exception, another exception occurred:
(skyrl_entrypoint pid=3659716)
(skyrl_entrypoint pid=3659716) Traceback (most recent call last):
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/loguru/_handler.py", line 204, in emit
(skyrl_entrypoint pid=3659716) self._queue.put(str_record)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/queues.py", line 393, in put
(skyrl_entrypoint pid=3659716) obj = _ForkingPickler.dumps(obj)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
(skyrl_entrypoint pid=3659716) cls(buf, protocol).dump(obj)
(skyrl_entrypoint pid=3659716) _pickle.PicklingError: Can't pickle <class 'ray.exceptions.RayTaskError(ActorDiedError)'>: attribute lookup RayTaskError(ActorDiedError) on ray.exceptions failed
(skyrl_entrypoint pid=3659716) --- End of logging error ---
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.907 | INFO  | skyrl_train.inference_engines.inference_engine_client_http_endpoint:shutdown_server:203 - Server shut down after 2 seconds
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.908 | INFO  | skyrl_train.trainer:_guarded_sync:226 - HTTP endpoint shutdown complete
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.908 | INFO  | examples.terminal_bench.terminal_bench_generator:shutdown:328 - Shutting down shared QueueOrchestrator...
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.910 | INFO  | harbor.orchestrators.queue:shutdown:377 - [terminal_bench_generator:236] Shutdown complete. Total completed: 0
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.910 | INFO  | examples.terminal_bench.terminal_bench_generator:shutdown:330 - QueueOrchestrator shutdown complete
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.910 | INFO  | skyrl_train.trainer:_guarded_async:215 - Generator shutdown complete
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.920 | WARNING  | skyrl_train.trainer:_guarded_async:219 - Inference engine teardown error (non-fatal): The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400bfefd3080>)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(skyrl_entrypoint pid=3659716) return self.__get_result()
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(skyrl_entrypoint pid=3659716) raise self._exception
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
(skyrl_entrypoint pid=3659716) super().__init__(*args, **kwargs)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
(skyrl_entrypoint pid=3659716) self.llm = self._create_engine(*args, **kwargs)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine
(skyrl_entrypoint pid=3659716) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(skyrl_entrypoint pid=3659716) return cls(
(skyrl_entrypoint pid=3659716) ^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(skyrl_entrypoint pid=3659716) self.engine_core = EngineCoreClient.make_async_mp_client(
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(skyrl_entrypoint pid=3659716) return AsyncMPClient(*client_args)
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(skyrl_entrypoint pid=3659716) super().__init__(
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(skyrl_entrypoint pid=3659716) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(skyrl_entrypoint pid=3659716) next(self.gen)
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(skyrl_entrypoint pid=3659716) wait_for_engine_startup(
(skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(skyrl_entrypoint pid=3659716) raise RuntimeError(
(skyrl_entrypoint pid=3659716) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.920 | INFO  | skyrl_train.trainer:_kill_ray_actors:248 - Killing policy_model actors...
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) (EngineCore_DP0 pid=3295221) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) (EngineCore_DP0 pid=3295221) warnings.warn("No model update group to destroy")
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.936 | INFO  | skyrl_train.trainer:_kill_ray_actors:267 - Killed 48 inference engine actor(s)
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.936 | INFO  | skyrl_train.trainer:_guarded_sync:226 - Ray actor cleanup complete
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.937 | INFO  | skyrl_train.trainer:_kill_ray_actors:248 - Killing policy_model actors...
(skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.941 | INFO  | skyrl_train.trainer:_kill_ray_actors:267 - Killed 48 inference engine actor(s)
2026-05-23 15:20:10.953 | ERROR | __main__:main:125 - Training failed: ray::skyrl_entrypoint() (pid=3659716, ip=10.128.17.210)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 96, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 403, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state
ray.get(
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400df41fe420>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state
results = await asyncio.gather(*tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator
rank_offset_count += engine.tp_size() * engine.pp_size()
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size
return ray.get(self.inference_engine_actor.tp_size.remote())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400bfefd3080>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
super().__init__(*args, **kwargs)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
self.llm = self._create_engine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine
engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
return cls(
^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
self.engine_core = EngineCoreClient.make_async_mp_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
return AsyncMPClient(*client_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
super().__init__(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
with launch_core_engines(vllm_config, executor_class, log_stats) as (
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
wait_for_engine_startup(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
2026-05-23 15:20:10.954 | INFO | __main__:main:128 - Shutting down Ray on head node...
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy [repeated 37x across cluster]
(AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420) warnings.warn("No model update group to destroy") [repeated 37x across cluster]
(skyrl_entrypoint pid=3659716) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:02 [arg_utils.py:1256] The global random seed is set to 42. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:03 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:03 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:03 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) WARNING 05-23 15:19:08 [arg_utils.py:1256] The global random seed is set to 47. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [model.py:1549] Using max model len 32768 [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) WARNING 05-23 15:19:08 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) WARNING 05-23 15:19:07 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) WARNING 05-23 15:19:07 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) INFO 05-23 15:19:12 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=45, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [arg_utils.py:1256] The global random seed is set to 62. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [model.py:1549] Using max model len 32768 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) INFO 05-23 15:19:15 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) (EngineCore_DP0 pid=523846) INFO 05-23 15:19:17 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=48, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) INFO 05-23 15:19:17 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.24.51:59091 backend=nccl
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:17 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(pid=2753436, ip=10.128.17.217) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:18 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) WARNING 05-23 15:19:18 [arg_utils.py:1256] The global random seed is set to 70. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [model.py:1549] Using max model len 32768 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) WARNING 05-23 15:19:18 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) WARNING 05-23 15:19:18 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) WARNING 05-23 15:19:18 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:19 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] EngineCore failed to start.
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] super().__init__(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self._init_executor()
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] init_distributed_environment(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] return TCPStore(
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 34697, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40003896c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #1: <unknown function> + 0x5f29220 (0x400018b09220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #2: <unknown function> + 0x5f4326c (0x400018b2326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #3: <unknown function> + 0x5f49074 (0x400018b29074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #4: <unknown function> + 0x5f49138 (0x400018b29138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #5: <unknown function> + 0x5f2ccc4 (0x400018b0ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x400018b11564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #7: <unknown function> + 0x109a094 (0x4000128ba094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #8: <unknown function> + 0x113236c (0x40001295236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #9: <unknown function> + 0x5d6d60 (0x400011df6d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #10: <unknown function> + 0x1b7a38 (0xaaaaca047a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #12: <unknown function> + 0x169f50 (0xaaaac9ff9f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #13: <unknown function> + 0x1682e4 (0xaaaac9ff82e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #14: <unknown function> + 0x1e0ce8 (0xaaaaca070ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #15: <unknown function> + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #16: <unknown function> + 0x646b0c (0x400011e66b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #19: <unknown function> + 0x1808c0 (0xaaaaca0108c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #20: <unknown function> + 0x182bf8 (0xaaaaca012bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #21: <unknown function> + 0x25fd30 (0xaaaaca0efd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #22: <unknown function> + 0x1b7d20 (0xaaaaca047d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #23: PyObject_Vectorcall + 0x54 (0xaaaac9ff60e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaac9ff7f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #27: <unknown function> + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #28: <unknown function> + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaac9ff7fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #33: <unknown function> + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #34: <unknown function> + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #35: _PyObject_Call + 0x68 (0xaaaac9ff8488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaaca0fd8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #37: PyEval_EvalCode + 0xb4 (0xaaaaca102eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #38: <unknown function> + 0x2ccdcc (0xaaaaca15cdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #39: <unknown function> + 0x2ccef4 (0xaaaaca15cef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #40: PyRun_StringFlags + 0x90 (0xaaaaca161050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaaca1610f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #42: Py_RunMain + 0x2c8 (0xaaaaca189190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #43: Py_BytesMain + 0x64 (0xaaaaca189fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #44: <unknown function> + 0x27540 (0x400000e67540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #45: __libc_start_main + 0x98 (0x400000e67618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #46: <unknown function> + 0x10e0c0 (0xaaaac9f9e0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006]
(AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) INFO 05-23 15:19:19 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582) INFO 05-23 15:19:22 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=61, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) (EngineCore_DP0 pid=523858) INFO 05-23 15:19:22 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.17.222:58805 backend=nccl [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) (EngineCore_DP0 pid=523858) INFO 05-23 15:19:22 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) (EngineCore_DP0 pid=523862) INFO 05-23 15:19:23 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [arg_utils.py:1256] The global random seed is set to 76. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [model.py:1549] Using max model len 32768 [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=3294996, ip=10.128.17.219) (EngineCore_DP0 pid=3295247) INFO 05-23 15:19:21 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582) INFO 05-23 15:19:24 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 8x across cluster]
(pid=2753509, ip=10.128.17.217) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) (EngineCore_DP0 pid=1440912) INFO 05-23 15:19:27 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=72, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:27 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.17.216:50115 backend=nccl [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:27 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [arg_utils.py:1256] The global random seed is set to 86. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [model.py:1549] Using max model len 32768 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:29 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) (EngineCore_DP0 pid=1440912) INFO 05-23 15:19:30 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 18x across cluster]
(pid=3659917) ⚙️ Running in WANDB offline mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:31 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=83, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918) INFO 05-23 15:19:32 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.24.53:39961 backend=nccl [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:32 [default_loader.py:293] Loading weights took 13.48 seconds
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918) INFO 05-23 15:19:32 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:33 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.270659 seconds
(AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918) INFO 05-23 15:19:32 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929) INFO 05-23 15:19:33 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:35 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:35 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:35 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:34 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:36 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216) INFO 05-23 15:19:36 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.76 seconds
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) WARNING 05-23 15:19:36 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:36 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) WARNING 05-23 15:19:36 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:36 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) WARNING 05-23 15:19:36 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:37 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=87, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.24.50:55605 backend=nccl [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:37 [default_loader.py:293] Loading weights took 8.24 seconds [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:37 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 8.907134 seconds [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:35 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:39 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:39 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:39 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998) INFO 05-23 15:19:39 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=200757, ip=10.128.17.221) (EngineCore_DP0 pid=200995) INFO 05-23 15:19:40 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:40 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) WARNING 05-23 15:19:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:40 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) WARNING 05-23 15:19:40 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:40 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) WARNING 05-23 15:19:40 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 10x across cluster]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) NCCL version 2.27.7+cuda13.0
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/model_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/extra_state_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/optim_world_size_8_rank_0.pt
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:40 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.17.214:50715 backend=nccl [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:43 [default_loader.py:293] Loading weights took 13.72 seconds [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:40 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:43 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.989906 seconds [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:41 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998) INFO 05-23 15:19:41 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:45 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:45 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:45 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:40 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.73 seconds [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) WARNING 05-23 15:19:46 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) WARNING 05-23 15:19:46 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=3123494, ip=10.128.17.215) WARNING 05-23 15:19:42 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:49 [default_loader.py:293] Loading weights took 13.85 seconds [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:49 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.998130 seconds [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605) INFO 05-23 15:19:51 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605) INFO 05-23 15:19:51 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605) INFO 05-23 15:19:51 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929) INFO 05-23 15:19:50 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929) INFO 05-23 15:19:50 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) WARNING 05-23 15:19:51 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:51 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) WARNING 05-23 15:19:51 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:51 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) WARNING 05-23 15:19:51 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:56 [default_loader.py:293] Loading weights took 14.98 seconds
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:52 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.78 seconds [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:56 [default_loader.py:293] Loading weights took 14.91 seconds
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:57 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 16.066204 seconds
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) WARNING 05-23 15:19:52 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:52 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) WARNING 05-23 15:19:52 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:52 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:53 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 12x across cluster]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Successfully loaded model state dict
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:59 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:59 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:59 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:20:00 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:20:00 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:19:56 [default_loader.py:293] Loading weights took 15.06 seconds [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:19:57 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 16.070195 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) WARNING 05-23 15:20:00 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:20:00 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) WARNING 05-23 15:20:00 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:20:00 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) WARNING 05-23 15:20:00 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Successfully loaded optimizer state
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Successfully loaded scheduler state
(FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy
(AsyncVLLMInferenceEngine pid=523482, ip=10.128.17.222) ERROR 05-23 15:20:10 [core_client.py:616] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
(AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) ERROR 05-23 15:20:10 [core_client.py:616] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client. [repeated 12x across cluster]
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=token_mean', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/a3-rl-DCAgent_code-contests-noblock', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=true', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=a3-rl-DCAgent_code-contests-noblock', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=450', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/code-contests-noblock"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=1800', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=900', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.model_info.max_input_tokens=32768', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 133, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 123, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ActorDiedError): ray::skyrl_entrypoint() (pid=3659716, ip=10.128.17.210)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 96, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 403, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state
ray.get(
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400df41fe420>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state
results = await asyncio.gather(*tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator
rank_offset_count += engine.tp_size() * engine.pp_size()
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size
return ray.get(self.inference_engine_actor.tp_size.remote())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400bfefd3080>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
super().__init__(*args, **kwargs)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
self.llm = self._create_engine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine
engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
return cls(
^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
self.engine_core = EngineCoreClient.make_async_mp_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
return AsyncMPClient(*client_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
super().__init__(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
with launch_core_engines(vllm_config, executor_class, log_stats) as (
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
wait_for_engine_startup(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Stopping Ray cluster...
Ray cluster stopped
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/a3-rl-DCAgent_code-contests-noblock
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock
episodes: last
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/logs/a3-rl-DCAgent_code-contests-noblock_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/ray_logs/
Collecting Ray logs from worker jpbo-010-36...
Collecting Ray logs from worker jpbo-010-38...
Collecting Ray logs from worker jpbo-010-39...
Collecting Ray logs from worker jpbo-010-40...
Collecting Ray logs from worker jpbo-010-41...
Collecting Ray logs from worker jpbo-010-43...
Collecting Ray logs from worker jpbo-010-44...
Collecting Ray logs from worker jpbo-010-45...
Collecting Ray logs from worker jpbo-010-46...
Collecting Ray logs from worker jpbo-022-02...
Collecting Ray logs from worker jpbo-022-03...
Collecting Ray logs from worker jpbo-022-04...
Collecting Ray logs from worker jpbo-022-05...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink