laion/a3-rl-DCAgent_code-contests-noblock-5-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
a3-rl-DCAgent_code-contests…/training_logs/a3-rl-DCAgent_code-contests-noblock_485102.out
ModelHub XC a508caa68d 初始化项目，由ModelHub XC社区提供模型 
Model: laion/a3-rl-DCAgent_code-contests-noblock-5-8B
Source: Original Platform
2026-08-07 12:29:36 +08:00

1631 lines
268 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_485102
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_485102
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_485102
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.43.81 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_485102.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_485102.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.43.81:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/configs/a3-rl-DCAgent_code-contests-noblock_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: a3-rl-DCAgent_code-contests-noblock ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-078-33 (10.128.43.81)
Ray port: 6379
============================
Starting Ray head on jpbo-078-33 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-078-33.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.81 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.43.81 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-078-33
Starting Ray worker on jpbo-078-36 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.84 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.84 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-078-36
Starting Ray worker on jpbo-078-37 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.85 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.85 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-078-37
Starting Ray worker on jpbo-078-38 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.86 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.86 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-078-38
Starting Ray worker on jpbo-078-39 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.87 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.87 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-078-39
Starting Ray worker on jpbo-078-40 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-40.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.88 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.88 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-078-40
Starting Ray worker on jpbo-078-42 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.90 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.90 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-078-42
Starting Ray worker on jpbo-078-43 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.91 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.91 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-078-43
Starting Ray worker on jpbo-078-45 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.93 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.93 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-078-45
Starting Ray worker on jpbo-078-46 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.94 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.94 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-078-46
Starting Ray worker on jpbo-078-47 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-47.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.95 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.95 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-078-47
Starting Ray worker on jpbo-078-48 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-078-48.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.96 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-078-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.96 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-078-48
Starting Ray worker on jpbo-079-07 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-079-07.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.103 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-079-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.103 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-079-07
Starting Ray worker on jpbo-079-12 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-079-12.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.43.108 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-079-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.43.81:6379 --node-ip-address=10.128.43.108 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-079-12
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.43.81:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'memory': 10624069140480.0, 'GPU': 56.0, 'node:__internal_head__': 1.0, 'object_store_memory': 601295421440.0, 'node:10.128.43.81': 1.0, 'CPU': 4032.0, 'accelerator_type:GH200': 14.0, 'node:10.128.43.96': 1.0, 'node:10.128.43.84': 1.0, 'node:10.128.43.90': 1.0, 'node:10.128.43.103': 1.0, 'node:10.128.43.85': 1.0, 'node:10.128.43.86': 1.0, 'node:10.128.43.95': 1.0, 'node:10.128.43.87': 1.0, 'node:10.128.43.93': 1.0, 'node:10.128.43.94': 1.0, 'node:10.128.43.88': 1.0, 'node:10.128.43.91': 1.0, 'node:10.128.43.108': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
Address: 10.128.43.81:6379
Total GPUs: 56
=========================
Ray cluster ready at 10.128.43.81:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
Running SkyRL:
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
Args: 119 Hydra arguments
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=token_mean trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/a3-rl-DCAgent_code-contests-noblock ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=true trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=a3-rl-DCAgent_code-contests-noblock trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=450 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/code-contests-noblock"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=1800 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=900 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.model_info.max_input_tokens=32768 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_485102.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-05-23 01:51:48.647 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:590 - Exporting wandb api key to ray runtime env
2026-05-23 01:51:48.648 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:609 - Exporting RAY_ADDRESS to ray runtime env
2026-05-23 01:51:48,648 INFO worker.py:1680 -- Using address 10.128.43.81:6379 set in the environment variable RAY_ADDRESS
2026-05-23 01:51:48,679 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.43.81:6379...
2026-05-23 01:51:48,689 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.43.87) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-05-23 01:51:51.188 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
(skyrl_entrypoint pid=1461281) 2026-05-23 01:52:01.959 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:199 - SkyRL log level set to: INFO
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(skyrl_entrypoint pid=1461281) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=1461281) 2026-05-23 01:52:02.310 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/code-contests-noblock
[2026-05-23 01:52:18,917 E 1460804 1461255] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RegistryActor pid=1657870, ip=10.128.43.87) [2026-05-23 01:52:19,865 E 1657870 1657910] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(skyrl_entrypoint pid=1461281) 2026-05-23 01:53:55.602 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 8728 valid task directories out of 8728 total directories
(skyrl_entrypoint pid=1461281) 2026-05-23 01:53:55.602 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 8728 task paths
(skyrl_entrypoint pid=1461281) 2026-05-23 01:53:55.613 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:352 - data:
(skyrl_entrypoint pid=1461281) train_data:
(skyrl_entrypoint pid=1461281) - /e/scratch/jureap59/feuer1/tasks/code-contests-noblock
(skyrl_entrypoint pid=1461281) val_data: []
(skyrl_entrypoint pid=1461281) trainer:
(skyrl_entrypoint pid=1461281) placement:
(skyrl_entrypoint pid=1461281) colocate_all: false
(skyrl_entrypoint pid=1461281) colocate_policy_ref: true
(skyrl_entrypoint pid=1461281) policy_num_nodes: 2
(skyrl_entrypoint pid=1461281) policy_num_gpus_per_node: 4
(skyrl_entrypoint pid=1461281) critic_num_nodes: 1
(skyrl_entrypoint pid=1461281) critic_num_gpus_per_node: 4
(skyrl_entrypoint pid=1461281) ref_num_nodes: 2
(skyrl_entrypoint pid=1461281) ref_num_gpus_per_node: 4
(skyrl_entrypoint pid=1461281) sequence_parallel_backend: ulysses
(skyrl_entrypoint pid=1461281) strategy: fsdp2
(skyrl_entrypoint pid=1461281) policy:
(skyrl_entrypoint pid=1461281) model:
(skyrl_entrypoint pid=1461281) path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=1461281) lora:
(skyrl_entrypoint pid=1461281) rank: 0
(skyrl_entrypoint pid=1461281) alpha: 16
(skyrl_entrypoint pid=1461281) dropout: 0
(skyrl_entrypoint pid=1461281) lora_sync_path: /tmp/skyrl_lora_sync
(skyrl_entrypoint pid=1461281) target_modules: all-linear
(skyrl_entrypoint pid=1461281) exclude_modules: null
(skyrl_entrypoint pid=1461281) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=1461281) optimizer_config:
(skyrl_entrypoint pid=1461281) optimizer: AdamW
(skyrl_entrypoint pid=1461281) lr: 8.0e-06
(skyrl_entrypoint pid=1461281) adam_betas:
(skyrl_entrypoint pid=1461281) - 0.9
(skyrl_entrypoint pid=1461281) - 0.999
(skyrl_entrypoint pid=1461281) weight_decay: 0.0
(skyrl_entrypoint pid=1461281) max_grad_norm: 0.9
(skyrl_entrypoint pid=1461281) offload_after_step: true
(skyrl_entrypoint pid=1461281) num_warmup_steps: 0
(skyrl_entrypoint pid=1461281) scheduler: constant_with_warmup
(skyrl_entrypoint pid=1461281) optimizer_kwargs: {}
(skyrl_entrypoint pid=1461281) fsdp_config:
(skyrl_entrypoint pid=1461281) cpu_offload: false
(skyrl_entrypoint pid=1461281) reshard_after_forward: true
(skyrl_entrypoint pid=1461281) fsdp_size: 4
(skyrl_entrypoint pid=1461281) sequence_parallel_size: 1
(skyrl_entrypoint pid=1461281) use_torch_compile: false
(skyrl_entrypoint pid=1461281) record_memory: false
(skyrl_entrypoint pid=1461281) megatron_config:
(skyrl_entrypoint pid=1461281) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=1461281) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=1461281) context_parallel_size: 1
(skyrl_entrypoint pid=1461281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=1461281) expert_tensor_parallel_size: null
(skyrl_entrypoint pid=1461281) ddp_config:
(skyrl_entrypoint pid=1461281) grad_reduce_in_fp32: true
(skyrl_entrypoint pid=1461281) overlap_grad_reduce: false
(skyrl_entrypoint pid=1461281) overlap_param_gather: false
(skyrl_entrypoint pid=1461281) average_in_collective: true
(skyrl_entrypoint pid=1461281) model_config_kwargs: {}
(skyrl_entrypoint pid=1461281) torch_profiler_config:
(skyrl_entrypoint pid=1461281) enable: false
(skyrl_entrypoint pid=1461281) ranks: []
(skyrl_entrypoint pid=1461281) save_path: null
(skyrl_entrypoint pid=1461281) optimizer_config_kwargs:
(skyrl_entrypoint pid=1461281) overlap_cpu_optimizer_d2h_h2d: false
(skyrl_entrypoint pid=1461281) use_precision_aware_optimizer: false
(skyrl_entrypoint pid=1461281) optimizer_cpu_offload: false
(skyrl_entrypoint pid=1461281) optimizer_offload_fraction: 0.0
(skyrl_entrypoint pid=1461281) transformer_config_kwargs:
(skyrl_entrypoint pid=1461281) recompute_granularity: full
(skyrl_entrypoint pid=1461281) recompute_modules:
(skyrl_entrypoint pid=1461281) - core_attn
(skyrl_entrypoint pid=1461281) recompute_method: uniform
(skyrl_entrypoint pid=1461281) recompute_num_layers: 1
(skyrl_entrypoint pid=1461281) empty_cuda_cache: true
(skyrl_entrypoint pid=1461281) ref:
(skyrl_entrypoint pid=1461281) model:
(skyrl_entrypoint pid=1461281) path: ${trainer.policy.model.path}
(skyrl_entrypoint pid=1461281) sequence_parallel_size: 1
(skyrl_entrypoint pid=1461281) deepspeed_config: ${deepspeed_config.eval}
(skyrl_entrypoint pid=1461281) fsdp_config:
(skyrl_entrypoint pid=1461281) cpu_offload: false
(skyrl_entrypoint pid=1461281) reshard_after_forward: true
(skyrl_entrypoint pid=1461281) fsdp_size: 4
(skyrl_entrypoint pid=1461281) megatron_config:
(skyrl_entrypoint pid=1461281) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=1461281) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=1461281) context_parallel_size: 1
(skyrl_entrypoint pid=1461281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=1461281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=1461281) model_config_kwargs: {}
(skyrl_entrypoint pid=1461281) transformer_config_kwargs: {}
(skyrl_entrypoint pid=1461281) critic:
(skyrl_entrypoint pid=1461281) model:
(skyrl_entrypoint pid=1461281) path: null
(skyrl_entrypoint pid=1461281) lora:
(skyrl_entrypoint pid=1461281) rank: 0
(skyrl_entrypoint pid=1461281) alpha: 16
(skyrl_entrypoint pid=1461281) dropout: 0
(skyrl_entrypoint pid=1461281) target_modules: all-linear
(skyrl_entrypoint pid=1461281) exclude_modules: null
(skyrl_entrypoint pid=1461281) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=1461281) optimizer_config:
(skyrl_entrypoint pid=1461281) optimizer: AdamW
(skyrl_entrypoint pid=1461281) lr: 5.0e-06
(skyrl_entrypoint pid=1461281) adam_betas:
(skyrl_entrypoint pid=1461281) - 0.9
(skyrl_entrypoint pid=1461281) - 0.999
(skyrl_entrypoint pid=1461281) weight_decay: 0.01
(skyrl_entrypoint pid=1461281) max_grad_norm: 1.0
(skyrl_entrypoint pid=1461281) offload_after_step: true
(skyrl_entrypoint pid=1461281) num_warmup_steps: 0
(skyrl_entrypoint pid=1461281) scheduler: constant_with_warmup
(skyrl_entrypoint pid=1461281) optimizer_kwargs: {}
(skyrl_entrypoint pid=1461281) fsdp_config:
(skyrl_entrypoint pid=1461281) cpu_offload: false
(skyrl_entrypoint pid=1461281) reshard_after_forward: true
(skyrl_entrypoint pid=1461281) fsdp_size: -1
(skyrl_entrypoint pid=1461281) sequence_parallel_size: 1
(skyrl_entrypoint pid=1461281) algorithm:
(skyrl_entrypoint pid=1461281) advantage_estimator: rloo_n
(skyrl_entrypoint pid=1461281) kl_ctrl:
(skyrl_entrypoint pid=1461281) type: fixed
(skyrl_entrypoint pid=1461281) kl_target: 0.1
(skyrl_entrypoint pid=1461281) horizon: 10000
(skyrl_entrypoint pid=1461281) kl_estimator_type: k3
(skyrl_entrypoint pid=1461281) use_kl_estimator_k3: false
(skyrl_entrypoint pid=1461281) use_abs_kl: false
(skyrl_entrypoint pid=1461281) use_kl_in_reward: false
(skyrl_entrypoint pid=1461281) use_kl_loss: false
(skyrl_entrypoint pid=1461281) kl_loss_coef: 0.0
(skyrl_entrypoint pid=1461281) use_entropy_loss: false
(skyrl_entrypoint pid=1461281) entropy_loss_coef: 0.01
(skyrl_entrypoint pid=1461281) advantage_batch_normalize: false
(skyrl_entrypoint pid=1461281) value_head_prefix: value_head
(skyrl_entrypoint pid=1461281) policy_loss_type: regular
(skyrl_entrypoint pid=1461281) loss_reduction: token_mean
(skyrl_entrypoint pid=1461281) grpo_norm_by_std: true
(skyrl_entrypoint pid=1461281) rloo_n_min_group_size: 4
(skyrl_entrypoint pid=1461281) rloo_n_filter_zero_reward_groups: true
(skyrl_entrypoint pid=1461281) lambd: 1.0
(skyrl_entrypoint pid=1461281) gamma: 1.0
(skyrl_entrypoint pid=1461281) eps_clip_low: 0.2
(skyrl_entrypoint pid=1461281) eps_clip_high: 0.05
(skyrl_entrypoint pid=1461281) clip_ratio_c: 3.0
(skyrl_entrypoint pid=1461281) tis_imp_ratio_cap: -1.0
(skyrl_entrypoint pid=1461281) use_tis: false
(skyrl_entrypoint pid=1461281) sapo:
(skyrl_entrypoint pid=1461281) tau_pos: 1.0
(skyrl_entrypoint pid=1461281) tau_neg: 1.05
(skyrl_entrypoint pid=1461281) value_clip: 0.2
(skyrl_entrypoint pid=1461281) dynamic_sampling:
(skyrl_entrypoint pid=1461281) type: null
(skyrl_entrypoint pid=1461281) max_sample_batches: 30
(skyrl_entrypoint pid=1461281) min_replace_ratio: 0.3
(skyrl_entrypoint pid=1461281) clip_cov:
(skyrl_entrypoint pid=1461281) clip_ratio: 0.0002
(skyrl_entrypoint pid=1461281) clip_cov_lb: 1.0
(skyrl_entrypoint pid=1461281) clip_cov_ub: 5.0
(skyrl_entrypoint pid=1461281) kl_cov:
(skyrl_entrypoint pid=1461281) kl_cov_frac: 0.2
(skyrl_entrypoint pid=1461281) ppo_kl_coef: 1.0
(skyrl_entrypoint pid=1461281) cispo:
(skyrl_entrypoint pid=1461281) cispo_eps_clip_low: 0
(skyrl_entrypoint pid=1461281) cispo_eps_clip_high: 5
(skyrl_entrypoint pid=1461281) z_clip:
(skyrl_entrypoint pid=1461281) enabled: false
(skyrl_entrypoint pid=1461281) alpha: 0.97
(skyrl_entrypoint pid=1461281) z_thresh: 2.5
(skyrl_entrypoint pid=1461281) warmup_steps: 25
(skyrl_entrypoint pid=1461281) mode: zscore
(skyrl_entrypoint pid=1461281) clip_option: adaptive_scaling
(skyrl_entrypoint pid=1461281) clip_factor: 1.0
(skyrl_entrypoint pid=1461281) skip_update_on_spike: false
(skyrl_entrypoint pid=1461281) stale_clip:
(skyrl_entrypoint pid=1461281) enabled: false
(skyrl_entrypoint pid=1461281) alpha: 0.3
(skyrl_entrypoint pid=1461281) entropy_threshold: 0.15
(skyrl_entrypoint pid=1461281) entropy_window: 10
(skyrl_entrypoint pid=1461281) min_lr_scale: 0.1
(skyrl_entrypoint pid=1461281) max_seq_len: 1004095
(skyrl_entrypoint pid=1461281) fully_async:
(skyrl_entrypoint pid=1461281) max_staleness_steps: 16
(skyrl_entrypoint pid=1461281) num_parallel_generation_workers: 450
(skyrl_entrypoint pid=1461281) gradient_checkpointing: true
(skyrl_entrypoint pid=1461281) gradient_checkpointing_use_reentrant: false
(skyrl_entrypoint pid=1461281) seed: 42
(skyrl_entrypoint pid=1461281) resume_mode: latest
(skyrl_entrypoint pid=1461281) resume_path: null
(skyrl_entrypoint pid=1461281) ckpt_path: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints
(skyrl_entrypoint pid=1461281) max_ckpts_to_keep: -1
(skyrl_entrypoint pid=1461281) ckpt_interval: 2
(skyrl_entrypoint pid=1461281) hf_save_interval: 5
(skyrl_entrypoint pid=1461281) export_path: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports
(skyrl_entrypoint pid=1461281) bf16: true
(skyrl_entrypoint pid=1461281) epochs: 2
(skyrl_entrypoint pid=1461281) max_steps: 80
(skyrl_entrypoint pid=1461281) update_epochs_per_batch: 1
(skyrl_entrypoint pid=1461281) train_batch_size: 64
(skyrl_entrypoint pid=1461281) policy_mini_batch_size: 64
(skyrl_entrypoint pid=1461281) critic_mini_batch_size: 256
(skyrl_entrypoint pid=1461281) micro_train_batch_size_per_gpu: 1
(skyrl_entrypoint pid=1461281) micro_forward_batch_size_per_gpu: 4
(skyrl_entrypoint pid=1461281) update_ref_every_epoch: false
(skyrl_entrypoint pid=1461281) use_sample_packing: true
(skyrl_entrypoint pid=1461281) eval_batch_size: 64
(skyrl_entrypoint pid=1461281) eval_before_train: false
(skyrl_entrypoint pid=1461281) eval_interval: 999999
(skyrl_entrypoint pid=1461281) max_prompt_length: 999999
(skyrl_entrypoint pid=1461281) flash_attn: true
(skyrl_entrypoint pid=1461281) disable_fast_tokenizer: false
(skyrl_entrypoint pid=1461281) target_modules: null
(skyrl_entrypoint pid=1461281) exclude_modules: null
(skyrl_entrypoint pid=1461281) project_name: OpenThoughts-Agent
(skyrl_entrypoint pid=1461281) run_name: a3-rl-DCAgent_code-contests-noblock
(skyrl_entrypoint pid=1461281) logger: console
(skyrl_entrypoint pid=1461281) tracker_commit_each_step: true
(skyrl_entrypoint pid=1461281) dump_data_batch: false
(skyrl_entrypoint pid=1461281) dump_eval_results: true
(skyrl_entrypoint pid=1461281) log_level: INFO
(skyrl_entrypoint pid=1461281) rope_scaling: null
(skyrl_entrypoint pid=1461281) rope_theta: null
(skyrl_entrypoint pid=1461281) step_wise_training: false
(skyrl_entrypoint pid=1461281) hf_hub_repo_id: laion/a3-rl-DCAgent_code-contests-noblock
(skyrl_entrypoint pid=1461281) hf_hub_private: false
(skyrl_entrypoint pid=1461281) hf_hub_revision: main
(skyrl_entrypoint pid=1461281) enable_db_registration: true
(skyrl_entrypoint pid=1461281) generator:
(skyrl_entrypoint pid=1461281) model_name: ${trainer.policy.model.path}
(skyrl_entrypoint pid=1461281) model_dtype: bfloat16
(skyrl_entrypoint pid=1461281) timeout_multiplier: 1.0
(skyrl_entrypoint pid=1461281) run_engines_locally: true
(skyrl_entrypoint pid=1461281) num_inference_engines: 48
(skyrl_entrypoint pid=1461281) backend: vllm
(skyrl_entrypoint pid=1461281) weight_sync_backend: nccl
(skyrl_entrypoint pid=1461281) fuse_weights: false
(skyrl_entrypoint pid=1461281) weight_transfer_threshold_cuda_ipc_GB: 1.0
(skyrl_entrypoint pid=1461281) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=1461281) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=1461281) inference_engine_expert_parallel_size: 1
(skyrl_entrypoint pid=1461281) inference_engine_data_parallel_size: 1
(skyrl_entrypoint pid=1461281) n_samples_per_prompt: 8
(skyrl_entrypoint pid=1461281) async_engine: true
(skyrl_entrypoint pid=1461281) batched: false
(skyrl_entrypoint pid=1461281) max_input_length: ${trainer.max_prompt_length}
(skyrl_entrypoint pid=1461281) vllm_v1_disable_multiproc: true
(skyrl_entrypoint pid=1461281) enable_prefix_caching: true
(skyrl_entrypoint pid=1461281) enable_chunked_prefill: true
(skyrl_entrypoint pid=1461281) max_num_batched_tokens: 65536
(skyrl_entrypoint pid=1461281) enforce_eager: true
(skyrl_entrypoint pid=1461281) fully_sharded_loras: false
(skyrl_entrypoint pid=1461281) enable_ray_prometheus_stats: false
(skyrl_entrypoint pid=1461281) vllm_stats_interval: 1
(skyrl_entrypoint pid=1461281) gpu_memory_utilization: 0.75
(skyrl_entrypoint pid=1461281) max_num_seqs: 24
(skyrl_entrypoint pid=1461281) remote_inference_engine_urls:
(skyrl_entrypoint pid=1461281) - 127.0.0.1:8001
(skyrl_entrypoint pid=1461281) enable_http_endpoint: true
(skyrl_entrypoint pid=1461281) http_endpoint_host: 127.0.0.1
(skyrl_entrypoint pid=1461281) http_endpoint_port: 8000
(skyrl_entrypoint pid=1461281) max_turns: 999999
(skyrl_entrypoint pid=1461281) chat_template:
(skyrl_entrypoint pid=1461281) source: name
(skyrl_entrypoint pid=1461281) name_or_path: null
(skyrl_entrypoint pid=1461281) chat_template_kwargs: {}
(skyrl_entrypoint pid=1461281) engine_init_kwargs:
(skyrl_entrypoint pid=1461281) max_model_len: 32768
(skyrl_entrypoint pid=1461281) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=1461281) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=1461281) override_existing_update_group: disable
(skyrl_entrypoint pid=1461281) sampling_params:
(skyrl_entrypoint pid=1461281) max_generate_length: 4096
(skyrl_entrypoint pid=1461281) repetition_penalty: 1.0
(skyrl_entrypoint pid=1461281) temperature: 0.7
(skyrl_entrypoint pid=1461281) top_p: 0.95
(skyrl_entrypoint pid=1461281) min_p: 0.0
(skyrl_entrypoint pid=1461281) top_k: 20
(skyrl_entrypoint pid=1461281) logprobs: null
(skyrl_entrypoint pid=1461281) stop: null
(skyrl_entrypoint pid=1461281) use_conversation_multi_turn: true
(skyrl_entrypoint pid=1461281) append_eos_token_after_stop_str_in_multi_turn: true
(skyrl_entrypoint pid=1461281) eval_sampling_params:
(skyrl_entrypoint pid=1461281) max_generate_length: ${generator.sampling_params.max_generate_length}
(skyrl_entrypoint pid=1461281) repetition_penalty: 1.0
(skyrl_entrypoint pid=1461281) temperature: 0.0
(skyrl_entrypoint pid=1461281) top_p: 1.0
(skyrl_entrypoint pid=1461281) min_p: 0.0
(skyrl_entrypoint pid=1461281) top_k: -1
(skyrl_entrypoint pid=1461281) logprobs: null
(skyrl_entrypoint pid=1461281) stop: null
(skyrl_entrypoint pid=1461281) eval_n_samples_per_prompt: 8
(skyrl_entrypoint pid=1461281) zero_reward_on_non_stop: false
(skyrl_entrypoint pid=1461281) apply_overlong_filtering: false
(skyrl_entrypoint pid=1461281) rope_scaling: ${trainer.rope_scaling}
(skyrl_entrypoint pid=1461281) rope_theta: ${trainer.rope_theta}
(skyrl_entrypoint pid=1461281) teacher:
(skyrl_entrypoint pid=1461281) model_path: null
(skyrl_entrypoint pid=1461281) top_k_logprobs: 256
(skyrl_entrypoint pid=1461281) num_inference_engines: 1
(skyrl_entrypoint pid=1461281) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=1461281) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=1461281) gpu_memory_utilization: 0.9
(skyrl_entrypoint pid=1461281) enforce_eager: false
(skyrl_entrypoint pid=1461281) backend: vllm
(skyrl_entrypoint pid=1461281) engine_init_kwargs: {}
(skyrl_entrypoint pid=1461281) environment:
(skyrl_entrypoint pid=1461281) env_class: gsm8k
(skyrl_entrypoint pid=1461281) skyrl_gym:
(skyrl_entrypoint pid=1461281) max_env_workers: 32
(skyrl_entrypoint pid=1461281) text2sql:
(skyrl_entrypoint pid=1461281) db_path: /home/ray/default/sql_data
(skyrl_entrypoint pid=1461281) llm_as_a_judge:
(skyrl_entrypoint pid=1461281) model: gpt-4o-mini
(skyrl_entrypoint pid=1461281) base_url: null
(skyrl_entrypoint pid=1461281) search:
(skyrl_entrypoint pid=1461281) log_requests: false
(skyrl_entrypoint pid=1461281) search_url: http://127.0.0.1:8000/retrieve
(skyrl_entrypoint pid=1461281) topk: 3
(skyrl_entrypoint pid=1461281) timeout: 30
(skyrl_entrypoint pid=1461281) deepspeed_config:
(skyrl_entrypoint pid=1461281) train:
(skyrl_entrypoint pid=1461281) zero_optimization:
(skyrl_entrypoint pid=1461281) stage: 3
(skyrl_entrypoint pid=1461281) offload_param:
(skyrl_entrypoint pid=1461281) device: none
(skyrl_entrypoint pid=1461281) offload_optimizer:
(skyrl_entrypoint pid=1461281) device: none
(skyrl_entrypoint pid=1461281) pin_memory: true
(skyrl_entrypoint pid=1461281) sub_group_size: auto
(skyrl_entrypoint pid=1461281) reduce_bucket_size: auto
(skyrl_entrypoint pid=1461281) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=1461281) stage3_prefetch_bucket_size: auto
(skyrl_entrypoint pid=1461281) stage3_max_live_parameters: auto
(skyrl_entrypoint pid=1461281) stage3_max_reuse_distance: auto
(skyrl_entrypoint pid=1461281) round_robin_gradients: true
(skyrl_entrypoint pid=1461281) zero_hpz_partition_size: 1
(skyrl_entrypoint pid=1461281) zero_quantized_weights: false
(skyrl_entrypoint pid=1461281) zero_quantized_gradients: false
(skyrl_entrypoint pid=1461281) torch_autocast:
(skyrl_entrypoint pid=1461281) enabled: true
(skyrl_entrypoint pid=1461281) dtype: bfloat16
(skyrl_entrypoint pid=1461281) disable_trace_cache: false
(skyrl_entrypoint pid=1461281) data_types:
(skyrl_entrypoint pid=1461281) grad_accum_dtype: fp32
(skyrl_entrypoint pid=1461281) gradient_clipping: 1.0
(skyrl_entrypoint pid=1461281) wall_clock_breakdown: false
(skyrl_entrypoint pid=1461281) prescale_gradient: false
(skyrl_entrypoint pid=1461281) eval:
(skyrl_entrypoint pid=1461281) zero_optimization:
(skyrl_entrypoint pid=1461281) stage: 3
(skyrl_entrypoint pid=1461281) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=1461281) offload_param:
(skyrl_entrypoint pid=1461281) device: cpu
(skyrl_entrypoint pid=1461281) pin_memory: true
(skyrl_entrypoint pid=1461281) torch_autocast:
(skyrl_entrypoint pid=1461281) enabled: true
(skyrl_entrypoint pid=1461281) dtype: bfloat16
(skyrl_entrypoint pid=1461281) gradient_clipping: 1.0
(skyrl_entrypoint pid=1461281) prescale_gradient: false
(skyrl_entrypoint pid=1461281) wall_clock_breakdown: false
(skyrl_entrypoint pid=1461281) terminal_bench_config:
(skyrl_entrypoint pid=1461281) trials_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs
(skyrl_entrypoint pid=1461281) harbor:
(skyrl_entrypoint pid=1461281) name: terminus-2
(skyrl_entrypoint pid=1461281) max_episodes: 999999
(skyrl_entrypoint pid=1461281) enable_summarize: false
(skyrl_entrypoint pid=1461281) store_all_messages: true
(skyrl_entrypoint pid=1461281) trajectory_config:
(skyrl_entrypoint pid=1461281) raw_content: true
(skyrl_entrypoint pid=1461281) enable_episode_logging: false
(skyrl_entrypoint pid=1461281) record_terminal_session: false
(skyrl_entrypoint pid=1461281) enable_pane_logging: false
(skyrl_entrypoint pid=1461281) strict_json_parser: true
(skyrl_entrypoint pid=1461281) interleaved_thinking: true
(skyrl_entrypoint pid=1461281) extra_body:
(skyrl_entrypoint pid=1461281) chat_template_kwargs:
(skyrl_entrypoint pid=1461281) enable_thinking: true
(skyrl_entrypoint pid=1461281) override_timeout_sec: 1800
(skyrl_entrypoint pid=1461281) override_cpus: 1
(skyrl_entrypoint pid=1461281) override_memory_mb: 2048
(skyrl_entrypoint pid=1461281) override_storage_mb: 2048
(skyrl_entrypoint pid=1461281) auto_snapshot: true
(skyrl_entrypoint pid=1461281) verifier_override_timeout_sec: 120
(skyrl_entrypoint pid=1461281) max_retries: 3
(skyrl_entrypoint pid=1461281) min_wait_sec: 60.0
(skyrl_entrypoint pid=1461281) max_wait_sec: 600.0
(skyrl_entrypoint pid=1461281) wait_multiplier: 2.0
(skyrl_entrypoint pid=1461281) exclude_exceptions:
(skyrl_entrypoint pid=1461281) - VerifierTimeoutError
(skyrl_entrypoint pid=1461281) - VerifierRuntimeError
(skyrl_entrypoint pid=1461281) - RewardFileNotFoundError
(skyrl_entrypoint pid=1461281) - RewardFileEmptyError
(skyrl_entrypoint pid=1461281) - VerifierOutputParseError
(skyrl_entrypoint pid=1461281) n_concurrent_trials: 900
(skyrl_entrypoint pid=1461281) log_level: INFO
(skyrl_entrypoint pid=1461281) enable_reward_shaping: false
(skyrl_entrypoint pid=1461281) enable_error_classification: true
(skyrl_entrypoint pid=1461281) mask_exceptions:
(skyrl_entrypoint pid=1461281) - DaytonaError
(skyrl_entrypoint pid=1461281) - EnvironmentStartTimeoutError
(skyrl_entrypoint pid=1461281) - NetworkError
(skyrl_entrypoint pid=1461281) - ConnectionError
(skyrl_entrypoint pid=1461281) - RewardFileNotFoundError
(skyrl_entrypoint pid=1461281) - RewardFileEmptyError
(skyrl_entrypoint pid=1461281) - AgentEnvironmentTimeoutError
(skyrl_entrypoint pid=1461281) default_error_treatment: zero
(skyrl_entrypoint pid=1461281) passthrough_exceptions:
(skyrl_entrypoint pid=1461281) - AgentTimeoutError
(skyrl_entrypoint pid=1461281) - ContextLengthExceededError
(skyrl_entrypoint pid=1461281) model_info:
(skyrl_entrypoint pid=1461281) max_input_tokens: 32768
(skyrl_entrypoint pid=1461281) max_output_tokens: 4096
(skyrl_entrypoint pid=1461281) archiving:
(skyrl_entrypoint pid=1461281) enabled: false
(skyrl_entrypoint pid=1461281) trace_upload:
(skyrl_entrypoint pid=1461281) enabled: true
(skyrl_entrypoint pid=1461281) repo_org: DCAgent
(skyrl_entrypoint pid=1461281) episodes: last
(skyrl_entrypoint pid=1461281) dataset_type: SFT
(skyrl_entrypoint pid=1461281) cleanup: true
(skyrl_entrypoint pid=1461281) 
(skyrl_entrypoint pid=1461281) [2026-05-23 01:52:22,342 E 1461281 1461324] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster]
(skyrl_entrypoint pid=1461281) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(skyrl_entrypoint pid=1461281) No module named 'vllm._version'
(skyrl_entrypoint pid=1461281) from .version import __version__, __version_tuple__ # isort:skip
(skyrl_entrypoint pid=1461281) W0523 01:54:12.721000 1461281 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.43.84) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster]
(pid=873920, ip=10.128.43.93) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=873920, ip=10.128.43.93) No module named 'vllm._version'
(pid=873920, ip=10.128.43.93) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.43.108) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 98x across cluster]
(pid=874053, ip=10.128.43.93) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster]
(pid=874053, ip=10.128.43.93) No module named 'vllm._version' [repeated 15x across cluster]
(pid=874053, ip=10.128.43.93) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster]
(raylet, ip=10.128.43.95) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 72x across cluster]
(pid=900781, ip=10.128.43.91) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 9x across cluster]
(pid=900781, ip=10.128.43.91) No module named 'vllm._version' [repeated 9x across cluster]
(pid=900781, ip=10.128.43.91) from .version import __version__, __version_tuple__ # isort:skip [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:35.893 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:35.895 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:35.895 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:37.262 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:37.280 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:37.280 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:37.280 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:37.280 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) 2026-05-23 01:54:37.311 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.82s to avoid port collisions
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(raylet, ip=10.128.43.90) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 106x across cluster]
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=874054, ip=10.128.43.93) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=874054, ip=10.128.43.93) No module named 'vllm._version' [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=874054, ip=10.128.43.93) from .version import __version__, __version_tuple__ # isort:skip [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) 2026-05-23 01:54:39.678 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) 2026-05-23 01:54:39.678 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) 2026-05-23 01:54:39.678 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 7x across cluster]
(skyrl_entrypoint pid=1461281) [2026-05-23 01:54:41] INFO inference_engine_client_http_endpoint.py:350: Starting server on 127.0.0.1:8000
(skyrl_entrypoint pid=1461281) [2026-05-23 01:54:41] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
(skyrl_entrypoint pid=1461281) [2026-05-23 01:54:42] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.194 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:960 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.194 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines.
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.196 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:213 - Harbor logging level set to INFO
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.197 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:113 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 900. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.210 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:129 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.211 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 272
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.211 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:351 - Length of train_dataloader: 8728
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.211 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:352 - Number of steps per epoch: 136
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:42.211 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:353 - Total training steps: 80
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) 2026-05-23 01:54:41.065 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=1658031, ip=10.128.43.87) 2026-05-23 01:54:42.409 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1658031, ip=10.128.43.87) 2026-05-23 01:54:42.409 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1658031, ip=10.128.43.87) 2026-05-23 01:54:42.409 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1658031, ip=10.128.43.87) 2026-05-23 01:54:42.409 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) 2026-05-23 01:54:41.128 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.24s to avoid port collisions [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 69x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=937426, ip=10.128.43.84) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=937426, ip=10.128.43.84) No module named 'vllm._version' [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=937426, ip=10.128.43.84) from .version import __version__, __version_tuple__ # isort:skip [repeated 14x across cluster]
(pid=851205, ip=10.128.43.85) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) 2026-05-23 01:54:45.957 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) 2026-05-23 01:54:45.958 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) 2026-05-23 01:54:45.958 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 15x across cluster]
(bundle_reservation_check_func pid=1461399) [2026-05-23 01:54:47,098 E 1461399 1461441] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) 2026-05-23 01:54:47.356 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) 2026-05-23 01:54:47.380 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) 2026-05-23 01:54:47.380 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) 2026-05-23 01:54:47.380 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) 2026-05-23 01:54:47.380 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) 2026-05-23 01:54:47.403 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.57s to avoid port collisions [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 30x across cluster]
(AsyncVLLMInferenceEngine pid=900916, ip=10.128.43.91) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 32x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) No module named 'vllm._version' [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:49.703 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:49.704 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:49.704 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) (EngineCore_DP0 pid=874281) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) (EngineCore_DP0 pid=874281) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=937299, ip=10.128.43.84) [2026-05-23 01:54:51,552 E 937299 937401] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:50.968 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:50.989 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:50.989 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:50.989 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:50.990 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:54:51.045 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.40s to avoid port collisions [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=874052, ip=10.128.43.93) (EngineCore_DP0 pid=874286) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) [W523 01:54:54.227983559 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-078-45.jupiter.internal]:49029 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) [W523 01:54:54.231619642 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) [rank0]:[W523 01:54:54.236680175 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) 2026-05-23 01:54:54 INFO [ipv4-debug] hostname=jpbo-078-37.jupiter.internal
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) 2026-05-23 01:54:54 INFO [ipv4-debug] _global_node.node_ip_address=10.128.43.85
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) 2026-05-23 01:54:54 INFO [ipv4-debug] get_node_ip_address()=10.128.43.85
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 16x across cluster]
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 20x across cluster]
(skyrl_entrypoint pid=1461281) 2026-05-23 01:54:54.377 | INFO  | skyrl_train.workers.worker:_initiate_actors:496 - Initializing process group for RayActorGroup
(AsyncVLLMInferenceEngine pid=874054, ip=10.128.43.93) (EngineCore_DP0 pid=874292) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=937427, ip=10.128.43.84) (EngineCore_DP0 pid=937657) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) No module named 'vllm._version' [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) (EngineCore_DP0 pid=937670) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=937299, ip=10.128.43.84) (EngineCore_DP0 pid=937662) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:56.483 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:56.484 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:56.484 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) (EngineCore_DP0 pid=1658393) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=839106, ip=10.128.43.94) (EngineCore_DP0 pid=839345) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1658031, ip=10.128.43.87) (EngineCore_DP0 pid=1658398) _C._set_float32_matmul_precision(precision)
(pid=851282, ip=10.128.43.85) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=876508, ip=10.128.43.96) (EngineCore_DP0 pid=876740) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=937426, ip=10.128.43.84) (EngineCore_DP0 pid=937681) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) [2026-05-23 01:54:56,430 E 876379 876482] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) (EngineCore_DP0 pid=839337) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=839108, ip=10.128.43.94) (EngineCore_DP0 pid=839350) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1658164, ip=10.128.43.87) (EngineCore_DP0 pid=1658411) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:57.854 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:57.879 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:57.879 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:57.879 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:57.879 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) 2026-05-23 01:54:57.910 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.00s to avoid port collisions [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=839107, ip=10.128.43.94) (EngineCore_DP0 pid=839358) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=876509, ip=10.128.43.96) (EngineCore_DP0 pid=876761) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) Process EngineCore_DP0:
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) self.run()
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 108, in run
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) self._target(*self._args, **self._kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1010, in run_engine_core
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) raise e
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) super().__init__(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) self._init_executor()
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) init_distributed_environment(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) return TCPStore(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 36099, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40004b8fc700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #1: <unknown function> + 0x5f29220 (0x40002ba99220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #2: <unknown function> + 0x5f4326c (0x40002bab326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #3: <unknown function> + 0x5f49074 (0x40002bab9074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #4: <unknown function> + 0x5f49138 (0x40002bab9138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #5: <unknown function> + 0x5f2ccc4 (0x40002ba9ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x40002baa1564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #7: <unknown function> + 0x109a094 (0x40002584a094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #8: <unknown function> + 0x113236c (0x4000258e236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #9: <unknown function> + 0x5d6d60 (0x400024d86d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #10: <unknown function> + 0x1b7a38 (0xaaaab65b7a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaab6565db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #12: <unknown function> + 0x169f50 (0xaaaab6569f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #13: <unknown function> + 0x1682e4 (0xaaaab65682e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #14: <unknown function> + 0x1e0ce8 (0xaaaab65e0ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #15: <unknown function> + 0x1d7ddc (0xaaaab65d7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #16: <unknown function> + 0x646b0c (0x400024df6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaab6565db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaab666ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #19: <unknown function> + 0x1808c0 (0xaaaab65808c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #20: <unknown function> + 0x182bf8 (0xaaaab6582bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #21: <unknown function> + 0x25fd30 (0xaaaab665fd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #22: <unknown function> + 0x1b7d20 (0xaaaab65b7d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #23: PyObject_Vectorcall + 0x54 (0xaaaab65660e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaab666ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaab6567f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaab65681ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #27: <unknown function> + 0x1e0d80 (0xaaaab65e0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #28: <unknown function> + 0x1d7ddc (0xaaaab65d7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaab6565db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaab666ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaab6567fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaab65681ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #33: <unknown function> + 0x1e0d80 (0xaaaab65e0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #34: <unknown function> + 0x1d7ddc (0xaaaab65d7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #35: _PyObject_Call + 0x68 (0xaaaab6568488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaab666d8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #37: PyEval_EvalCode + 0xb4 (0xaaaab6672eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #38: <unknown function> + 0x2ccdcc (0xaaaab66ccdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #39: <unknown function> + 0x2ccef4 (0xaaaab66ccef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #40: PyRun_StringFlags + 0x90 (0xaaaab66d1050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaab66d10f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #42: Py_RunMain + 0x2c8 (0xaaaab66f9190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #43: Py_BytesMain + 0x64 (0xaaaab66f9fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #44: <unknown function> + 0x27540 (0x400013df7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #45: __libc_start_main + 0x98 (0x400013df7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) frame #46: <unknown function> + 0x10e0c0 (0xaaaab650e0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748)
(AsyncVLLMInferenceEngine pid=876509, ip=10.128.43.96) [W523 01:54:59.553303224 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-078-48.jupiter.internal]:54877 (errno: 97 - Address family not supported by protocol). [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=876509, ip=10.128.43.96) [W523 01:54:59.553793743 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=876509, ip=10.128.43.96) [rank0]:[W523 01:54:59.556104773 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 58x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) No module named 'vllm._version' [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) from .version import __version__, __version_tuple__ # isort:skip [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=900915, ip=10.128.43.91) (EngineCore_DP0 pid=901149) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.32s/it]
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=876507, ip=10.128.43.96, actor_id=58494295fdaf9ec8226b54f502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400be192c230>)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) return self.__get_result()
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) raise self._exception
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) super().__init__(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) self.llm = self._create_engine(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1081, in _create_engine
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) return cls(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) self.engine_core = EngineCoreClient.make_async_mp_client(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) return AsyncMPClient(*client_args)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) super().__init__(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) next(self.gen)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) wait_for_engine_startup(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) raise RuntimeError(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) (EngineCore_DP0 pid=1003267) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) (EngineCore_DP0 pid=901162) _C._set_float32_matmul_precision(precision)
(pid=1461509) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1003033, ip=10.128.43.108) (EngineCore_DP0 pid=1003273) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1003033, ip=10.128.43.108) (EngineCore_DP0 pid=1003273) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) [2026-05-23 01:55:02,094 E 2095386 2095488] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=900916, ip=10.128.43.91) (EngineCore_DP0 pid=901154) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1003034, ip=10.128.43.108) (EngineCore_DP0 pid=1003287) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1003034, ip=10.128.43.108) [W523 01:55:03.708851322 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-079-12.jupiter.internal]:56203 (errno: 97 - Address family not supported by protocol). [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1003034, ip=10.128.43.108) [W523 01:55:03.709312144 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1003034, ip=10.128.43.108) [rank0]:[W523 01:55:03.711459426 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1002904, ip=10.128.43.108) (EngineCore_DP0 pid=1003263) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) No module named 'vllm._version' [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) from .version import __version__, __version_tuple__ # isort:skip [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) (EngineCore_DP0 pid=1003267)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:01<00:05, 1.83s/it] [repeated 26x across cluster]
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:06.182 | INFO  | skyrl_train.workers.worker:_initiate_actors:498 - Initialized process group for RayActorGroup
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:06.191 | INFO  | skyrl_train.workers.worker:_initiate_actors:500 - Mesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1)]
(FSDPPolicyWorkerBase pid=1461510) `torch_dtype` is deprecated! Use `dtype` instead!
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85)
Loading checkpoint shards: 50%|█████ | 2/4 [00:00<00:00, 9.42it/s]
(AsyncVLLMInferenceEngine pid=1002904, ip=10.128.43.108) (EngineCore_DP0 pid=1003263)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095513, ip=10.128.43.86) (EngineCore_DP0 pid=2095757) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2552032, ip=10.128.43.88) (EngineCore_DP0 pid=2552262) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2552032, ip=10.128.43.88) (EngineCore_DP0 pid=2552262) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) [2026-05-23 01:55:06,382 E 940292 940457] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) (EngineCore_DP0 pid=2095768) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=940291, ip=10.128.43.95) (EngineCore_DP0 pid=940522) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=1461510)
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00, 2.50it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00, 2.83it/s]
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=940164, ip=10.128.43.95) (EngineCore_DP0 pid=940526) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) (EngineCore_DP0 pid=2095747) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) (EngineCore_DP0 pid=2095751) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) [W523 01:55:09.885732158 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-078-38.jupiter.internal]:46833 (errno: 97 - Address family not supported by protocol). [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) [W523 01:55:09.886241428 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) [rank0]:[W523 01:55:09.888544871 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753)
(AsyncVLLMInferenceEngine pid=876508, ip=10.128.43.96) (EngineCore_DP0 pid=876740)
(AsyncVLLMInferenceEngine pid=876509, ip=10.128.43.96) (EngineCore_DP0 pid=876761)
(AsyncVLLMInferenceEngine pid=2551904, ip=10.128.43.88) (EngineCore_DP0 pid=2552263) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=940293, ip=10.128.43.95) (EngineCore_DP0 pid=940540) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2552034, ip=10.128.43.88) (EngineCore_DP0 pid=2552274) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307)
(AsyncVLLMInferenceEngine pid=874052, ip=10.128.43.93) (EngineCore_DP0 pid=874286)
(AsyncVLLMInferenceEngine pid=874054, ip=10.128.43.93) (EngineCore_DP0 pid=874292)
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) (EngineCore_DP0 pid=874281)
(AsyncVLLMInferenceEngine pid=1002904, ip=10.128.43.108) (EngineCore_DP0 pid=1003263)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:05<00:02, 2.18s/it] [repeated 45x across cluster]
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) (EngineCore_DP0 pid=869188) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=851281, ip=10.128.43.85) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=1461512)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) (EngineCore_DP0 pid=869183) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=851281, ip=10.128.43.85)
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 3.08it/s] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1002904, ip=10.128.43.108) (EngineCore_DP0 pid=1003263)
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) (EngineCore_DP0 pid=869183) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) [2026-05-23 01:55:12,049 E 975888 976055] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=1003033, ip=10.128.43.108) (EngineCore_DP0 pid=1003273)
(AsyncVLLMInferenceEngine pid=1003034, ip=10.128.43.108) (EngineCore_DP0 pid=1003287)
(AsyncVLLMInferenceEngine pid=1003032, ip=10.128.43.108) (EngineCore_DP0 pid=1003267)
(AsyncVLLMInferenceEngine pid=868823, ip=10.128.43.90) (EngineCore_DP0 pid=869195) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) (EngineCore_DP0 pid=839337)
(AsyncVLLMInferenceEngine pid=839106, ip=10.128.43.94) (EngineCore_DP0 pid=839345)
(AsyncVLLMInferenceEngine pid=839107, ip=10.128.43.94) (EngineCore_DP0 pid=839358)
(AsyncVLLMInferenceEngine pid=839108, ip=10.128.43.94) (EngineCore_DP0 pid=839350)
(AsyncVLLMInferenceEngine pid=937299, ip=10.128.43.84) (EngineCore_DP0 pid=937662)
(AsyncVLLMInferenceEngine pid=937426, ip=10.128.43.84) (EngineCore_DP0 pid=937681)
(AsyncVLLMInferenceEngine pid=937427, ip=10.128.43.84) (EngineCore_DP0 pid=937657)
(AsyncVLLMInferenceEngine pid=937554, ip=10.128.43.84) (EngineCore_DP0 pid=937670)
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(FSDPPolicyWorkerBase pid=851281, ip=10.128.43.85)
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00, 2.30it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00, 2.73it/s] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=975889, ip=10.128.43.103) (EngineCore_DP0 pid=976138) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) (EngineCore_DP0 pid=869207) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) 2026-05-23 01:55:13.517 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1115 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) (EngineCore_DP0 pid=976133) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1658031, ip=10.128.43.87) (EngineCore_DP0 pid=1658398)
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) (EngineCore_DP0 pid=1658393)
(AsyncVLLMInferenceEngine pid=1658164, ip=10.128.43.87) (EngineCore_DP0 pid=1658411)
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405)
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) [W523 01:55:14.722835558 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-079-07.jupiter.internal]:43683 (errno: 97 - Address family not supported by protocol). [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) [W523 01:55:14.723282685 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) [rank0]:[W523 01:55:14.727560583 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.11s/it] [repeated 63x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 6x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [2026-05-23 01:55:13,239 E 851205 851247] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141)
(AsyncVLLMInferenceEngine pid=900916, ip=10.128.43.91) (EngineCore_DP0 pid=901154)
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) (EngineCore_DP0 pid=901162)
(AsyncVLLMInferenceEngine pid=900915, ip=10.128.43.91) (EngineCore_DP0 pid=901149)
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) (EngineCore_DP0 pid=1658393) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) 2026-05-23 01:55:18.058 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1115 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) [W523 01:55:14.098232925 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-079-07.jupiter.internal]:58483 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) [W523 01:55:14.098713651 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(FSDPPolicyWorkerBase pid=1461509) [rank7]:[W523 01:55:18.669215806 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster]
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.565 | INFO  | skyrl_train.trainer:build_models:802 - init policy/ref/critic models done
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.567 | INFO  | harbor.orchestrators.queue:start:262 - [terminal_bench_generator:236] Started 900 workers (status every 120.0s, 0.75s launch grace period)
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.567 | INFO  | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:257 - QueueOrchestrator created and started with n_concurrent_trials=900, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.568 | INFO  | examples.terminal_bench.terminal_bench_generator:startup:226 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=900
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.568 | INFO  | skyrl_train.fully_async_trainer:train:428 - Generator startup complete
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.568 | INFO  | skyrl_train.fully_async_trainer:_train_loop:453 - Started: 'load_checkpoints'
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.694 | INFO  | skyrl_train.trainer:load_checkpoints:1595 - Loading checkpoint from: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.694 | INFO  | skyrl_train.trainer:load_checkpoints:1601 - Resuming from global_step: 18
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.727 | INFO  | skyrl_train.trainer:load_checkpoints:1617 - Successfully loaded trainer state
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.761 | INFO  | skyrl_train.trainer:load_checkpoints:1627 - Successfully loaded dataloader state
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:20.762 | INFO  | skyrl_train.trainer:load_checkpoints:1636 - Loading policy checkpoint from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.13s/it] [repeated 32x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) 2026-05-23 01:55:21.838 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1115 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) (EngineCore_DP0 pid=2095751)
(AsyncVLLMInferenceEngine pid=2095513, ip=10.128.43.86) (EngineCore_DP0 pid=2095757)
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) (EngineCore_DP0 pid=2095768)
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) (EngineCore_DP0 pid=2095747)
(AsyncVLLMInferenceEngine pid=2551904, ip=10.128.43.88) (EngineCore_DP0 pid=2552263)
(AsyncVLLMInferenceEngine pid=2552034, ip=10.128.43.88) (EngineCore_DP0 pid=2552274)
(AsyncVLLMInferenceEngine pid=2552032, ip=10.128.43.88) (EngineCore_DP0 pid=2552262)
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267)
(FSDPPolicyWorkerBase pid=851282, ip=10.128.43.85) [2026-05-23 01:55:25,133 E 851282 851395] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=940164, ip=10.128.43.95) (EngineCore_DP0 pid=940526)
(AsyncVLLMInferenceEngine pid=940291, ip=10.128.43.95) (EngineCore_DP0 pid=940522)
(AsyncVLLMInferenceEngine pid=940293, ip=10.128.43.95) (EngineCore_DP0 pid=940540)
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531)
(AsyncVLLMInferenceEngine pid=868823, ip=10.128.43.90) (EngineCore_DP0 pid=869195)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.18s/it] [repeated 45x across cluster]
(AsyncVLLMInferenceEngine pid=2551904, ip=10.128.43.88) (EngineCore_DP0 pid=2552263) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) 2026-05-23 01:55:28.153 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1115 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=868823, ip=10.128.43.90) (EngineCore_DP0 pid=869195)
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) (EngineCore_DP0 pid=869207)
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) (EngineCore_DP0 pid=869183)
(AsyncVLLMInferenceEngine pid=868951, ip=10.128.43.90) (EngineCore_DP0 pid=869188)
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117)
(AsyncVLLMInferenceEngine pid=975889, ip=10.128.43.103) (EngineCore_DP0 pid=976138)
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) (EngineCore_DP0 pid=976133)
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122)
(FSDPPolicyWorkerBase pid=1461509) [2026-05-23 01:55:25,304 E 1461509 1461738] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00, 3.50s/it] [repeated 23x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) 2026-05-23 01:55:33.841 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1115 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 13x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) warnings.warn( # warn only once
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:38.114 | INFO  | skyrl_train.trainer:load_checkpoints:1646 - Successfully loaded policy checkpoint
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:38.114 | INFO  | skyrl_train.trainer:load_checkpoints:1662 - Successfully loaded complete checkpoint state from global_step_18
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:38.114 | INFO  | skyrl_train.fully_async_trainer:_train_loop:455 - Resumed training from global_step 18
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:38.132 | INFO  | skyrl_train.utils.data_tracker:load_state:97 - Loaded data tracker state: epoch=0, consumed_in_epoch=1144, total_consumed=1144
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:38.133 | WARNING  | skyrl_train.fully_async_trainer:_train_loop:480 - Data consumption count mismatch on resume: expected 1152, got 1144. This can happen after epoch boundary transitions or error recovery.
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:38.133 | INFO  | skyrl_train.fully_async_trainer:_train_loop:453 - Finished: 'load_checkpoints', time cost: 17.57s
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:38.133 | INFO  | skyrl_train.fully_async_trainer:_train_loop:487 - Started: 'init_weight_sync_state'
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) No module named 'vllm._version'
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) from .version import __version__, __version_tuple__ # isort:skip
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) 2026-05-23 01:55:44.331 | INFO  | logging:info:2216 - [weight-sync] Using master_addr=10.128.43.85, master_port=38311
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) 2026-05-23 01:55:33.884 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1115 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) 2026-05-23 01:55:44.331 | INFO  | logging:info:2216 - [weight-sync] get_node_ip_address()=10.128.43.85 [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=1461509) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=1461509) No module named 'vllm._version' [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=1461509) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank0]:[W523 01:55:44.016298483 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-078-37-interconnect-1.jupiter.internal]:38311 (errno: 97 - Address family not supported by protocol).
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:44.431 | INFO  | skyrl_train.fully_async_trainer:_train_loop:487 - Finished: 'init_weight_sync_state', time cost: 6.30s
(skyrl_entrypoint pid=1461281) --- Logging error in Loguru Handler #2 ---
(skyrl_entrypoint pid=1461281) Record was: {'elapsed': datetime.timedelta(seconds=226, microseconds=919551), 'exception': (type=<class 'ray.exceptions.RayTaskError(ActorDiedError)'>, value=RayTaskError(ActorDiedError)(ActorDiedError(RayTaskError('__init__', 'Traceback (most recent call last):\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n File "python/ray/_raylet.pyx", line 4329, in async_func\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\n return method(__ray_actor, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\n return method(self, *_args, **_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__\n super().__init__(*args, **kwargs)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__\n self.llm = self._create_engine(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\n return method(self, *_args, **_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1081, in _create_engine\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\n return cls(\n ^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\n self.engine_core = EngineCoreClient.make_async_mp_client(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\n return AsyncMPClient(*client_args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\n super().__init__(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\n next(self.gen)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\n wait_for_engine_startup(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\n raise RuntimeError(\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\n', RuntimeError('Engine core initialization failed. See root cause above. Failed core proc(s): {}'), 'ray::AsyncVLLMInferenceEngine.__init__', None, None))), traceback=<traceback object at 0x400c723cbec0>), 'extra': {}, 'file': (name='fully_async_trainer.py', path='/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py'), 'function': 'train', 'level': (name='ERROR', no=40, icon='❌'), 'line': 436, 'message': 'Train loop failed at global_step 18: \x1b[36mray::FSDPPolicyWorkerBase.init_weight_sync_state()\x1b[39m (pid=851205, ip=10.128.43.85, actor_id=8b87233f106bcf2285f12e1602000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400dfe746e10>)\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state\n results = await asyncio.gather(*tasks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator\n rank_offset_count += engine.tp_size() * engine.pp_size()\n ^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size\n return ray.get(self.inference_engine_actor.tp_size.remote())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^\nray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, \x1b[36mray::AsyncVLLMInferenceEngine.__init__()\x1b[39m (pid=876507, ip=10.128.43.96, actor_id=58494295fdaf9ec8226b54f502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400be192c230>)\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__\n super().__init__(*args, **kwargs)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__\n self.llm = self._create_engine(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1081, in _create_engine\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\n return cls(\n ^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\n self.engine_core = EngineCoreClient.make_async_mp_client(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\n return AsyncMPClient(*client_args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\n super().__init__(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\n next(self.gen)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\n wait_for_engine_startup(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\n raise RuntimeError(\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}', 'module': 'fully_async_trainer', 'name': 'skyrl_train.fully_async_trainer', 'process': (id=1461281, name='MainProcess'), 'thread': (id=70369743691552, name='MainThread'), 'time': datetime(2026, 5, 23, 1, 55, 44, 431622, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200), 'CEST'))}
(skyrl_entrypoint pid=1461281) Traceback (most recent call last):
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train
(skyrl_entrypoint pid=1461281) await self._train_loop()
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop
(skyrl_entrypoint pid=1461281) self.init_weight_sync_state()
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state
(skyrl_entrypoint pid=1461281) ray.get(
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(skyrl_entrypoint pid=1461281) return fn(*args, **kwargs)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
(skyrl_entrypoint pid=1461281) return func(*args, **kwargs)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
(skyrl_entrypoint pid=1461281) values, debugger_breakpoint = worker.get_objects(
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
(skyrl_entrypoint pid=1461281) raise value.as_instanceof_cause()
(skyrl_entrypoint pid=1461281) ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=851205, ip=10.128.43.85, actor_id=8b87233f106bcf2285f12e1602000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400dfe746e10>)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(skyrl_entrypoint pid=1461281) return self.__get_result()
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(skyrl_entrypoint pid=1461281) raise self._exception
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state
(skyrl_entrypoint pid=1461281) results = await asyncio.gather(*tasks)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator
(skyrl_entrypoint pid=1461281) rank_offset_count += engine.tp_size() * engine.pp_size()
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size
(skyrl_entrypoint pid=1461281) return ray.get(self.inference_engine_actor.tp_size.remote())
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=876507, ip=10.128.43.96, actor_id=58494295fdaf9ec8226b54f502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400be192c230>)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(skyrl_entrypoint pid=1461281) return self.__get_result()
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(skyrl_entrypoint pid=1461281) raise self._exception
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
(skyrl_entrypoint pid=1461281) super().__init__(*args, **kwargs)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
(skyrl_entrypoint pid=1461281) self.llm = self._create_engine(*args, **kwargs)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1081, in _create_engine
(skyrl_entrypoint pid=1461281) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(skyrl_entrypoint pid=1461281) return cls(
(skyrl_entrypoint pid=1461281) ^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(skyrl_entrypoint pid=1461281) self.engine_core = EngineCoreClient.make_async_mp_client(
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(skyrl_entrypoint pid=1461281) return AsyncMPClient(*client_args)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(skyrl_entrypoint pid=1461281) super().__init__(
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(skyrl_entrypoint pid=1461281) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(skyrl_entrypoint pid=1461281) next(self.gen)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(skyrl_entrypoint pid=1461281) wait_for_engine_startup(
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(skyrl_entrypoint pid=1461281) raise RuntimeError(
(skyrl_entrypoint pid=1461281) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(skyrl_entrypoint pid=1461281)
(skyrl_entrypoint pid=1461281) During handling of the above exception, another exception occurred:
(skyrl_entrypoint pid=1461281)
(skyrl_entrypoint pid=1461281) Traceback (most recent call last):
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/loguru/_handler.py", line 204, in emit
(skyrl_entrypoint pid=1461281) self._queue.put(str_record)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/queues.py", line 393, in put
(skyrl_entrypoint pid=1461281) obj = _ForkingPickler.dumps(obj)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
(skyrl_entrypoint pid=1461281) cls(buf, protocol).dump(obj)
(skyrl_entrypoint pid=1461281) _pickle.PicklingError: Can't pickle <class 'ray.exceptions.RayTaskError(ActorDiedError)'>: attribute lookup RayTaskError(ActorDiedError) on ray.exceptions failed
(skyrl_entrypoint pid=1461281) --- End of logging error ---
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) warnings.warn("No model update group to destroy")
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.513 | INFO  | skyrl_train.inference_engines.inference_engine_client_http_endpoint:shutdown_server:203 - Server shut down after 2 seconds
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.513 | INFO  | skyrl_train.trainer:_guarded_sync:226 - HTTP endpoint shutdown complete
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.513 | INFO  | examples.terminal_bench.terminal_bench_generator:shutdown:328 - Shutting down shared QueueOrchestrator...
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.516 | INFO  | harbor.orchestrators.queue:shutdown:377 - [terminal_bench_generator:236] Shutdown complete. Total completed: 0
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.516 | INFO  | examples.terminal_bench.terminal_bench_generator:shutdown:330 - QueueOrchestrator shutdown complete
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.516 | INFO  | skyrl_train.trainer:_guarded_async:215 - Generator shutdown complete
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.526 | WARNING  | skyrl_train.trainer:_guarded_async:219 - Inference engine teardown error (non-fatal): The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=876507, ip=10.128.43.96, actor_id=58494295fdaf9ec8226b54f502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400be192c230>)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(skyrl_entrypoint pid=1461281) return self.__get_result()
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(skyrl_entrypoint pid=1461281) raise self._exception
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
(skyrl_entrypoint pid=1461281) super().__init__(*args, **kwargs)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
(skyrl_entrypoint pid=1461281) self.llm = self._create_engine(*args, **kwargs)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1081, in _create_engine
(skyrl_entrypoint pid=1461281) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(skyrl_entrypoint pid=1461281) return cls(
(skyrl_entrypoint pid=1461281) ^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(skyrl_entrypoint pid=1461281) self.engine_core = EngineCoreClient.make_async_mp_client(
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(skyrl_entrypoint pid=1461281) return AsyncMPClient(*client_args)
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(skyrl_entrypoint pid=1461281) super().__init__(
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(skyrl_entrypoint pid=1461281) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(skyrl_entrypoint pid=1461281) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(skyrl_entrypoint pid=1461281) next(self.gen)
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(skyrl_entrypoint pid=1461281) wait_for_engine_startup(
(skyrl_entrypoint pid=1461281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(skyrl_entrypoint pid=1461281) raise RuntimeError(
(skyrl_entrypoint pid=1461281) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.526 | INFO  | skyrl_train.trainer:_kill_ray_actors:248 - Killing policy_model actors...
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.540 | INFO  | skyrl_train.trainer:_kill_ray_actors:267 - Killed 48 inference engine actor(s)
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.541 | INFO  | skyrl_train.trainer:_guarded_sync:226 - Ray actor cleanup complete
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.542 | INFO  | skyrl_train.trainer:_kill_ray_actors:248 - Killing policy_model actors...
(skyrl_entrypoint pid=1461281) 2026-05-23 01:55:45.544 | INFO  | skyrl_train.trainer:_kill_ray_actors:267 - Killed 48 inference engine actor(s)
2026-05-23 01:55:45.559 | ERROR | __main__:main:125 - Training failed: ray::skyrl_entrypoint() (pid=1461281, ip=10.128.43.81)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 96, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 403, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state
ray.get(
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=851205, ip=10.128.43.85, actor_id=8b87233f106bcf2285f12e1602000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400dfe746e10>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state
results = await asyncio.gather(*tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator
rank_offset_count += engine.tp_size() * engine.pp_size()
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size
return ray.get(self.inference_engine_actor.tp_size.remote())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=876507, ip=10.128.43.96, actor_id=58494295fdaf9ec8226b54f502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400be192c230>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
super().__init__(*args, **kwargs)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
self.llm = self._create_engine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1081, in _create_engine
engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
return cls(
^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
self.engine_core = EngineCoreClient.make_async_mp_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
return AsyncMPClient(*client_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
super().__init__(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
with launch_core_engines(vllm_config, executor_class, log_stats) as (
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
wait_for_engine_startup(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
2026-05-23 01:55:45.559 | INFO | __main__:main:128 - Shutting down Ray on head node...
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy [repeated 21x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) warnings.warn("No model update group to destroy") [repeated 21x across cluster]
(skyrl_entrypoint pid=1461281) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) WARNING 05-23 01:54:39 [arg_utils.py:1256] The global random seed is set to 57. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) INFO 05-23 01:54:39 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) INFO 05-23 01:54:39 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) INFO 05-23 01:54:39 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) INFO 05-23 01:54:39 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) INFO 05-23 01:54:39 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) WARNING 05-23 01:54:39 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) INFO 05-23 01:54:39 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) WARNING 05-23 01:54:39 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) WARNING 05-23 01:54:39 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) WARNING 05-23 01:54:44 [arg_utils.py:1256] The global random seed is set to 46. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) INFO 05-23 01:54:44 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) INFO 05-23 01:54:44 [model.py:1549] Using max model len 32768 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) INFO 05-23 01:54:44 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) INFO 05-23 01:54:44 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) INFO 05-23 01:54:44 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) WARNING 05-23 01:54:44 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) INFO 05-23 01:54:44 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) WARNING 05-23 01:54:44 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) WARNING 05-23 01:54:44 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) (EngineCore_DP0 pid=874281) INFO 05-23 01:54:48 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=57, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) WARNING 05-23 01:54:49 [arg_utils.py:1256] The global random seed is set to 80. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) INFO 05-23 01:54:49 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) INFO 05-23 01:54:49 [model.py:1549] Using max model len 32768 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) INFO 05-23 01:54:49 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) INFO 05-23 01:54:49 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) INFO 05-23 01:54:49 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) WARNING 05-23 01:54:49 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) INFO 05-23 01:54:49 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) WARNING 05-23 01:54:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=900914, ip=10.128.43.91) WARNING 05-23 01:54:49 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=874053, ip=10.128.43.93) (EngineCore_DP0 pid=874281) INFO 05-23 01:54:52 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=838979, ip=10.128.43.94) (EngineCore_DP0 pid=839337) INFO 05-23 01:54:53 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=46, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307) INFO 05-23 01:54:54 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.43.93:49029 backend=nccl
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307) INFO 05-23 01:54:54 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) WARNING 05-23 01:54:54 [arg_utils.py:1256] The global random seed is set to 68. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) INFO 05-23 01:54:54 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) INFO 05-23 01:54:54 [model.py:1549] Using max model len 32768 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) INFO 05-23 01:54:54 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) INFO 05-23 01:54:54 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) INFO 05-23 01:54:54 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) WARNING 05-23 01:54:54 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) INFO 05-23 01:54:54 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster]
(pid=851205, ip=10.128.43.85) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307) INFO 05-23 01:54:54 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) WARNING 05-23 01:54:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) WARNING 05-23 01:54:54 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=873920, ip=10.128.43.93) (EngineCore_DP0 pid=874307) INFO 05-23 01:54:56 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) INFO 05-23 01:54:57 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=900916, ip=10.128.43.91) (EngineCore_DP0 pid=901154) INFO 05-23 01:54:58 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=79, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] EngineCore failed to start.
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] super().__init__(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] self._init_executor()
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] init_distributed_environment(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] return TCPStore(
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 36099, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40004b8fc700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #1: <unknown function> + 0x5f29220 (0x40002ba99220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #2: <unknown function> + 0x5f4326c (0x40002bab326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #3: <unknown function> + 0x5f49074 (0x40002bab9074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #4: <unknown function> + 0x5f49138 (0x40002bab9138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #5: <unknown function> + 0x5f2ccc4 (0x40002ba9ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x40002baa1564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #7: <unknown function> + 0x109a094 (0x40002584a094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #8: <unknown function> + 0x113236c (0x4000258e236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #9: <unknown function> + 0x5d6d60 (0x400024d86d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #10: <unknown function> + 0x1b7a38 (0xaaaab65b7a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaab6565db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #12: <unknown function> + 0x169f50 (0xaaaab6569f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #13: <unknown function> + 0x1682e4 (0xaaaab65682e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #14: <unknown function> + 0x1e0ce8 (0xaaaab65e0ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #15: <unknown function> + 0x1d7ddc (0xaaaab65d7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #16: <unknown function> + 0x646b0c (0x400024df6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaab6565db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaab666ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #19: <unknown function> + 0x1808c0 (0xaaaab65808c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #20: <unknown function> + 0x182bf8 (0xaaaab6582bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #21: <unknown function> + 0x25fd30 (0xaaaab665fd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #22: <unknown function> + 0x1b7d20 (0xaaaab65b7d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #23: PyObject_Vectorcall + 0x54 (0xaaaab65660e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaab666ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaab6567f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaab65681ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #27: <unknown function> + 0x1e0d80 (0xaaaab65e0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #28: <unknown function> + 0x1d7ddc (0xaaaab65d7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaab6565db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaab666ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaab6567fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaab65681ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #33: <unknown function> + 0x1e0d80 (0xaaaab65e0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #34: <unknown function> + 0x1d7ddc (0xaaaab65d7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #35: _PyObject_Call + 0x68 (0xaaaab6568488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaab666d8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #37: PyEval_EvalCode + 0xb4 (0xaaaab6672eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #38: <unknown function> + 0x2ccdcc (0xaaaab66ccdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #39: <unknown function> + 0x2ccef4 (0xaaaab66ccef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #40: PyRun_StringFlags + 0x90 (0xaaaab66d1050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaab66d10f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #42: Py_RunMain + 0x2c8 (0xaaaab66f9190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #43: Py_BytesMain + 0x64 (0xaaaab66f9fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #44: <unknown function> + 0x27540 (0x400013df7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #45: __libc_start_main + 0x98 (0x400013df7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006] frame #46: <unknown function> + 0x10e0c0 (0xaaaab650e0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) ERROR 05-23 01:54:59 [core.py:1006]
(AsyncVLLMInferenceEngine pid=876507, ip=10.128.43.96) (EngineCore_DP0 pid=876748) INFO 05-23 01:54:59 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.43.96:36099 backend=nccl [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=876509, ip=10.128.43.96) (EngineCore_DP0 pid=876761) INFO 05-23 01:54:59 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) WARNING 05-23 01:54:59 [arg_utils.py:1256] The global random seed is set to 89. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) INFO 05-23 01:54:59 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) INFO 05-23 01:54:59 [model.py:1549] Using max model len 32768 [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) INFO 05-23 01:54:59 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) INFO 05-23 01:54:59 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) INFO 05-23 01:54:59 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) WARNING 05-23 01:54:59 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) INFO 05-23 01:54:59 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=876509, ip=10.128.43.96) (EngineCore_DP0 pid=876761) INFO 05-23 01:54:59 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) WARNING 05-23 01:54:59 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) WARNING 05-23 01:54:59 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:00 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 18x across cluster]
(pid=851281, ip=10.128.43.85) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=1003033, ip=10.128.43.108) (EngineCore_DP0 pid=1003273) INFO 05-23 01:55:02 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1003034, ip=10.128.43.108) (EngineCore_DP0 pid=1003287) INFO 05-23 01:55:00 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=64, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1003034, ip=10.128.43.108) (EngineCore_DP0 pid=1003287) INFO 05-23 01:55:03 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.43.108:56203 backend=nccl [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1002904, ip=10.128.43.108) (EngineCore_DP0 pid=1003263) INFO 05-23 01:55:04 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) WARNING 05-23 01:55:00 [arg_utils.py:1256] The global random seed is set to 84. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) INFO 05-23 01:55:00 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) INFO 05-23 01:55:00 [model.py:1549] Using max model len 32768 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) INFO 05-23 01:55:00 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) INFO 05-23 01:55:00 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) INFO 05-23 01:55:00 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) WARNING 05-23 01:55:00 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) INFO 05-23 01:55:00 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1002904, ip=10.128.43.108) (EngineCore_DP0 pid=1003263) INFO 05-23 01:55:04 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) WARNING 05-23 01:55:00 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) WARNING 05-23 01:55:00 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1002904, ip=10.128.43.108) (EngineCore_DP0 pid=1003263) INFO 05-23 01:55:05 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 8x across cluster]
(pid=1461509) ⚙️ Running in WANDB offline mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2095514, ip=10.128.43.86) (EngineCore_DP0 pid=2095768) INFO 05-23 01:55:07 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117) INFO 05-23 01:55:08 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=86, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2095642, ip=10.128.43.86) (EngineCore_DP0 pid=2095747) INFO 05-23 01:55:08 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.43.86:50391 backend=nccl [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:09 [default_loader.py:293] Loading weights took 8.80 seconds
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) (EngineCore_DP0 pid=2095751) INFO 05-23 01:55:09 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:09 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 9.163299 seconds
(AsyncVLLMInferenceEngine pid=2095386, ip=10.128.43.86) (EngineCore_DP0 pid=2095751) INFO 05-23 01:55:09 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2552034, ip=10.128.43.88) (EngineCore_DP0 pid=2552274) INFO 05-23 01:55:11 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:12 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:12 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:12 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=868952, ip=10.128.43.90) (EngineCore_DP0 pid=869183) INFO 05-23 01:55:11 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:12 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:12 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.11 seconds
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) WARNING 05-23 01:55:13 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:13 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) WARNING 05-23 01:55:13 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) (EngineCore_DP0 pid=876753) INFO 05-23 01:55:13 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=876379, ip=10.128.43.96) WARNING 05-23 01:55:13 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(AsyncVLLMInferenceEngine pid=868953, ip=10.128.43.90) (EngineCore_DP0 pid=869207) INFO 05-23 01:55:09 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=84, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) INFO 05-23 01:55:14 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.43.103:43683 backend=nccl [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) INFO 05-23 01:55:14 [default_loader.py:293] Loading weights took 13.88 seconds [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117) INFO 05-23 01:55:14 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) INFO 05-23 01:55:14 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.858314 seconds [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117) INFO 05-23 01:55:14 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) INFO 05-23 01:55:15 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) (EngineCore_DP0 pid=1658393) INFO 05-23 01:55:17 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) (EngineCore_DP0 pid=1658393) INFO 05-23 01:55:17 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) (EngineCore_DP0 pid=1658393) INFO 05-23 01:55:17 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117) INFO 05-23 01:55:14 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) (EngineCore_DP0 pid=1658393) INFO 05-23 01:55:17 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) INFO 05-23 01:55:17 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.76 seconds [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) WARNING 05-23 01:55:17 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) INFO 05-23 01:55:17 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) WARNING 05-23 01:55:17 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=1658162, ip=10.128.43.87) (EngineCore_DP0 pid=1658405) INFO 05-23 01:55:17 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 22x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) NCCL version 2.27.7+cuda13.0
(AsyncVLLMInferenceEngine pid=1658163, ip=10.128.43.87) WARNING 05-23 01:55:18 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=975760, ip=10.128.43.103) (EngineCore_DP0 pid=976117) INFO 05-23 01:55:14 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.43.103:58483 backend=nccl
(AsyncVLLMInferenceEngine pid=900915, ip=10.128.43.91) (EngineCore_DP0 pid=901149) INFO 05-23 01:55:17 [default_loader.py:293] Loading weights took 13.60 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900915, ip=10.128.43.91) (EngineCore_DP0 pid=901149) INFO 05-23 01:55:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.850248 seconds [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/model_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/extra_state_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/optim_world_size_8_rank_0.pt
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) INFO 05-23 01:55:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) INFO 05-23 01:55:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) INFO 05-23 01:55:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900915, ip=10.128.43.91) (EngineCore_DP0 pid=901149) INFO 05-23 01:55:21 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) INFO 05-23 01:55:21 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) WARNING 05-23 01:55:21 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) INFO 05-23 01:55:21 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) WARNING 05-23 01:55:21 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) (EngineCore_DP0 pid=901141) INFO 05-23 01:55:21 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=900781, ip=10.128.43.91) WARNING 05-23 01:55:21 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=940164, ip=10.128.43.95) (EngineCore_DP0 pid=940526) INFO 05-23 01:55:25 [default_loader.py:293] Loading weights took 13.84 seconds [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267) INFO 05-23 01:55:25 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.506031 seconds [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531) INFO 05-23 01:55:28 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531) INFO 05-23 01:55:28 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531) INFO 05-23 01:55:28 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531) INFO 05-23 01:55:29 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) (EngineCore_DP0 pid=940531) INFO 05-23 01:55:29 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.88 seconds [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267) WARNING 05-23 01:55:28 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267) INFO 05-23 01:55:28 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267) WARNING 05-23 01:55:28 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2552033, ip=10.128.43.88) (EngineCore_DP0 pid=2552267) INFO 05-23 01:55:28 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=940292, ip=10.128.43.95) WARNING 05-23 01:55:29 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) INFO 05-23 01:55:30 [default_loader.py:293] Loading weights took 14.02 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) INFO 05-23 01:55:30 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.199574 seconds [repeated 12x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank-0]: Successfully loaded model state dict
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) (EngineCore_DP0 pid=976133) INFO 05-23 01:55:32 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) (EngineCore_DP0 pid=976133) INFO 05-23 01:55:32 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) (EngineCore_DP0 pid=976133) INFO 05-23 01:55:32 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) (EngineCore_DP0 pid=976133) INFO 05-23 01:55:33 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) INFO 05-23 01:55:33 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.75 seconds [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) WARNING 05-23 01:55:33 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) INFO 05-23 01:55:33 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) WARNING 05-23 01:55:33 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=975888, ip=10.128.43.103) (EngineCore_DP0 pid=976122) INFO 05-23 01:55:33 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=975890, ip=10.128.43.103) WARNING 05-23 01:55:33 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 8x across cluster]
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank-0]: Successfully loaded optimizer state
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank-0]: Successfully loaded scheduler state
(FSDPPolicyWorkerBase pid=851205, ip=10.128.43.85) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy
(AsyncVLLMInferenceEngine pid=937427, ip=10.128.43.84) ERROR 05-23 01:55:45 [core_client.py:616] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=token_mean', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/a3-rl-DCAgent_code-contests-noblock', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=true', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=a3-rl-DCAgent_code-contests-noblock', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=450', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/code-contests-noblock"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=1800', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=900', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.model_info.max_input_tokens=32768', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 133, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 123, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ActorDiedError): ray::skyrl_entrypoint() (pid=1461281, ip=10.128.43.81)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 96, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 403, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state
ray.get(
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=851205, ip=10.128.43.85, actor_id=8b87233f106bcf2285f12e1602000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400dfe746e10>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state
results = await asyncio.gather(*tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator
rank_offset_count += engine.tp_size() * engine.pp_size()
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size
return ray.get(self.inference_engine_actor.tp_size.remote())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=876507, ip=10.128.43.96, actor_id=58494295fdaf9ec8226b54f502000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400be192c230>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__
super().__init__(*args, **kwargs)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__
self.llm = self._create_engine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1081, in _create_engine
engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
return cls(
^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
self.engine_core = EngineCoreClient.make_async_mp_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
return AsyncMPClient(*client_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
super().__init__(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
with launch_core_engines(vllm_config, executor_class, log_stats) as (
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
wait_for_engine_startup(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Stopping Ray cluster...
Ray cluster stopped
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/a3-rl-DCAgent_code-contests-noblock
job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock
episodes: last
log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/logs/a3-rl-DCAgent_code-contests-noblock_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/ray_logs/
Collecting Ray logs from worker jpbo-078-36...
Collecting Ray logs from worker jpbo-078-37...
Collecting Ray logs from worker jpbo-078-38...
Collecting Ray logs from worker jpbo-078-39...
Collecting Ray logs from worker jpbo-078-40...
Collecting Ray logs from worker jpbo-078-42...
Collecting Ray logs from worker jpbo-078-43...
Collecting Ray logs from worker jpbo-078-45...
Collecting Ray logs from worker jpbo-078-46...
Collecting Ray logs from worker jpbo-078-47...
Collecting Ray logs from worker jpbo-078-48...
Collecting Ray logs from worker jpbo-079-07...
Collecting Ray logs from worker jpbo-079-12...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink