The following have been reloaded with a version change: 1) GCCcore/.14.3.0 => GCCcore/14.3.0 Lmod is automatically replacing "GCC/14.3.0" with "nvidia-compilers/25.9-CUDA-13". Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python [ray] RAY_TMPDIR=/tmp/ray/ray_496419 [triton_cache] Triton cache: /tmp/triton_cache_feuer1_496419 [triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_496419 [proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 [proxy] Setting up SSH tunnel to jpbl-s01-01 [proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc [proxy] Tunnel port: 7003 [proxy] Node IP: 10.128.17.210 (workers will connect here) [proxy] ✓ SSH tunnel started successfully [proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496419.conf [proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT [proxy] - External traffic (internet) → PROXY via tunnel [proxy] ✓ Daytona timeout settings configured [proxy] Testing proxy connectivity... [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496419.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary) [proxy] ⚠ Tunnel not accessible at 10.128.17.210:7003 (workers may fail) [proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers) [container_runtime] Using cloud backend: daytona (no local container setup) === Universal RL Training Runner === Config: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/configs/a3-rl-DCAgent_code-contests-noblock_rl_config.json Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python version: Python 3.12.12 Proxy: DISABLED (direct internet or not configured) ======================================== === RLJobRunner: a3-rl-DCAgent_code-contests-noblock === [wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb [wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb HF_TOKEN=****pDbg HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config) Environment configured: TENSOR_PARALLEL_SIZE=1 NUM_INFERENCE_ENGINES=56 POLICY_NUM_NODES=14 WANDB_DIR=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/wandb Starting Ray cluster with 14 nodes, 4 GPUs/node Cleaning up existing Ray instances... === Starting Ray Cluster === Nodes: 14 GPUs per node: 4 CPUs per node: 288 Head node: jpbo-010-34 (10.128.17.210) Ray port: 6379 ============================ Starting Ray head on jpbo-010-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-010-34.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.210 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.17.210 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray head on jpbo-010-34 Starting Ray worker on jpbo-010-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-36.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.212 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.212 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 1 on jpbo-010-36 Starting Ray worker on jpbo-010-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-38.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.214 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.214 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 2 on jpbo-010-38 Starting Ray worker on jpbo-010-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-39.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.215 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.215 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 3 on jpbo-010-39 Starting Ray worker on jpbo-010-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-40.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.216 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.216 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 4 on jpbo-010-40 Starting Ray worker on jpbo-010-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-41.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.217 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.217 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 5 on jpbo-010-41 Starting Ray worker on jpbo-010-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-43.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.219 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.219 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 6 on jpbo-010-43 Starting Ray worker on jpbo-010-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-44.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.220 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.220 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 7 on jpbo-010-44 Starting Ray worker on jpbo-010-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-45.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.221 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.221 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 8 on jpbo-010-45 Starting Ray worker on jpbo-010-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-010-46.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.17.222 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-010-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.17.222 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 9 on jpbo-010-46 Starting Ray worker on jpbo-022-02 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-02.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.50 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.50 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 10 on jpbo-022-02 Starting Ray worker on jpbo-022-03 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-03.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.51 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.51 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 11 on jpbo-022-03 Starting Ray worker on jpbo-022-04 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-04.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.52 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.52 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 12 on jpbo-022-04 Starting Ray worker on jpbo-022-05 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-022-05.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.24.53 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-022-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.17.210:6379 --node-ip-address=10.128.24.53 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 13 on jpbo-022-05 Waiting for cluster (56 GPUs, 14 nodes)... Connecting to Ray at 10.128.17.210:6379 (expecting 14 nodes, 56.0 GPUs) Ray connection established, polling for resources... [Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'accelerator_type:GH200': 14.0, 'CPU': 4032.0, 'GPU': 56.0, 'node:10.128.17.216': 1.0, 'object_store_memory': 601295421440.0, 'memory': 10779218477056.0, 'node:10.128.17.222': 1.0, 'node:10.128.17.221': 1.0, 'node:10.128.24.50': 1.0, 'node:10.128.24.52': 1.0, 'node:10.128.17.215': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.17.210': 1.0, 'node:10.128.17.214': 1.0, 'node:10.128.24.51': 1.0, 'node:10.128.24.53': 1.0, 'node:10.128.17.217': 1.0, 'node:10.128.17.220': 1.0, 'node:10.128.17.219': 1.0, 'node:10.128.17.212': 1.0} ✓ Ray cluster ready === Ray Cluster Ready === Address: 10.128.17.210:6379 Total GPUs: 56 ========================= Ray cluster ready at 10.128.17.210:6379 Total GPUs available: 56 [RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona) [RLJobRunner] No Pinggy tunnel needed, using local vLLM Running SkyRL: Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Entrypoint: examples.terminal_bench.entrypoints.main_tbench Args: 119 Hydra arguments Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=token_mean trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/a3-rl-DCAgent_code-contests-noblock ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=true trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=a3-rl-DCAgent_code-contests-noblock trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=450 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/code-contests-noblock"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=1800 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=900 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.model_info.max_input_tokens=32768 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_496419.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e 2026-05-23 15:16:29.498 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:590 - Exporting wandb api key to ray runtime env 2026-05-23 15:16:29.500 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:609 - Exporting RAY_ADDRESS to ray runtime env 2026-05-23 15:16:29,500 INFO worker.py:1680 -- Using address 10.128.17.210:6379 set in the environment variable RAY_ADDRESS 2026-05-23 15:16:29,536 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.17.210:6379... 2026-05-23 15:16:29,547 INFO worker.py:2007 -- Connected to Ray cluster. /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( (raylet, ip=10.128.24.53) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e 2026-05-23 15:16:32.151 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor (skyrl_entrypoint pid=3659716) 2026-05-23 15:16:41.392 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:199 - SkyRL log level set to: INFO (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (skyrl_entrypoint pid=3659716) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (skyrl_entrypoint pid=3659716) 2026-05-23 15:16:41.632 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/code-contests-noblock [2026-05-23 15:16:59,779 E 3659268 3659688] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (RegistryActor pid=1440387, ip=10.128.24.53) [2026-05-23 15:17:00,782 E 1440387 1440427] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (skyrl_entrypoint pid=3659716) 2026-05-23 15:18:34.306 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 8728 valid task directories out of 8728 total directories (skyrl_entrypoint pid=3659716) 2026-05-23 15:18:34.306 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 8728 task paths (skyrl_entrypoint pid=3659716) 2026-05-23 15:18:34.317 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:352 - data: (skyrl_entrypoint pid=3659716) train_data: (skyrl_entrypoint pid=3659716) - /e/scratch/jureap59/feuer1/tasks/code-contests-noblock (skyrl_entrypoint pid=3659716) val_data: [] (skyrl_entrypoint pid=3659716) trainer: (skyrl_entrypoint pid=3659716) placement: (skyrl_entrypoint pid=3659716) colocate_all: false (skyrl_entrypoint pid=3659716) colocate_policy_ref: true (skyrl_entrypoint pid=3659716) policy_num_nodes: 2 (skyrl_entrypoint pid=3659716) policy_num_gpus_per_node: 4 (skyrl_entrypoint pid=3659716) critic_num_nodes: 1 (skyrl_entrypoint pid=3659716) critic_num_gpus_per_node: 4 (skyrl_entrypoint pid=3659716) ref_num_nodes: 2 (skyrl_entrypoint pid=3659716) ref_num_gpus_per_node: 4 (skyrl_entrypoint pid=3659716) sequence_parallel_backend: ulysses (skyrl_entrypoint pid=3659716) strategy: fsdp2 (skyrl_entrypoint pid=3659716) policy: (skyrl_entrypoint pid=3659716) model: (skyrl_entrypoint pid=3659716) path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 (skyrl_entrypoint pid=3659716) lora: (skyrl_entrypoint pid=3659716) rank: 0 (skyrl_entrypoint pid=3659716) alpha: 16 (skyrl_entrypoint pid=3659716) dropout: 0 (skyrl_entrypoint pid=3659716) lora_sync_path: /tmp/skyrl_lora_sync (skyrl_entrypoint pid=3659716) target_modules: all-linear (skyrl_entrypoint pid=3659716) exclude_modules: null (skyrl_entrypoint pid=3659716) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=3659716) optimizer_config: (skyrl_entrypoint pid=3659716) optimizer: AdamW (skyrl_entrypoint pid=3659716) lr: 8.0e-06 (skyrl_entrypoint pid=3659716) adam_betas: (skyrl_entrypoint pid=3659716) - 0.9 (skyrl_entrypoint pid=3659716) - 0.999 (skyrl_entrypoint pid=3659716) weight_decay: 0.0 (skyrl_entrypoint pid=3659716) max_grad_norm: 0.9 (skyrl_entrypoint pid=3659716) offload_after_step: true (skyrl_entrypoint pid=3659716) num_warmup_steps: 0 (skyrl_entrypoint pid=3659716) scheduler: constant_with_warmup (skyrl_entrypoint pid=3659716) optimizer_kwargs: {} (skyrl_entrypoint pid=3659716) fsdp_config: (skyrl_entrypoint pid=3659716) cpu_offload: false (skyrl_entrypoint pid=3659716) reshard_after_forward: true (skyrl_entrypoint pid=3659716) fsdp_size: 4 (skyrl_entrypoint pid=3659716) sequence_parallel_size: 1 (skyrl_entrypoint pid=3659716) use_torch_compile: false (skyrl_entrypoint pid=3659716) record_memory: false (skyrl_entrypoint pid=3659716) megatron_config: (skyrl_entrypoint pid=3659716) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=3659716) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=3659716) context_parallel_size: 1 (skyrl_entrypoint pid=3659716) expert_model_parallel_size: 1 (skyrl_entrypoint pid=3659716) expert_tensor_parallel_size: null (skyrl_entrypoint pid=3659716) ddp_config: (skyrl_entrypoint pid=3659716) grad_reduce_in_fp32: true (skyrl_entrypoint pid=3659716) overlap_grad_reduce: false (skyrl_entrypoint pid=3659716) overlap_param_gather: false (skyrl_entrypoint pid=3659716) average_in_collective: true (skyrl_entrypoint pid=3659716) model_config_kwargs: {} (skyrl_entrypoint pid=3659716) torch_profiler_config: (skyrl_entrypoint pid=3659716) enable: false (skyrl_entrypoint pid=3659716) ranks: [] (skyrl_entrypoint pid=3659716) save_path: null (skyrl_entrypoint pid=3659716) optimizer_config_kwargs: (skyrl_entrypoint pid=3659716) overlap_cpu_optimizer_d2h_h2d: false (skyrl_entrypoint pid=3659716) use_precision_aware_optimizer: false (skyrl_entrypoint pid=3659716) optimizer_cpu_offload: false (skyrl_entrypoint pid=3659716) optimizer_offload_fraction: 0.0 (skyrl_entrypoint pid=3659716) transformer_config_kwargs: (skyrl_entrypoint pid=3659716) recompute_granularity: full (skyrl_entrypoint pid=3659716) recompute_modules: (skyrl_entrypoint pid=3659716) - core_attn (skyrl_entrypoint pid=3659716) recompute_method: uniform (skyrl_entrypoint pid=3659716) recompute_num_layers: 1 (skyrl_entrypoint pid=3659716) empty_cuda_cache: true (skyrl_entrypoint pid=3659716) ref: (skyrl_entrypoint pid=3659716) model: (skyrl_entrypoint pid=3659716) path: ${trainer.policy.model.path} (skyrl_entrypoint pid=3659716) sequence_parallel_size: 1 (skyrl_entrypoint pid=3659716) deepspeed_config: ${deepspeed_config.eval} (skyrl_entrypoint pid=3659716) fsdp_config: (skyrl_entrypoint pid=3659716) cpu_offload: false (skyrl_entrypoint pid=3659716) reshard_after_forward: true (skyrl_entrypoint pid=3659716) fsdp_size: 4 (skyrl_entrypoint pid=3659716) megatron_config: (skyrl_entrypoint pid=3659716) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=3659716) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=3659716) context_parallel_size: 1 (skyrl_entrypoint pid=3659716) expert_model_parallel_size: 1 (skyrl_entrypoint pid=3659716) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=3659716) model_config_kwargs: {} (skyrl_entrypoint pid=3659716) transformer_config_kwargs: {} (skyrl_entrypoint pid=3659716) critic: (skyrl_entrypoint pid=3659716) model: (skyrl_entrypoint pid=3659716) path: null (skyrl_entrypoint pid=3659716) lora: (skyrl_entrypoint pid=3659716) rank: 0 (skyrl_entrypoint pid=3659716) alpha: 16 (skyrl_entrypoint pid=3659716) dropout: 0 (skyrl_entrypoint pid=3659716) target_modules: all-linear (skyrl_entrypoint pid=3659716) exclude_modules: null (skyrl_entrypoint pid=3659716) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=3659716) optimizer_config: (skyrl_entrypoint pid=3659716) optimizer: AdamW (skyrl_entrypoint pid=3659716) lr: 5.0e-06 (skyrl_entrypoint pid=3659716) adam_betas: (skyrl_entrypoint pid=3659716) - 0.9 (skyrl_entrypoint pid=3659716) - 0.999 (skyrl_entrypoint pid=3659716) weight_decay: 0.01 (skyrl_entrypoint pid=3659716) max_grad_norm: 1.0 (skyrl_entrypoint pid=3659716) offload_after_step: true (skyrl_entrypoint pid=3659716) num_warmup_steps: 0 (skyrl_entrypoint pid=3659716) scheduler: constant_with_warmup (skyrl_entrypoint pid=3659716) optimizer_kwargs: {} (skyrl_entrypoint pid=3659716) fsdp_config: (skyrl_entrypoint pid=3659716) cpu_offload: false (skyrl_entrypoint pid=3659716) reshard_after_forward: true (skyrl_entrypoint pid=3659716) fsdp_size: -1 (skyrl_entrypoint pid=3659716) sequence_parallel_size: 1 (skyrl_entrypoint pid=3659716) algorithm: (skyrl_entrypoint pid=3659716) advantage_estimator: rloo_n (skyrl_entrypoint pid=3659716) kl_ctrl: (skyrl_entrypoint pid=3659716) type: fixed (skyrl_entrypoint pid=3659716) kl_target: 0.1 (skyrl_entrypoint pid=3659716) horizon: 10000 (skyrl_entrypoint pid=3659716) kl_estimator_type: k3 (skyrl_entrypoint pid=3659716) use_kl_estimator_k3: false (skyrl_entrypoint pid=3659716) use_abs_kl: false (skyrl_entrypoint pid=3659716) use_kl_in_reward: false (skyrl_entrypoint pid=3659716) use_kl_loss: false (skyrl_entrypoint pid=3659716) kl_loss_coef: 0.0 (skyrl_entrypoint pid=3659716) use_entropy_loss: false (skyrl_entrypoint pid=3659716) entropy_loss_coef: 0.01 (skyrl_entrypoint pid=3659716) advantage_batch_normalize: false (skyrl_entrypoint pid=3659716) value_head_prefix: value_head (skyrl_entrypoint pid=3659716) policy_loss_type: regular (skyrl_entrypoint pid=3659716) loss_reduction: token_mean (skyrl_entrypoint pid=3659716) grpo_norm_by_std: true (skyrl_entrypoint pid=3659716) rloo_n_min_group_size: 4 (skyrl_entrypoint pid=3659716) rloo_n_filter_zero_reward_groups: true (skyrl_entrypoint pid=3659716) lambd: 1.0 (skyrl_entrypoint pid=3659716) gamma: 1.0 (skyrl_entrypoint pid=3659716) eps_clip_low: 0.2 (skyrl_entrypoint pid=3659716) eps_clip_high: 0.05 (skyrl_entrypoint pid=3659716) clip_ratio_c: 3.0 (skyrl_entrypoint pid=3659716) tis_imp_ratio_cap: -1.0 (skyrl_entrypoint pid=3659716) use_tis: false (skyrl_entrypoint pid=3659716) sapo: (skyrl_entrypoint pid=3659716) tau_pos: 1.0 (skyrl_entrypoint pid=3659716) tau_neg: 1.05 (skyrl_entrypoint pid=3659716) value_clip: 0.2 (skyrl_entrypoint pid=3659716) dynamic_sampling: (skyrl_entrypoint pid=3659716) type: null (skyrl_entrypoint pid=3659716) max_sample_batches: 30 (skyrl_entrypoint pid=3659716) min_replace_ratio: 0.3 (skyrl_entrypoint pid=3659716) clip_cov: (skyrl_entrypoint pid=3659716) clip_ratio: 0.0002 (skyrl_entrypoint pid=3659716) clip_cov_lb: 1.0 (skyrl_entrypoint pid=3659716) clip_cov_ub: 5.0 (skyrl_entrypoint pid=3659716) kl_cov: (skyrl_entrypoint pid=3659716) kl_cov_frac: 0.2 (skyrl_entrypoint pid=3659716) ppo_kl_coef: 1.0 (skyrl_entrypoint pid=3659716) cispo: (skyrl_entrypoint pid=3659716) cispo_eps_clip_low: 0 (skyrl_entrypoint pid=3659716) cispo_eps_clip_high: 5 (skyrl_entrypoint pid=3659716) z_clip: (skyrl_entrypoint pid=3659716) enabled: false (skyrl_entrypoint pid=3659716) alpha: 0.97 (skyrl_entrypoint pid=3659716) z_thresh: 2.5 (skyrl_entrypoint pid=3659716) warmup_steps: 25 (skyrl_entrypoint pid=3659716) mode: zscore (skyrl_entrypoint pid=3659716) clip_option: adaptive_scaling (skyrl_entrypoint pid=3659716) clip_factor: 1.0 (skyrl_entrypoint pid=3659716) skip_update_on_spike: false (skyrl_entrypoint pid=3659716) stale_clip: (skyrl_entrypoint pid=3659716) enabled: false (skyrl_entrypoint pid=3659716) alpha: 0.3 (skyrl_entrypoint pid=3659716) entropy_threshold: 0.15 (skyrl_entrypoint pid=3659716) entropy_window: 10 (skyrl_entrypoint pid=3659716) min_lr_scale: 0.1 (skyrl_entrypoint pid=3659716) max_seq_len: 1004095 (skyrl_entrypoint pid=3659716) fully_async: (skyrl_entrypoint pid=3659716) max_staleness_steps: 16 (skyrl_entrypoint pid=3659716) num_parallel_generation_workers: 450 (skyrl_entrypoint pid=3659716) gradient_checkpointing: true (skyrl_entrypoint pid=3659716) gradient_checkpointing_use_reentrant: false (skyrl_entrypoint pid=3659716) seed: 42 (skyrl_entrypoint pid=3659716) resume_mode: latest (skyrl_entrypoint pid=3659716) resume_path: null (skyrl_entrypoint pid=3659716) ckpt_path: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints (skyrl_entrypoint pid=3659716) max_ckpts_to_keep: -1 (skyrl_entrypoint pid=3659716) ckpt_interval: 2 (skyrl_entrypoint pid=3659716) hf_save_interval: 5 (skyrl_entrypoint pid=3659716) export_path: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports (skyrl_entrypoint pid=3659716) bf16: true (skyrl_entrypoint pid=3659716) epochs: 2 (skyrl_entrypoint pid=3659716) max_steps: 80 (skyrl_entrypoint pid=3659716) update_epochs_per_batch: 1 (skyrl_entrypoint pid=3659716) train_batch_size: 64 (skyrl_entrypoint pid=3659716) policy_mini_batch_size: 64 (skyrl_entrypoint pid=3659716) critic_mini_batch_size: 256 (skyrl_entrypoint pid=3659716) micro_train_batch_size_per_gpu: 1 (skyrl_entrypoint pid=3659716) micro_forward_batch_size_per_gpu: 4 (skyrl_entrypoint pid=3659716) update_ref_every_epoch: false (skyrl_entrypoint pid=3659716) use_sample_packing: true (skyrl_entrypoint pid=3659716) eval_batch_size: 64 (skyrl_entrypoint pid=3659716) eval_before_train: false (skyrl_entrypoint pid=3659716) eval_interval: 999999 (skyrl_entrypoint pid=3659716) max_prompt_length: 999999 (skyrl_entrypoint pid=3659716) flash_attn: true (skyrl_entrypoint pid=3659716) disable_fast_tokenizer: false (skyrl_entrypoint pid=3659716) target_modules: null (skyrl_entrypoint pid=3659716) exclude_modules: null (skyrl_entrypoint pid=3659716) project_name: OpenThoughts-Agent (skyrl_entrypoint pid=3659716) run_name: a3-rl-DCAgent_code-contests-noblock (skyrl_entrypoint pid=3659716) logger: console (skyrl_entrypoint pid=3659716) tracker_commit_each_step: true (skyrl_entrypoint pid=3659716) dump_data_batch: false (skyrl_entrypoint pid=3659716) dump_eval_results: true (skyrl_entrypoint pid=3659716) log_level: INFO (skyrl_entrypoint pid=3659716) rope_scaling: null (skyrl_entrypoint pid=3659716) rope_theta: null (skyrl_entrypoint pid=3659716) step_wise_training: false (skyrl_entrypoint pid=3659716) hf_hub_repo_id: laion/a3-rl-DCAgent_code-contests-noblock (skyrl_entrypoint pid=3659716) hf_hub_private: false (skyrl_entrypoint pid=3659716) hf_hub_revision: main (skyrl_entrypoint pid=3659716) enable_db_registration: true (skyrl_entrypoint pid=3659716) generator: (skyrl_entrypoint pid=3659716) model_name: ${trainer.policy.model.path} (skyrl_entrypoint pid=3659716) model_dtype: bfloat16 (skyrl_entrypoint pid=3659716) timeout_multiplier: 1.0 (skyrl_entrypoint pid=3659716) run_engines_locally: true (skyrl_entrypoint pid=3659716) num_inference_engines: 48 (skyrl_entrypoint pid=3659716) backend: vllm (skyrl_entrypoint pid=3659716) weight_sync_backend: nccl (skyrl_entrypoint pid=3659716) fuse_weights: false (skyrl_entrypoint pid=3659716) weight_transfer_threshold_cuda_ipc_GB: 1.0 (skyrl_entrypoint pid=3659716) inference_engine_tensor_parallel_size: 1 (skyrl_entrypoint pid=3659716) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=3659716) inference_engine_expert_parallel_size: 1 (skyrl_entrypoint pid=3659716) inference_engine_data_parallel_size: 1 (skyrl_entrypoint pid=3659716) n_samples_per_prompt: 8 (skyrl_entrypoint pid=3659716) async_engine: true (skyrl_entrypoint pid=3659716) batched: false (skyrl_entrypoint pid=3659716) max_input_length: ${trainer.max_prompt_length} (skyrl_entrypoint pid=3659716) vllm_v1_disable_multiproc: true (skyrl_entrypoint pid=3659716) enable_prefix_caching: true (skyrl_entrypoint pid=3659716) enable_chunked_prefill: true (skyrl_entrypoint pid=3659716) max_num_batched_tokens: 65536 (skyrl_entrypoint pid=3659716) enforce_eager: true (skyrl_entrypoint pid=3659716) fully_sharded_loras: false (skyrl_entrypoint pid=3659716) enable_ray_prometheus_stats: false (skyrl_entrypoint pid=3659716) vllm_stats_interval: 1 (skyrl_entrypoint pid=3659716) gpu_memory_utilization: 0.75 (skyrl_entrypoint pid=3659716) max_num_seqs: 24 (skyrl_entrypoint pid=3659716) remote_inference_engine_urls: (skyrl_entrypoint pid=3659716) - 127.0.0.1:8001 (skyrl_entrypoint pid=3659716) enable_http_endpoint: true (skyrl_entrypoint pid=3659716) http_endpoint_host: 127.0.0.1 (skyrl_entrypoint pid=3659716) http_endpoint_port: 8000 (skyrl_entrypoint pid=3659716) max_turns: 999999 (skyrl_entrypoint pid=3659716) chat_template: (skyrl_entrypoint pid=3659716) source: name (skyrl_entrypoint pid=3659716) name_or_path: null (skyrl_entrypoint pid=3659716) chat_template_kwargs: {} (skyrl_entrypoint pid=3659716) engine_init_kwargs: (skyrl_entrypoint pid=3659716) max_model_len: 32768 (skyrl_entrypoint pid=3659716) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=3659716) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 (skyrl_entrypoint pid=3659716) override_existing_update_group: disable (skyrl_entrypoint pid=3659716) sampling_params: (skyrl_entrypoint pid=3659716) max_generate_length: 4096 (skyrl_entrypoint pid=3659716) repetition_penalty: 1.0 (skyrl_entrypoint pid=3659716) temperature: 0.7 (skyrl_entrypoint pid=3659716) top_p: 0.95 (skyrl_entrypoint pid=3659716) min_p: 0.0 (skyrl_entrypoint pid=3659716) top_k: 20 (skyrl_entrypoint pid=3659716) logprobs: null (skyrl_entrypoint pid=3659716) stop: null (skyrl_entrypoint pid=3659716) use_conversation_multi_turn: true (skyrl_entrypoint pid=3659716) append_eos_token_after_stop_str_in_multi_turn: true (skyrl_entrypoint pid=3659716) eval_sampling_params: (skyrl_entrypoint pid=3659716) max_generate_length: ${generator.sampling_params.max_generate_length} (skyrl_entrypoint pid=3659716) repetition_penalty: 1.0 (skyrl_entrypoint pid=3659716) temperature: 0.0 (skyrl_entrypoint pid=3659716) top_p: 1.0 (skyrl_entrypoint pid=3659716) min_p: 0.0 (skyrl_entrypoint pid=3659716) top_k: -1 (skyrl_entrypoint pid=3659716) logprobs: null (skyrl_entrypoint pid=3659716) stop: null (skyrl_entrypoint pid=3659716) eval_n_samples_per_prompt: 8 (skyrl_entrypoint pid=3659716) zero_reward_on_non_stop: false (skyrl_entrypoint pid=3659716) apply_overlong_filtering: false (skyrl_entrypoint pid=3659716) rope_scaling: ${trainer.rope_scaling} (skyrl_entrypoint pid=3659716) rope_theta: ${trainer.rope_theta} (skyrl_entrypoint pid=3659716) teacher: (skyrl_entrypoint pid=3659716) model_path: null (skyrl_entrypoint pid=3659716) top_k_logprobs: 256 (skyrl_entrypoint pid=3659716) num_inference_engines: 1 (skyrl_entrypoint pid=3659716) inference_engine_tensor_parallel_size: 1 (skyrl_entrypoint pid=3659716) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=3659716) gpu_memory_utilization: 0.9 (skyrl_entrypoint pid=3659716) enforce_eager: false (skyrl_entrypoint pid=3659716) backend: vllm (skyrl_entrypoint pid=3659716) engine_init_kwargs: {} (skyrl_entrypoint pid=3659716) environment: (skyrl_entrypoint pid=3659716) env_class: gsm8k (skyrl_entrypoint pid=3659716) skyrl_gym: (skyrl_entrypoint pid=3659716) max_env_workers: 32 (skyrl_entrypoint pid=3659716) text2sql: (skyrl_entrypoint pid=3659716) db_path: /home/ray/default/sql_data (skyrl_entrypoint pid=3659716) llm_as_a_judge: (skyrl_entrypoint pid=3659716) model: gpt-4o-mini (skyrl_entrypoint pid=3659716) base_url: null (skyrl_entrypoint pid=3659716) search: (skyrl_entrypoint pid=3659716) log_requests: false (skyrl_entrypoint pid=3659716) search_url: http://127.0.0.1:8000/retrieve (skyrl_entrypoint pid=3659716) topk: 3 (skyrl_entrypoint pid=3659716) timeout: 30 (skyrl_entrypoint pid=3659716) deepspeed_config: (skyrl_entrypoint pid=3659716) train: (skyrl_entrypoint pid=3659716) zero_optimization: (skyrl_entrypoint pid=3659716) stage: 3 (skyrl_entrypoint pid=3659716) offload_param: (skyrl_entrypoint pid=3659716) device: none (skyrl_entrypoint pid=3659716) offload_optimizer: (skyrl_entrypoint pid=3659716) device: none (skyrl_entrypoint pid=3659716) pin_memory: true (skyrl_entrypoint pid=3659716) sub_group_size: auto (skyrl_entrypoint pid=3659716) reduce_bucket_size: auto (skyrl_entrypoint pid=3659716) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=3659716) stage3_prefetch_bucket_size: auto (skyrl_entrypoint pid=3659716) stage3_max_live_parameters: auto (skyrl_entrypoint pid=3659716) stage3_max_reuse_distance: auto (skyrl_entrypoint pid=3659716) round_robin_gradients: true (skyrl_entrypoint pid=3659716) zero_hpz_partition_size: 1 (skyrl_entrypoint pid=3659716) zero_quantized_weights: false (skyrl_entrypoint pid=3659716) zero_quantized_gradients: false (skyrl_entrypoint pid=3659716) torch_autocast: (skyrl_entrypoint pid=3659716) enabled: true (skyrl_entrypoint pid=3659716) dtype: bfloat16 (skyrl_entrypoint pid=3659716) disable_trace_cache: false (skyrl_entrypoint pid=3659716) data_types: (skyrl_entrypoint pid=3659716) grad_accum_dtype: fp32 (skyrl_entrypoint pid=3659716) gradient_clipping: 1.0 (skyrl_entrypoint pid=3659716) wall_clock_breakdown: false (skyrl_entrypoint pid=3659716) prescale_gradient: false (skyrl_entrypoint pid=3659716) eval: (skyrl_entrypoint pid=3659716) zero_optimization: (skyrl_entrypoint pid=3659716) stage: 3 (skyrl_entrypoint pid=3659716) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=3659716) offload_param: (skyrl_entrypoint pid=3659716) device: cpu (skyrl_entrypoint pid=3659716) pin_memory: true (skyrl_entrypoint pid=3659716) torch_autocast: (skyrl_entrypoint pid=3659716) enabled: true (skyrl_entrypoint pid=3659716) dtype: bfloat16 (skyrl_entrypoint pid=3659716) gradient_clipping: 1.0 (skyrl_entrypoint pid=3659716) prescale_gradient: false (skyrl_entrypoint pid=3659716) wall_clock_breakdown: false (skyrl_entrypoint pid=3659716) terminal_bench_config: (skyrl_entrypoint pid=3659716) trials_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs (skyrl_entrypoint pid=3659716) harbor: (skyrl_entrypoint pid=3659716) name: terminus-2 (skyrl_entrypoint pid=3659716) max_episodes: 999999 (skyrl_entrypoint pid=3659716) enable_summarize: false (skyrl_entrypoint pid=3659716) store_all_messages: true (skyrl_entrypoint pid=3659716) trajectory_config: (skyrl_entrypoint pid=3659716) raw_content: true (skyrl_entrypoint pid=3659716) enable_episode_logging: false (skyrl_entrypoint pid=3659716) record_terminal_session: false (skyrl_entrypoint pid=3659716) enable_pane_logging: false (skyrl_entrypoint pid=3659716) strict_json_parser: true (skyrl_entrypoint pid=3659716) interleaved_thinking: true (skyrl_entrypoint pid=3659716) extra_body: (skyrl_entrypoint pid=3659716) chat_template_kwargs: (skyrl_entrypoint pid=3659716) enable_thinking: true (skyrl_entrypoint pid=3659716) override_timeout_sec: 1800 (skyrl_entrypoint pid=3659716) override_cpus: 1 (skyrl_entrypoint pid=3659716) override_memory_mb: 2048 (skyrl_entrypoint pid=3659716) override_storage_mb: 2048 (skyrl_entrypoint pid=3659716) auto_snapshot: true (skyrl_entrypoint pid=3659716) verifier_override_timeout_sec: 120 (skyrl_entrypoint pid=3659716) max_retries: 3 (skyrl_entrypoint pid=3659716) min_wait_sec: 60.0 (skyrl_entrypoint pid=3659716) max_wait_sec: 600.0 (skyrl_entrypoint pid=3659716) wait_multiplier: 2.0 (skyrl_entrypoint pid=3659716) exclude_exceptions: (skyrl_entrypoint pid=3659716) - VerifierTimeoutError (skyrl_entrypoint pid=3659716) - VerifierRuntimeError (skyrl_entrypoint pid=3659716) - RewardFileNotFoundError (skyrl_entrypoint pid=3659716) - RewardFileEmptyError (skyrl_entrypoint pid=3659716) - VerifierOutputParseError (skyrl_entrypoint pid=3659716) n_concurrent_trials: 900 (skyrl_entrypoint pid=3659716) log_level: INFO (skyrl_entrypoint pid=3659716) enable_reward_shaping: false (skyrl_entrypoint pid=3659716) enable_error_classification: true (skyrl_entrypoint pid=3659716) mask_exceptions: (skyrl_entrypoint pid=3659716) - DaytonaError (skyrl_entrypoint pid=3659716) - EnvironmentStartTimeoutError (skyrl_entrypoint pid=3659716) - NetworkError (skyrl_entrypoint pid=3659716) - ConnectionError (skyrl_entrypoint pid=3659716) - RewardFileNotFoundError (skyrl_entrypoint pid=3659716) - RewardFileEmptyError (skyrl_entrypoint pid=3659716) - AgentEnvironmentTimeoutError (skyrl_entrypoint pid=3659716) default_error_treatment: zero (skyrl_entrypoint pid=3659716) passthrough_exceptions: (skyrl_entrypoint pid=3659716) - AgentTimeoutError (skyrl_entrypoint pid=3659716) - ContextLengthExceededError (skyrl_entrypoint pid=3659716) model_info: (skyrl_entrypoint pid=3659716) max_input_tokens: 32768 (skyrl_entrypoint pid=3659716) max_output_tokens: 4096 (skyrl_entrypoint pid=3659716) archiving: (skyrl_entrypoint pid=3659716) enabled: false (skyrl_entrypoint pid=3659716) trace_upload: (skyrl_entrypoint pid=3659716) enabled: true (skyrl_entrypoint pid=3659716) repo_org: DCAgent (skyrl_entrypoint pid=3659716) episodes: last (skyrl_entrypoint pid=3659716) dataset_type: SFT (skyrl_entrypoint pid=3659716) cleanup: true (skyrl_entrypoint pid=3659716)  (skyrl_entrypoint pid=3659716) [2026-05-23 15:17:03,082 E 3659716 3659759] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster] (skyrl_entrypoint pid=3659716) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (skyrl_entrypoint pid=3659716) No module named 'vllm._version' (skyrl_entrypoint pid=3659716) from .version import __version__, __version_tuple__ # isort:skip (skyrl_entrypoint pid=3659716) W0523 15:18:39.734000 3659716 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (raylet, ip=10.128.24.51) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster] (pid=3294862, ip=10.128.17.219) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (pid=3294862, ip=10.128.17.219) No module named 'vllm._version' (pid=3294862, ip=10.128.17.219) from .version import __version__, __version_tuple__ # isort:skip (raylet, ip=10.128.17.216) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 57x across cluster] (pid=2397096, ip=10.128.24.51) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster] (pid=2397096, ip=10.128.24.51) No module named 'vllm._version' [repeated 8x across cluster] (pid=2397096, ip=10.128.24.51) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster] (raylet, ip=10.128.17.220) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 101x across cluster] (pid=3123494, ip=10.128.17.215) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster] (pid=3123494, ip=10.128.17.215) No module named 'vllm._version' [repeated 15x across cluster] (pid=3123494, ip=10.128.17.215) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:18:59.704 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING= (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:18:59.706 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:18:59.706 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.677 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:00.999 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) 2026-05-23 15:19:01.027 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 1.88s (attempt 1/5) to avoid port collisions (raylet, ip=10.128.24.52) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 64x across cluster] (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. (pid=1440680, ip=10.128.24.53) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster] (pid=1440680, ip=10.128.24.53) No module named 'vllm._version' [repeated 8x across cluster] (pid=1440680, ip=10.128.24.53) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:04.230 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:04.288 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:04.288 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.504 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.792 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.793 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.793 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.793 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) 2026-05-23 15:19:05.823 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 2.65s (attempt 1/5) to avoid port collisions [repeated 11x across cluster] (skyrl_entrypoint pid=3659716) [2026-05-23 15:19:06] INFO inference_engine_client_http_endpoint.py:350: Starting server on 127.0.0.1:8000 (skyrl_entrypoint pid=3659716) [2026-05-23 15:19:06] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint... (raylet, ip=10.128.17.214) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 118x across cluster] (skyrl_entrypoint pid=3659716) [2026-05-23 15:19:07] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds) (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.125 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:960 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000 (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.125 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines. (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.127 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:213 - Harbor logging level set to INFO (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.128 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:113 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 900. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.130 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:129 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 272 (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:351 - Length of train_dataloader: 8728 (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:352 - Number of steps per epoch: 136 (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:07.131 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:353 - Total training steps: 80 (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 18x across cluster] (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) No module named 'vllm._version' [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) from .version import __version__, __version_tuple__ # isort:skip [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) 2026-05-23 15:19:10.098 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) 2026-05-23 15:19:10.100 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) 2026-05-23 15:19:10.100 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 5x across cluster] (pid=2753436, ip=10.128.17.217) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) 2026-05-23 15:19:08.832 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.876 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.877 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.877 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.877 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) 2026-05-23 15:19:10.898 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 2.48s (attempt 1/5) to avoid port collisions [repeated 5x across cluster] (bundle_reservation_check_func pid=3659816) [2026-05-23 15:19:11,518 E 3659816 3659856] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 21x across cluster] (AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 17x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) No module named 'vllm._version' [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) from .version import __version__, __version_tuple__ # isort:skip [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) (AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:14.386 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:14.387 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:14.387 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) 2026-05-23 15:19:15.611 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.277 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) 2026-05-23 15:19:14.309 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 1.81s (attempt 1/5) to avoid port collisions [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) [2026-05-23 15:19:16,018 E 2396835 2396937] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 34x across cluster] (AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) [W523 15:19:17.519222234 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-022-03-interconnect-1.jupiter.internal]:40421 (errno: 97 - Address family not supported by protocol). (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) [W523 15:19:17.519601842 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) [rank0]:[W523 15:19:17.521578185 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) _C._set_float32_matmul_precision(precision) (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:19:18 INFO [ipv4-debug] hostname=jpbo-010-41.jupiter.internal (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:19:18 INFO [ipv4-debug] _global_node.node_ip_address=10.128.17.217 (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) 2026-05-23 15:19:18 INFO [ipv4-debug] get_node_ip_address()=10.128.17.217 (skyrl_entrypoint pid=3659716) 2026-05-23 15:19:18.138 | INFO  | skyrl_train.workers.worker:_initiate_actors:496 - Initializing process group for RayActorGroup (AsyncVLLMInferenceEngine pid=1440679, ip=10.128.24.53) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 29x across cluster] (AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) No module named 'vllm._version' [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=372386, ip=10.128.17.220) from .version import __version__, __version_tuple__ # isort:skip [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3294996, ip=10.128.17.219) (EngineCore_DP0 pid=3295247) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3294998, ip=10.128.17.219) (EngineCore_DP0 pid=3295229) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) (EngineCore_DP0 pid=3295221) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00, std::allocator >) + 0xb0 (0x40003896c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #1: + 0x5f29220 (0x400018b09220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #2: + 0x5f4326c (0x400018b2326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #3: + 0x5f49074 (0x400018b29074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #4: + 0x5f49138 (0x400018b29138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #5: + 0x5f2ccc4 (0x400018b0ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string, std::allocator >, c10d::TCPStoreOptions const&) + 0x104 (0x400018b11564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #7: + 0x109a094 (0x4000128ba094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #8: + 0x113236c (0x40001295236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #9: + 0x5d6d60 (0x400011df6d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #10: + 0x1b7a38 (0xaaaaca047a38 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #12: + 0x169f50 (0xaaaac9ff9f50 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #13: + 0x1682e4 (0xaaaac9ff82e4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #14: + 0x1e0ce8 (0xaaaaca070ce8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #15: + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #16: + 0x646b0c (0x400011e66b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #19: + 0x1808c0 (0xaaaaca0108c0 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #20: + 0x182bf8 (0xaaaaca012bf8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #21: + 0x25fd30 (0xaaaaca0efd30 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #22: + 0x1b7d20 (0xaaaaca047d20 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #23: PyObject_Vectorcall + 0x54 (0xaaaac9ff60e4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaac9ff7f20 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #27: + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #28: + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaac9ff7fac in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #33: + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #34: + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #35: _PyObject_Call + 0x68 (0xaaaac9ff8488 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaaca0fd8f4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #37: PyEval_EvalCode + 0xb4 (0xaaaaca102eb4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #38: + 0x2ccdcc (0xaaaaca15cdcc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #39: + 0x2ccef4 (0xaaaaca15cef4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #40: PyRun_StringFlags + 0x90 (0xaaaaca161050 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaaca1610f8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #42: Py_RunMain + 0x2c8 (0xaaaaca189190 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #43: Py_BytesMain + 0x64 (0xaaaaca189fb4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #44: + 0x27540 (0x400000e67540 in /lib64/libc.so.6) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #45: __libc_start_main + 0x98 (0x400000e67618 in /lib64/libc.so.6) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) frame #46: + 0x10e0c0 (0xaaaac9f9e0c0 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:16.152 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:16.152 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:16.152 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 4x across cluster] (pid=2753511, ip=10.128.17.217) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) (EngineCore_DP0 pid=523846) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.465 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.766 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) 2026-05-23 15:19:17.796 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 2.12s (attempt 1/5) to avoid port collisions [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) return self.__get_result() (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) raise self._exception (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) super().__init__(*args, **kwargs) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) self.llm = self._create_engine(*args, **kwargs) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) return cls( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) self.engine_core = EngineCoreClient.make_async_mp_client( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) return AsyncMPClient(*client_args) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) super().__init__( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) with launch_core_engines(vllm_config, executor_class, log_stats) as ( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) next(self.gen) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) wait_for_engine_startup( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) raise RuntimeError( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) [2026-05-23 15:19:20,076 E 1909038 1909140] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=523482, ip=10.128.17.222) (EngineCore_DP0 pid=523842) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) (EngineCore_DP0 pid=523858) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 58x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) [W523 15:19:22.212466561 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-010-46.jupiter.internal]:58805 (errno: 97 - Address family not supported by protocol). [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) [W523 15:19:22.215869301 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) [rank0]:[W523 15:19:22.220727209 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) (EngineCore_DP0 pid=523862) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:11, 3.75s/it] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 14x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) (EngineCore_DP0 pid=3123723) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) No module named 'vllm._version' [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3294996, ip=10.128.17.219) (EngineCore_DP0 pid=3295247) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:22.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:22.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3222225, ip=10.128.17.212) (EngineCore_DP0 pid=3222586) _C._set_float32_matmul_precision(precision) (pid=3659917) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=200756, ip=10.128.17.221) (EngineCore_DP0 pid=200991) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3123493, ip=10.128.17.215) (EngineCore_DP0 pid=3123736) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:23.980 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) 2026-05-23 15:19:24.008 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) 2026-05-23 15:19:24.038 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1099 - Engine startup stagger: sleeping 1.65s (attempt 1/5) to avoid port collisions [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3222352, ip=10.128.17.212) (EngineCore_DP0 pid=3222606) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3123492, ip=10.128.17.215) (EngineCore_DP0 pid=3123744) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=200625, ip=10.128.17.221) [2026-05-23 15:19:25,287 E 200625 200728] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=3222353, ip=10.128.17.212) (EngineCore_DP0 pid=3222598) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=372387, ip=10.128.17.220) (EngineCore_DP0 pid=372619) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) (EngineCore_DP0 pid=1909400) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) [W523 15:19:27.784140679 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-010-40-interconnect-1.jupiter.internal]:45605 (errno: 97 - Address family not supported by protocol). [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) [W523 15:19:27.784665083 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=1909167, ip=10.128.17.216) [rank0]:[W523 15:19:27.786653454 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=200757, ip=10.128.17.221) (EngineCore_DP0 pid=200995) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216) Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.15s/it] [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=372514, ip=10.128.17.220) (EngineCore_DP0 pid=372624) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) No module named 'vllm._version' [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) from .version import __version__, __version_tuple__ # isort:skip [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=372258, ip=10.128.17.220) (EngineCore_DP0 pid=372628) _C._set_float32_matmul_precision(precision) (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00, value=RayTaskError(ActorDiedError)(ActorDiedError(RayTaskError('__init__', 'Traceback (most recent call last):\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n File "python/ray/_raylet.pyx", line 4329, in async_func\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\n return method(__ray_actor, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\n return method(self, *_args, **_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__\n super().__init__(*args, **kwargs)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__\n self.llm = self._create_engine(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\n return method(self, *_args, **_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\n return cls(\n ^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\n self.engine_core = EngineCoreClient.make_async_mp_client(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\n return AsyncMPClient(*client_args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\n super().__init__(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\n next(self.gen)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\n wait_for_engine_startup(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\n raise RuntimeError(\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\n', RuntimeError('Engine core initialization failed. See root cause above. Failed core proc(s): {}'), 'ray::AsyncVLLMInferenceEngine.__init__', None, None))), traceback=), 'extra': {}, 'file': (name='fully_async_trainer.py', path='/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py'), 'function': 'train', 'level': (name='ERROR', no=40, icon='❌'), 'line': 436, 'message': 'Train loop failed at global_step 18: \x1b[36mray::FSDPPolicyWorkerBase.init_weight_sync_state()\x1b[39m (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=)\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state\n results = await asyncio.gather(*tasks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator\n rank_offset_count += engine.tp_size() * engine.pp_size()\n ^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size\n return ray.get(self.inference_engine_actor.tp_size.remote())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^\nray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, \x1b[36mray::AsyncVLLMInferenceEngine.__init__()\x1b[39m (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=)\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__\n super().__init__(*args, **kwargs)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__\n self.llm = self._create_engine(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\n return cls(\n ^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\n self.engine_core = EngineCoreClient.make_async_mp_client(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\n return AsyncMPClient(*client_args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\n super().__init__(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\n next(self.gen)\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\n wait_for_engine_startup(\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\n raise RuntimeError(\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}', 'module': 'fully_async_trainer', 'name': 'skyrl_train.fully_async_trainer', 'process': (id=3659716, name='MainProcess'), 'thread': (id=70368994352832, name='MainThread'), 'time': datetime(2026, 5, 23, 15, 20, 9, 843010, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200), 'CEST'))} (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank0]:[W523 15:20:09.372907026 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-010-41.jupiter.internal]:47881 (errno: 97 - Address family not supported by protocol). (skyrl_entrypoint pid=3659716) Traceback (most recent call last): (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train (skyrl_entrypoint pid=3659716) await self._train_loop() (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop (skyrl_entrypoint pid=3659716) self.init_weight_sync_state() (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state (skyrl_entrypoint pid=3659716) ray.get( (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper (skyrl_entrypoint pid=3659716) return fn(*args, **kwargs) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper (skyrl_entrypoint pid=3659716) return func(*args, **kwargs) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get (skyrl_entrypoint pid=3659716) values, debugger_breakpoint = worker.get_objects( (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects (skyrl_entrypoint pid=3659716) raise value.as_instanceof_cause() (skyrl_entrypoint pid=3659716) ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result (skyrl_entrypoint pid=3659716) return self.__get_result() (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result (skyrl_entrypoint pid=3659716) raise self._exception (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state (skyrl_entrypoint pid=3659716) results = await asyncio.gather(*tasks) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator (skyrl_entrypoint pid=3659716) rank_offset_count += engine.tp_size() * engine.pp_size() (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size (skyrl_entrypoint pid=3659716) return ray.get(self.inference_engine_actor.tp_size.remote()) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result (skyrl_entrypoint pid=3659716) return self.__get_result() (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result (skyrl_entrypoint pid=3659716) raise self._exception (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__ (skyrl_entrypoint pid=3659716) super().__init__(*args, **kwargs) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__ (skyrl_entrypoint pid=3659716) self.llm = self._create_engine(*args, **kwargs) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine (skyrl_entrypoint pid=3659716) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args (skyrl_entrypoint pid=3659716) return cls( (skyrl_entrypoint pid=3659716) ^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__ (skyrl_entrypoint pid=3659716) self.engine_core = EngineCoreClient.make_async_mp_client( (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client (skyrl_entrypoint pid=3659716) return AsyncMPClient(*client_args) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__ (skyrl_entrypoint pid=3659716) super().__init__( (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__ (skyrl_entrypoint pid=3659716) with launch_core_engines(vllm_config, executor_class, log_stats) as ( (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__ (skyrl_entrypoint pid=3659716) next(self.gen) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines (skyrl_entrypoint pid=3659716) wait_for_engine_startup( (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup (skyrl_entrypoint pid=3659716) raise RuntimeError( (skyrl_entrypoint pid=3659716) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} (skyrl_entrypoint pid=3659716) (skyrl_entrypoint pid=3659716) During handling of the above exception, another exception occurred: (skyrl_entrypoint pid=3659716) (skyrl_entrypoint pid=3659716) Traceback (most recent call last): (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/loguru/_handler.py", line 204, in emit (skyrl_entrypoint pid=3659716) self._queue.put(str_record) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/queues.py", line 393, in put (skyrl_entrypoint pid=3659716) obj = _ForkingPickler.dumps(obj) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps (skyrl_entrypoint pid=3659716) cls(buf, protocol).dump(obj) (skyrl_entrypoint pid=3659716) _pickle.PicklingError: Can't pickle : attribute lookup RayTaskError(ActorDiedError) on ray.exceptions failed (skyrl_entrypoint pid=3659716) --- End of logging error --- (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.907 | INFO  | skyrl_train.inference_engines.inference_engine_client_http_endpoint:shutdown_server:203 - Server shut down after 2 seconds (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.908 | INFO  | skyrl_train.trainer:_guarded_sync:226 - HTTP endpoint shutdown complete (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.908 | INFO  | examples.terminal_bench.terminal_bench_generator:shutdown:328 - Shutting down shared QueueOrchestrator... (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.910 | INFO  | harbor.orchestrators.queue:shutdown:377 - [terminal_bench_generator:236] Shutdown complete. Total completed: 0 (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.910 | INFO  | examples.terminal_bench.terminal_bench_generator:shutdown:330 - QueueOrchestrator shutdown complete (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.910 | INFO  | skyrl_train.trainer:_guarded_async:215 - Generator shutdown complete (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.920 | WARNING  | skyrl_train.trainer:_guarded_async:219 - Inference engine teardown error (non-fatal): The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result (skyrl_entrypoint pid=3659716) return self.__get_result() (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result (skyrl_entrypoint pid=3659716) raise self._exception (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__ (skyrl_entrypoint pid=3659716) super().__init__(*args, **kwargs) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__ (skyrl_entrypoint pid=3659716) self.llm = self._create_engine(*args, **kwargs) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine (skyrl_entrypoint pid=3659716) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args (skyrl_entrypoint pid=3659716) return cls( (skyrl_entrypoint pid=3659716) ^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__ (skyrl_entrypoint pid=3659716) self.engine_core = EngineCoreClient.make_async_mp_client( (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client (skyrl_entrypoint pid=3659716) return AsyncMPClient(*client_args) (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__ (skyrl_entrypoint pid=3659716) super().__init__( (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__ (skyrl_entrypoint pid=3659716) with launch_core_engines(vllm_config, executor_class, log_stats) as ( (skyrl_entrypoint pid=3659716) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__ (skyrl_entrypoint pid=3659716) next(self.gen) (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines (skyrl_entrypoint pid=3659716) wait_for_engine_startup( (skyrl_entrypoint pid=3659716) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup (skyrl_entrypoint pid=3659716) raise RuntimeError( (skyrl_entrypoint pid=3659716) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.920 | INFO  | skyrl_train.trainer:_kill_ray_actors:248 - Killing policy_model actors... (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) (EngineCore_DP0 pid=3295221) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) (EngineCore_DP0 pid=3295221) warnings.warn("No model update group to destroy") (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.936 | INFO  | skyrl_train.trainer:_kill_ray_actors:267 - Killed 48 inference engine actor(s) (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.936 | INFO  | skyrl_train.trainer:_guarded_sync:226 - Ray actor cleanup complete (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.937 | INFO  | skyrl_train.trainer:_kill_ray_actors:248 - Killing policy_model actors... (skyrl_entrypoint pid=3659716) 2026-05-23 15:20:10.941 | INFO  | skyrl_train.trainer:_kill_ray_actors:267 - Killed 48 inference engine actor(s) 2026-05-23 15:20:10.953 | ERROR | __main__:main:125 - Training failed: ray::skyrl_entrypoint() (pid=3659716, ip=10.128.17.210) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 96, in skyrl_entrypoint exp.run() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 403, in run asyncio.run(trainer.train()) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train await self._train_loop() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop self.init_weight_sync_state() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state ray.get( ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state results = await asyncio.gather(*tasks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator rank_offset_count += engine.tp_size() * engine.pp_size() ^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size return ray.get(self.inference_engine_actor.tp_size.remote()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__ super().__init__(*args, **kwargs) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__ self.llm = self._create_engine(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args return cls( ^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client return AsyncMPClient(*client_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__ super().__init__( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__ with launch_core_engines(vllm_config, executor_class, log_stats) as ( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines wait_for_engine_startup( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup raise RuntimeError( RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} 2026-05-23 15:20:10.954 | INFO | __main__:main:128 - Shutting down Ray on head node... (AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy [repeated 37x across cluster] (AsyncVLLMInferenceEngine pid=1909166, ip=10.128.17.216) (EngineCore_DP0 pid=1909420) warnings.warn("No model update group to destroy") [repeated 37x across cluster] (skyrl_entrypoint pid=3659716) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:02 [arg_utils.py:1256] The global random seed is set to 42. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [model.py:529] Resolved architecture: Qwen3ForCausalLM (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [model.py:1549] Using max model len 32768 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [vllm.py:690] Asynchronous scheduling is enabled. (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:03 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) INFO 05-23 15:19:03 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:03 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=3294862, ip=10.128.17.219) WARNING 05-23 15:19:03 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) WARNING 05-23 15:19:08 [arg_utils.py:1256] The global random seed is set to 47. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [model.py:1549] Using max model len 32768 [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) WARNING 05-23 15:19:08 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) INFO 05-23 15:19:08 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) WARNING 05-23 15:19:07 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) WARNING 05-23 15:19:07 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) INFO 05-23 15:19:12 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=45, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [arg_utils.py:1256] The global random seed is set to 62. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [model.py:1549] Using max model len 32768 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) INFO 05-23 15:19:13 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) WARNING 05-23 15:19:13 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) INFO 05-23 15:19:15 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=523614, ip=10.128.17.222) (EngineCore_DP0 pid=523846) INFO 05-23 15:19:17 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=48, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2396962, ip=10.128.24.51) (EngineCore_DP0 pid=2397192) INFO 05-23 15:19:17 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.24.51:59091 backend=nccl (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:17 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A (pid=2753436, ip=10.128.17.217) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:18 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) WARNING 05-23 15:19:18 [arg_utils.py:1256] The global random seed is set to 70. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [model.py:1549] Using max model len 32768 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) WARNING 05-23 15:19:18 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) INFO 05-23 15:19:18 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) WARNING 05-23 15:19:18 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) WARNING 05-23 15:19:18 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:19 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] EngineCore failed to start. (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] Traceback (most recent call last): (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] super().__init__( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self.model_executor = executor_class(vllm_config) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self._init_executor() (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self.driver_worker.init_device() (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] self.worker.init_device() # type: ignore (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] init_worker_distributed_environment( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] init_distributed_environment( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] torch.distributed.init_process_group( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] return func(*args, **kwargs) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] func_return = func(*args, **kwargs) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] store, rank, world_size = next(rendezvous_iterator) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] store = _create_c10d_store( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^^^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] return TCPStore( (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] ^^^^^^^^^ (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 34697, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first): (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0xb0 (0x40003896c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #1: + 0x5f29220 (0x400018b09220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #2: + 0x5f4326c (0x400018b2326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #3: + 0x5f49074 (0x400018b29074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #4: + 0x5f49138 (0x400018b29138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #5: + 0x5f2ccc4 (0x400018b0ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string, std::allocator >, c10d::TCPStoreOptions const&) + 0x104 (0x400018b11564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #7: + 0x109a094 (0x4000128ba094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #8: + 0x113236c (0x40001295236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #9: + 0x5d6d60 (0x400011df6d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #10: + 0x1b7a38 (0xaaaaca047a38 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #12: + 0x169f50 (0xaaaac9ff9f50 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #13: + 0x1682e4 (0xaaaac9ff82e4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #14: + 0x1e0ce8 (0xaaaaca070ce8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #15: + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #16: + 0x646b0c (0x400011e66b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #19: + 0x1808c0 (0xaaaaca0108c0 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #20: + 0x182bf8 (0xaaaaca012bf8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #21: + 0x25fd30 (0xaaaaca0efd30 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #22: + 0x1b7d20 (0xaaaaca047d20 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #23: PyObject_Vectorcall + 0x54 (0xaaaac9ff60e4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaac9ff7f20 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #27: + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #28: + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaac9ff5db8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaaca0fae54 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaac9ff7fac in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaac9ff81ec in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #33: + 0x1e0d80 (0xaaaaca070d80 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #34: + 0x1d7ddc (0xaaaaca067ddc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #35: _PyObject_Call + 0x68 (0xaaaac9ff8488 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaaca0fd8f4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #37: PyEval_EvalCode + 0xb4 (0xaaaaca102eb4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #38: + 0x2ccdcc (0xaaaaca15cdcc in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #39: + 0x2ccef4 (0xaaaaca15cef4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #40: PyRun_StringFlags + 0x90 (0xaaaaca161050 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaaca1610f8 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #42: Py_RunMain + 0x2c8 (0xaaaaca189190 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #43: Py_BytesMain + 0x64 (0xaaaaca189fb4 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #44: + 0x27540 (0x400000e67540 in /lib64/libc.so.6) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #45: __libc_start_main + 0x98 (0x400000e67618 in /lib64/libc.so.6) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] frame #46: + 0x10e0c0 (0xaaaac9f9e0c0 in VLLM::EngineCore) (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) ERROR 05-23 15:19:19 [core.py:1006] (AsyncVLLMInferenceEngine pid=3294997, ip=10.128.17.219) (EngineCore_DP0 pid=3295225) INFO 05-23 15:19:19 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582) INFO 05-23 15:19:22 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=61, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) (EngineCore_DP0 pid=523858) INFO 05-23 15:19:22 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.17.222:58805 backend=nccl [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=523613, ip=10.128.17.222) (EngineCore_DP0 pid=523858) INFO 05-23 15:19:22 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=523615, ip=10.128.17.222) (EngineCore_DP0 pid=523862) INFO 05-23 15:19:23 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [arg_utils.py:1256] The global random seed is set to 76. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [model.py:1549] Using max model len 32768 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) INFO 05-23 15:19:20 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:20 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=3294996, ip=10.128.17.219) (EngineCore_DP0 pid=3295247) INFO 05-23 15:19:21 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=3222354, ip=10.128.17.212) (EngineCore_DP0 pid=3222582) INFO 05-23 15:19:24 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 8x across cluster] (pid=2753509, ip=10.128.17.217) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) (EngineCore_DP0 pid=1440912) INFO 05-23 15:19:27 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=72, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:27 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.17.216:50115 backend=nccl [repeated 13x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:27 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 16x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [arg_utils.py:1256] The global random seed is set to 86. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [model.py:1549] Using max model len 32768 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) INFO 05-23 15:19:26 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) WARNING 05-23 15:19:26 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:29 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) (EngineCore_DP0 pid=1440912) INFO 05-23 15:19:30 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 18x across cluster] (pid=3659917) ⚙️ Running in WANDB offline mode [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:31 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=83, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918) INFO 05-23 15:19:32 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.24.53:39961 backend=nccl [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:32 [default_loader.py:293] Loading weights took 13.48 seconds (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918) INFO 05-23 15:19:32 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:33 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.270659 seconds (AsyncVLLMInferenceEngine pid=1440552, ip=10.128.24.53) (EngineCore_DP0 pid=1440918) INFO 05-23 15:19:32 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929) INFO 05-23 15:19:33 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:35 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:35 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens (AsyncVLLMInferenceEngine pid=2396963, ip=10.128.24.51) (EngineCore_DP0 pid=2397201) INFO 05-23 15:19:35 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:34 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:36 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (AsyncVLLMInferenceEngine pid=2397096, ip=10.128.24.51) (EngineCore_DP0 pid=2397216) INFO 05-23 15:19:36 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.76 seconds (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) WARNING 05-23 15:19:36 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:36 [vllm.py:690] Asynchronous scheduling is enabled. (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) WARNING 05-23 15:19:36 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) (EngineCore_DP0 pid=2397197) INFO 05-23 15:19:36 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=2396835, ip=10.128.24.51) WARNING 05-23 15:19:36 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. (AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:37 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=87, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.24.50:55605 backend=nccl [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:37 [default_loader.py:293] Loading weights took 8.24 seconds [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:37 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 8.907134 seconds [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:35 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:39 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:39 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:39 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998) INFO 05-23 15:19:39 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=200757, ip=10.128.17.221) (EngineCore_DP0 pid=200995) INFO 05-23 15:19:40 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:40 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) WARNING 05-23 15:19:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:40 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) WARNING 05-23 15:19:40 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) (EngineCore_DP0 pid=201001) INFO 05-23 15:19:40 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=200758, ip=10.128.17.221) WARNING 05-23 15:19:40 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 10x across cluster] (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) NCCL version 2.27.7+cuda13.0 (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/model_world_size_8_rank_0.pt (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/extra_state_world_size_8_rank_0.pt (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy/optim_world_size_8_rank_0.pt (AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:40 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.17.214:50715 backend=nccl [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:43 [default_loader.py:293] Loading weights took 13.72 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:40 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1909165, ip=10.128.17.216) (EngineCore_DP0 pid=1909404) INFO 05-23 15:19:43 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.989906 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:41 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=3051740, ip=10.128.17.214) (EngineCore_DP0 pid=3051998) INFO 05-23 15:19:41 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:45 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:45 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:45 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:40 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.73 seconds [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) WARNING 05-23 15:19:46 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) WARNING 05-23 15:19:46 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=1909038, ip=10.128.17.216) (EngineCore_DP0 pid=1909396) INFO 05-23 15:19:46 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=3123494, ip=10.128.17.215) WARNING 05-23 15:19:42 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:49 [default_loader.py:293] Loading weights took 13.85 seconds [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:49 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.998130 seconds [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605) INFO 05-23 15:19:51 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605) INFO 05-23 15:19:51 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=1503222, ip=10.128.24.50) (EngineCore_DP0 pid=1503605) INFO 05-23 15:19:51 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929) INFO 05-23 15:19:50 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1440680, ip=10.128.24.53) (EngineCore_DP0 pid=1440929) INFO 05-23 15:19:50 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) WARNING 05-23 15:19:51 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:51 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) WARNING 05-23 15:19:51 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=1503354, ip=10.128.24.50) (EngineCore_DP0 pid=1503581) INFO 05-23 15:19:51 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=1440681, ip=10.128.24.53) WARNING 05-23 15:19:51 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:56 [default_loader.py:293] Loading weights took 14.98 seconds (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) (EngineCore_DP0 pid=2173022) INFO 05-23 15:19:52 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:52 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.78 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3051742, ip=10.128.17.214) (EngineCore_DP0 pid=3051996) INFO 05-23 15:19:56 [default_loader.py:293] Loading weights took 14.91 seconds (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:57 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 16.066204 seconds (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) WARNING 05-23 15:19:52 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:52 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) WARNING 05-23 15:19:52 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2172774, ip=10.128.24.52) (EngineCore_DP0 pid=2173007) INFO 05-23 15:19:52 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2172773, ip=10.128.24.52) WARNING 05-23 15:19:53 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 12x across cluster] (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Successfully loaded model state dict (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:59 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:59 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:19:59 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:20:00 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:20:00 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:19:56 [default_loader.py:293] Loading weights took 15.06 seconds [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) (EngineCore_DP0 pid=3051990) INFO 05-23 15:19:57 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 16.070195 seconds [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) WARNING 05-23 15:20:00 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:20:00 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) WARNING 05-23 15:20:00 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051613, ip=10.128.17.214) (EngineCore_DP0 pid=3052003) INFO 05-23 15:20:00 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3051741, ip=10.128.17.214) WARNING 05-23 15:20:00 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 4x across cluster] (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Successfully loaded optimizer state (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Successfully loaded scheduler state (FSDPPolicyWorkerBase pid=2753436, ip=10.128.17.217) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints/global_step_18/policy (AsyncVLLMInferenceEngine pid=523482, ip=10.128.17.222) ERROR 05-23 15:20:10 [core_client.py:616] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client. (AsyncVLLMInferenceEngine pid=3123358, ip=10.128.17.215) ERROR 05-23 15:20:10 [core_client.py:616] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client. [repeated 12x across cluster] Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=token_mean', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/a3-rl-DCAgent_code-contests-noblock', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=true', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=a3-rl-DCAgent_code-contests-noblock', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=450', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/code-contests-noblock"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=1800', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=900', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.model_info.max_input_tokens=32768', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true'] Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 133, in main() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() ^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( ^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run _ = ret.return_value ^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) ^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 123, in main ray.get(skyrl_entrypoint.remote(cfg)) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get values, debugger_breakpoint = worker.get_objects( ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects raise value.as_instanceof_cause() ray.exceptions.RayTaskError(ActorDiedError): ray::skyrl_entrypoint() (pid=3659716, ip=10.128.17.210) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 96, in skyrl_entrypoint exp.run() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 403, in run asyncio.run(trainer.train()) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 434, in train await self._train_loop() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 488, in _train_loop self.init_weight_sync_state() File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 808, in init_weight_sync_state ray.get( ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ ray.exceptions.RayTaskError(ActorDiedError): ray::FSDPPolicyWorkerBase.init_weight_sync_state() (pid=2753436, ip=10.128.17.217, actor_id=7804ef248912326214f6164a02000000, repr=) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 299, in init_weight_sync_state results = await asyncio.gather(*tasks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/inference_engine_client.py", line 655, in init_weight_update_communicator rank_offset_count += engine.tp_size() * engine.pp_size() ^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/ray_wrapped_inference_engine.py", line 26, in tp_size return ray.get(self.inference_engine_actor.tp_size.remote()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=3294997, ip=10.128.17.219, actor_id=3099a3967d816aa5fa00928502000000, repr=) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 456, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1028, in __init__ super().__init__(*args, **kwargs) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 500, in __init__ self.llm = self._create_engine(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1105, in _create_engine engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args return cls( ^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client return AsyncMPClient(*client_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__ super().__init__( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__ with launch_core_engines(vllm_config, executor_class, log_stats) as ( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines wait_for_engine_startup( File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup raise RuntimeError( RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} Stopping Ray cluster... Ray cluster stopped [RLJobRunner] Launching trace upload (training exit code: 1): repo_id: DCAgent/a3-rl-DCAgent_code-contests-noblock job_dir: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/a3-rl-DCAgent_code-contests-noblock episodes: last log: /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/logs/a3-rl-DCAgent_code-contests-noblock_trace_upload.log [RLJobRunner] Waiting for trace upload to complete... [RLJobRunner] Trace upload failed with exit code 1. Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a3-rl-DCAgent_code-contests-noblock/ray_logs/ Collecting Ray logs from worker jpbo-010-36... Collecting Ray logs from worker jpbo-010-38... Collecting Ray logs from worker jpbo-010-39... Collecting Ray logs from worker jpbo-010-40... Collecting Ray logs from worker jpbo-010-41... Collecting Ray logs from worker jpbo-010-43... Collecting Ray logs from worker jpbo-010-44... Collecting Ray logs from worker jpbo-010-45... Collecting Ray logs from worker jpbo-010-46... Collecting Ray logs from worker jpbo-022-02... Collecting Ray logs from worker jpbo-022-03... Collecting Ray logs from worker jpbo-022-04... Collecting Ray logs from worker jpbo-022-05... Ray log preservation complete