laion/ablation-pymethods2test-seqmean-arm0-30-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0_630126.out

1666 lines
318 KiB
Plaintext
Raw Permalink Normal View History

初始化项目，由ModelHub XC社区提供模型 Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B Source: Original Platform
2026-07-18 18:02:02 +08:00
The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_630126
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630126
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630126
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.32.33 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630126.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630126.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.32.33:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-041-33 (10.128.32.33)
Ray port: 6379
============================
Starting Ray head on jpbo-041-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-33.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-041-33
Starting Ray worker on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-34.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-041-34
Starting Ray worker on jpbo-041-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-35.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-041-35
Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-041-36
Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-041-37
Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-041-38
Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-041-39
Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-041-40
Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-041-41
Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-041-42
Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-041-43
Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-041-44
Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-041-45
Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-041-46
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.32.33:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'GPU': 56.0, 'memory': 10652272558080.0, 'accelerator_type:GH200': 14.0, 'object_store_memory': 601295421440.0, 'CPU': 4032.0, 'node:10.128.32.35': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.39': 1.0, 'node:10.128.32.41': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.37': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
Address: 10.128.32.33:6379
Total GPUs: 56
=========================
Ray cluster ready at 10.128.32.33:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
Running SkyRL:
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
Args: 120 Hydra arguments
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=sequence_mean trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0 ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqmean-arm0 trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.har
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630126.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 02:20:53.422 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:686 - Exporting wandb api key to ray runtime env
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:705 - Exporting RAY_ADDRESS to ray runtime env
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
2026-06-07 02:20:53,423 INFO worker.py:1680 -- Using address 10.128.32.33:6379 set in the environment variable RAY_ADDRESS
2026-06-07 02:20:53,456 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.33:6379...
2026-06-07 02:20:53,467 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 02:20:55.689 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:02.936 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:212 - SkyRL log level set to: INFO
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(skyrl_entrypoint pid=294281) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:03.364 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:06.531 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 5000 valid task directories out of 5000 total directories
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:06.532 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 5000 task paths
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:06.545 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:405 - data:
(skyrl_entrypoint pid=294281) train_data:
(skyrl_entrypoint pid=294281) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=294281) val_data: []
(skyrl_entrypoint pid=294281) trainer:
(skyrl_entrypoint pid=294281) placement:
(skyrl_entrypoint pid=294281) colocate_all: false
(skyrl_entrypoint pid=294281) colocate_policy_ref: true
(skyrl_entrypoint pid=294281) policy_num_nodes: 2
(skyrl_entrypoint pid=294281) policy_num_gpus_per_node: 4
(skyrl_entrypoint pid=294281) critic_num_nodes: 1
(skyrl_entrypoint pid=294281) critic_num_gpus_per_node: 4
(skyrl_entrypoint pid=294281) ref_num_nodes: 2
(skyrl_entrypoint pid=294281) ref_num_gpus_per_node: 4
(skyrl_entrypoint pid=294281) policy_strict_spread_pg: false
(skyrl_entrypoint pid=294281) policy_per_gpu_bundles: false
(skyrl_entrypoint pid=294281) sequence_parallel_backend: ulysses
(skyrl_entrypoint pid=294281) strategy: fsdp2
(skyrl_entrypoint pid=294281) policy:
(skyrl_entrypoint pid=294281) model:
(skyrl_entrypoint pid=294281) path: /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=294281) lora:
(skyrl_entrypoint pid=294281) rank: 0
(skyrl_entrypoint pid=294281) alpha: 16
(skyrl_entrypoint pid=294281) dropout: 0
(skyrl_entrypoint pid=294281) lora_sync_path: /tmp/skyrl_lora_sync
(skyrl_entrypoint pid=294281) target_modules: all-linear
(skyrl_entrypoint pid=294281) exclude_modules: null
(skyrl_entrypoint pid=294281) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=294281) optimizer_config:
(skyrl_entrypoint pid=294281) optimizer: AdamW
(skyrl_entrypoint pid=294281) lr: 8.0e-06
(skyrl_entrypoint pid=294281) adam_betas:
(skyrl_entrypoint pid=294281) - 0.9
(skyrl_entrypoint pid=294281) - 0.999
(skyrl_entrypoint pid=294281) weight_decay: 0.0
(skyrl_entrypoint pid=294281) max_grad_norm: 0.9
(skyrl_entrypoint pid=294281) offload_after_step: true
(skyrl_entrypoint pid=294281) num_warmup_steps: 0
(skyrl_entrypoint pid=294281) scheduler: constant_with_warmup
(skyrl_entrypoint pid=294281) optimizer_kwargs: {}
(skyrl_entrypoint pid=294281) fsdp_config:
(skyrl_entrypoint pid=294281) cpu_offload: false
(skyrl_entrypoint pid=294281) reshard_after_forward: true
(skyrl_entrypoint pid=294281) fsdp_size: 4
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=294281) moe_router_replay: false
(skyrl_entrypoint pid=294281) moe_grouped_gemm: false
(skyrl_entrypoint pid=294281) ep_comm_backend: torch
(skyrl_entrypoint pid=294281) deepep_num_sms: 20
(skyrl_entrypoint pid=294281) deepep_token_chunk_size: null
(skyrl_entrypoint pid=294281) sequence_parallel_size: 1
(skyrl_entrypoint pid=294281) use_torch_compile: false
(skyrl_entrypoint pid=294281) record_memory: false
(skyrl_entrypoint pid=294281) megatron_config:
(skyrl_entrypoint pid=294281) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=294281) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=294281) context_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: null
(skyrl_entrypoint pid=294281) ddp_config:
(skyrl_entrypoint pid=294281) grad_reduce_in_fp32: true
(skyrl_entrypoint pid=294281) overlap_grad_reduce: false
(skyrl_entrypoint pid=294281) overlap_param_gather: false
(skyrl_entrypoint pid=294281) average_in_collective: true
(skyrl_entrypoint pid=294281) model_config_kwargs: {}
(skyrl_entrypoint pid=294281) torch_profiler_config:
(skyrl_entrypoint pid=294281) enable: false
(skyrl_entrypoint pid=294281) ranks: []
(skyrl_entrypoint pid=294281) save_path: null
(skyrl_entrypoint pid=294281) optimizer_config_kwargs:
(skyrl_entrypoint pid=294281) overlap_cpu_optimizer_d2h_h2d: false
(skyrl_entrypoint pid=294281) use_precision_aware_optimizer: false
(skyrl_entrypoint pid=294281) optimizer_cpu_offload: false
(skyrl_entrypoint pid=294281) optimizer_offload_fraction: 0.0
(skyrl_entrypoint pid=294281) transformer_config_kwargs:
(skyrl_entrypoint pid=294281) recompute_granularity: full
(skyrl_entrypoint pid=294281) recompute_modules:
(skyrl_entrypoint pid=294281) - core_attn
(skyrl_entrypoint pid=294281) recompute_method: uniform
(skyrl_entrypoint pid=294281) recompute_num_layers: 1
(skyrl_entrypoint pid=294281) empty_cuda_cache: true
(skyrl_entrypoint pid=294281) ref:
(skyrl_entrypoint pid=294281) model:
(skyrl_entrypoint pid=294281) path: ${trainer.policy.model.path}
(skyrl_entrypoint pid=294281) sequence_parallel_size: 1
(skyrl_entrypoint pid=294281) deepspeed_config: ${deepspeed_config.eval}
(skyrl_entrypoint pid=294281) fsdp_config:
(skyrl_entrypoint pid=294281) cpu_offload: false
(skyrl_entrypoint pid=294281) reshard_after_forward: true
(skyrl_entrypoint pid=294281) fsdp_size: 4
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=294281) moe_router_replay: false
(skyrl_entrypoint pid=294281) moe_grouped_gemm: false
(skyrl_entrypoint pid=294281) ep_comm_backend: torch
(skyrl_entrypoint pid=294281) deepep_num_sms: 20
(skyrl_entrypoint pid=294281) deepep_token_chunk_size: null
(skyrl_entrypoint pid=294281) megatron_config:
(skyrl_entrypoint pid=294281) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=294281) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=294281) context_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) model_config_kwargs: {}
(skyrl_entrypoint pid=294281) transformer_config_kwargs: {}
(skyrl_entrypoint pid=294281) critic:
(skyrl_entrypoint pid=294281) model:
(skyrl_entrypoint pid=294281) path: null
(skyrl_entrypoint pid=294281) lora:
(skyrl_entrypoint pid=294281) rank: 0
(skyrl_entrypoint pid=294281) alpha: 16
(skyrl_entrypoint pid=294281) dropout: 0
(skyrl_entrypoint pid=294281) target_modules: all-linear
(skyrl_entrypoint pid=294281) exclude_modules: null
(skyrl_entrypoint pid=294281) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=294281) optimizer_config:
(skyrl_entrypoint pid=294281) optimizer: AdamW
(skyrl_entrypoint pid=294281) lr: 5.0e-06
(skyrl_entrypoint pid=294281) adam_betas:
(skyrl_entrypoint pid=294281) - 0.9
(skyrl_entrypoint pid=294281) - 0.999
(skyrl_entrypoint pid=294281) weight_decay: 0.01
(skyrl_entrypoint pid=294281) max_grad_norm: 1.0
(skyrl_entrypoint pid=294281) offload_after_step: true
(skyrl_entrypoint pid=294281) num_warmup_steps: 0
(skyrl_entrypoint pid=294281) scheduler: constant_with_warmup
(skyrl_entrypoint pid=294281) optimizer_kwargs: {}
(skyrl_entrypoint pid=294281) fsdp_config:
(skyrl_entrypoint pid=294281) cpu_offload: false
(skyrl_entrypoint pid=294281) reshard_after_forward: true
(skyrl_entrypoint pid=294281) fsdp_size: -1
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=294281) moe_router_replay: false
(skyrl_entrypoint pid=294281) moe_grouped_gemm: false
(skyrl_entrypoint pid=294281) ep_comm_backend: torch
(skyrl_entrypoint pid=294281) deepep_num_sms: 20
(skyrl_entrypoint pid=294281) deepep_token_chunk_size: null
(skyrl_entrypoint pid=294281) sequence_parallel_size: 1
(skyrl_entrypoint pid=294281) algorithm:
(skyrl_entrypoint pid=294281) advantage_estimator: rloo_n
(skyrl_entrypoint pid=294281) kl_ctrl:
(skyrl_entrypoint pid=294281) type: fixed
(skyrl_entrypoint pid=294281) kl_target: 0.1
(skyrl_entrypoint pid=294281) horizon: 10000
(skyrl_entrypoint pid=294281) kl_estimator_type: k3
(skyrl_entrypoint pid=294281) use_kl_estimator_k3: false
(skyrl_entrypoint pid=294281) use_abs_kl: false
(skyrl_entrypoint pid=294281) use_kl_in_reward: false
(skyrl_entrypoint pid=294281) use_kl_loss: false
(skyrl_entrypoint pid=294281) kl_loss_coef: 0.0
(skyrl_entrypoint pid=294281) use_entropy_loss: false
(skyrl_entrypoint pid=294281) entropy_loss_coef: 0.01
(skyrl_entrypoint pid=294281) advantage_batch_normalize: false
(skyrl_entrypoint pid=294281) value_head_prefix: value_head
(skyrl_entrypoint pid=294281) policy_loss_type: regular
(skyrl_entrypoint pid=294281) loss_reduction: sequence_mean
(skyrl_entrypoint pid=294281) global_loss_denom: null
(skyrl_entrypoint pid=294281) grpo_norm_by_std: true
(skyrl_entrypoint pid=294281) rloo_n_min_group_size: 4
(skyrl_entrypoint pid=294281) rloo_n_filter_zero_reward_groups: true
(skyrl_entrypoint pid=294281) lambd: 1.0
(skyrl_entrypoint pid=294281) gamma: 1.0
(skyrl_entrypoint pid=294281) eps_clip_low: 0.2
(skyrl_entrypoint pid=294281) eps_clip_high: 0.05
(skyrl_entrypoint pid=294281) clip_ratio_c: 3.0
(skyrl_entrypoint pid=294281) tis_imp_ratio_cap: -1.0
(skyrl_entrypoint pid=294281) use_tis: false
(skyrl_entrypoint pid=294281) sapo:
(skyrl_entrypoint pid=294281) tau_pos: 1.0
(skyrl_entrypoint pid=294281) tau_neg: 1.05
(skyrl_entrypoint pid=294281) value_clip: 0.2
(skyrl_entrypoint pid=294281) dynamic_sampling:
(skyrl_entrypoint pid=294281) type: null
(skyrl_entrypoint pid=294281) max_sample_batches: 30
(skyrl_entrypoint pid=294281) min_replace_ratio: 0.3
(skyrl_entrypoint pid=294281) clip_cov:
(skyrl_entrypoint pid=294281) clip_ratio: 0.0002
(skyrl_entrypoint pid=294281) clip_cov_lb: 1.0
(skyrl_entrypoint pid=294281) clip_cov_ub: 5.0
(skyrl_entrypoint pid=294281) kl_cov:
(skyrl_entrypoint pid=294281) kl_cov_frac: 0.2
(skyrl_entrypoint pid=294281) ppo_kl_coef: 1.0
(skyrl_entrypoint pid=294281) cispo:
(skyrl_entrypoint pid=294281) cispo_eps_clip_low: 0
(skyrl_entrypoint pid=294281) cispo_eps_clip_high: 5
(skyrl_entrypoint pid=294281) z_clip:
(skyrl_entrypoint pid=294281) enabled: false
(skyrl_entrypoint pid=294281) alpha: 0.97
(skyrl_entrypoint pid=294281) z_thresh: 2.5
(skyrl_entrypoint pid=294281) warmup_steps: 25
(skyrl_entrypoint pid=294281) mode: zscore
(skyrl_entrypoint pid=294281) clip_option: adaptive_scaling
(skyrl_entrypoint pid=294281) clip_factor: 1.0
(skyrl_entrypoint pid=294281) skip_update_on_spike: false
(skyrl_entrypoint pid=294281) stale_clip:
(skyrl_entrypoint pid=294281) enabled: false
(skyrl_entrypoint pid=294281) alpha: 0.3
(skyrl_entrypoint pid=294281) entropy_threshold: 0.15
(skyrl_entrypoint pid=294281) entropy_window: 10
(skyrl_entrypoint pid=294281) min_lr_scale: 0.1
(skyrl_entrypoint pid=294281) max_seq_len: 1004095
(skyrl_entrypoint pid=294281) fully_async:
(skyrl_entrypoint pid=294281) max_staleness_steps: 16
(skyrl_entrypoint pid=294281) num_parallel_generation_workers: 338
(skyrl_entrypoint pid=294281) gradient_checkpointing: true
(skyrl_entrypoint pid=294281) gradient_checkpointing_use_reentrant: false
(skyrl_entrypoint pid=294281) seed: 42
(skyrl_entrypoint pid=294281) resume_mode: latest
(skyrl_entrypoint pid=294281) resume_path: null
(skyrl_entrypoint pid=294281) ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints
(skyrl_entrypoint pid=294281) max_ckpts_to_keep: -1
(skyrl_entrypoint pid=294281) ckpt_interval: 2
(skyrl_entrypoint pid=294281) hf_save_interval: 5
(skyrl_entrypoint pid=294281) hf_upload_mode: latest
(skyrl_entrypoint pid=294281) export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports
(skyrl_entrypoint pid=294281) bf16: true
(skyrl_entrypoint pid=294281) epochs: 2
(skyrl_entrypoint pid=294281) max_steps: 80
(skyrl_entrypoint pid=294281) update_epochs_per_batch: 1
(skyrl_entrypoint pid=294281) train_batch_size: 64
(skyrl_entrypoint pid=294281) policy_mini_batch_size: 64
(skyrl_entrypoint pid=294281) critic_mini_batch_size: 256
(skyrl_entrypoint pid=294281) micro_train_batch_size_per_gpu: 1
(skyrl_entrypoint pid=294281) micro_forward_batch_size_per_gpu: 4
(skyrl_entrypoint pid=294281) update_ref_every_epoch: false
(skyrl_entrypoint pid=294281) use_sample_packing: true
(skyrl_entrypoint pid=294281) eval_batch_size: 64
(skyrl_entrypoint pid=294281) eval_before_train: false
(skyrl_entrypoint pid=294281) eval_interval: 999999
(skyrl_entrypoint pid=294281) max_prompt_length: 999999
(skyrl_entrypoint pid=294281) flash_attn: true
(skyrl_entrypoint pid=294281) disable_fast_tokenizer: false
(skyrl_entrypoint pid=294281) target_modules: null
(skyrl_entrypoint pid=294281) exclude_modules: null
(skyrl_entrypoint pid=294281) project_name: OpenThoughts-Agent
(skyrl_entrypoint pid=294281) run_name: ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=294281) logger: console
(skyrl_entrypoint pid=294281) tracker_commit_each_step: true
(skyrl_entrypoint pid=294281) dump_data_batch: false
(skyrl_entrypoint pid=294281) dump_eval_results: true
(skyrl_entrypoint pid=294281) log_level: INFO
(skyrl_entrypoint pid=294281) rope_scaling: null
(skyrl_entrypoint pid=294281) rope_theta: null
(skyrl_entrypoint pid=294281) step_wise_training: false
(skyrl_entrypoint pid=294281) hf_hub_repo_id: laion/ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=294281) hf_hub_private: false
(skyrl_entrypoint pid=294281) hf_hub_revision: main
(skyrl_entrypoint pid=294281) enable_db_registration: false
(skyrl_entrypoint pid=294281) generator:
(skyrl_entrypoint pid=294281) model_name: ${trainer.policy.model.path}
(skyrl_entrypoint pid=294281) model_dtype: bfloat16
(skyrl_entrypoint pid=294281) timeout_multiplier: 1.0
(skyrl_entrypoint pid=294281) run_engines_locally: true
(skyrl_entrypoint pid=294281) num_inference_engines: 48
(skyrl_entrypoint pid=294281) backend: vllm
(skyrl_entrypoint pid=294281) weight_sync_backend: nccl
(skyrl_entrypoint pid=294281) fuse_weights: false
(skyrl_entrypoint pid=294281) weight_transfer_threshold_cuda_ipc_GB: 1.0
(skyrl_entrypoint pid=294281) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_expert_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_data_parallel_size: 1
(skyrl_entrypoint pid=294281) n_samples_per_prompt: 8
(skyrl_entrypoint pid=294281) async_engine: true
(skyrl_entrypoint pid=294281) batched: false
(skyrl_entrypoint pid=294281) max_input_length: ${trainer.max_prompt_length}
(skyrl_entrypoint pid=294281) vllm_v1_disable_multiproc: true
(skyrl_entrypoint pid=294281) enable_prefix_caching: true
(skyrl_entrypoint pid=294281) enable_chunked_prefill: true
(skyrl_entrypoint pid=294281) max_num_batched_tokens: 65536
(skyrl_entrypoint pid=294281) enforce_eager: true
(skyrl_entrypoint pid=294281) fully_sharded_loras: false
(skyrl_entrypoint pid=294281) enable_ray_prometheus_stats: false
(skyrl_entrypoint pid=294281) vllm_stats_interval: 1
(skyrl_entrypoint pid=294281) gpu_memory_utilization: 0.75
(skyrl_entrypoint pid=294281) max_num_seqs: 24
(skyrl_entrypoint pid=294281) remote_inference_engine_urls:
(skyrl_entrypoint pid=294281) - 127.0.0.1:8001
(skyrl_entrypoint pid=294281) enable_http_endpoint: true
(skyrl_entrypoint pid=294281) http_endpoint_host: 127.0.0.1
(skyrl_entrypoint pid=294281) http_endpoint_port: 8000
(skyrl_entrypoint pid=294281) max_turns: 999999
(skyrl_entrypoint pid=294281) chat_template:
(skyrl_entrypoint pid=294281) source: name
(skyrl_entrypoint pid=294281) name_or_path: null
(skyrl_entrypoint pid=294281) chat_template_kwargs: {}
(skyrl_entrypoint pid=294281) engine_init_kwargs:
(skyrl_entrypoint pid=294281) max_model_len: 32768
(skyrl_entrypoint pid=294281) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=294281) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=294281) override_existing_update_group: disable
(skyrl_entrypoint pid=294281) sampling_params:
(skyrl_entrypoint pid=294281) max_generate_length: 4096
(skyrl_entrypoint pid=294281) repetition_penalty: 1.0
(skyrl_entrypoint pid=294281) temperature: 0.7
(skyrl_entrypoint pid=294281) top_p: 0.95
(skyrl_entrypoint pid=294281) min_p: 0.0
(skyrl_entrypoint pid=294281) top_k: 20
(skyrl_entrypoint pid=294281) logprobs: null
(skyrl_entrypoint pid=294281) stop: null
(skyrl_entrypoint pid=294281) use_conversation_multi_turn: true
(skyrl_entrypoint pid=294281) append_eos_token_after_stop_str_in_multi_turn: true
(skyrl_entrypoint pid=294281) eval_sampling_params:
(skyrl_entrypoint pid=294281) max_generate_length: ${generator.sampling_params.max_generate_length}
(skyrl_entrypoint pid=294281) repetition_penalty: 1.0
(skyrl_entrypoint pid=294281) temperature: 0.0
(skyrl_entrypoint pid=294281) top_p: 1.0
(skyrl_entrypoint pid=294281) min_p: 0.0
(skyrl_entrypoint pid=294281) top_k: -1
(skyrl_entrypoint pid=294281) logprobs: null
(skyrl_entrypoint pid=294281) stop: null
(skyrl_entrypoint pid=294281) eval_n_samples_per_prompt: 8
(skyrl_entrypoint pid=294281) zero_reward_on_non_stop: false
(skyrl_entrypoint pid=294281) apply_overlong_filtering: false
(skyrl_entrypoint pid=294281) rope_scaling: ${trainer.rope_scaling}
(skyrl_entrypoint pid=294281) rope_theta: ${trainer.rope_theta}
(skyrl_entrypoint pid=294281) teacher:
(skyrl_entrypoint pid=294281) model_path: null
(skyrl_entrypoint pid=294281) top_k_logprobs: 256
(skyrl_entrypoint pid=294281) num_inference_engines: 1
(skyrl_entrypoint pid=294281) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=294281) gpu_memory_utilization: 0.9
(skyrl_entrypoint pid=294281) enforce_eager: false
(skyrl_entrypoint pid=294281) backend: vllm
(skyrl_entrypoint pid=294281) engine_init_kwargs: {}
(skyrl_entrypoint pid=294281) environment:
(skyrl_entrypoint pid=294281) env_class: gsm8k
(skyrl_entrypoint pid=294281) skyrl_gym:
(skyrl_entrypoint pid=294281) max_env_workers: 32
(skyrl_entrypoint pid=294281) text2sql:
(skyrl_entrypoint pid=294281) db_path: /home/ray/default/sql_data
(skyrl_entrypoint pid=294281) llm_as_a_judge:
(skyrl_entrypoint pid=294281) model: gpt-4o-mini
(skyrl_entrypoint pid=294281) base_url: null
(skyrl_entrypoint pid=294281) search:
(skyrl_entrypoint pid=294281) log_requests: false
(skyrl_entrypoint pid=294281) search_url: http://127.0.0.1:8000/retrieve
(skyrl_entrypoint pid=294281) topk: 3
(skyrl_entrypoint pid=294281) timeout: 30
(skyrl_entrypoint pid=294281) rollout:
(skyrl_entrypoint pid=294281) fanout:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) num_coordinators: 4
(skyrl_entrypoint pid=294281) cpus_per_coordinator: 8
(skyrl_entrypoint pid=294281) deepspeed_config:
(skyrl_entrypoint pid=294281) train:
(skyrl_entrypoint pid=294281) zero_optimization:
(skyrl_entrypoint pid=294281) stage: 3
(skyrl_entrypoint pid=294281) offload_param:
(skyrl_entrypoint pid=294281) device: none
(skyrl_entrypoint pid=294281) offload_optimizer:
(skyrl_entrypoint pid=294281) device: none
(skyrl_entrypoint pid=294281) pin_memory: true
(skyrl_entrypoint pid=294281) sub_group_size: auto
(skyrl_entrypoint pid=294281) reduce_bucket_size: auto
(skyrl_entrypoint pid=294281) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=294281) stage3_prefetch_bucket_size: auto
(skyrl_entrypoint pid=294281) stage3_max_live_parameters: auto
(skyrl_entrypoint pid=294281) stage3_max_reuse_distance: auto
(skyrl_entrypoint pid=294281) round_robin_gradients: true
(skyrl_entrypoint pid=294281) zero_hpz_partition_size: 1
(skyrl_entrypoint pid=294281) zero_quantized_weights: false
(skyrl_entrypoint pid=294281) zero_quantized_gradients: false
(skyrl_entrypoint pid=294281) torch_autocast:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) dtype: bfloat16
(skyrl_entrypoint pid=294281) disable_trace_cache: false
(skyrl_entrypoint pid=294281) data_types:
(skyrl_entrypoint pid=294281) grad_accum_dtype: fp32
(skyrl_entrypoint pid=294281) gradient_clipping: 1.0
(skyrl_entrypoint pid=294281) wall_clock_breakdown: false
(skyrl_entrypoint pid=294281) prescale_gradient: false
(skyrl_entrypoint pid=294281) eval:
(skyrl_entrypoint pid=294281) zero_optimization:
(skyrl_entrypoint pid=294281) stage: 3
(skyrl_entrypoint pid=294281) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=294281) offload_param:
(skyrl_entrypoint pid=294281) device: cpu
(skyrl_entrypoint pid=294281) pin_memory: true
(skyrl_entrypoint pid=294281) torch_autocast:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) dtype: bfloat16
(skyrl_entrypoint pid=294281) gradient_clipping: 1.0
(skyrl_entrypoint pid=294281) prescale_gradient: false
(skyrl_entrypoint pid=294281) wall_clock_breakdown: false
(skyrl_entrypoint pid=294281) terminal_bench_config:
(skyrl_entrypoint pid=294281) trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs
(skyrl_entrypoint pid=294281) harbor:
(skyrl_entrypoint pid=294281) name: terminus-2
(skyrl_entrypoint pid=294281) max_episodes: 999999
(skyrl_entrypoint pid=294281) enable_summarize: false
(skyrl_entrypoint pid=294281) store_all_messages: true
(skyrl_entrypoint pid=294281) trajectory_config:
(skyrl_entrypoint pid=294281) raw_content: true
(skyrl_entrypoint pid=294281) enable_episode_logging: false
(skyrl_entrypoint pid=294281) record_terminal_session: false
(skyrl_entrypoint pid=294281) enable_pane_logging: false
(skyrl_entrypoint pid=294281) strict_json_parser: true
(skyrl_entrypoint pid=294281) interleaved_thinking: true
(skyrl_entrypoint pid=294281) extra_body:
(skyrl_entrypoint pid=294281) chat_template_kwargs:
(skyrl_entrypoint pid=294281) enable_thinking: true
(skyrl_entrypoint pid=294281) override_timeout_sec: 900
(skyrl_entrypoint pid=294281) override_cpus: 1
(skyrl_entrypoint pid=294281) override_memory_mb: 2048
(skyrl_entrypoint pid=294281) override_storage_mb: 2048
(skyrl_entrypoint pid=294281) auto_snapshot: true
(skyrl_entrypoint pid=294281) verifier_override_timeout_sec: 120
(skyrl_entrypoint pid=294281) max_retries: 3
(skyrl_entrypoint pid=294281) min_wait_sec: 60.0
(skyrl_entrypoint pid=294281) max_wait_sec: 600.0
(skyrl_entrypoint pid=294281) wait_multiplier: 2.0
(skyrl_entrypoint pid=294281) exclude_exceptions:
(skyrl_entrypoint pid=294281) - VerifierTimeoutError
(skyrl_entrypoint pid=294281) - VerifierRuntimeError
(skyrl_entrypoint pid=294281) - RewardFileNotFoundError
(skyrl_entrypoint pid=294281) - RewardFileEmptyError
(skyrl_entrypoint pid=294281) - VerifierOutputParseError
(skyrl_entrypoint pid=294281) n_concurrent_trials: 675
(skyrl_entrypoint pid=294281) log_level: INFO
(skyrl_entrypoint pid=294281) enable_reward_shaping: false
(skyrl_entrypoint pid=294281) enable_error_classification: true
(skyrl_entrypoint pid=294281) mask_exceptions:
(skyrl_entrypoint pid=294281) - DaytonaError
(skyrl_entrypoint pid=294281) - EnvironmentStartTimeoutError
(skyrl_entrypoint pid=294281) - NetworkError
(skyrl_entrypoint pid=294281) - ConnectionError
(skyrl_entrypoint pid=294281) - RewardFileNotFoundError
(skyrl_entrypoint pid=294281) - RewardFileEmptyError
(skyrl_entrypoint pid=294281) - AgentEnvironmentTimeoutError
(skyrl_entrypoint pid=294281) - ContextLengthExceededError
(skyrl_entrypoint pid=294281) default_error_treatment: zero
(skyrl_entrypoint pid=294281) passthrough_exceptions:
(skyrl_entrypoint pid=294281) - AgentTimeoutError
(skyrl_entrypoint pid=294281) zero_exceptions: []
(skyrl_entrypoint pid=294281) model_info:
(skyrl_entrypoint pid=294281) max_input_tokens: 32000
(skyrl_entrypoint pid=294281) max_output_tokens: 4096
(skyrl_entrypoint pid=294281) archiving:
(skyrl_entrypoint pid=294281) enabled: false
(skyrl_entrypoint pid=294281) trace_upload:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) repo_org: DCAgent
(skyrl_entrypoint pid=294281) episodes: last
(skyrl_entrypoint pid=294281) dataset_type: SFT
(skyrl_entrypoint pid=294281) cleanup: true
(skyrl_entrypoint pid=294281) 
(skyrl_entrypoint pid=294281) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(skyrl_entrypoint pid=294281) No module named 'vllm._version'
(skyrl_entrypoint pid=294281) from .version import __version__, __version_tuple__ # isort:skip
(skyrl_entrypoint pid=294281) W0607 02:21:22.569000 294281 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
[2026-06-07 02:21:23,669 E 293833 294255] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RegistryActor pid=987007, ip=10.128.32.37) [2026-06-07 02:21:24,551 E 987007 987047] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:27,013 E 294281 294324] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster]
(pid=987079, ip=10.128.32.37) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=987079, ip=10.128.32.37) No module named 'vllm._version'
(pid=987079, ip=10.128.32.37) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster]
(pid=2596339, ip=10.128.32.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(pid=2596339, ip=10.128.32.36) No module named 'vllm._version' [repeated 7x across cluster]
(pid=2596339, ip=10.128.32.36) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 111x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:40.638 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:40.638 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:40.639 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.259 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.284 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.284 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.285 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.285 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.305 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.18s (attempt 1/5) to avoid port collisions
(pid=2704166, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 13x across cluster]
(pid=2704166, ip=10.128.32.34) No module named 'vllm._version' [repeated 13x across cluster]
(pid=2704166, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip [repeated 13x across cluster]
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 69x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:45.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:45.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:45.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:21:46.027 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:21:46.028 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:21:46.028 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.972 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:47.003 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.81s (attempt 1/5) to avoid port collisions
(pid=2958820, ip=10.128.32.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 10x across cluster]
(pid=2958820, ip=10.128.32.39) No module named 'vllm._version' [repeated 10x across cluster]
(pid=2958820, ip=10.128.32.39) from .version import __version__, __version_tuple__ # isort:skip [repeated 10x across cluster]
(raylet, ip=10.128.32.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 63x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:50] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:50] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:50.121 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:50.122 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:50.122 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 17x across cluster]
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:51] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.391 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:969 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.391 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines.
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.394 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.395 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.408 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 156
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:357 - Length of train_dataloader: 5000
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:358 - Number of steps per epoch: 78
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:359 - Total training steps: 80
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.407 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.452 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.18s (attempt 1/5) to avoid port collisions [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) No module named 'vllm._version' [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) from .version import __version__, __version_tuple__ # isort:skip [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 120x across cluster]
(AsyncVLLMInferenceEngine pid=3079062, ip=10.128.32.42) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 32x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 17x across cluster]
(pid=2662729, ip=10.128.32.43) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) 2026-06-07 02:21:55.378 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) 2026-06-07 02:21:55.380 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) 2026-06-07 02:21:55.380 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) [W607 02:21:56.447607181 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-37.jupiter.internal]:41139 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) [W607 02:21:56.449335629 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) [rank0]:[W607 02:21:56.451880032 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.650 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.667 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.60s (attempt 1/5) to avoid port collisions [repeated 9x across cluster]
(bundle_reservation_check_func pid=294359) [2026-06-07 02:21:57,860 E 294359 294399] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) No module named 'vllm._version' [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:21:59 INFO [ipv4-debug] hostname=jpbo-041-43.jupiter.internal
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:21:59 INFO [ipv4-debug] _global_node.node_ip_address=10.128.32.43
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:21:59 INFO [ipv4-debug] get_node_ip_address()=10.128.32.43
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:00.059 | INFO  | skyrl_train.workers.worker:_initiate_actors:563 - Initializing process group for RayActorGroup
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) (EngineCore_DP0 pid=987460) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2541995, ip=10.128.32.40) (EngineCore_DP0 pid=2542222) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:21:58.938 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:21:58.939 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:21:58.939 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) [W607 02:22:01.433309818 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-37.jupiter.internal]:50831 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) [W607 02:22:01.433831729 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) [rank0]:[W607 02:22:01.436403587 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:02<00:08, 2.86s/it]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.236 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.294 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.55s (attempt 1/5) to avoid port collisions [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234) _C._set_float32_matmul_precision(precision)
(pid=2662807, ip=10.128.32.43) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) [2026-06-07 02:22:01,958 E 2541854 2541963] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2541994, ip=10.128.32.40) (EngineCore_DP0 pid=2542240) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) No module named 'vllm._version' [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 72x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 26x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) (EngineCore_DP0 pid=3079301) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2596211, ip=10.128.32.36) (EngineCore_DP0 pid=2596590) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:01.690 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:01.691 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:01.691 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) [W607 02:22:06.608280396 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-36.jupiter.internal]:36675 (errno: 97 - Address family not supported by protocol). [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) [W607 02:22:06.608742019 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) [rank0]:[W607 02:22:06.611067956 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:07<00:02, 2.64s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.951 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.975 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.975 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.975 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.976 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:03.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.80s (attempt 1/5) to avoid port collisions [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3078934, ip=10.128.32.42) (EngineCore_DP0 pid=3079317) _C._set_float32_matmul_precision(precision)
(pid=294462) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) [2026-06-07 02:22:07,570 E 2669552 2669716] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079062, ip=10.128.32.42) (EngineCore_DP0 pid=3079296) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418)
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452)
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) (EngineCore_DP0 pid=987460)
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447)
(AsyncVLLMInferenceEngine pid=2669550, ip=10.128.32.38) (EngineCore_DP0 pid=2669785) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) No module named 'vllm._version' [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 8x across cluster]
(FSDPPolicyWorkerBase pid=294462) [W607 02:22:11.696602899 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-43-interconnect-1.jupiter.internal]:36595 (errno: 97 - Address family not supported by protocol). [repeated 16x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [W607 02:22:11.005700994 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) [rank0]:[W607 02:22:10.881961700 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354) Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:04<00:04, 2.41s/it] [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2796151, ip=10.128.32.45) (EngineCore_DP0 pid=2796398) _C._set_float32_matmul_precision(precision)
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:12.293 | INFO  | skyrl_train.workers.worker:_initiate_actors:565 - Initialized process group for RayActorGroup
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:12.304 | INFO  | skyrl_train.workers.worker:_initiate_actors:567 - Mesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1)]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) `torch_dtype` is deprecated! Use `dtype` instead!
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) (EngineCore_DP0 pid=2704543) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704295, ip=10.128.32.34) (EngineCore_DP0 pid=2704527) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704296, ip=10.128.32.34) (EngineCore_DP0 pid=2704525) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=294465) Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
(FSDPPolicyWorkerBase pid=2662807, ip=10.128.32.43) Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:00, 9.50it/s]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) (EngineCore_DP0 pid=2626901) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:22:12.897 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) [2026-06-07 02:22:12,169 E 2796016 2796119] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 10x across cluster]
(FSDPPolicyWorkerBase pid=294465) Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 8.17it/s] Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 10.83it/s]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 10.75it/s]
(AsyncVLLMInferenceEngine pid=2958818, ip=10.128.32.39) (EngineCore_DP0 pid=2959050) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) (EngineCore_DP0 pid=2626885) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:13.430 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) (EngineCore_DP0 pid=2626889) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2958819, ip=10.128.32.39) (EngineCore_DP0 pid=2959046) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.790 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.816 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.816 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.817 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.817 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.850 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.65s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2975429, ip=10.128.32.44) (EngineCore_DP0 pid=2975795) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704295, ip=10.128.32.34) (EngineCore_DP0 pid=2704527) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) Process EngineCore_DP0:
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.run()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 108, in run
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self._target(*self._args, **self._kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1010, in run_engine_core
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) raise e
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) super().__init__(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self._init_executor()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) init_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) return TCPStore(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 44571, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40006f4ac700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #1: <unknown function> + 0x5f29220 (0x40004f649220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #2: <unknown function> + 0x5f4326c (0x40004f66326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #3: <unknown function> + 0x5f49074 (0x40004f669074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #4: <unknown function> + 0x5f49138 (0x40004f669138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #5: <unknown function> + 0x5f2ccc4 (0x40004f64ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x40004f651564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #7: <unknown function> + 0x109a094 (0x4000493fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #8: <unknown function> + 0x113236c (0x40004949236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #9: <unknown function> + 0x5d6d60 (0x400048936d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #10: <unknown function> + 0x1b7a38 (0xaaaab8697a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #12: <unknown function> + 0x169f50 (0xaaaab8649f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #13: <unknown function> + 0x1682e4 (0xaaaab86482e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #14: <unknown function> + 0x1e0ce8 (0xaaaab86c0ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #15: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #16: <unknown function> + 0x646b0c (0x4000489a6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #19: <unknown function> + 0x1808c0 (0xaaaab86608c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #20: <unknown function> + 0x182bf8 (0xaaaab8662bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #21: <unknown function> + 0x25fd30 (0xaaaab873fd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #22: <unknown function> + 0x1b7d20 (0xaaaab8697d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #23: PyObject_Vectorcall + 0x54 (0xaaaab86460e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaab8647f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #27: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #28: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaab8647fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #33: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #34: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #35: _PyObject_Call + 0x68 (0xaaaab8648488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaab874d8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #37: PyEval_EvalCode + 0xb4 (0xaaaab8752eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #38: <unknown function> + 0x2ccdcc (0xaaaab87acdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #39: <unknown function> + 0x2ccef4 (0xaaaab87acef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #40: PyRun_StringFlags + 0x90 (0xaaaab87b1050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaab87b10f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #42: Py_RunMain + 0x2c8 (0xaaaab87d9190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #43: Py_BytesMain + 0x64 (0xaaaab87d9fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #44: <unknown function> + 0x27540 (0x4000379a7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #45: __libc_start_main + 0x98 (0x4000379a7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #46: <unknown function> + 0x10e0c0 (0xaaaab85ee0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) [pynccl] dumped 0 entries to /e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace_pynccl_pid2959070 (reason=atexit)
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [W607 02:22:16.342198091 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-44.jupiter.internal]:41207 (errno: 97 - Address family not supported by protocol). [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [W607 02:22:16.342706915 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [rank0]:[W607 02:22:16.344851394 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=2958691, ip=10.128.32.39, actor_id=9f591a871969d2b79a9e310202000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400c1389d1f0>)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) return self.__get_result()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) raise self._exception
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) super().__init__(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) self.llm = self._create_engine(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) return cls(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) self.engine_core = EngineCoreClient.make_async_mp_client(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) return AsyncMPClient(*client_args)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) super().__init__(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) next(self.gen)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) wait_for_engine_startup(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) raise RuntimeError(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(AsyncVLLMInferenceEngine pid=2796151, ip=10.128.32.45) (EngineCore_DP0 pid=2796398) Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:03<00:01, 1.26s/it] [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) [pynccl] dumped 0 entries to /e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace_pynccl_pid2958691 (reason=atexit)
(FSDPPolicyWorkerBase pid=294462) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2796151, ip=10.128.32.45) (EngineCore_DP0 pid=2796398)
(FSDPPolicyWorkerBase pid=294463) Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s] [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=2662808, ip=10.128.32.43) Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 7.89it/s] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2578178, ip=10.128.32.41) (EngineCore_DP0 pid=2578410) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:22:12.972 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [2026-06-07 02:22:17,355 E 863950 864052] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 11x across cluster]
(FSDPPolicyWorkerBase pid=294462) Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 14.05it/s] Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 18.02it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391)
(FSDPPolicyWorkerBase pid=2662808, ip=10.128.32.43) Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 10.58it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:13.430 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230)
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2541994, ip=10.128.32.40) (EngineCore_DP0 pid=2542240)
(AsyncVLLMInferenceEngine pid=2541995, ip=10.128.32.40) (EngineCore_DP0 pid=2542222)
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.788 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.850 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.60s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2596211, ip=10.128.32.36) (EngineCore_DP0 pid=2596590)
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579)
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570)
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) [W607 02:22:20.901124860 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-41-interconnect-1.jupiter.internal]:43419 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) [W607 02:22:20.901611923 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) [rank0]:[W607 02:22:20.903638798 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.04s/it] [repeated 60x across cluster]
(AsyncVLLMInferenceEngine pid=3078934, ip=10.128.32.42) (EngineCore_DP0 pid=3079317)
(AsyncVLLMInferenceEngine pid=3079062, ip=10.128.32.42) (EngineCore_DP0 pid=3079296)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292)
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) (EngineCore_DP0 pid=3079301)
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) 2026-06-07 02:22:23.033 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 5x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [2026-06-07 02:22:22,373 E 2662729 2662769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802)
(AsyncVLLMInferenceEngine pid=2669550, ip=10.128.32.38) (EngineCore_DP0 pid=2669785)
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786)
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784)
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 4x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:26.990 | INFO  | skyrl_train.trainer:build_models:831 - init policy/ref/critic models done
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:26.998 | INFO  | skyrl_train.fully_async_trainer:_maybe_enable_rollout_fanout:456 - Rollout fan-out ENABLED: replacing single-process generator with RolloutDispatcher (K=4, cpus_per_coordinator=8).
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:27.021 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:387 - [RolloutDispatcher] fan-out path: overriding inference host 127.0.0.1 -> 10.128.32.33 (routable head IP) for coordinator litellm base_url connectivity
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:27.021 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:405 - [RolloutDispatcher] configured num_coordinators=4, cpus_per_coordinator=8
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:27.059 | INFO  | examples.terminal_bench.rollout_coordinator:startup:434 - [RolloutDispatcher] PlacementGroup ready: 4 bundles x 8 CPU (SPREAD)
(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(FSDPPolicyWorkerBase pid=294462) [rank7]:[W607 02:22:24.789186688 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:11<00:03, 3.91s/it] [repeated 46x across cluster]
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054)
(AsyncVLLMInferenceEngine pid=2958818, ip=10.128.32.39) (EngineCore_DP0 pid=2959050)
(AsyncVLLMInferenceEngine pid=2958819, ip=10.128.32.39) (EngineCore_DP0 pid=2959046)
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) (EngineCore_DP0 pid=2704543)
(AsyncVLLMInferenceEngine pid=2704295, ip=10.128.32.34) (EngineCore_DP0 pid=2704527)
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539)
(AsyncVLLMInferenceEngine pid=2704296, ip=10.128.32.34) (EngineCore_DP0 pid=2704525)
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) (EngineCore_DP0 pid=2626901)
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) 2026-06-07 02:22:27.290 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) (EngineCore_DP0 pid=2626889)
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) (EngineCore_DP0 pid=2626885)
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905)
(AsyncVLLMInferenceEngine pid=2975429, ip=10.128.32.44) (EngineCore_DP0 pid=2975795)
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804)
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787)
(FSDPPolicyWorkerBase pid=2662807, ip=10.128.32.43) [2026-06-07 02:22:30,830 E 2662807 2662947] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.049 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 0/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.265 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.265 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.265 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 0] startup complete
(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:32.266 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 1/4 started
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.28s/it] [repeated 38x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:22:32.541 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406)
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420)
(AsyncVLLMInferenceEngine pid=2578178, ip=10.128.32.41) (EngineCore_DP0 pid=2578410)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [W607 02:22:35.858670330 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-35.jupiter.internal]:33029 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [W607 02:22:35.862483909 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [rank0]:[W607 02:22:35.868173189 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 16x across cluster]
(FSDPPolicyWorkerBase pid=294462) [2026-06-07 02:22:31,188 E 294462 294693] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:12<00:00, 3.21s/it] [repeated 12x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.718 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:38.742 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:39.941 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 2/4 started
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 1/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.940 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.940 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.940 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 1] startup complete
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [W607 02:22:35.208951015 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-35-interconnect-1.jupiter.internal]:40633 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [W607 02:22:35.209434236 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [rank0]:[W607 02:22:35.211722249 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 5x across cluster]
(raylet, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.13s/it] [repeated 5x across cluster]
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.037 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.257 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.257 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.258 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.258 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 2/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.260 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.261 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.261 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 2] startup complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:48.262 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 3/4 started
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312)
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.36s/it] [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324)
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313)
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:54.394 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.126 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.36s/it] [repeated 6x across cluster]
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 4x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 4/4 started
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | examples.terminal_bench.rollout_coordinator:startup:477 - [RolloutDispatcher] 4 coordinators started
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | skyrl_train.fully_async_trainer:train:485 - Generator startup complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Started: 'load_checkpoints'
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.363 | INFO  | skyrl_train.trainer:load_checkpoints:1664 - Loading checkpoint from: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.363 | INFO  | skyrl_train.trainer:load_checkpoints:1670 - Resuming from global_step: 28
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.383 | INFO  | skyrl_train.trainer:load_checkpoints:1686 - Successfully loaded trainer state
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.349 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.350 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.350 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.350 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 3/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.353 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.353 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.353 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 3] startup complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.392 | INFO  | skyrl_train.trainer:load_checkpoints:1696 - Successfully loaded dataloader state
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.392 | INFO  | skyrl_train.trainer:load_checkpoints:1705 - Loading policy checkpoint from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [2026-06-07 02:22:57,810 E 2669976 2670068] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RolloutCoordinator pid=987693, ip=10.128.32.37) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) 2026-06-07 02:22:54.477 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [2026-06-07 02:23:05,084 E 2542476 2542516] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [2026-06-07 02:23:12,865 E 2596821 2596861] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) warnings.warn( # warn only once
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.648 | INFO  | skyrl_train.trainer:load_checkpoints:1715 - Successfully loaded policy checkpoint
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.648 | INFO  | skyrl_train.trainer:load_checkpoints:1731 - Successfully loaded complete checkpoint state from global_step_28
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.648 | INFO  | skyrl_train.fully_async_trainer:_train_loop:512 - Resumed training from global_step 28
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.663 | INFO  | skyrl_train.utils.data_tracker:load_state:97 - Loaded data tracker state: epoch=0, consumed_in_epoch=1792, total_consumed=1792
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.664 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Finished: 'load_checkpoints', time cost: 19.31s
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.664 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Started: 'init_weight_sync_state'
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) No module named 'vllm._version'
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:23:19.911 | INFO  | logging:info:2216 - [weight-sync] Using master_addr=10.128.32.43, master_port=46453
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank0]:[W607 02:23:19.180220203 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-43.jupiter.internal]:46453 (errno: 97 - Address family not supported by protocol).
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:20.003 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Finished: 'init_weight_sync_state', time cost: 4.34s
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:20.003 | ERROR  | skyrl_train.fully_async_trainer:train:493 - Train loop failed at global_step 28: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exi
(skyrl_entrypoint pid=294281) Traceback (most recent call last):
(skyrl_entrypoint pid=294281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
(skyrl_entrypoint pid=294281) await self._train_loop()
(skyrl_entrypoint pid=294281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
(skyrl_entrypoint pid=294281) self.init_weight_sync_state()
(skyrl_entrypoint pid=294281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
(skyrl_entrypoint pid=294281) raise RuntimeError(
(skyrl_entrypoint pid=294281) RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", l
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) warnings.warn("No model update group to destroy")
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:23:19.910 | INFO  | logging:info:2216 - [weight-sync] get_node_ip_address()=10.128.32.43 [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=294462) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=294462) No module named 'vllm._version' [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=294462) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:363 - Shutting down shared QueueOrchestrator...
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:365 - QueueOrchestrator shutdown complete
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.rollout_coordinator:shutdown:258 - [RolloutCoordinator 1] shutdown complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:21.029 | INFO  | skyrl_train.inference_engines.inference_engine_client_http_endpoint:shutdown_server:203 - Server shut down after 2 seconds
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:21.029 | INFO  | skyrl_train.trainer:_guarded_sync:226 - HTTP endpoint shutdown complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:21.033 | INFO  | skyrl_train.trainer:_guarded_async:215 - Generator shutdown complete
2026-06-07 02:23:21.090 | ERROR | __main__:main:134 - Training failed: ray::skyrl_entrypoint() (pid=294281, ip=10.128.32.33)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 483, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
raise RuntimeError(
RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_st
2026-06-07 02:23:21.091 | INFO | __main__:main:137 - Shutting down Ray on head node...
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054) warnings.warn("No model update group to destroy") [repeated 27x across cluster]
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:363 - Shutting down shared QueueOrchestrator... [repeated 3x across cluster]
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:365 - QueueOrchestrator shutdown complete [repeated 3x across cluster]
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.rollout_coordinator:shutdown:258 - [RolloutCoordinator 3] shutdown complete [repeated 3x across cluster]
(skyrl_entrypoint pid=294281) [fd-monitor] Started monitoring (every 120s)
(skyrl_entrypoint pid=294281) [fd-monitor] [02:21:02] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=294281) [fd-monitor] [02:21:02] OK: RSS 1.38 GiB | node mem 129.1/858.0 GiB used (15.0%), avail 728.9 GiB
(skyrl_entrypoint pid=294281) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=294281) INFO 06-07 02:21:22 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(pid=987079, ip=10.128.32.37) INFO 06-07 02:21:39 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:43 [arg_utils.py:1256] The global random seed is set to 42. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:43 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:44 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:44 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(pid=987212, ip=10.128.32.37) INFO 06-07 02:21:44 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) WARNING 06-07 02:21:48 [arg_utils.py:1256] The global random seed is set to 56. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:48 [arg_utils.py:1256] The global random seed is set to 55. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [model.py:1549] Using max model len 32768 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:48 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) WARNING 06-07 02:21:48 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) WARNING 06-07 02:21:48 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:49 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:48 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:51 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=42, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [arg_utils.py:1256] The global random seed is set to 61. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [model.py:1549] Using max model len 32768 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 16x across cluster]
(pid=2626785, ip=10.128.32.46) INFO 06-07 02:21:55 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:56 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:56 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.37:41139 backend=nccl
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:56 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:21:57 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=56, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:57 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:58 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) WARNING 06-07 02:21:58 [arg_utils.py:1256] The global random seed is set to 69. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [model.py:1549] Using max model len 32768 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) WARNING 06-07 02:21:58 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) WARNING 06-07 02:21:59 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) WARNING 06-07 02:21:59 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster]
(pid=2662729, ip=10.128.32.43) ⚙️ Running in WANDB offline mode
(pid=2578180, ip=10.128.32.41) INFO 06-07 02:22:00 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 26x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) INFO 06-07 02:22:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:01 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.37:50831 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:01 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2669550, ip=10.128.32.38) (EngineCore_DP0 pid=2669785) INFO 06-07 02:22:01 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=51, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:02 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:02 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:02 [arg_utils.py:1256] The global random seed is set to 84. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [model.py:1549] Using max model len 32768 [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:02 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:03 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:03 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) INFO 06-07 02:22:05 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292) INFO 06-07 02:22:06 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566) INFO 06-07 02:22:06 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.36:37763 backend=nccl [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566) INFO 06-07 02:22:06 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539) INFO 06-07 02:22:05 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=64, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570) INFO 06-07 02:22:07 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 9x across cluster]
(pid=2662809, ip=10.128.32.43) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:08 [default_loader.py:293] Loading weights took 9.29 seconds
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 6.143796 seconds
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570) INFO 06-07 02:22:08 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:05 [arg_utils.py:1256] The global random seed is set to 89. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [model.py:1549] Using max model len 32768 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:06 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:06 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:06 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975429, ip=10.128.32.44) INFO 06-07 02:22:10 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:10 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:11 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:11 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:11 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:10 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.45:48431 backend=nccl [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:10 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:12 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:12 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.50 seconds
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:12 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=84, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:11 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) WARNING 06-07 02:22:12 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:22:12 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(pid=294462) ⚙️ Running in WANDB offline mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:08 [default_loader.py:293] Loading weights took 6.05 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 6.835432 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:13 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) WARNING 06-07 02:22:12 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] EngineCore failed to start.
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] super().__init__(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self._init_executor()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] init_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] return TCPStore(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 44571, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40006f4ac700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #1: <unknown function> + 0x5f29220 (0x40004f649220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #2: <unknown function> + 0x5f4326c (0x40004f66326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #3: <unknown function> + 0x5f49074 (0x40004f669074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #4: <unknown function> + 0x5f49138 (0x40004f669138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #5: <unknown function> + 0x5f2ccc4 (0x40004f64ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x40004f651564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #7: <unknown function> + 0x109a094 (0x4000493fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #8: <unknown function> + 0x113236c (0x40004949236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #9: <unknown function> + 0x5d6d60 (0x400048936d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #10: <unknown function> + 0x1b7a38 (0xaaaab8697a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #12: <unknown function> + 0x169f50 (0xaaaab8649f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #13: <unknown function> + 0x1682e4 (0xaaaab86482e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #14: <unknown function> + 0x1e0ce8 (0xaaaab86c0ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #15: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #16: <unknown function> + 0x646b0c (0x4000489a6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #19: <unknown function> + 0x1808c0 (0xaaaab86608c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #20: <unknown function> + 0x182bf8 (0xaaaab8662bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #21: <unknown function> + 0x25fd30 (0xaaaab873fd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #22: <unknown function> + 0x1b7d20 (0xaaaab8697d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #23: PyObject_Vectorcall + 0x54 (0xaaaab86460e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaab8647f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #27: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #28: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaab8647fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #33: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #34: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #35: _PyObject_Call + 0x68 (0xaaaab8648488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaab874d8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #37: PyEval_EvalCode + 0xb4 (0xaaaab8752eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #38: <unknown function> + 0x2ccdcc (0xaaaab87acdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #39: <unknown function> + 0x2ccef4 (0xaaaab87acef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #40: PyRun_StringFlags + 0x90 (0xaaaab87b1050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaab87b10f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #42: Py_RunMain + 0x2c8 (0xaaaab87d9190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #43: Py_BytesMain + 0x64 (0xaaaab87d9fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #44: <unknown function> + 0x27540 (0x4000379a7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #45: __libc_start_main + 0x98 (0x4000379a7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #46: <unknown function> + 0x10e0c0 (0xaaaab85ee0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) INFO 06-07 02:22:15 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) WARNING 06-07 02:22:16 [arg_utils.py:1256] The global random seed is set to 77. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) WARNING 06-07 02:22:16 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:11 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:11 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:11 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.44:41207 backend=nccl [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) WARNING 06-07 02:22:16 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.49 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406) INFO 06-07 02:22:16 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=86, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) WARNING 06-07 02:22:12 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) WARNING 06-07 02:22:12 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:18 [default_loader.py:293] Loading weights took 4.18 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 4.518817 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:17 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:19 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:19 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) WARNING 06-07 02:22:19 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) INFO 06-07 02:22:15 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:17 [arg_utils.py:1256] The global random seed is set to 76. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:17 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:20 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:20 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.41:43419 backend=nccl [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:20 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:17 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) INFO 06-07 02:22:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) INFO 06-07 02:22:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:21 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234) WARNING 06-07 02:22:22 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) WARNING 06-07 02:22:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:23 [default_loader.py:293] Loading weights took 13.40 seconds [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802) INFO 06-07 02:22:23 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.574883 seconds [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:21 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) NCCL version 2.27.7+cuda13.0
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) INFO 06-07 02:22:25 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) INFO 06-07 02:22:25 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) WARNING 06-07 02:22:25 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:26 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) WARNING 06-07 02:22:27 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) WARNING 06-07 02:22:27 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:28 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=76, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:28 [default_loader.py:293] Loading weights took 13.02 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539) INFO 06-07 02:22:28 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.076531 seconds [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:27 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:27 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) WARNING 06-07 02:22:27 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 11x across cluster]
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [fd-monitor] [02:22:32] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [fd-monitor] [02:22:32] OK: RSS 0.78 GiB | node mem 385.3/858.0 GiB used (44.9%), avail 472.7 GiB
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) WARNING 06-07 02:22:32 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) WARNING 06-07 02:22:32 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:28 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=74, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) INFO 06-07 02:22:30 [default_loader.py:293] Loading weights took 13.13 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) INFO 06-07 02:22:30 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.203295 seconds [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:34 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.35:33029 backend=nccl
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) INFO 06-07 02:22:34 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) INFO 06-07 02:22:34 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) WARNING 06-07 02:22:34 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:35 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) INFO 06-07 02:22:37 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) INFO 06-07 02:22:37 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) INFO 06-07 02:22:37 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) INFO 06-07 02:22:33 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) INFO 06-07 02:22:33 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406) WARNING 06-07 02:22:38 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) WARNING 06-07 02:22:34 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 4x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [fd-monitor] [02:22:39] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [fd-monitor] [02:22:39] OK: RSS 0.78 GiB | node mem 406.2/858.0 GiB used (47.3%), avail 451.7 GiB
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:34 [default_loader.py:293] Loading weights took 12.85 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:35 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 13.617507 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.35:40633 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) WARNING 06-07 02:22:38 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:37 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:37 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:37 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) WARNING 06-07 02:22:38 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) WARNING 06-07 02:22:38 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 4x across cluster]
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [fd-monitor] [02:22:48] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [fd-monitor] [02:22:48] OK: RSS 0.78 GiB | node mem 394.8/858.0 GiB used (46.0%), avail 463.2 GiB
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:50 [default_loader.py:293] Loading weights took 13.45 seconds
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:50 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.577902 seconds
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:53 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:53 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:53 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.72 seconds
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) WARNING 06-07 02:22:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:54 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) WARNING 06-07 02:22:54 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:54 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) WARNING 06-07 02:22:54 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(RolloutCoordinator pid=987693, ip=10.128.32.37) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=987693, ip=10.128.32.37) [fd-monitor] [02:22:56] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=987693, ip=10.128.32.37) [fd-monitor] [02:22:56] OK: RSS 0.78 GiB | node mem 390.4/858.0 GiB used (45.5%), avail 467.6 GiB
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:50 [default_loader.py:293] Loading weights took 13.45 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:50 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.715511 seconds [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/model_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/extra_state_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/optim_world_size_8_rank_0.pt
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) WARNING 06-07 02:22:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:54 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) WARNING 06-07 02:22:54 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:54 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:54 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(skyrl_entrypoint pid=294281) [fd-monitor] [02:23:02] OK: 102 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=294281) [fd-monitor] [02:23:02] OK: RSS 1.64 GiB | node mem 233.1/858.0 GiB used (27.2%), avail 624.9 GiB
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Successfully loaded model state dict
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Successfully loaded optimizer state
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Successfully loaded scheduler state
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) INFO 06-07 02:23:18 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(FSDPPolicyWorkerBase pid=294462) INFO 06-07 02:23:19 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 7x across cluster]
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=sequence_mean', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqmean-arm0', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harb
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::skyrl_entrypoint() (pid=294281, ip=10.128.32.33)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 483, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
raise RuntimeError(
RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_st
Stopping Ray cluster...
Ray cluster stopped
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
Collecting Ray logs from worker jpbo-041-34...
Collecting Ray logs from worker jpbo-041-35...
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
[RLJobRunner] Crash-time Ray log preservation complete.
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
episodes: last
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
Collecting Ray logs from worker jpbo-041-34...
Collecting Ray logs from worker jpbo-041-35...
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
Ray log preservation complete
Reference in New Issue Copy Permalink