laion/ablation-pymethods2test-seqmean-arm0-30-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0_630126.out
ModelHub XC 0c293fc953 初始化项目，由ModelHub XC社区提供模型 
Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B
Source: Original Platform
2026-07-18 18:02:02 +08:00

1666 lines
318 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_630126
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630126
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630126
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.32.33 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630126.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630126.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.32.33:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-041-33 (10.128.32.33)
Ray port: 6379
============================
Starting Ray head on jpbo-041-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-33.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-041-33
Starting Ray worker on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-34.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-041-34
Starting Ray worker on jpbo-041-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-35.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-041-35
Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-041-36
Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-041-37
Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-041-38
Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-041-39
Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-041-40
Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-041-41
Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-041-42
Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-041-43
Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-041-44
Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-041-45
Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-041-46
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.32.33:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'GPU': 56.0, 'memory': 10652272558080.0, 'accelerator_type:GH200': 14.0, 'object_store_memory': 601295421440.0, 'CPU': 4032.0, 'node:10.128.32.35': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.39': 1.0, 'node:10.128.32.41': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.37': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
Address: 10.128.32.33:6379
Total GPUs: 56
=========================
Ray cluster ready at 10.128.32.33:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
Running SkyRL:
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
Args: 120 Hydra arguments
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=sequence_mean trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0 ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqmean-arm0 trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630126.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 02:20:53.422 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:686 - Exporting wandb api key to ray runtime env
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:705 - Exporting RAY_ADDRESS to ray runtime env
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
2026-06-07 02:20:53.423 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
2026-06-07 02:20:53,423 INFO worker.py:1680 -- Using address 10.128.32.33:6379 set in the environment variable RAY_ADDRESS
2026-06-07 02:20:53,456 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.33:6379...
2026-06-07 02:20:53,467 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 02:20:55.689 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:02.936 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:212 - SkyRL log level set to: INFO
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(skyrl_entrypoint pid=294281) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:03.364 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:06.531 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 5000 valid task directories out of 5000 total directories
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:06.532 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 5000 task paths
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:06.545 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:405 - data:
(skyrl_entrypoint pid=294281) train_data:
(skyrl_entrypoint pid=294281) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=294281) val_data: []
(skyrl_entrypoint pid=294281) trainer:
(skyrl_entrypoint pid=294281) placement:
(skyrl_entrypoint pid=294281) colocate_all: false
(skyrl_entrypoint pid=294281) colocate_policy_ref: true
(skyrl_entrypoint pid=294281) policy_num_nodes: 2
(skyrl_entrypoint pid=294281) policy_num_gpus_per_node: 4
(skyrl_entrypoint pid=294281) critic_num_nodes: 1
(skyrl_entrypoint pid=294281) critic_num_gpus_per_node: 4
(skyrl_entrypoint pid=294281) ref_num_nodes: 2
(skyrl_entrypoint pid=294281) ref_num_gpus_per_node: 4
(skyrl_entrypoint pid=294281) policy_strict_spread_pg: false
(skyrl_entrypoint pid=294281) policy_per_gpu_bundles: false
(skyrl_entrypoint pid=294281) sequence_parallel_backend: ulysses
(skyrl_entrypoint pid=294281) strategy: fsdp2
(skyrl_entrypoint pid=294281) policy:
(skyrl_entrypoint pid=294281) model:
(skyrl_entrypoint pid=294281) path: /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=294281) lora:
(skyrl_entrypoint pid=294281) rank: 0
(skyrl_entrypoint pid=294281) alpha: 16
(skyrl_entrypoint pid=294281) dropout: 0
(skyrl_entrypoint pid=294281) lora_sync_path: /tmp/skyrl_lora_sync
(skyrl_entrypoint pid=294281) target_modules: all-linear
(skyrl_entrypoint pid=294281) exclude_modules: null
(skyrl_entrypoint pid=294281) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=294281) optimizer_config:
(skyrl_entrypoint pid=294281) optimizer: AdamW
(skyrl_entrypoint pid=294281) lr: 8.0e-06
(skyrl_entrypoint pid=294281) adam_betas:
(skyrl_entrypoint pid=294281) - 0.9
(skyrl_entrypoint pid=294281) - 0.999
(skyrl_entrypoint pid=294281) weight_decay: 0.0
(skyrl_entrypoint pid=294281) max_grad_norm: 0.9
(skyrl_entrypoint pid=294281) offload_after_step: true
(skyrl_entrypoint pid=294281) num_warmup_steps: 0
(skyrl_entrypoint pid=294281) scheduler: constant_with_warmup
(skyrl_entrypoint pid=294281) optimizer_kwargs: {}
(skyrl_entrypoint pid=294281) fsdp_config:
(skyrl_entrypoint pid=294281) cpu_offload: false
(skyrl_entrypoint pid=294281) reshard_after_forward: true
(skyrl_entrypoint pid=294281) fsdp_size: 4
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=294281) moe_router_replay: false
(skyrl_entrypoint pid=294281) moe_grouped_gemm: false
(skyrl_entrypoint pid=294281) ep_comm_backend: torch
(skyrl_entrypoint pid=294281) deepep_num_sms: 20
(skyrl_entrypoint pid=294281) deepep_token_chunk_size: null
(skyrl_entrypoint pid=294281) sequence_parallel_size: 1
(skyrl_entrypoint pid=294281) use_torch_compile: false
(skyrl_entrypoint pid=294281) record_memory: false
(skyrl_entrypoint pid=294281) megatron_config:
(skyrl_entrypoint pid=294281) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=294281) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=294281) context_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: null
(skyrl_entrypoint pid=294281) ddp_config:
(skyrl_entrypoint pid=294281) grad_reduce_in_fp32: true
(skyrl_entrypoint pid=294281) overlap_grad_reduce: false
(skyrl_entrypoint pid=294281) overlap_param_gather: false
(skyrl_entrypoint pid=294281) average_in_collective: true
(skyrl_entrypoint pid=294281) model_config_kwargs: {}
(skyrl_entrypoint pid=294281) torch_profiler_config:
(skyrl_entrypoint pid=294281) enable: false
(skyrl_entrypoint pid=294281) ranks: []
(skyrl_entrypoint pid=294281) save_path: null
(skyrl_entrypoint pid=294281) optimizer_config_kwargs:
(skyrl_entrypoint pid=294281) overlap_cpu_optimizer_d2h_h2d: false
(skyrl_entrypoint pid=294281) use_precision_aware_optimizer: false
(skyrl_entrypoint pid=294281) optimizer_cpu_offload: false
(skyrl_entrypoint pid=294281) optimizer_offload_fraction: 0.0
(skyrl_entrypoint pid=294281) transformer_config_kwargs:
(skyrl_entrypoint pid=294281) recompute_granularity: full
(skyrl_entrypoint pid=294281) recompute_modules:
(skyrl_entrypoint pid=294281) - core_attn
(skyrl_entrypoint pid=294281) recompute_method: uniform
(skyrl_entrypoint pid=294281) recompute_num_layers: 1
(skyrl_entrypoint pid=294281) empty_cuda_cache: true
(skyrl_entrypoint pid=294281) ref:
(skyrl_entrypoint pid=294281) model:
(skyrl_entrypoint pid=294281) path: ${trainer.policy.model.path}
(skyrl_entrypoint pid=294281) sequence_parallel_size: 1
(skyrl_entrypoint pid=294281) deepspeed_config: ${deepspeed_config.eval}
(skyrl_entrypoint pid=294281) fsdp_config:
(skyrl_entrypoint pid=294281) cpu_offload: false
(skyrl_entrypoint pid=294281) reshard_after_forward: true
(skyrl_entrypoint pid=294281) fsdp_size: 4
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=294281) moe_router_replay: false
(skyrl_entrypoint pid=294281) moe_grouped_gemm: false
(skyrl_entrypoint pid=294281) ep_comm_backend: torch
(skyrl_entrypoint pid=294281) deepep_num_sms: 20
(skyrl_entrypoint pid=294281) deepep_token_chunk_size: null
(skyrl_entrypoint pid=294281) megatron_config:
(skyrl_entrypoint pid=294281) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=294281) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=294281) context_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) model_config_kwargs: {}
(skyrl_entrypoint pid=294281) transformer_config_kwargs: {}
(skyrl_entrypoint pid=294281) critic:
(skyrl_entrypoint pid=294281) model:
(skyrl_entrypoint pid=294281) path: null
(skyrl_entrypoint pid=294281) lora:
(skyrl_entrypoint pid=294281) rank: 0
(skyrl_entrypoint pid=294281) alpha: 16
(skyrl_entrypoint pid=294281) dropout: 0
(skyrl_entrypoint pid=294281) target_modules: all-linear
(skyrl_entrypoint pid=294281) exclude_modules: null
(skyrl_entrypoint pid=294281) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=294281) optimizer_config:
(skyrl_entrypoint pid=294281) optimizer: AdamW
(skyrl_entrypoint pid=294281) lr: 5.0e-06
(skyrl_entrypoint pid=294281) adam_betas:
(skyrl_entrypoint pid=294281) - 0.9
(skyrl_entrypoint pid=294281) - 0.999
(skyrl_entrypoint pid=294281) weight_decay: 0.01
(skyrl_entrypoint pid=294281) max_grad_norm: 1.0
(skyrl_entrypoint pid=294281) offload_after_step: true
(skyrl_entrypoint pid=294281) num_warmup_steps: 0
(skyrl_entrypoint pid=294281) scheduler: constant_with_warmup
(skyrl_entrypoint pid=294281) optimizer_kwargs: {}
(skyrl_entrypoint pid=294281) fsdp_config:
(skyrl_entrypoint pid=294281) cpu_offload: false
(skyrl_entrypoint pid=294281) reshard_after_forward: true
(skyrl_entrypoint pid=294281) fsdp_size: -1
(skyrl_entrypoint pid=294281) expert_model_parallel_size: 1
(skyrl_entrypoint pid=294281) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=294281) moe_router_replay: false
(skyrl_entrypoint pid=294281) moe_grouped_gemm: false
(skyrl_entrypoint pid=294281) ep_comm_backend: torch
(skyrl_entrypoint pid=294281) deepep_num_sms: 20
(skyrl_entrypoint pid=294281) deepep_token_chunk_size: null
(skyrl_entrypoint pid=294281) sequence_parallel_size: 1
(skyrl_entrypoint pid=294281) algorithm:
(skyrl_entrypoint pid=294281) advantage_estimator: rloo_n
(skyrl_entrypoint pid=294281) kl_ctrl:
(skyrl_entrypoint pid=294281) type: fixed
(skyrl_entrypoint pid=294281) kl_target: 0.1
(skyrl_entrypoint pid=294281) horizon: 10000
(skyrl_entrypoint pid=294281) kl_estimator_type: k3
(skyrl_entrypoint pid=294281) use_kl_estimator_k3: false
(skyrl_entrypoint pid=294281) use_abs_kl: false
(skyrl_entrypoint pid=294281) use_kl_in_reward: false
(skyrl_entrypoint pid=294281) use_kl_loss: false
(skyrl_entrypoint pid=294281) kl_loss_coef: 0.0
(skyrl_entrypoint pid=294281) use_entropy_loss: false
(skyrl_entrypoint pid=294281) entropy_loss_coef: 0.01
(skyrl_entrypoint pid=294281) advantage_batch_normalize: false
(skyrl_entrypoint pid=294281) value_head_prefix: value_head
(skyrl_entrypoint pid=294281) policy_loss_type: regular
(skyrl_entrypoint pid=294281) loss_reduction: sequence_mean
(skyrl_entrypoint pid=294281) global_loss_denom: null
(skyrl_entrypoint pid=294281) grpo_norm_by_std: true
(skyrl_entrypoint pid=294281) rloo_n_min_group_size: 4
(skyrl_entrypoint pid=294281) rloo_n_filter_zero_reward_groups: true
(skyrl_entrypoint pid=294281) lambd: 1.0
(skyrl_entrypoint pid=294281) gamma: 1.0
(skyrl_entrypoint pid=294281) eps_clip_low: 0.2
(skyrl_entrypoint pid=294281) eps_clip_high: 0.05
(skyrl_entrypoint pid=294281) clip_ratio_c: 3.0
(skyrl_entrypoint pid=294281) tis_imp_ratio_cap: -1.0
(skyrl_entrypoint pid=294281) use_tis: false
(skyrl_entrypoint pid=294281) sapo:
(skyrl_entrypoint pid=294281) tau_pos: 1.0
(skyrl_entrypoint pid=294281) tau_neg: 1.05
(skyrl_entrypoint pid=294281) value_clip: 0.2
(skyrl_entrypoint pid=294281) dynamic_sampling:
(skyrl_entrypoint pid=294281) type: null
(skyrl_entrypoint pid=294281) max_sample_batches: 30
(skyrl_entrypoint pid=294281) min_replace_ratio: 0.3
(skyrl_entrypoint pid=294281) clip_cov:
(skyrl_entrypoint pid=294281) clip_ratio: 0.0002
(skyrl_entrypoint pid=294281) clip_cov_lb: 1.0
(skyrl_entrypoint pid=294281) clip_cov_ub: 5.0
(skyrl_entrypoint pid=294281) kl_cov:
(skyrl_entrypoint pid=294281) kl_cov_frac: 0.2
(skyrl_entrypoint pid=294281) ppo_kl_coef: 1.0
(skyrl_entrypoint pid=294281) cispo:
(skyrl_entrypoint pid=294281) cispo_eps_clip_low: 0
(skyrl_entrypoint pid=294281) cispo_eps_clip_high: 5
(skyrl_entrypoint pid=294281) z_clip:
(skyrl_entrypoint pid=294281) enabled: false
(skyrl_entrypoint pid=294281) alpha: 0.97
(skyrl_entrypoint pid=294281) z_thresh: 2.5
(skyrl_entrypoint pid=294281) warmup_steps: 25
(skyrl_entrypoint pid=294281) mode: zscore
(skyrl_entrypoint pid=294281) clip_option: adaptive_scaling
(skyrl_entrypoint pid=294281) clip_factor: 1.0
(skyrl_entrypoint pid=294281) skip_update_on_spike: false
(skyrl_entrypoint pid=294281) stale_clip:
(skyrl_entrypoint pid=294281) enabled: false
(skyrl_entrypoint pid=294281) alpha: 0.3
(skyrl_entrypoint pid=294281) entropy_threshold: 0.15
(skyrl_entrypoint pid=294281) entropy_window: 10
(skyrl_entrypoint pid=294281) min_lr_scale: 0.1
(skyrl_entrypoint pid=294281) max_seq_len: 1004095
(skyrl_entrypoint pid=294281) fully_async:
(skyrl_entrypoint pid=294281) max_staleness_steps: 16
(skyrl_entrypoint pid=294281) num_parallel_generation_workers: 338
(skyrl_entrypoint pid=294281) gradient_checkpointing: true
(skyrl_entrypoint pid=294281) gradient_checkpointing_use_reentrant: false
(skyrl_entrypoint pid=294281) seed: 42
(skyrl_entrypoint pid=294281) resume_mode: latest
(skyrl_entrypoint pid=294281) resume_path: null
(skyrl_entrypoint pid=294281) ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints
(skyrl_entrypoint pid=294281) max_ckpts_to_keep: -1
(skyrl_entrypoint pid=294281) ckpt_interval: 2
(skyrl_entrypoint pid=294281) hf_save_interval: 5
(skyrl_entrypoint pid=294281) hf_upload_mode: latest
(skyrl_entrypoint pid=294281) export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports
(skyrl_entrypoint pid=294281) bf16: true
(skyrl_entrypoint pid=294281) epochs: 2
(skyrl_entrypoint pid=294281) max_steps: 80
(skyrl_entrypoint pid=294281) update_epochs_per_batch: 1
(skyrl_entrypoint pid=294281) train_batch_size: 64
(skyrl_entrypoint pid=294281) policy_mini_batch_size: 64
(skyrl_entrypoint pid=294281) critic_mini_batch_size: 256
(skyrl_entrypoint pid=294281) micro_train_batch_size_per_gpu: 1
(skyrl_entrypoint pid=294281) micro_forward_batch_size_per_gpu: 4
(skyrl_entrypoint pid=294281) update_ref_every_epoch: false
(skyrl_entrypoint pid=294281) use_sample_packing: true
(skyrl_entrypoint pid=294281) eval_batch_size: 64
(skyrl_entrypoint pid=294281) eval_before_train: false
(skyrl_entrypoint pid=294281) eval_interval: 999999
(skyrl_entrypoint pid=294281) max_prompt_length: 999999
(skyrl_entrypoint pid=294281) flash_attn: true
(skyrl_entrypoint pid=294281) disable_fast_tokenizer: false
(skyrl_entrypoint pid=294281) target_modules: null
(skyrl_entrypoint pid=294281) exclude_modules: null
(skyrl_entrypoint pid=294281) project_name: OpenThoughts-Agent
(skyrl_entrypoint pid=294281) run_name: ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=294281) logger: console
(skyrl_entrypoint pid=294281) tracker_commit_each_step: true
(skyrl_entrypoint pid=294281) dump_data_batch: false
(skyrl_entrypoint pid=294281) dump_eval_results: true
(skyrl_entrypoint pid=294281) log_level: INFO
(skyrl_entrypoint pid=294281) rope_scaling: null
(skyrl_entrypoint pid=294281) rope_theta: null
(skyrl_entrypoint pid=294281) step_wise_training: false
(skyrl_entrypoint pid=294281) hf_hub_repo_id: laion/ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=294281) hf_hub_private: false
(skyrl_entrypoint pid=294281) hf_hub_revision: main
(skyrl_entrypoint pid=294281) enable_db_registration: false
(skyrl_entrypoint pid=294281) generator:
(skyrl_entrypoint pid=294281) model_name: ${trainer.policy.model.path}
(skyrl_entrypoint pid=294281) model_dtype: bfloat16
(skyrl_entrypoint pid=294281) timeout_multiplier: 1.0
(skyrl_entrypoint pid=294281) run_engines_locally: true
(skyrl_entrypoint pid=294281) num_inference_engines: 48
(skyrl_entrypoint pid=294281) backend: vllm
(skyrl_entrypoint pid=294281) weight_sync_backend: nccl
(skyrl_entrypoint pid=294281) fuse_weights: false
(skyrl_entrypoint pid=294281) weight_transfer_threshold_cuda_ipc_GB: 1.0
(skyrl_entrypoint pid=294281) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_expert_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_data_parallel_size: 1
(skyrl_entrypoint pid=294281) n_samples_per_prompt: 8
(skyrl_entrypoint pid=294281) async_engine: true
(skyrl_entrypoint pid=294281) batched: false
(skyrl_entrypoint pid=294281) max_input_length: ${trainer.max_prompt_length}
(skyrl_entrypoint pid=294281) vllm_v1_disable_multiproc: true
(skyrl_entrypoint pid=294281) enable_prefix_caching: true
(skyrl_entrypoint pid=294281) enable_chunked_prefill: true
(skyrl_entrypoint pid=294281) max_num_batched_tokens: 65536
(skyrl_entrypoint pid=294281) enforce_eager: true
(skyrl_entrypoint pid=294281) fully_sharded_loras: false
(skyrl_entrypoint pid=294281) enable_ray_prometheus_stats: false
(skyrl_entrypoint pid=294281) vllm_stats_interval: 1
(skyrl_entrypoint pid=294281) gpu_memory_utilization: 0.75
(skyrl_entrypoint pid=294281) max_num_seqs: 24
(skyrl_entrypoint pid=294281) remote_inference_engine_urls:
(skyrl_entrypoint pid=294281) - 127.0.0.1:8001
(skyrl_entrypoint pid=294281) enable_http_endpoint: true
(skyrl_entrypoint pid=294281) http_endpoint_host: 127.0.0.1
(skyrl_entrypoint pid=294281) http_endpoint_port: 8000
(skyrl_entrypoint pid=294281) max_turns: 999999
(skyrl_entrypoint pid=294281) chat_template:
(skyrl_entrypoint pid=294281) source: name
(skyrl_entrypoint pid=294281) name_or_path: null
(skyrl_entrypoint pid=294281) chat_template_kwargs: {}
(skyrl_entrypoint pid=294281) engine_init_kwargs:
(skyrl_entrypoint pid=294281) max_model_len: 32768
(skyrl_entrypoint pid=294281) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=294281) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=294281) override_existing_update_group: disable
(skyrl_entrypoint pid=294281) sampling_params:
(skyrl_entrypoint pid=294281) max_generate_length: 4096
(skyrl_entrypoint pid=294281) repetition_penalty: 1.0
(skyrl_entrypoint pid=294281) temperature: 0.7
(skyrl_entrypoint pid=294281) top_p: 0.95
(skyrl_entrypoint pid=294281) min_p: 0.0
(skyrl_entrypoint pid=294281) top_k: 20
(skyrl_entrypoint pid=294281) logprobs: null
(skyrl_entrypoint pid=294281) stop: null
(skyrl_entrypoint pid=294281) use_conversation_multi_turn: true
(skyrl_entrypoint pid=294281) append_eos_token_after_stop_str_in_multi_turn: true
(skyrl_entrypoint pid=294281) eval_sampling_params:
(skyrl_entrypoint pid=294281) max_generate_length: ${generator.sampling_params.max_generate_length}
(skyrl_entrypoint pid=294281) repetition_penalty: 1.0
(skyrl_entrypoint pid=294281) temperature: 0.0
(skyrl_entrypoint pid=294281) top_p: 1.0
(skyrl_entrypoint pid=294281) min_p: 0.0
(skyrl_entrypoint pid=294281) top_k: -1
(skyrl_entrypoint pid=294281) logprobs: null
(skyrl_entrypoint pid=294281) stop: null
(skyrl_entrypoint pid=294281) eval_n_samples_per_prompt: 8
(skyrl_entrypoint pid=294281) zero_reward_on_non_stop: false
(skyrl_entrypoint pid=294281) apply_overlong_filtering: false
(skyrl_entrypoint pid=294281) rope_scaling: ${trainer.rope_scaling}
(skyrl_entrypoint pid=294281) rope_theta: ${trainer.rope_theta}
(skyrl_entrypoint pid=294281) teacher:
(skyrl_entrypoint pid=294281) model_path: null
(skyrl_entrypoint pid=294281) top_k_logprobs: 256
(skyrl_entrypoint pid=294281) num_inference_engines: 1
(skyrl_entrypoint pid=294281) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=294281) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=294281) gpu_memory_utilization: 0.9
(skyrl_entrypoint pid=294281) enforce_eager: false
(skyrl_entrypoint pid=294281) backend: vllm
(skyrl_entrypoint pid=294281) engine_init_kwargs: {}
(skyrl_entrypoint pid=294281) environment:
(skyrl_entrypoint pid=294281) env_class: gsm8k
(skyrl_entrypoint pid=294281) skyrl_gym:
(skyrl_entrypoint pid=294281) max_env_workers: 32
(skyrl_entrypoint pid=294281) text2sql:
(skyrl_entrypoint pid=294281) db_path: /home/ray/default/sql_data
(skyrl_entrypoint pid=294281) llm_as_a_judge:
(skyrl_entrypoint pid=294281) model: gpt-4o-mini
(skyrl_entrypoint pid=294281) base_url: null
(skyrl_entrypoint pid=294281) search:
(skyrl_entrypoint pid=294281) log_requests: false
(skyrl_entrypoint pid=294281) search_url: http://127.0.0.1:8000/retrieve
(skyrl_entrypoint pid=294281) topk: 3
(skyrl_entrypoint pid=294281) timeout: 30
(skyrl_entrypoint pid=294281) rollout:
(skyrl_entrypoint pid=294281) fanout:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) num_coordinators: 4
(skyrl_entrypoint pid=294281) cpus_per_coordinator: 8
(skyrl_entrypoint pid=294281) deepspeed_config:
(skyrl_entrypoint pid=294281) train:
(skyrl_entrypoint pid=294281) zero_optimization:
(skyrl_entrypoint pid=294281) stage: 3
(skyrl_entrypoint pid=294281) offload_param:
(skyrl_entrypoint pid=294281) device: none
(skyrl_entrypoint pid=294281) offload_optimizer:
(skyrl_entrypoint pid=294281) device: none
(skyrl_entrypoint pid=294281) pin_memory: true
(skyrl_entrypoint pid=294281) sub_group_size: auto
(skyrl_entrypoint pid=294281) reduce_bucket_size: auto
(skyrl_entrypoint pid=294281) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=294281) stage3_prefetch_bucket_size: auto
(skyrl_entrypoint pid=294281) stage3_max_live_parameters: auto
(skyrl_entrypoint pid=294281) stage3_max_reuse_distance: auto
(skyrl_entrypoint pid=294281) round_robin_gradients: true
(skyrl_entrypoint pid=294281) zero_hpz_partition_size: 1
(skyrl_entrypoint pid=294281) zero_quantized_weights: false
(skyrl_entrypoint pid=294281) zero_quantized_gradients: false
(skyrl_entrypoint pid=294281) torch_autocast:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) dtype: bfloat16
(skyrl_entrypoint pid=294281) disable_trace_cache: false
(skyrl_entrypoint pid=294281) data_types:
(skyrl_entrypoint pid=294281) grad_accum_dtype: fp32
(skyrl_entrypoint pid=294281) gradient_clipping: 1.0
(skyrl_entrypoint pid=294281) wall_clock_breakdown: false
(skyrl_entrypoint pid=294281) prescale_gradient: false
(skyrl_entrypoint pid=294281) eval:
(skyrl_entrypoint pid=294281) zero_optimization:
(skyrl_entrypoint pid=294281) stage: 3
(skyrl_entrypoint pid=294281) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=294281) offload_param:
(skyrl_entrypoint pid=294281) device: cpu
(skyrl_entrypoint pid=294281) pin_memory: true
(skyrl_entrypoint pid=294281) torch_autocast:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) dtype: bfloat16
(skyrl_entrypoint pid=294281) gradient_clipping: 1.0
(skyrl_entrypoint pid=294281) prescale_gradient: false
(skyrl_entrypoint pid=294281) wall_clock_breakdown: false
(skyrl_entrypoint pid=294281) terminal_bench_config:
(skyrl_entrypoint pid=294281) trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs
(skyrl_entrypoint pid=294281) harbor:
(skyrl_entrypoint pid=294281) name: terminus-2
(skyrl_entrypoint pid=294281) max_episodes: 999999
(skyrl_entrypoint pid=294281) enable_summarize: false
(skyrl_entrypoint pid=294281) store_all_messages: true
(skyrl_entrypoint pid=294281) trajectory_config:
(skyrl_entrypoint pid=294281) raw_content: true
(skyrl_entrypoint pid=294281) enable_episode_logging: false
(skyrl_entrypoint pid=294281) record_terminal_session: false
(skyrl_entrypoint pid=294281) enable_pane_logging: false
(skyrl_entrypoint pid=294281) strict_json_parser: true
(skyrl_entrypoint pid=294281) interleaved_thinking: true
(skyrl_entrypoint pid=294281) extra_body:
(skyrl_entrypoint pid=294281) chat_template_kwargs:
(skyrl_entrypoint pid=294281) enable_thinking: true
(skyrl_entrypoint pid=294281) override_timeout_sec: 900
(skyrl_entrypoint pid=294281) override_cpus: 1
(skyrl_entrypoint pid=294281) override_memory_mb: 2048
(skyrl_entrypoint pid=294281) override_storage_mb: 2048
(skyrl_entrypoint pid=294281) auto_snapshot: true
(skyrl_entrypoint pid=294281) verifier_override_timeout_sec: 120
(skyrl_entrypoint pid=294281) max_retries: 3
(skyrl_entrypoint pid=294281) min_wait_sec: 60.0
(skyrl_entrypoint pid=294281) max_wait_sec: 600.0
(skyrl_entrypoint pid=294281) wait_multiplier: 2.0
(skyrl_entrypoint pid=294281) exclude_exceptions:
(skyrl_entrypoint pid=294281) - VerifierTimeoutError
(skyrl_entrypoint pid=294281) - VerifierRuntimeError
(skyrl_entrypoint pid=294281) - RewardFileNotFoundError
(skyrl_entrypoint pid=294281) - RewardFileEmptyError
(skyrl_entrypoint pid=294281) - VerifierOutputParseError
(skyrl_entrypoint pid=294281) n_concurrent_trials: 675
(skyrl_entrypoint pid=294281) log_level: INFO
(skyrl_entrypoint pid=294281) enable_reward_shaping: false
(skyrl_entrypoint pid=294281) enable_error_classification: true
(skyrl_entrypoint pid=294281) mask_exceptions:
(skyrl_entrypoint pid=294281) - DaytonaError
(skyrl_entrypoint pid=294281) - EnvironmentStartTimeoutError
(skyrl_entrypoint pid=294281) - NetworkError
(skyrl_entrypoint pid=294281) - ConnectionError
(skyrl_entrypoint pid=294281) - RewardFileNotFoundError
(skyrl_entrypoint pid=294281) - RewardFileEmptyError
(skyrl_entrypoint pid=294281) - AgentEnvironmentTimeoutError
(skyrl_entrypoint pid=294281) - ContextLengthExceededError
(skyrl_entrypoint pid=294281) default_error_treatment: zero
(skyrl_entrypoint pid=294281) passthrough_exceptions:
(skyrl_entrypoint pid=294281) - AgentTimeoutError
(skyrl_entrypoint pid=294281) zero_exceptions: []
(skyrl_entrypoint pid=294281) model_info:
(skyrl_entrypoint pid=294281) max_input_tokens: 32000
(skyrl_entrypoint pid=294281) max_output_tokens: 4096
(skyrl_entrypoint pid=294281) archiving:
(skyrl_entrypoint pid=294281) enabled: false
(skyrl_entrypoint pid=294281) trace_upload:
(skyrl_entrypoint pid=294281) enabled: true
(skyrl_entrypoint pid=294281) repo_org: DCAgent
(skyrl_entrypoint pid=294281) episodes: last
(skyrl_entrypoint pid=294281) dataset_type: SFT
(skyrl_entrypoint pid=294281) cleanup: true
(skyrl_entrypoint pid=294281) 
(skyrl_entrypoint pid=294281) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(skyrl_entrypoint pid=294281) No module named 'vllm._version'
(skyrl_entrypoint pid=294281) from .version import __version__, __version_tuple__ # isort:skip
(skyrl_entrypoint pid=294281) W0607 02:21:22.569000 294281 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
[2026-06-07 02:21:23,669 E 293833 294255] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RegistryActor pid=987007, ip=10.128.32.37) [2026-06-07 02:21:24,551 E 987007 987047] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:27,013 E 294281 294324] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster]
(pid=987079, ip=10.128.32.37) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=987079, ip=10.128.32.37) No module named 'vllm._version'
(pid=987079, ip=10.128.32.37) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster]
(pid=2596339, ip=10.128.32.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(pid=2596339, ip=10.128.32.36) No module named 'vllm._version' [repeated 7x across cluster]
(pid=2596339, ip=10.128.32.36) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 111x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:40.638 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:40.638 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:40.639 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.259 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.284 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.284 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.285 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.285 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) 2026-06-07 02:21:41.305 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.18s (attempt 1/5) to avoid port collisions
(pid=2704166, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 13x across cluster]
(pid=2704166, ip=10.128.32.34) No module named 'vllm._version' [repeated 13x across cluster]
(pid=2704166, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip [repeated 13x across cluster]
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 69x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:45.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:45.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:45.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:21:46.027 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:21:46.028 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:21:46.028 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.972 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:46.976 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:21:47.003 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.81s (attempt 1/5) to avoid port collisions
(pid=2958820, ip=10.128.32.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 10x across cluster]
(pid=2958820, ip=10.128.32.39) No module named 'vllm._version' [repeated 10x across cluster]
(pid=2958820, ip=10.128.32.39) from .version import __version__, __version_tuple__ # isort:skip [repeated 10x across cluster]
(raylet, ip=10.128.32.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 63x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:50] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:50] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:50.121 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:50.122 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:50.122 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 17x across cluster]
(skyrl_entrypoint pid=294281) [2026-06-07 02:21:51] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.391 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:969 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.391 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines.
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.394 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.395 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.408 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 156
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:357 - Length of train_dataloader: 5000
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:358 - Number of steps per epoch: 78
(skyrl_entrypoint pid=294281) 2026-06-07 02:21:51.409 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:359 - Total training steps: 80
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.407 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.427 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) 2026-06-07 02:21:51.452 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.18s (attempt 1/5) to avoid port collisions [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) No module named 'vllm._version' [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) from .version import __version__, __version_tuple__ # isort:skip [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 120x across cluster]
(AsyncVLLMInferenceEngine pid=3079062, ip=10.128.32.42) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 32x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 17x across cluster]
(pid=2662729, ip=10.128.32.43) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) 2026-06-07 02:21:55.378 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) 2026-06-07 02:21:55.380 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) 2026-06-07 02:21:55.380 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) [W607 02:21:56.447607181 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-37.jupiter.internal]:41139 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) [W607 02:21:56.449335629 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) [rank0]:[W607 02:21:56.451880032 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.650 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.652 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:21:56.667 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.60s (attempt 1/5) to avoid port collisions [repeated 9x across cluster]
(bundle_reservation_check_func pid=294359) [2026-06-07 02:21:57,860 E 294359 294399] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) No module named 'vllm._version' [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:21:59 INFO [ipv4-debug] hostname=jpbo-041-43.jupiter.internal
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:21:59 INFO [ipv4-debug] _global_node.node_ip_address=10.128.32.43
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:21:59 INFO [ipv4-debug] get_node_ip_address()=10.128.32.43
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:00.059 | INFO  | skyrl_train.workers.worker:_initiate_actors:563 - Initializing process group for RayActorGroup
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) (EngineCore_DP0 pid=987460) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2541995, ip=10.128.32.40) (EngineCore_DP0 pid=2542222) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:21:58.938 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:21:58.939 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:21:58.939 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) [W607 02:22:01.433309818 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-37.jupiter.internal]:50831 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) [W607 02:22:01.433831729 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) [rank0]:[W607 02:22:01.436403587 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:02<00:08, 2.86s/it]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.236 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.261 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) 2026-06-07 02:22:00.294 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.55s (attempt 1/5) to avoid port collisions [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234) _C._set_float32_matmul_precision(precision)
(pid=2662807, ip=10.128.32.43) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) [2026-06-07 02:22:01,958 E 2541854 2541963] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2541994, ip=10.128.32.40) (EngineCore_DP0 pid=2542240) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) No module named 'vllm._version' [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 72x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 26x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) (EngineCore_DP0 pid=3079301) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2596211, ip=10.128.32.36) (EngineCore_DP0 pid=2596590) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:01.690 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:01.691 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:01.691 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) [W607 02:22:06.608280396 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-36.jupiter.internal]:36675 (errno: 97 - Address family not supported by protocol). [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) [W607 02:22:06.608742019 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) [rank0]:[W607 02:22:06.611067956 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:07<00:02, 2.64s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.951 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.975 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.975 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.975 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:02.976 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:03.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.80s (attempt 1/5) to avoid port collisions [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3078934, ip=10.128.32.42) (EngineCore_DP0 pid=3079317) _C._set_float32_matmul_precision(precision)
(pid=294462) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) [2026-06-07 02:22:07,570 E 2669552 2669716] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=3079062, ip=10.128.32.42) (EngineCore_DP0 pid=3079296) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418)
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452)
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) (EngineCore_DP0 pid=987460)
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447)
(AsyncVLLMInferenceEngine pid=2669550, ip=10.128.32.38) (EngineCore_DP0 pid=2669785) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) No module named 'vllm._version' [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 8x across cluster]
(FSDPPolicyWorkerBase pid=294462) [W607 02:22:11.696602899 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-43-interconnect-1.jupiter.internal]:36595 (errno: 97 - Address family not supported by protocol). [repeated 16x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [W607 02:22:11.005700994 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) [rank0]:[W607 02:22:10.881961700 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:04<00:04, 2.41s/it] [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2796151, ip=10.128.32.45) (EngineCore_DP0 pid=2796398) _C._set_float32_matmul_precision(precision)
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:12.293 | INFO  | skyrl_train.workers.worker:_initiate_actors:565 - Initialized process group for RayActorGroup
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:12.304 | INFO  | skyrl_train.workers.worker:_initiate_actors:567 - Mesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1)]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) `torch_dtype` is deprecated! Use `dtype` instead!
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) (EngineCore_DP0 pid=2704543) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704295, ip=10.128.32.34) (EngineCore_DP0 pid=2704527) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704296, ip=10.128.32.34) (EngineCore_DP0 pid=2704525) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=294465)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
(FSDPPolicyWorkerBase pid=2662807, ip=10.128.32.43)
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:00, 9.50it/s]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) (EngineCore_DP0 pid=2626901) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) 2026-06-07 02:22:12.897 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) [2026-06-07 02:22:12,169 E 2796016 2796119] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 10x across cluster]
(FSDPPolicyWorkerBase pid=294465)
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 8.17it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 10.83it/s]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43)
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 10.75it/s]
(AsyncVLLMInferenceEngine pid=2958818, ip=10.128.32.39) (EngineCore_DP0 pid=2959050) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) (EngineCore_DP0 pid=2626885) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:13.430 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) (EngineCore_DP0 pid=2626889) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2958819, ip=10.128.32.39) (EngineCore_DP0 pid=2959046) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.790 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.816 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.816 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.817 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.817 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:14.850 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.65s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2975429, ip=10.128.32.44) (EngineCore_DP0 pid=2975795) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2704295, ip=10.128.32.34) (EngineCore_DP0 pid=2704527)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) Process EngineCore_DP0:
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.run()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 108, in run
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self._target(*self._args, **self._kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1010, in run_engine_core
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) raise e
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) super().__init__(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self._init_executor()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) init_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) return TCPStore(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 44571, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40006f4ac700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #1: <unknown function> + 0x5f29220 (0x40004f649220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #2: <unknown function> + 0x5f4326c (0x40004f66326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #3: <unknown function> + 0x5f49074 (0x40004f669074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #4: <unknown function> + 0x5f49138 (0x40004f669138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #5: <unknown function> + 0x5f2ccc4 (0x40004f64ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x40004f651564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #7: <unknown function> + 0x109a094 (0x4000493fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #8: <unknown function> + 0x113236c (0x40004949236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #9: <unknown function> + 0x5d6d60 (0x400048936d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #10: <unknown function> + 0x1b7a38 (0xaaaab8697a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #12: <unknown function> + 0x169f50 (0xaaaab8649f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #13: <unknown function> + 0x1682e4 (0xaaaab86482e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #14: <unknown function> + 0x1e0ce8 (0xaaaab86c0ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #15: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #16: <unknown function> + 0x646b0c (0x4000489a6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #19: <unknown function> + 0x1808c0 (0xaaaab86608c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #20: <unknown function> + 0x182bf8 (0xaaaab8662bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #21: <unknown function> + 0x25fd30 (0xaaaab873fd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #22: <unknown function> + 0x1b7d20 (0xaaaab8697d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #23: PyObject_Vectorcall + 0x54 (0xaaaab86460e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaab8647f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #27: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #28: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaab8647fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #33: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #34: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #35: _PyObject_Call + 0x68 (0xaaaab8648488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaab874d8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #37: PyEval_EvalCode + 0xb4 (0xaaaab8752eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #38: <unknown function> + 0x2ccdcc (0xaaaab87acdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #39: <unknown function> + 0x2ccef4 (0xaaaab87acef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #40: PyRun_StringFlags + 0x90 (0xaaaab87b1050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaab87b10f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #42: Py_RunMain + 0x2c8 (0xaaaab87d9190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #43: Py_BytesMain + 0x64 (0xaaaab87d9fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #44: <unknown function> + 0x27540 (0x4000379a7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #45: __libc_start_main + 0x98 (0x4000379a7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) frame #46: <unknown function> + 0x10e0c0 (0xaaaab85ee0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) [pynccl] dumped 0 entries to /e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace_pynccl_pid2959070 (reason=atexit)
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [W607 02:22:16.342198091 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-44.jupiter.internal]:41207 (errno: 97 - Address family not supported by protocol). [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [W607 02:22:16.342706915 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) [rank0]:[W607 02:22:16.344851394 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=2958691, ip=10.128.32.39, actor_id=9f591a871969d2b79a9e310202000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400c1389d1f0>)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) return self.__get_result()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) raise self._exception
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) super().__init__(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) self.llm = self._create_engine(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) return cls(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) self.engine_core = EngineCoreClient.make_async_mp_client(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) return AsyncMPClient(*client_args)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) super().__init__(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) next(self.gen)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) wait_for_engine_startup(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) raise RuntimeError(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(AsyncVLLMInferenceEngine pid=2796151, ip=10.128.32.45) (EngineCore_DP0 pid=2796398)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:03<00:01, 1.26s/it] [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) [pynccl] dumped 0 entries to /e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace_pynccl_pid2958691 (reason=atexit)
(FSDPPolicyWorkerBase pid=294462) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2796151, ip=10.128.32.45) (EngineCore_DP0 pid=2796398)
(FSDPPolicyWorkerBase pid=294463)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s] [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=2662808, ip=10.128.32.43)
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 7.89it/s] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2578178, ip=10.128.32.41) (EngineCore_DP0 pid=2578410) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) 2026-06-07 02:22:12.972 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [2026-06-07 02:22:17,355 E 863950 864052] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 11x across cluster]
(FSDPPolicyWorkerBase pid=294462)
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:00<00:00, 14.05it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 18.02it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391)
(FSDPPolicyWorkerBase pid=2662808, ip=10.128.32.43)
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 10.58it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:13.430 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:13.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230)
(AsyncVLLMInferenceEngine pid=2796016, ip=10.128.32.45) (EngineCore_DP0 pid=2796354) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2541994, ip=10.128.32.40) (EngineCore_DP0 pid=2542240)
(AsyncVLLMInferenceEngine pid=2541995, ip=10.128.32.40) (EngineCore_DP0 pid=2542222)
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.788 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.816 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) 2026-06-07 02:22:14.850 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.60s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2596211, ip=10.128.32.36) (EngineCore_DP0 pid=2596590)
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579)
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570)
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) [W607 02:22:20.901124860 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-41-interconnect-1.jupiter.internal]:43419 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) [W607 02:22:20.901611923 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) [rank0]:[W607 02:22:20.903638798 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.04s/it] [repeated 60x across cluster]
(AsyncVLLMInferenceEngine pid=3078934, ip=10.128.32.42) (EngineCore_DP0 pid=3079317)
(AsyncVLLMInferenceEngine pid=3079062, ip=10.128.32.42) (EngineCore_DP0 pid=3079296)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292)
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) (EngineCore_DP0 pid=3079301)
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) 2026-06-07 02:22:23.033 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 5x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [2026-06-07 02:22:22,373 E 2662729 2662769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802)
(AsyncVLLMInferenceEngine pid=2669550, ip=10.128.32.38) (EngineCore_DP0 pid=2669785)
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786)
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784)
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 4x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:26.990 | INFO  | skyrl_train.trainer:build_models:831 - init policy/ref/critic models done
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:26.998 | INFO  | skyrl_train.fully_async_trainer:_maybe_enable_rollout_fanout:456 - Rollout fan-out ENABLED: replacing single-process generator with RolloutDispatcher (K=4, cpus_per_coordinator=8).
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:27.021 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:387 - [RolloutDispatcher] fan-out path: overriding inference host 127.0.0.1 -> 10.128.32.33 (routable head IP) for coordinator litellm base_url connectivity
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:27.021 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:405 - [RolloutDispatcher] configured num_coordinators=4, cpus_per_coordinator=8
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:27.059 | INFO  | examples.terminal_bench.rollout_coordinator:startup:434 - [RolloutDispatcher] PlacementGroup ready: 4 bundles x 8 CPU (SPREAD)
(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(FSDPPolicyWorkerBase pid=294462) [rank7]:[W607 02:22:24.789186688 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:11<00:03, 3.91s/it] [repeated 46x across cluster]
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054)
(AsyncVLLMInferenceEngine pid=2958818, ip=10.128.32.39) (EngineCore_DP0 pid=2959050)
(AsyncVLLMInferenceEngine pid=2958819, ip=10.128.32.39) (EngineCore_DP0 pid=2959046)
(AsyncVLLMInferenceEngine pid=2704166, ip=10.128.32.34) (EngineCore_DP0 pid=2704543)
(AsyncVLLMInferenceEngine pid=2704295, ip=10.128.32.34) (EngineCore_DP0 pid=2704527)
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539)
(AsyncVLLMInferenceEngine pid=2704296, ip=10.128.32.34) (EngineCore_DP0 pid=2704525)
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) (EngineCore_DP0 pid=2626901)
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) 2026-06-07 02:22:27.290 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) (EngineCore_DP0 pid=2626889)
(AsyncVLLMInferenceEngine pid=2626656, ip=10.128.32.46) (EngineCore_DP0 pid=2626885)
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905)
(AsyncVLLMInferenceEngine pid=2975429, ip=10.128.32.44) (EngineCore_DP0 pid=2975795)
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800)
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804)
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787)
(FSDPPolicyWorkerBase pid=2662807, ip=10.128.32.43) [2026-06-07 02:22:30,830 E 2662807 2662947] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.049 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.261 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 0/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.265 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.265 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2669976, ip=10.128.32.38) 2026-06-07 02:22:32.265 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 0] startup complete
(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:32.266 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 1/4 started
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.28s/it] [repeated 38x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) 2026-06-07 02:22:32.541 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406)
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420)
(AsyncVLLMInferenceEngine pid=2578178, ip=10.128.32.41) (EngineCore_DP0 pid=2578410)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416)
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [W607 02:22:35.858670330 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-35.jupiter.internal]:33029 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [W607 02:22:35.862483909 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) [rank0]:[W607 02:22:35.868173189 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 16x across cluster]
(FSDPPolicyWorkerBase pid=294462) [2026-06-07 02:22:31,188 E 294462 294693] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:12<00:00, 3.21s/it] [repeated 12x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.718 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) 2026-06-07 02:22:38.742 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:39.941 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 2/4 started
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.937 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 1/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.940 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.940 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:22:39.940 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 1] startup complete
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [W607 02:22:35.208951015 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-35-interconnect-1.jupiter.internal]:40633 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [W607 02:22:35.209434236 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) [rank0]:[W607 02:22:35.211722249 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 5x across cluster]
(raylet, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.13s/it] [repeated 5x across cluster]
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.037 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.257 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.257 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.258 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.258 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 2/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.260 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.261 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2596821, ip=10.128.32.36) 2026-06-07 02:22:48.261 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 2] startup complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:48.262 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 3/4 started
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312)
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.36s/it] [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324)
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313)
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) 2026-06-07 02:22:54.394 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.126 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.36s/it] [repeated 6x across cluster]
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 4x across cluster]
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 4/4 started
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | examples.terminal_bench.rollout_coordinator:startup:477 - [RolloutDispatcher] 4 coordinators started
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | skyrl_train.fully_async_trainer:train:485 - Generator startup complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.354 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Started: 'load_checkpoints'
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.363 | INFO  | skyrl_train.trainer:load_checkpoints:1664 - Loading checkpoint from: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.363 | INFO  | skyrl_train.trainer:load_checkpoints:1670 - Resuming from global_step: 28
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.383 | INFO  | skyrl_train.trainer:load_checkpoints:1686 - Successfully loaded trainer state
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.349 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.350 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.350 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.350 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 3/4] constructed (http=10.128.32.33:8000)
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.353 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.353 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:22:56.353 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 3] startup complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.392 | INFO  | skyrl_train.trainer:load_checkpoints:1696 - Successfully loaded dataloader state
(skyrl_entrypoint pid=294281) 2026-06-07 02:22:56.392 | INFO  | skyrl_train.trainer:load_checkpoints:1705 - Loading policy checkpoint from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [2026-06-07 02:22:57,810 E 2669976 2670068] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RolloutCoordinator pid=987693, ip=10.128.32.37) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) 2026-06-07 02:22:54.477 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [2026-06-07 02:23:05,084 E 2542476 2542516] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [2026-06-07 02:23:12,865 E 2596821 2596861] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) warnings.warn( # warn only once
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.648 | INFO  | skyrl_train.trainer:load_checkpoints:1715 - Successfully loaded policy checkpoint
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.648 | INFO  | skyrl_train.trainer:load_checkpoints:1731 - Successfully loaded complete checkpoint state from global_step_28
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.648 | INFO  | skyrl_train.fully_async_trainer:_train_loop:512 - Resumed training from global_step 28
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.663 | INFO  | skyrl_train.utils.data_tracker:load_state:97 - Loaded data tracker state: epoch=0, consumed_in_epoch=1792, total_consumed=1792
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.664 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Finished: 'load_checkpoints', time cost: 19.31s
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:15.664 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Started: 'init_weight_sync_state'
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) No module named 'vllm._version'
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:23:19.911 | INFO  | logging:info:2216 - [weight-sync] Using master_addr=10.128.32.43, master_port=46453
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank0]:[W607 02:23:19.180220203 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-43.jupiter.internal]:46453 (errno: 97 - Address family not supported by protocol).
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:20.003 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Finished: 'init_weight_sync_state', time cost: 4.34s
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:20.003 | ERROR  | skyrl_train.fully_async_trainer:train:493 - Train loop failed at global_step 28: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
(skyrl_entrypoint pid=294281) Traceback (most recent call last):
(skyrl_entrypoint pid=294281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
(skyrl_entrypoint pid=294281) await self._train_loop()
(skyrl_entrypoint pid=294281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
(skyrl_entrypoint pid=294281) self.init_weight_sync_state()
(skyrl_entrypoint pid=294281) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
(skyrl_entrypoint pid=294281) raise RuntimeError(
(skyrl_entrypoint pid=294281) RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) warnings.warn("No model update group to destroy")
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) 2026-06-07 02:23:19.910 | INFO  | logging:info:2216 - [weight-sync] get_node_ip_address()=10.128.32.43 [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=294462) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=294462) No module named 'vllm._version' [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=294462) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:363 - Shutting down shared QueueOrchestrator...
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:365 - QueueOrchestrator shutdown complete
(RolloutCoordinator pid=2542476, ip=10.128.32.40) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.rollout_coordinator:shutdown:258 - [RolloutCoordinator 1] shutdown complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:21.029 | INFO  | skyrl_train.inference_engines.inference_engine_client_http_endpoint:shutdown_server:203 - Server shut down after 2 seconds
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:21.029 | INFO  | skyrl_train.trainer:_guarded_sync:226 - HTTP endpoint shutdown complete
(skyrl_entrypoint pid=294281) 2026-06-07 02:23:21.033 | INFO  | skyrl_train.trainer:_guarded_async:215 - Generator shutdown complete
2026-06-07 02:23:21.090 | ERROR | __main__:main:134 - Training failed: ray::skyrl_entrypoint() (pid=294281, ip=10.128.32.33)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 483, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
raise RuntimeError(
RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
2026-06-07 02:23:21.091 | INFO | __main__:main:137 - Shutting down Ray on head node...
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=2958820, ip=10.128.32.39) (EngineCore_DP0 pid=2959054) warnings.warn("No model update group to destroy") [repeated 27x across cluster]
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:363 - Shutting down shared QueueOrchestrator... [repeated 3x across cluster]
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:365 - QueueOrchestrator shutdown complete [repeated 3x across cluster]
(RolloutCoordinator pid=987693, ip=10.128.32.37) 2026-06-07 02:23:21.030 | INFO | examples.terminal_bench.rollout_coordinator:shutdown:258 - [RolloutCoordinator 3] shutdown complete [repeated 3x across cluster]
(skyrl_entrypoint pid=294281) [fd-monitor] Started monitoring (every 120s)
(skyrl_entrypoint pid=294281) [fd-monitor] [02:21:02] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=294281) [fd-monitor] [02:21:02] OK: RSS 1.38 GiB | node mem 129.1/858.0 GiB used (15.0%), avail 728.9 GiB
(skyrl_entrypoint pid=294281) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=294281) INFO 06-07 02:21:22 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(pid=987079, ip=10.128.32.37) INFO 06-07 02:21:39 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:43 [arg_utils.py:1256] The global random seed is set to 42. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:43 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) INFO 06-07 02:21:43 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:44 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) WARNING 06-07 02:21:44 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(pid=987212, ip=10.128.32.37) INFO 06-07 02:21:44 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) WARNING 06-07 02:21:48 [arg_utils.py:1256] The global random seed is set to 56. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:48 [arg_utils.py:1256] The global random seed is set to 55. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [model.py:1549] Using max model len 32768 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:48 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) INFO 06-07 02:21:48 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) WARNING 06-07 02:21:48 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) WARNING 06-07 02:21:48 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:21:49 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:48 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:51 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=42, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [arg_utils.py:1256] The global random seed is set to 61. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [model.py:1549] Using max model len 32768 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) INFO 06-07 02:21:53 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=3079064, ip=10.128.32.42) WARNING 06-07 02:21:53 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 16x across cluster]
(pid=2626785, ip=10.128.32.46) INFO 06-07 02:21:55 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:56 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:56 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.37:41139 backend=nccl
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:56 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:21:57 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=56, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:57 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:21:58 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) WARNING 06-07 02:21:58 [arg_utils.py:1256] The global random seed is set to 69. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [model.py:1549] Using max model len 32768 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) WARNING 06-07 02:21:58 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626657, ip=10.128.32.46) INFO 06-07 02:21:58 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) WARNING 06-07 02:21:59 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2626529, ip=10.128.32.46) WARNING 06-07 02:21:59 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster]
(pid=2662729, ip=10.128.32.43) ⚙️ Running in WANDB offline mode
(pid=2578180, ip=10.128.32.41) INFO 06-07 02:22:00 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 26x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) INFO 06-07 02:22:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:01 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.37:50831 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:01 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2669550, ip=10.128.32.38) (EngineCore_DP0 pid=2669785) INFO 06-07 02:22:01 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=51, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:02 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:02 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:02 [arg_utils.py:1256] The global random seed is set to 84. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [model.py:1549] Using max model len 32768 [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:02 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) INFO 06-07 02:22:02 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:03 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) WARNING 06-07 02:22:03 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) INFO 06-07 02:22:05 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=3079063, ip=10.128.32.42) (EngineCore_DP0 pid=3079292) INFO 06-07 02:22:06 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566) INFO 06-07 02:22:06 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.36:37763 backend=nccl [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2596339, ip=10.128.32.36) (EngineCore_DP0 pid=2596566) INFO 06-07 02:22:06 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539) INFO 06-07 02:22:05 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=64, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570) INFO 06-07 02:22:07 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 9x across cluster]
(pid=2662809, ip=10.128.32.43) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:08 [default_loader.py:293] Loading weights took 9.29 seconds
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) (EngineCore_DP0 pid=987452) INFO 06-07 02:22:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 6.143796 seconds
(AsyncVLLMInferenceEngine pid=2596340, ip=10.128.32.36) (EngineCore_DP0 pid=2596570) INFO 06-07 02:22:08 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:05 [arg_utils.py:1256] The global random seed is set to 89. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [model.py:1549] Using max model len 32768 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:06 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) INFO 06-07 02:22:06 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:06 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) WARNING 06-07 02:22:06 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975429, ip=10.128.32.44) INFO 06-07 02:22:10 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:10 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:11 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:11 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:11 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:10 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.45:48431 backend=nccl [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:10 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:12 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) INFO 06-07 02:22:12 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.50 seconds
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:12 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=84, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:11 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=987079, ip=10.128.32.37) (EngineCore_DP0 pid=987418) WARNING 06-07 02:22:12 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=987212, ip=10.128.32.37) WARNING 06-07 02:22:12 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(pid=294462) ⚙️ Running in WANDB offline mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:08 [default_loader.py:293] Loading weights took 6.05 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 6.835432 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:13 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) WARNING 06-07 02:22:12 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] EngineCore failed to start.
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] super().__init__(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self._init_executor()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] init_distributed_environment(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] return TCPStore(
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 44571, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x40006f4ac700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #1: <unknown function> + 0x5f29220 (0x40004f649220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #2: <unknown function> + 0x5f4326c (0x40004f66326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #3: <unknown function> + 0x5f49074 (0x40004f669074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #4: <unknown function> + 0x5f49138 (0x40004f669138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #5: <unknown function> + 0x5f2ccc4 (0x40004f64ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x40004f651564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #7: <unknown function> + 0x109a094 (0x4000493fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #8: <unknown function> + 0x113236c (0x40004949236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #9: <unknown function> + 0x5d6d60 (0x400048936d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #10: <unknown function> + 0x1b7a38 (0xaaaab8697a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #12: <unknown function> + 0x169f50 (0xaaaab8649f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #13: <unknown function> + 0x1682e4 (0xaaaab86482e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #14: <unknown function> + 0x1e0ce8 (0xaaaab86c0ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #15: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #16: <unknown function> + 0x646b0c (0x4000489a6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #19: <unknown function> + 0x1808c0 (0xaaaab86608c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #20: <unknown function> + 0x182bf8 (0xaaaab8662bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #21: <unknown function> + 0x25fd30 (0xaaaab873fd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #22: <unknown function> + 0x1b7d20 (0xaaaab8697d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #23: PyObject_Vectorcall + 0x54 (0xaaaab86460e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaab8647f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #27: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #28: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaab8645db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaab874ae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaab8647fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaab86481ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #33: <unknown function> + 0x1e0d80 (0xaaaab86c0d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #34: <unknown function> + 0x1d7ddc (0xaaaab86b7ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #35: _PyObject_Call + 0x68 (0xaaaab8648488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaab874d8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #37: PyEval_EvalCode + 0xb4 (0xaaaab8752eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #38: <unknown function> + 0x2ccdcc (0xaaaab87acdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #39: <unknown function> + 0x2ccef4 (0xaaaab87acef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #40: PyRun_StringFlags + 0x90 (0xaaaab87b1050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaab87b10f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #42: Py_RunMain + 0x2c8 (0xaaaab87d9190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #43: Py_BytesMain + 0x64 (0xaaaab87d9fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #44: <unknown function> + 0x27540 (0x4000379a7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #45: __libc_start_main + 0x98 (0x4000379a7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006] frame #46: <unknown function> + 0x10e0c0 (0xaaaab85ee0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2958691, ip=10.128.32.39) (EngineCore_DP0 pid=2959070) ERROR 06-07 02:22:15 [core.py:1006]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) INFO 06-07 02:22:15 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) WARNING 06-07 02:22:16 [arg_utils.py:1256] The global random seed is set to 77. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:16 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) WARNING 06-07 02:22:16 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:11 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:11 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:11 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.44:41207 backend=nccl [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) WARNING 06-07 02:22:16 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) INFO 06-07 02:22:12 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.49 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406) INFO 06-07 02:22:16 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=86, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:16 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 17x across cluster]
(AsyncVLLMInferenceEngine pid=987213, ip=10.128.32.37) (EngineCore_DP0 pid=987447) WARNING 06-07 02:22:12 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=987214, ip=10.128.32.37) WARNING 06-07 02:22:12 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:18 [default_loader.py:293] Loading weights took 4.18 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 4.518817 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975565, ip=10.128.32.44) (EngineCore_DP0 pid=2975804) INFO 06-07 02:22:17 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:19 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) INFO 06-07 02:22:19 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2796150, ip=10.128.32.45) (EngineCore_DP0 pid=2796384) WARNING 06-07 02:22:19 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) INFO 06-07 02:22:15 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:17 [arg_utils.py:1256] The global random seed is set to 76. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:17 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:17 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:20 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2796149, ip=10.128.32.45) (EngineCore_DP0 pid=2796391) INFO 06-07 02:22:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:20 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.41:43419 backend=nccl [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:20 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:17 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) INFO 06-07 02:22:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) (EngineCore_DP0 pid=2542230) INFO 06-07 02:22:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:21 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2542123, ip=10.128.32.40) (EngineCore_DP0 pid=2542234) WARNING 06-07 02:22:22 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2541854, ip=10.128.32.40) WARNING 06-07 02:22:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:23 [default_loader.py:293] Loading weights took 13.40 seconds [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669422, ip=10.128.32.38) (EngineCore_DP0 pid=2669802) INFO 06-07 02:22:23 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.574883 seconds [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:21 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) NCCL version 2.27.7+cuda13.0
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) INFO 06-07 02:22:25 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) INFO 06-07 02:22:25 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2596338, ip=10.128.32.36) (EngineCore_DP0 pid=2596579) WARNING 06-07 02:22:25 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) (EngineCore_DP0 pid=2669786) INFO 06-07 02:22:26 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:26 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) WARNING 06-07 02:22:27 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) INFO 06-07 02:22:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2669551, ip=10.128.32.38) WARNING 06-07 02:22:27 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:28 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=76, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:28 [default_loader.py:293] Loading weights took 13.02 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2704294, ip=10.128.32.34) (EngineCore_DP0 pid=2704539) INFO 06-07 02:22:28 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.076531 seconds [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:27 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) INFO 06-07 02:22:27 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2669552, ip=10.128.32.38) (EngineCore_DP0 pid=2669784) WARNING 06-07 02:22:27 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 11x across cluster]
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [fd-monitor] [02:22:32] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2669976, ip=10.128.32.38) [fd-monitor] [02:22:32] OK: RSS 0.78 GiB | node mem 385.3/858.0 GiB used (44.9%), avail 472.7 GiB
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) INFO 06-07 02:22:31 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) (EngineCore_DP0 pid=2626905) WARNING 06-07 02:22:32 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) INFO 06-07 02:22:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2626785, ip=10.128.32.46) WARNING 06-07 02:22:32 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:28 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=74, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) INFO 06-07 02:22:30 [default_loader.py:293] Loading weights took 13.13 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) INFO 06-07 02:22:30 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.203295 seconds [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:34 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.35:33029 backend=nccl
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) INFO 06-07 02:22:34 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) INFO 06-07 02:22:34 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) WARNING 06-07 02:22:34 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:35 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) INFO 06-07 02:22:37 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) INFO 06-07 02:22:37 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2578179, ip=10.128.32.41) (EngineCore_DP0 pid=2578420) INFO 06-07 02:22:37 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2975564, ip=10.128.32.44) (EngineCore_DP0 pid=2975800) INFO 06-07 02:22:33 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) (EngineCore_DP0 pid=2975787) INFO 06-07 02:22:33 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578050, ip=10.128.32.41) (EngineCore_DP0 pid=2578406) WARNING 06-07 02:22:38 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2975563, ip=10.128.32.44) WARNING 06-07 02:22:34 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 4x across cluster]
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [fd-monitor] [02:22:39] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2542476, ip=10.128.32.40) [fd-monitor] [02:22:39] OK: RSS 0.78 GiB | node mem 406.2/858.0 GiB used (47.3%), avail 451.7 GiB
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:34 [default_loader.py:293] Loading weights took 12.85 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:35 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 13.617507 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.35:40633 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) WARNING 06-07 02:22:38 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:35 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:36 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:37 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:37 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:37 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) INFO 06-07 02:22:38 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) (EngineCore_DP0 pid=2578416) WARNING 06-07 02:22:38 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2578180, ip=10.128.32.41) WARNING 06-07 02:22:38 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 4x across cluster]
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [fd-monitor] [02:22:48] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2596821, ip=10.128.32.36) [fd-monitor] [02:22:48] OK: RSS 0.78 GiB | node mem 394.8/858.0 GiB used (46.0%), avail 463.2 GiB
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:50 [default_loader.py:293] Loading weights took 13.45 seconds
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) (EngineCore_DP0 pid=864312) INFO 06-07 02:22:50 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.577902 seconds
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:53 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:53 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:53 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:53 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.72 seconds
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) WARNING 06-07 02:22:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:54 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) WARNING 06-07 02:22:54 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=864077, ip=10.128.32.35) (EngineCore_DP0 pid=864319) INFO 06-07 02:22:54 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=863950, ip=10.128.32.35) WARNING 06-07 02:22:54 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(RolloutCoordinator pid=987693, ip=10.128.32.37) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=987693, ip=10.128.32.37) [fd-monitor] [02:22:56] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=987693, ip=10.128.32.37) [fd-monitor] [02:22:56] OK: RSS 0.78 GiB | node mem 390.4/858.0 GiB used (45.5%), avail 467.6 GiB
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:50 [default_loader.py:293] Loading weights took 13.45 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:50 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.715511 seconds [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/model_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/extra_state_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/optim_world_size_8_rank_0.pt
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) (EngineCore_DP0 pid=864324) INFO 06-07 02:22:53 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) WARNING 06-07 02:22:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:54 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) WARNING 06-07 02:22:54 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864079, ip=10.128.32.35) (EngineCore_DP0 pid=864313) INFO 06-07 02:22:54 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=864078, ip=10.128.32.35) WARNING 06-07 02:22:54 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(skyrl_entrypoint pid=294281) [fd-monitor] [02:23:02] OK: 102 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=294281) [fd-monitor] [02:23:02] OK: RSS 1.64 GiB | node mem 233.1/858.0 GiB used (27.2%), avail 624.9 GiB
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Successfully loaded model state dict
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Successfully loaded optimizer state
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Successfully loaded scheduler state
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy
(FSDPPolicyWorkerBase pid=2662729, ip=10.128.32.43) INFO 06-07 02:23:18 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(FSDPPolicyWorkerBase pid=294462) INFO 06-07 02:23:19 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 7x across cluster]
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=sequence_mean', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqmean-arm0', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::skyrl_entrypoint() (pid=294281, ip=10.128.32.33)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 483, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
raise RuntimeError(
RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
Stopping Ray cluster...
Ray cluster stopped
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
Collecting Ray logs from worker jpbo-041-34...
Collecting Ray logs from worker jpbo-041-35...
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
[RLJobRunner] Crash-time Ray log preservation complete.
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
episodes: last
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
Collecting Ray logs from worker jpbo-041-34...
Collecting Ray logs from worker jpbo-041-35...
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink