laion/ablation-pymethods2test-seqmean-arm0-30-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0_630129.out
ModelHub XC 0c293fc953 初始化项目，由ModelHub XC社区提供模型 
Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B
Source: Original Platform
2026-07-18 18:02:02 +08:00

2055 lines
411 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_630129
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630129
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630129
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.32.33 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630129.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630129.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.32.33:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-041-33 (10.128.32.33)
Ray port: 6379
============================
Starting Ray head on jpbo-041-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-33.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-041-33
Starting Ray worker on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-34.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-041-34
Starting Ray worker on jpbo-041-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-35.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-041-35
Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-041-36
Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-041-37
Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-041-38
Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-041-39
Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-041-40
Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-041-41
Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-041-42
Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-041-43
Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-041-44
Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-041-45
Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-041-46
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.32.33:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'accelerator_type:GH200': 12.0, 'GPU': 48.0, 'CPU': 3456.0, 'memory': 9095679574016.0, 'object_store_memory': 515396075520.0, 'node:10.128.32.35': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.41': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.39': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'object_store_memory': 558345748480.0, 'memory': 9872149315584.0, 'CPU': 3744.0, 'node:10.128.32.35': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.41': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.39': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'object_store_memory': 558345748480.0, 'memory': 9872149315584.0, 'CPU': 3744.0, 'node:10.128.32.35': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.41': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.39': 1.0}
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'accelerator_type:GH200': 13.0, 'GPU': 52.0, 'CPU': 3744.0, 'memory': 9872149315584.0, 'object_store_memory': 558345748480.0, 'node:10.128.32.35': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.41': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.39': 1.0}
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'accelerator_type:GH200': 14.0, 'GPU': 56.0, 'object_store_memory': 601295421440.0, 'memory': 10634851450880.0, 'node:10.128.32.41': 1.0, 'CPU': 4032.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.35': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.37': 1.0, 'node:10.128.32.39': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.38': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
Address: 10.128.32.33:6379
Total GPUs: 56
=========================
Ray cluster ready at 10.128.32.33:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
Running SkyRL:
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
Args: 120 Hydra arguments
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=sequence_mean trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0 ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqmean-arm0 trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630129.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 16:02:29.538 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:753 - Exporting wandb api key to ray runtime env
2026-06-07 16:02:29.538 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:772 - Exporting RAY_ADDRESS to ray runtime env
2026-06-07 16:02:29.538 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
2026-06-07 16:02:29.538 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
2026-06-07 16:02:29.538 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
2026-06-07 16:02:29,539 INFO worker.py:1680 -- Using address 10.128.32.33:6379 set in the environment variable RAY_ADDRESS
2026-06-07 16:02:29,574 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.33:6379...
2026-06-07 16:02:29,585 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.32.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 16:02:32.338 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
(skyrl_entrypoint pid=391889) 2026-06-07 16:02:41.646 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:212 - SkyRL log level set to: INFO
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(skyrl_entrypoint pid=391889) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=391889) 2026-06-07 16:02:42.007 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
[2026-06-07 16:02:59,789 E 391407 391863] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RegistryActor pid=2613231, ip=10.128.32.41) [2026-06-07 16:03:00,820 E 2613231 2613271] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(skyrl_entrypoint pid=391889) 2026-06-07 16:03:39.460 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 5000 valid task directories out of 5000 total directories
(skyrl_entrypoint pid=391889) 2026-06-07 16:03:39.461 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 5000 task paths
(skyrl_entrypoint pid=391889) 2026-06-07 16:03:39.473 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:405 - data:
(skyrl_entrypoint pid=391889) train_data:
(skyrl_entrypoint pid=391889) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=391889) val_data: []
(skyrl_entrypoint pid=391889) trainer:
(skyrl_entrypoint pid=391889) placement:
(skyrl_entrypoint pid=391889) colocate_all: false
(skyrl_entrypoint pid=391889) colocate_policy_ref: true
(skyrl_entrypoint pid=391889) policy_num_nodes: 2
(skyrl_entrypoint pid=391889) policy_num_gpus_per_node: 4
(skyrl_entrypoint pid=391889) critic_num_nodes: 1
(skyrl_entrypoint pid=391889) critic_num_gpus_per_node: 4
(skyrl_entrypoint pid=391889) ref_num_nodes: 2
(skyrl_entrypoint pid=391889) ref_num_gpus_per_node: 4
(skyrl_entrypoint pid=391889) policy_strict_spread_pg: false
(skyrl_entrypoint pid=391889) policy_per_gpu_bundles: false
(skyrl_entrypoint pid=391889) policy_force_cvd_mask: false
(skyrl_entrypoint pid=391889) sequence_parallel_backend: ulysses
(skyrl_entrypoint pid=391889) strategy: fsdp2
(skyrl_entrypoint pid=391889) policy:
(skyrl_entrypoint pid=391889) model:
(skyrl_entrypoint pid=391889) path: /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=391889) lora:
(skyrl_entrypoint pid=391889) rank: 0
(skyrl_entrypoint pid=391889) alpha: 16
(skyrl_entrypoint pid=391889) dropout: 0
(skyrl_entrypoint pid=391889) lora_sync_path: /tmp/skyrl_lora_sync
(skyrl_entrypoint pid=391889) target_modules: all-linear
(skyrl_entrypoint pid=391889) exclude_modules: null
(skyrl_entrypoint pid=391889) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=391889) optimizer_config:
(skyrl_entrypoint pid=391889) optimizer: AdamW
(skyrl_entrypoint pid=391889) lr: 8.0e-06
(skyrl_entrypoint pid=391889) adam_betas:
(skyrl_entrypoint pid=391889) - 0.9
(skyrl_entrypoint pid=391889) - 0.999
(skyrl_entrypoint pid=391889) weight_decay: 0.0
(skyrl_entrypoint pid=391889) max_grad_norm: 0.9
(skyrl_entrypoint pid=391889) offload_after_step: true
(skyrl_entrypoint pid=391889) num_warmup_steps: 0
(skyrl_entrypoint pid=391889) scheduler: constant_with_warmup
(skyrl_entrypoint pid=391889) optimizer_kwargs: {}
(skyrl_entrypoint pid=391889) fsdp_config:
(skyrl_entrypoint pid=391889) cpu_offload: false
(skyrl_entrypoint pid=391889) reshard_after_forward: true
(skyrl_entrypoint pid=391889) fsdp_size: 4
(skyrl_entrypoint pid=391889) expert_model_parallel_size: 1
(skyrl_entrypoint pid=391889) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=391889) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=391889) moe_router_replay: false
(skyrl_entrypoint pid=391889) moe_grouped_gemm: false
(skyrl_entrypoint pid=391889) ep_comm_backend: torch
(skyrl_entrypoint pid=391889) deepep_num_sms: 20
(skyrl_entrypoint pid=391889) deepep_token_chunk_size: null
(skyrl_entrypoint pid=391889) sequence_parallel_size: 1
(skyrl_entrypoint pid=391889) use_torch_compile: false
(skyrl_entrypoint pid=391889) record_memory: false
(skyrl_entrypoint pid=391889) megatron_config:
(skyrl_entrypoint pid=391889) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=391889) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=391889) context_parallel_size: 1
(skyrl_entrypoint pid=391889) expert_model_parallel_size: 1
(skyrl_entrypoint pid=391889) expert_tensor_parallel_size: null
(skyrl_entrypoint pid=391889) ddp_config:
(skyrl_entrypoint pid=391889) grad_reduce_in_fp32: true
(skyrl_entrypoint pid=391889) overlap_grad_reduce: false
(skyrl_entrypoint pid=391889) overlap_param_gather: false
(skyrl_entrypoint pid=391889) average_in_collective: true
(skyrl_entrypoint pid=391889) model_config_kwargs: {}
(skyrl_entrypoint pid=391889) torch_profiler_config:
(skyrl_entrypoint pid=391889) enable: false
(skyrl_entrypoint pid=391889) ranks: []
(skyrl_entrypoint pid=391889) save_path: null
(skyrl_entrypoint pid=391889) optimizer_config_kwargs:
(skyrl_entrypoint pid=391889) overlap_cpu_optimizer_d2h_h2d: false
(skyrl_entrypoint pid=391889) use_precision_aware_optimizer: false
(skyrl_entrypoint pid=391889) optimizer_cpu_offload: false
(skyrl_entrypoint pid=391889) optimizer_offload_fraction: 0.0
(skyrl_entrypoint pid=391889) transformer_config_kwargs:
(skyrl_entrypoint pid=391889) recompute_granularity: full
(skyrl_entrypoint pid=391889) recompute_modules:
(skyrl_entrypoint pid=391889) - core_attn
(skyrl_entrypoint pid=391889) recompute_method: uniform
(skyrl_entrypoint pid=391889) recompute_num_layers: 1
(skyrl_entrypoint pid=391889) empty_cuda_cache: true
(skyrl_entrypoint pid=391889) ref:
(skyrl_entrypoint pid=391889) model:
(skyrl_entrypoint pid=391889) path: ${trainer.policy.model.path}
(skyrl_entrypoint pid=391889) sequence_parallel_size: 1
(skyrl_entrypoint pid=391889) deepspeed_config: ${deepspeed_config.eval}
(skyrl_entrypoint pid=391889) fsdp_config:
(skyrl_entrypoint pid=391889) cpu_offload: false
(skyrl_entrypoint pid=391889) reshard_after_forward: true
(skyrl_entrypoint pid=391889) fsdp_size: 4
(skyrl_entrypoint pid=391889) expert_model_parallel_size: 1
(skyrl_entrypoint pid=391889) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=391889) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=391889) moe_router_replay: false
(skyrl_entrypoint pid=391889) moe_grouped_gemm: false
(skyrl_entrypoint pid=391889) ep_comm_backend: torch
(skyrl_entrypoint pid=391889) deepep_num_sms: 20
(skyrl_entrypoint pid=391889) deepep_token_chunk_size: null
(skyrl_entrypoint pid=391889) megatron_config:
(skyrl_entrypoint pid=391889) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=391889) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=391889) context_parallel_size: 1
(skyrl_entrypoint pid=391889) expert_model_parallel_size: 1
(skyrl_entrypoint pid=391889) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=391889) model_config_kwargs: {}
(skyrl_entrypoint pid=391889) transformer_config_kwargs: {}
(skyrl_entrypoint pid=391889) critic:
(skyrl_entrypoint pid=391889) model:
(skyrl_entrypoint pid=391889) path: null
(skyrl_entrypoint pid=391889) lora:
(skyrl_entrypoint pid=391889) rank: 0
(skyrl_entrypoint pid=391889) alpha: 16
(skyrl_entrypoint pid=391889) dropout: 0
(skyrl_entrypoint pid=391889) target_modules: all-linear
(skyrl_entrypoint pid=391889) exclude_modules: null
(skyrl_entrypoint pid=391889) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=391889) optimizer_config:
(skyrl_entrypoint pid=391889) optimizer: AdamW
(skyrl_entrypoint pid=391889) lr: 5.0e-06
(skyrl_entrypoint pid=391889) adam_betas:
(skyrl_entrypoint pid=391889) - 0.9
(skyrl_entrypoint pid=391889) - 0.999
(skyrl_entrypoint pid=391889) weight_decay: 0.01
(skyrl_entrypoint pid=391889) max_grad_norm: 1.0
(skyrl_entrypoint pid=391889) offload_after_step: true
(skyrl_entrypoint pid=391889) num_warmup_steps: 0
(skyrl_entrypoint pid=391889) scheduler: constant_with_warmup
(skyrl_entrypoint pid=391889) optimizer_kwargs: {}
(skyrl_entrypoint pid=391889) fsdp_config:
(skyrl_entrypoint pid=391889) cpu_offload: false
(skyrl_entrypoint pid=391889) reshard_after_forward: true
(skyrl_entrypoint pid=391889) fsdp_size: -1
(skyrl_entrypoint pid=391889) expert_model_parallel_size: 1
(skyrl_entrypoint pid=391889) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=391889) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=391889) moe_router_replay: false
(skyrl_entrypoint pid=391889) moe_grouped_gemm: false
(skyrl_entrypoint pid=391889) ep_comm_backend: torch
(skyrl_entrypoint pid=391889) deepep_num_sms: 20
(skyrl_entrypoint pid=391889) deepep_token_chunk_size: null
(skyrl_entrypoint pid=391889) sequence_parallel_size: 1
(skyrl_entrypoint pid=391889) algorithm:
(skyrl_entrypoint pid=391889) advantage_estimator: rloo_n
(skyrl_entrypoint pid=391889) kl_ctrl:
(skyrl_entrypoint pid=391889) type: fixed
(skyrl_entrypoint pid=391889) kl_target: 0.1
(skyrl_entrypoint pid=391889) horizon: 10000
(skyrl_entrypoint pid=391889) kl_estimator_type: k3
(skyrl_entrypoint pid=391889) use_kl_estimator_k3: false
(skyrl_entrypoint pid=391889) use_abs_kl: false
(skyrl_entrypoint pid=391889) use_kl_in_reward: false
(skyrl_entrypoint pid=391889) use_kl_loss: false
(skyrl_entrypoint pid=391889) kl_loss_coef: 0.0
(skyrl_entrypoint pid=391889) use_entropy_loss: false
(skyrl_entrypoint pid=391889) entropy_loss_coef: 0.01
(skyrl_entrypoint pid=391889) advantage_batch_normalize: false
(skyrl_entrypoint pid=391889) value_head_prefix: value_head
(skyrl_entrypoint pid=391889) policy_loss_type: regular
(skyrl_entrypoint pid=391889) loss_reduction: sequence_mean
(skyrl_entrypoint pid=391889) global_loss_denom: null
(skyrl_entrypoint pid=391889) grpo_norm_by_std: true
(skyrl_entrypoint pid=391889) rloo_n_min_group_size: 4
(skyrl_entrypoint pid=391889) rloo_n_filter_zero_reward_groups: true
(skyrl_entrypoint pid=391889) lambd: 1.0
(skyrl_entrypoint pid=391889) gamma: 1.0
(skyrl_entrypoint pid=391889) eps_clip_low: 0.2
(skyrl_entrypoint pid=391889) eps_clip_high: 0.05
(skyrl_entrypoint pid=391889) clip_ratio_c: 3.0
(skyrl_entrypoint pid=391889) tis_imp_ratio_cap: -1.0
(skyrl_entrypoint pid=391889) use_tis: false
(skyrl_entrypoint pid=391889) sapo:
(skyrl_entrypoint pid=391889) tau_pos: 1.0
(skyrl_entrypoint pid=391889) tau_neg: 1.05
(skyrl_entrypoint pid=391889) value_clip: 0.2
(skyrl_entrypoint pid=391889) dynamic_sampling:
(skyrl_entrypoint pid=391889) type: null
(skyrl_entrypoint pid=391889) max_sample_batches: 30
(skyrl_entrypoint pid=391889) min_replace_ratio: 0.3
(skyrl_entrypoint pid=391889) clip_cov:
(skyrl_entrypoint pid=391889) clip_ratio: 0.0002
(skyrl_entrypoint pid=391889) clip_cov_lb: 1.0
(skyrl_entrypoint pid=391889) clip_cov_ub: 5.0
(skyrl_entrypoint pid=391889) kl_cov:
(skyrl_entrypoint pid=391889) kl_cov_frac: 0.2
(skyrl_entrypoint pid=391889) ppo_kl_coef: 1.0
(skyrl_entrypoint pid=391889) cispo:
(skyrl_entrypoint pid=391889) cispo_eps_clip_low: 0
(skyrl_entrypoint pid=391889) cispo_eps_clip_high: 5
(skyrl_entrypoint pid=391889) z_clip:
(skyrl_entrypoint pid=391889) enabled: false
(skyrl_entrypoint pid=391889) alpha: 0.97
(skyrl_entrypoint pid=391889) z_thresh: 2.5
(skyrl_entrypoint pid=391889) warmup_steps: 25
(skyrl_entrypoint pid=391889) mode: zscore
(skyrl_entrypoint pid=391889) clip_option: adaptive_scaling
(skyrl_entrypoint pid=391889) clip_factor: 1.0
(skyrl_entrypoint pid=391889) skip_update_on_spike: false
(skyrl_entrypoint pid=391889) stale_clip:
(skyrl_entrypoint pid=391889) enabled: false
(skyrl_entrypoint pid=391889) alpha: 0.3
(skyrl_entrypoint pid=391889) entropy_threshold: 0.15
(skyrl_entrypoint pid=391889) entropy_window: 10
(skyrl_entrypoint pid=391889) min_lr_scale: 0.1
(skyrl_entrypoint pid=391889) max_seq_len: 1004095
(skyrl_entrypoint pid=391889) fully_async:
(skyrl_entrypoint pid=391889) max_staleness_steps: 16
(skyrl_entrypoint pid=391889) num_parallel_generation_workers: 338
(skyrl_entrypoint pid=391889) gradient_checkpointing: true
(skyrl_entrypoint pid=391889) gradient_checkpointing_use_reentrant: false
(skyrl_entrypoint pid=391889) seed: 42
(skyrl_entrypoint pid=391889) resume_mode: latest
(skyrl_entrypoint pid=391889) resume_path: null
(skyrl_entrypoint pid=391889) ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints
(skyrl_entrypoint pid=391889) max_ckpts_to_keep: -1
(skyrl_entrypoint pid=391889) ckpt_interval: 2
(skyrl_entrypoint pid=391889) hf_save_interval: 5
(skyrl_entrypoint pid=391889) hf_upload_mode: latest
(skyrl_entrypoint pid=391889) export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports
(skyrl_entrypoint pid=391889) bf16: true
(skyrl_entrypoint pid=391889) epochs: 2
(skyrl_entrypoint pid=391889) max_steps: 80
(skyrl_entrypoint pid=391889) update_epochs_per_batch: 1
(skyrl_entrypoint pid=391889) train_batch_size: 64
(skyrl_entrypoint pid=391889) policy_mini_batch_size: 64
(skyrl_entrypoint pid=391889) critic_mini_batch_size: 256
(skyrl_entrypoint pid=391889) micro_train_batch_size_per_gpu: 1
(skyrl_entrypoint pid=391889) micro_forward_batch_size_per_gpu: 4
(skyrl_entrypoint pid=391889) update_ref_every_epoch: false
(skyrl_entrypoint pid=391889) use_sample_packing: true
(skyrl_entrypoint pid=391889) eval_batch_size: 64
(skyrl_entrypoint pid=391889) eval_before_train: false
(skyrl_entrypoint pid=391889) eval_interval: 999999
(skyrl_entrypoint pid=391889) max_prompt_length: 999999
(skyrl_entrypoint pid=391889) flash_attn: true
(skyrl_entrypoint pid=391889) disable_fast_tokenizer: false
(skyrl_entrypoint pid=391889) target_modules: null
(skyrl_entrypoint pid=391889) exclude_modules: null
(skyrl_entrypoint pid=391889) project_name: OpenThoughts-Agent
(skyrl_entrypoint pid=391889) run_name: ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=391889) logger: console
(skyrl_entrypoint pid=391889) tracker_commit_each_step: true
(skyrl_entrypoint pid=391889) dump_data_batch: false
(skyrl_entrypoint pid=391889) dump_eval_results: true
(skyrl_entrypoint pid=391889) log_level: INFO
(skyrl_entrypoint pid=391889) rope_scaling: null
(skyrl_entrypoint pid=391889) rope_theta: null
(skyrl_entrypoint pid=391889) step_wise_training: false
(skyrl_entrypoint pid=391889) hf_hub_repo_id: laion/ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=391889) hf_hub_private: false
(skyrl_entrypoint pid=391889) hf_hub_revision: main
(skyrl_entrypoint pid=391889) enable_db_registration: false
(skyrl_entrypoint pid=391889) generator:
(skyrl_entrypoint pid=391889) model_name: ${trainer.policy.model.path}
(skyrl_entrypoint pid=391889) model_dtype: bfloat16
(skyrl_entrypoint pid=391889) timeout_multiplier: 1.0
(skyrl_entrypoint pid=391889) run_engines_locally: true
(skyrl_entrypoint pid=391889) num_inference_engines: 48
(skyrl_entrypoint pid=391889) backend: vllm
(skyrl_entrypoint pid=391889) weight_sync_backend: nccl
(skyrl_entrypoint pid=391889) fuse_weights: false
(skyrl_entrypoint pid=391889) weight_transfer_threshold_cuda_ipc_GB: 1.0
(skyrl_entrypoint pid=391889) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=391889) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=391889) inference_engine_expert_parallel_size: 1
(skyrl_entrypoint pid=391889) inference_engine_data_parallel_size: 1
(skyrl_entrypoint pid=391889) n_samples_per_prompt: 8
(skyrl_entrypoint pid=391889) async_engine: true
(skyrl_entrypoint pid=391889) batched: false
(skyrl_entrypoint pid=391889) max_input_length: ${trainer.max_prompt_length}
(skyrl_entrypoint pid=391889) vllm_v1_disable_multiproc: true
(skyrl_entrypoint pid=391889) enable_prefix_caching: true
(skyrl_entrypoint pid=391889) enable_chunked_prefill: true
(skyrl_entrypoint pid=391889) max_num_batched_tokens: 65536
(skyrl_entrypoint pid=391889) enforce_eager: true
(skyrl_entrypoint pid=391889) fully_sharded_loras: false
(skyrl_entrypoint pid=391889) enable_ray_prometheus_stats: false
(skyrl_entrypoint pid=391889) vllm_stats_interval: 1
(skyrl_entrypoint pid=391889) gpu_memory_utilization: 0.75
(skyrl_entrypoint pid=391889) max_num_seqs: 24
(skyrl_entrypoint pid=391889) remote_inference_engine_urls:
(skyrl_entrypoint pid=391889) - 127.0.0.1:8001
(skyrl_entrypoint pid=391889) enable_http_endpoint: true
(skyrl_entrypoint pid=391889) http_endpoint_host: 127.0.0.1
(skyrl_entrypoint pid=391889) http_endpoint_port: 8000
(skyrl_entrypoint pid=391889) max_turns: 999999
(skyrl_entrypoint pid=391889) chat_template:
(skyrl_entrypoint pid=391889) source: name
(skyrl_entrypoint pid=391889) name_or_path: null
(skyrl_entrypoint pid=391889) chat_template_kwargs: {}
(skyrl_entrypoint pid=391889) engine_init_kwargs:
(skyrl_entrypoint pid=391889) max_model_len: 32768
(skyrl_entrypoint pid=391889) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=391889) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=391889) override_existing_update_group: disable
(skyrl_entrypoint pid=391889) sampling_params:
(skyrl_entrypoint pid=391889) max_generate_length: 4096
(skyrl_entrypoint pid=391889) repetition_penalty: 1.0
(skyrl_entrypoint pid=391889) temperature: 0.7
(skyrl_entrypoint pid=391889) top_p: 0.95
(skyrl_entrypoint pid=391889) min_p: 0.0
(skyrl_entrypoint pid=391889) top_k: 20
(skyrl_entrypoint pid=391889) logprobs: null
(skyrl_entrypoint pid=391889) stop: null
(skyrl_entrypoint pid=391889) use_conversation_multi_turn: true
(skyrl_entrypoint pid=391889) append_eos_token_after_stop_str_in_multi_turn: true
(skyrl_entrypoint pid=391889) eval_sampling_params:
(skyrl_entrypoint pid=391889) max_generate_length: ${generator.sampling_params.max_generate_length}
(skyrl_entrypoint pid=391889) repetition_penalty: 1.0
(skyrl_entrypoint pid=391889) temperature: 0.0
(skyrl_entrypoint pid=391889) top_p: 1.0
(skyrl_entrypoint pid=391889) min_p: 0.0
(skyrl_entrypoint pid=391889) top_k: -1
(skyrl_entrypoint pid=391889) logprobs: null
(skyrl_entrypoint pid=391889) stop: null
(skyrl_entrypoint pid=391889) eval_n_samples_per_prompt: 8
(skyrl_entrypoint pid=391889) zero_reward_on_non_stop: false
(skyrl_entrypoint pid=391889) apply_overlong_filtering: false
(skyrl_entrypoint pid=391889) rope_scaling: ${trainer.rope_scaling}
(skyrl_entrypoint pid=391889) rope_theta: ${trainer.rope_theta}
(skyrl_entrypoint pid=391889) teacher:
(skyrl_entrypoint pid=391889) model_path: null
(skyrl_entrypoint pid=391889) top_k_logprobs: 256
(skyrl_entrypoint pid=391889) num_inference_engines: 1
(skyrl_entrypoint pid=391889) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=391889) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=391889) gpu_memory_utilization: 0.9
(skyrl_entrypoint pid=391889) enforce_eager: false
(skyrl_entrypoint pid=391889) backend: vllm
(skyrl_entrypoint pid=391889) engine_init_kwargs: {}
(skyrl_entrypoint pid=391889) environment:
(skyrl_entrypoint pid=391889) env_class: gsm8k
(skyrl_entrypoint pid=391889) skyrl_gym:
(skyrl_entrypoint pid=391889) max_env_workers: 32
(skyrl_entrypoint pid=391889) text2sql:
(skyrl_entrypoint pid=391889) db_path: /home/ray/default/sql_data
(skyrl_entrypoint pid=391889) llm_as_a_judge:
(skyrl_entrypoint pid=391889) model: gpt-4o-mini
(skyrl_entrypoint pid=391889) base_url: null
(skyrl_entrypoint pid=391889) search:
(skyrl_entrypoint pid=391889) log_requests: false
(skyrl_entrypoint pid=391889) search_url: http://127.0.0.1:8000/retrieve
(skyrl_entrypoint pid=391889) topk: 3
(skyrl_entrypoint pid=391889) timeout: 30
(skyrl_entrypoint pid=391889) rollout:
(skyrl_entrypoint pid=391889) fanout:
(skyrl_entrypoint pid=391889) enabled: true
(skyrl_entrypoint pid=391889) num_coordinators: 4
(skyrl_entrypoint pid=391889) cpus_per_coordinator: 8
(skyrl_entrypoint pid=391889) deepspeed_config:
(skyrl_entrypoint pid=391889) train:
(skyrl_entrypoint pid=391889) zero_optimization:
(skyrl_entrypoint pid=391889) stage: 3
(skyrl_entrypoint pid=391889) offload_param:
(skyrl_entrypoint pid=391889) device: none
(skyrl_entrypoint pid=391889) offload_optimizer:
(skyrl_entrypoint pid=391889) device: none
(skyrl_entrypoint pid=391889) pin_memory: true
(skyrl_entrypoint pid=391889) sub_group_size: auto
(skyrl_entrypoint pid=391889) reduce_bucket_size: auto
(skyrl_entrypoint pid=391889) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=391889) stage3_prefetch_bucket_size: auto
(skyrl_entrypoint pid=391889) stage3_max_live_parameters: auto
(skyrl_entrypoint pid=391889) stage3_max_reuse_distance: auto
(skyrl_entrypoint pid=391889) round_robin_gradients: true
(skyrl_entrypoint pid=391889) zero_hpz_partition_size: 1
(skyrl_entrypoint pid=391889) zero_quantized_weights: false
(skyrl_entrypoint pid=391889) zero_quantized_gradients: false
(skyrl_entrypoint pid=391889) torch_autocast:
(skyrl_entrypoint pid=391889) enabled: true
(skyrl_entrypoint pid=391889) dtype: bfloat16
(skyrl_entrypoint pid=391889) disable_trace_cache: false
(skyrl_entrypoint pid=391889) data_types:
(skyrl_entrypoint pid=391889) grad_accum_dtype: fp32
(skyrl_entrypoint pid=391889) gradient_clipping: 1.0
(skyrl_entrypoint pid=391889) wall_clock_breakdown: false
(skyrl_entrypoint pid=391889) prescale_gradient: false
(skyrl_entrypoint pid=391889) eval:
(skyrl_entrypoint pid=391889) zero_optimization:
(skyrl_entrypoint pid=391889) stage: 3
(skyrl_entrypoint pid=391889) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=391889) offload_param:
(skyrl_entrypoint pid=391889) device: cpu
(skyrl_entrypoint pid=391889) pin_memory: true
(skyrl_entrypoint pid=391889) torch_autocast:
(skyrl_entrypoint pid=391889) enabled: true
(skyrl_entrypoint pid=391889) dtype: bfloat16
(skyrl_entrypoint pid=391889) gradient_clipping: 1.0
(skyrl_entrypoint pid=391889) prescale_gradient: false
(skyrl_entrypoint pid=391889) wall_clock_breakdown: false
(skyrl_entrypoint pid=391889) terminal_bench_config:
(skyrl_entrypoint pid=391889) trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs
(skyrl_entrypoint pid=391889) harbor:
(skyrl_entrypoint pid=391889) name: terminus-2
(skyrl_entrypoint pid=391889) max_episodes: 999999
(skyrl_entrypoint pid=391889) enable_summarize: false
(skyrl_entrypoint pid=391889) store_all_messages: true
(skyrl_entrypoint pid=391889) trajectory_config:
(skyrl_entrypoint pid=391889) raw_content: true
(skyrl_entrypoint pid=391889) enable_episode_logging: false
(skyrl_entrypoint pid=391889) record_terminal_session: false
(skyrl_entrypoint pid=391889) enable_pane_logging: false
(skyrl_entrypoint pid=391889) strict_json_parser: true
(skyrl_entrypoint pid=391889) interleaved_thinking: true
(skyrl_entrypoint pid=391889) extra_body:
(skyrl_entrypoint pid=391889) chat_template_kwargs:
(skyrl_entrypoint pid=391889) enable_thinking: true
(skyrl_entrypoint pid=391889) override_timeout_sec: 900
(skyrl_entrypoint pid=391889) override_cpus: 1
(skyrl_entrypoint pid=391889) override_memory_mb: 2048
(skyrl_entrypoint pid=391889) override_storage_mb: 2048
(skyrl_entrypoint pid=391889) auto_snapshot: true
(skyrl_entrypoint pid=391889) verifier_override_timeout_sec: 120
(skyrl_entrypoint pid=391889) max_retries: 3
(skyrl_entrypoint pid=391889) min_wait_sec: 60.0
(skyrl_entrypoint pid=391889) max_wait_sec: 600.0
(skyrl_entrypoint pid=391889) wait_multiplier: 2.0
(skyrl_entrypoint pid=391889) exclude_exceptions:
(skyrl_entrypoint pid=391889) - VerifierTimeoutError
(skyrl_entrypoint pid=391889) - VerifierRuntimeError
(skyrl_entrypoint pid=391889) - RewardFileNotFoundError
(skyrl_entrypoint pid=391889) - RewardFileEmptyError
(skyrl_entrypoint pid=391889) - VerifierOutputParseError
(skyrl_entrypoint pid=391889) n_concurrent_trials: 675
(skyrl_entrypoint pid=391889) log_level: INFO
(skyrl_entrypoint pid=391889) enable_reward_shaping: false
(skyrl_entrypoint pid=391889) enable_error_classification: true
(skyrl_entrypoint pid=391889) mask_exceptions:
(skyrl_entrypoint pid=391889) - DaytonaError
(skyrl_entrypoint pid=391889) - EnvironmentStartTimeoutError
(skyrl_entrypoint pid=391889) - NetworkError
(skyrl_entrypoint pid=391889) - ConnectionError
(skyrl_entrypoint pid=391889) - RewardFileNotFoundError
(skyrl_entrypoint pid=391889) - RewardFileEmptyError
(skyrl_entrypoint pid=391889) - AgentEnvironmentTimeoutError
(skyrl_entrypoint pid=391889) - ContextLengthExceededError
(skyrl_entrypoint pid=391889) default_error_treatment: zero
(skyrl_entrypoint pid=391889) passthrough_exceptions:
(skyrl_entrypoint pid=391889) - AgentTimeoutError
(skyrl_entrypoint pid=391889) zero_exceptions: []
(skyrl_entrypoint pid=391889) model_info:
(skyrl_entrypoint pid=391889) max_input_tokens: 32000
(skyrl_entrypoint pid=391889) max_output_tokens: 4096
(skyrl_entrypoint pid=391889) archiving:
(skyrl_entrypoint pid=391889) enabled: false
(skyrl_entrypoint pid=391889) trace_upload:
(skyrl_entrypoint pid=391889) enabled: true
(skyrl_entrypoint pid=391889) repo_org: DCAgent
(skyrl_entrypoint pid=391889) episodes: last
(skyrl_entrypoint pid=391889) dataset_type: SFT
(skyrl_entrypoint pid=391889) cleanup: true
(skyrl_entrypoint pid=391889) 
(skyrl_entrypoint pid=391889) [2026-06-07 16:03:03,420 E 391889 391932] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster]
(skyrl_entrypoint pid=391889) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(skyrl_entrypoint pid=391889) No module named 'vllm._version'
(skyrl_entrypoint pid=391889) from .version import __version__, __version_tuple__ # isort:skip
(skyrl_entrypoint pid=391889) W0607 16:06:28.744000 391889 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(get_all_env_variables pid=392160) [2026-06-07 16:07:18,837 E 392160 392200] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 29x across cluster]
(pid=3113603, ip=10.128.32.42) [2026-06-07 16:07:21,100 E 3113603 3113706] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 2x across cluster]
(pid=3113603, ip=10.128.32.42) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=3113603, ip=10.128.32.42) No module named 'vllm._version'
(pid=3113603, ip=10.128.32.42) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 10x across cluster]
(get_addr_port pid=1022558, ip=10.128.32.37) [2026-06-07 16:08:25,572 E 1022558 1022633] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(pid=1022559, ip=10.128.32.37) [2026-06-07 16:08:25,619 E 1022559 1022661] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(get_addr_port pid=899452, ip=10.128.32.35) [2026-06-07 16:13:12,407 E 899452 899534] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 23x across cluster]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:18.252 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:18.253 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:18.253 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=1022866, ip=10.128.32.37) [2026-06-07 16:13:12,454 E 1022866 1022968] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:18.863 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:19.203 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:19.203 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:19.203 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:19.203 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:16:19.249 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.96s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.32.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
(pid=2697284, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2697284, ip=10.128.32.43) No module named 'vllm._version'
(pid=2697284, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 81x across cluster]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) _C._set_float32_matmul_precision(precision)
(pid=3114114, ip=10.128.32.42) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 6x across cluster]
(pid=3114114, ip=10.128.32.42) No module named 'vllm._version' [repeated 6x across cluster]
(pid=3114114, ip=10.128.32.42) from .version import __version__, __version_tuple__ # isort:skip [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) [W607 16:18:19.012223938 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-42-interconnect-1.jupiter.internal]:48887 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) [W607 16:18:19.013894942 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) [rank0]:[W607 16:18:19.016313609 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:11, 3.70s/it]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:05<00:05, 2.63s/it]
(get_addr_port pid=2697285, ip=10.128.32.43) [2026-06-07 16:18:27,672 E 2697285 2697359] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:07<00:02, 2.29s/it]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00, 1.67s/it]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00, 2.05s/it]
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084)
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) 2026-06-07 16:18:32.160 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:37.079 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:37.080 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:37.080 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=3114114, ip=10.128.32.42) [2026-06-07 16:18:30,329 E 3114114 3114278] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:38.033 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:38.037 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:38.037 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:38.037 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:38.038 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:18:38.082 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.57s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:37.079 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:37.080 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:37.080 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:38.037 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:38.040 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:38.040 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:38.040 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:38.040 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:18:38.082 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.51s (attempt 1/5) to avoid port collisions [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) No module named 'vllm._version' [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) from .version import __version__, __version_tuple__ # isort:skip [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) [W607 16:18:52.506534225 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-42.jupiter.internal]:44549 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) [W607 16:18:52.507043782 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) [rank0]:[W607 16:18:52.509417811 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:01<00:03, 1.24s/it]
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) [W607 16:18:53.056472543 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-42.jupiter.internal]:51575 (errno: 97 - Address family not supported by protocol). [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) [W607 16:18:53.057000595 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) [rank0]:[W607 16:18:53.059114149 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382)
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390)
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394)
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.09s/it] [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) 2026-06-07 16:19:01.699 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) 2026-06-07 16:19:01.845 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 2x across cluster]
(pid=2739368, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2739368, ip=10.128.32.34) No module named 'vllm._version'
(pid=2739368, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip
(get_addr_port pid=2576534, ip=10.128.32.40) [2026-06-07 16:21:33,156 E 2576534 2576609] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 29x across cluster]
(pid=2739604, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=2739604, ip=10.128.32.34) No module named 'vllm._version' [repeated 3x across cluster]
(pid=2739604, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(pid=2739604, ip=10.128.32.34) [2026-06-07 16:21:33,411 E 2739604 2739772] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
(pid=2576535, ip=10.128.32.40) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2576535, ip=10.128.32.40) No module named 'vllm._version'
(pid=2576535, ip=10.128.32.40) from .version import __version__, __version_tuple__ # isort:skip
(get_addr_port pid=2631426, ip=10.128.32.36) [2026-06-07 16:25:01,686 E 2631426 2631501] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 27x across cluster]
(pid=2576838, ip=10.128.32.40) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=2576838, ip=10.128.32.40) No module named 'vllm._version' [repeated 3x across cluster]
(pid=2576838, ip=10.128.32.40) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(pid=2631427, ip=10.128.32.36) [2026-06-07 16:25:01,733 E 2631427 2631529] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=2631427, ip=10.128.32.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2631427, ip=10.128.32.36) No module named 'vllm._version'
(pid=2631427, ip=10.128.32.36) from .version import __version__, __version_tuple__ # isort:skip
(pid=2576838, ip=10.128.32.40) [2026-06-07 16:25:01,997 E 2576838 2577002] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(pid=899451, ip=10.128.32.35) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=899451, ip=10.128.32.35) No module named 'vllm._version'
(pid=899451, ip=10.128.32.35) from .version import __version__, __version_tuple__ # isort:skip
(pid=1022559, ip=10.128.32.37) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(pid=1022559, ip=10.128.32.37) No module named 'vllm._version' [repeated 4x across cluster]
(pid=1022559, ip=10.128.32.37) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(pid=1023166, ip=10.128.32.37) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=1023166, ip=10.128.32.37) No module named 'vllm._version' [repeated 3x across cluster]
(pid=1023166, ip=10.128.32.37) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:40.941 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:40.952 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:40.952 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(raylet, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 23x across cluster]
(pid=3011432, ip=10.128.32.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=3011432, ip=10.128.32.44) No module named 'vllm._version' [repeated 3x across cluster]
(pid=3011432, ip=10.128.32.44) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:42.189 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:42.404 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:42.404 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:42.404 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:42.404 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:28:42.452 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.34s (attempt 1/5) to avoid port collisions
(get_addr_port pid=2705576, ip=10.128.32.38) [2026-06-07 16:28:59,577 E 2705576 2705650] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:40.941 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:40.952 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:40.952 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:42.192 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:42.404 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:42.404 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:42.404 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:42.404 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:28:42.452 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.44s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(pid=3011432, ip=10.128.32.44) [2026-06-07 16:28:59,741 E 3011432 3011535] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(raylet, ip=10.128.32.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(get_addr_port pid=2614264, ip=10.128.32.41) [2026-06-07 16:31:12,398 E 2614264 2614340] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 28x across cluster]
(pid=2705803, ip=10.128.32.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(pid=2705803, ip=10.128.32.38) No module named 'vllm._version' [repeated 4x across cluster]
(pid=2705803, ip=10.128.32.38) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(pid=2705803, ip=10.128.32.38) [2026-06-07 16:31:12,700 E 2705803 2705968] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(raylet, ip=10.128.32.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
(pid=3011671, ip=10.128.32.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=3011671, ip=10.128.32.44) No module named 'vllm._version'
(pid=3011671, ip=10.128.32.44) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.32.39) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 53x across cluster]
(pid=2631820, ip=10.128.32.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=2631820, ip=10.128.32.36) No module named 'vllm._version' [repeated 3x across cluster]
(pid=2631820, ip=10.128.32.36) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(get_addr_port pid=2831549, ip=10.128.32.45) [2026-06-07 16:32:34,010 E 2831549 2831630] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.39) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 28x across cluster]
(pid=2831686, ip=10.128.32.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(pid=2831686, ip=10.128.32.45) No module named 'vllm._version' [repeated 4x across cluster]
(pid=2831686, ip=10.128.32.45) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(pid=2831684, ip=10.128.32.45) [2026-06-07 16:33:02,327 E 2831684 2831794] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:47.395 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:47.396 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:47.396 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=2994914, ip=10.128.32.39) [2026-06-07 16:33:02,705 E 2994914 2995017] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:48.608 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:48.637 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:48.637 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:48.637 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:48.637 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:37:48.667 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.38s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) from .version import __version__, __version_tuple__ # isort:skip
(skyrl_entrypoint pid=391889) [2026-06-07 16:38:35] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:47.395 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:47.396 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:47.396 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:48.609 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:48.636 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:48.637 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:48.637 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:48.637 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) 2026-06-07 16:37:48.668 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.17s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(skyrl_entrypoint pid=391889) [2026-06-07 16:38:35] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
(skyrl_entrypoint pid=391889) [2026-06-07 16:38:36] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.246 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:969 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.246 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines.
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.249 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.250 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.251 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.252 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 156
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.252 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:357 - Length of train_dataloader: 5000
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.252 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:358 - Number of steps per epoch: 78
(skyrl_entrypoint pid=391889) 2026-06-07 16:38:36.252 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:359 - Total training steps: 80
(pid=2995275, ip=10.128.32.39) [2026-06-07 16:39:06,386 E 2995275 2995386] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 25x across cluster]
(pid=2662625, ip=10.128.32.46) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(pid=2662625, ip=10.128.32.46) [2026-06-07 16:39:07,218 E 2662625 2662665] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:20.342 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:20.343 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:20.344 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:21.623 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:21.875 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:21.875 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:21.875 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:21.875 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:40:21.925 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.28s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2697412, ip=10.128.32.43) (EngineCore_DP0 pid=2698009) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2697414, ip=10.128.32.43) (EngineCore_DP0 pid=2698001) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) [W607 16:40:24.572522745 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-43-interconnect-1.jupiter.internal]:60183 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) [W607 16:40:24.576008342 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) [rank0]:[W607 16:40:24.582010146 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:20.342 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:20.343 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:20.344 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:21.623 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:21.875 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:21.875 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:21.875 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:21.875 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:40:21.925 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.51s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.17s/it]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) [W607 16:40:24.572461210 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-43-interconnect-1.jupiter.internal]:46619 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) [W607 16:40:24.576002902 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) [rank0]:[W607 16:40:24.582556632 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.12s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000)
(AsyncVLLMInferenceEngine pid=2697412, ip=10.128.32.43) (EngineCore_DP0 pid=2698009)
(AsyncVLLMInferenceEngine pid=2697414, ip=10.128.32.43) (EngineCore_DP0 pid=2698001)
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005)
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) 2026-06-07 16:40:45.443 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.44s/it] [repeated 11x across cluster]
(pid=2614265, ip=10.128.32.41) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2614265, ip=10.128.32.41) No module named 'vllm._version'
(pid=2614265, ip=10.128.32.41) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) 2026-06-07 16:40:45.422 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:50.391 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:50.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:50.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=2614410, ip=10.128.32.41) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=2614410, ip=10.128.32.41) No module named 'vllm._version' [repeated 3x across cluster]
(pid=2614410, ip=10.128.32.41) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:51.752 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:51.780 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:51.780 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:51.780 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:51.780 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:43:51.824 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.15s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:50.391 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:50.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:50.392 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) No module named 'vllm._version' [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:51.752 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:51.780 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:51.780 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:51.780 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:51.780 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:43:51.824 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.92s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) (EngineCore_DP0 pid=2577485) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2576837, ip=10.128.32.40) (EngineCore_DP0 pid=2577489) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) [W607 16:46:26.579304849 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-40-interconnect-1.jupiter.internal]:39565 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) [W607 16:46:26.580154880 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) [rank0]:[W607 16:46:26.586558911 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2576837, ip=10.128.32.40) (EngineCore_DP0 pid=2577489)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) [W607 16:46:26.579359856 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-40.jupiter.internal]:52891 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) [W607 16:46:26.580154720 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) [rank0]:[W607 16:46:26.586350275 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:10, 3.33s/it]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:11<00:04, 4.01s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500)
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) (EngineCore_DP0 pid=2577485)
(AsyncVLLMInferenceEngine pid=2576837, ip=10.128.32.40) (EngineCore_DP0 pid=2577489)
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495)
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) 2026-06-07 16:47:11.175 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.27s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:36.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:36.433 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:36.433 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) 2026-06-07 16:47:11.226 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:37.639 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:37.989 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:37.989 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:37.989 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:37.989 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:48:38.038 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.40s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:36.432 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:36.433 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:36.433 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:37.639 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:37.989 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:37.989 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:37.989 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:37.989 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:48:38.038 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.04s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739605, ip=10.128.32.34) (EngineCore_DP0 pid=2740434) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) [W607 16:49:22.529996202 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-34.jupiter.internal]:58359 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) [W607 16:49:22.533237515 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) [rank0]:[W607 16:49:22.539139353 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) [W607 16:49:22.530487424 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-34-interconnect-1.jupiter.internal]:58341 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) [W607 16:49:22.533253643 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) [rank0]:[W607 16:49:22.539392180 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:10, 3.55s/it]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:11<00:03, 3.94s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424)
(AsyncVLLMInferenceEngine pid=2739605, ip=10.128.32.34) (EngineCore_DP0 pid=2740434)
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438)
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428)
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) 2026-06-07 16:50:33.345 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:12<00:00, 3.22s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) 2026-06-07 16:50:33.384 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) (EngineCore_DP0 pid=2632450) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) [W607 16:50:43.765319760 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-36.jupiter.internal]:48845 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) [W607 16:50:43.767120941 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) [rank0]:[W607 16:50:43.772874492 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) [W607 16:50:43.107187519 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-36-interconnect-1.jupiter.internal]:42177 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) [W607 16:50:43.107636054 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) [rank0]:[W607 16:50:43.109791084 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:06.305 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:06.306 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:06.306 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.01s/it]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:07.518 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:07.792 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:07.792 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:07.792 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:07.792 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:52:07.835 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.77s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:06.305 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:06.306 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:06.306 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:08, 4.06s/it] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:07.518 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:07.791 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:07.791 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:07.791 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:07.791 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:52:07.835 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.87s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455)
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466)
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) (EngineCore_DP0 pid=2632450)
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471)
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.34s/it] [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) 2026-06-07 16:52:19.686 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(pid=2994914, ip=10.128.32.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2994914, ip=10.128.32.39) No module named 'vllm._version'
(pid=2994914, ip=10.128.32.39) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) 2026-06-07 16:52:19.639 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(pid=2995275, ip=10.128.32.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2995275, ip=10.128.32.39) No module named 'vllm._version'
(pid=2995275, ip=10.128.32.39) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) _C._set_float32_matmul_precision(precision)
(pid=2995274, ip=10.128.32.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 2x across cluster]
(pid=2995274, ip=10.128.32.39) No module named 'vllm._version' [repeated 2x across cluster]
(pid=2995274, ip=10.128.32.39) from .version import __version__, __version_tuple__ # isort:skip [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) (EngineCore_DP0 pid=3012342) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) (EngineCore_DP0 pid=3012343) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) [W607 16:54:12.735426276 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-44.jupiter.internal]:48903 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) [W607 16:54:12.739040171 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) [rank0]:[W607 16:54:12.745053709 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) [W607 16:54:12.735348261 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-44.jupiter.internal]:53241 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) [W607 16:54:12.739023275 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) [rank0]:[W607 16:54:12.745099244 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:11, 3.83s/it]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.15s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340)
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) (EngineCore_DP0 pid=3012342)
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) (EngineCore_DP0 pid=3012343)
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341)
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) 2026-06-07 16:55:23.546 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.35s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) 2026-06-07 16:55:23.611 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) (EngineCore_DP0 pid=2706735) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2705802, ip=10.128.32.38) (EngineCore_DP0 pid=2706743) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) [W607 16:55:33.477299204 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-38.jupiter.internal]:39711 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) [W607 16:55:33.479730539 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) [rank0]:[W607 16:55:33.486001944 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:38.696 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:38.697 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:38.697 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) [W607 16:55:33.476236797 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-38-interconnect-1.jupiter.internal]:41289 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) [W607 16:55:33.479705003 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) [rank0]:[W607 16:55:33.484923505 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:40.047 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:40.076 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:40.077 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:40.077 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:40.077 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 16:55:40.144 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.39s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:38.696 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:38.697 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:38.697 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:40.048 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:40.076 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:40.076 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:40.076 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:40.076 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) 2026-06-07 16:55:40.144 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.33s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2831685, ip=10.128.32.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=2831685, ip=10.128.32.45) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=2831685, ip=10.128.32.45) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:11, 3.97s/it]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.03s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759)
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) (EngineCore_DP0 pid=2706735)
(AsyncVLLMInferenceEngine pid=2705802, ip=10.128.32.38) (EngineCore_DP0 pid=2706743)
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739)
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) (EngineCore_DP0 pid=2706735) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) 2026-06-07 16:57:12.397 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.35s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2705802, ip=10.128.32.38) (EngineCore_DP0 pid=2706743) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) 2026-06-07 16:57:12.355 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831684, ip=10.128.32.45) (EngineCore_DP0 pid=2832749) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2831685, ip=10.128.32.45) (EngineCore_DP0 pid=2832737) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) [W607 17:00:00.617336904 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-45.jupiter.internal]:41873 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) [W607 17:00:00.621348216 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) [rank0]:[W607 17:00:00.627687930 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) [W607 17:00:00.617384583 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-45.jupiter.internal]:60113 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) [W607 17:00:00.621367768 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) [rank0]:[W607 17:00:00.627463967 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:11, 3.98s/it]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.08s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747)
(AsyncVLLMInferenceEngine pid=2831684, ip=10.128.32.45) (EngineCore_DP0 pid=2832749)
(AsyncVLLMInferenceEngine pid=2831685, ip=10.128.32.45) (EngineCore_DP0 pid=2832737)
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741)
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2831684, ip=10.128.32.45) 2026-06-07 17:01:11.571 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.39s/it] [repeated 11x across cluster]
(FSDPPolicyWorkerBase pid=2662625, ip=10.128.32.46) 2026-06-07 17:02:34 INFO [ipv4-debug] hostname=jpbo-041-46.jupiter.internal
(FSDPPolicyWorkerBase pid=2662625, ip=10.128.32.46) 2026-06-07 17:02:34 INFO [ipv4-debug] _global_node.node_ip_address=10.128.32.46
(FSDPPolicyWorkerBase pid=2662625, ip=10.128.32.46) 2026-06-07 17:02:34 INFO [ipv4-debug] get_node_ip_address()=10.128.32.46
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) 2026-06-07 17:01:11.656 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(FSDPPolicyWorkerBase pid=2662625, ip=10.128.32.46) [W607 17:02:34.731583451 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-46-interconnect-1.jupiter.internal]:41111 (errno: 97 - Address family not supported by protocol).
(skyrl_entrypoint pid=391889) 2026-06-07 17:02:34.937 | INFO  | skyrl_train.workers.worker:_initiate_actors:636 - Initializing process group for RayActorGroup
(pid=2663493, ip=10.128.32.46) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster]
(pid=394709) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 3x across cluster]
(pid=2663493, ip=10.128.32.46) [2026-06-07 17:03:05,886 E 2663493 2663634] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(FSDPPolicyWorkerBase pid=2663493, ip=10.128.32.46) [W607 17:03:10.567912961 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-46-interconnect-1.jupiter.internal]:41111 (errno: 97 - Address family not supported by protocol).
(FSDPPolicyWorkerBase pid=2663493, ip=10.128.32.46) [W607 17:03:10.570281892 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(pid=394710) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 3x across cluster]
2026-06-07 17:12:36.032 | ERROR | __main__:main:134 - Training failed: ray::skyrl_entrypoint() (pid=391889, ip=10.128.32.33)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 481, in run
trainer = self._setup_trainer()
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 450, in _setup_trainer
trainer.build_models(PolicyWorker, CriticWorker, RefWorker, policy_pg=self.policy_pg)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 768, in build_models
policy_model = PPORayActorGroup(
^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 511, in __init__
self._initiate_actors(pg, num_gpus_per_actor)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 637, in _initiate_actors
ray.get([actor.init_worker_process_group.remote() for actor in self._actor_handlers])
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(DistStoreError): ray::FSDPPolicyWorkerBase.init_worker_process_group() (pid=2662625, ip=10.128.32.46, actor_id=b5939e2b137362308272f3e202000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400df5e2cf50>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 149, in init_worker_process_group
torch.distributed.init_process_group(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
func_return = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
store, rank, world_size = next(rendezvous_iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 278, in _env_rendezvous_handler
store = _create_c10d_store(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
return TCPStore(
^^^^^^^^^
torch.distributed.DistStoreError: Timed out after 601 seconds waiting for clients. 4/8 clients joined.
Exception raised from waitForWorkers at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:396 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000bf09c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5e9c9c0 (0x400ae12fc9c0 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::waitForWorkers() + 0x350 (0x400ae1391410 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x468 (0x400ae13918c8 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x109a094 (0x4000be86a094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x113236c (0x4000be90236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x5d6d60 (0x4000bdda6d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x1b7a38 (0xaaaacb007a38 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #8: _PyObject_MakeTpCall + 0x98 (0xaaaacafb5db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #9: <unknown function> + 0x169f50 (0xaaaacafb9f50 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #10: <unknown function> + 0x1682e4 (0xaaaacafb82e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #11: <unknown function> + 0x1e0ce8 (0xaaaacb030ce8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #12: <unknown function> + 0x1d7ddc (0xaaaacb027ddc in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #13: <unknown function> + 0x646b0c (0x4000bde16b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #14: _PyObject_MakeTpCall + 0x98 (0xaaaacafb5db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #15: _PyEval_EvalFrameDefault + 0x280c (0xaaaacb0bae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #16: <unknown function> + 0x1808c0 (0xaaaacafd08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #17: <unknown function> + 0x182bf8 (0xaaaacafd2bf8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #18: <unknown function> + 0x25fd30 (0xaaaacb0afd30 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #19: <unknown function> + 0x1b7d20 (0xaaaacb007d20 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #20: PyObject_Vectorcall + 0x54 (0xaaaacafb60e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #21: _PyEval_EvalFrameDefault + 0x280c (0xaaaacb0bae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #22: <unknown function> + 0x1808c0 (0xaaaacafd08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #23: <unknown function> + 0x1835c8 (0xaaaacafd35c8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #24: <unknown function> + 0x84b708 (0x40000a80b708 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #25: <unknown function> + 0x865fe4 (0x40000a825fe4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #26: <unknown function> + 0x823470 (0x40000a7e3470 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #27: <unknown function> + 0x830110 (0x40000a7f0110 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #28: <unknown function> + 0x174a60 (0xaaaacafc4a60 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #29: PyObject_VectorcallMethod + 0xa4 (0xaaaacafb6270 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #30: PyIter_Send + 0xbc (0xaaaacaf9b62c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #31: <unknown function> + 0xaa04 (0x40000ca7aa04 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #32: <unknown function> + 0xbcc4 (0x40000ca7bcc4 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #33: _PyObject_MakeTpCall + 0x98 (0xaaaacafb5db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #34: <unknown function> + 0x28cfac (0xaaaacb0dcfac in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #35: <unknown function> + 0x1b7b68 (0xaaaacb007b68 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaacb0bd8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #37: <unknown function> + 0x83c564 (0x40000a7fc564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #38: _PyEval_EvalFrameDefault + 0x52ac (0xaaaacb0bd8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #39: <unknown function> + 0x169f88 (0xaaaacafb9f88 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #40: <unknown function> + 0x36002c (0xaaaacb1b002c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #41: <unknown function> + 0x2e4024 (0xaaaacb134024 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #42: <unknown function> + 0x80e00 (0x400009730e00 in /lib64/libc.so.6)
frame #43: <unknown function> + 0xeb49c (0x40000979b49c in /lib64/libc.so.6)
2026-06-07 17:12:36.033 | INFO | __main__:main:137 - Shutting down Ray on head node...
(pid=394710) [2026-06-07 17:03:06,091 E 394710 394918] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(FSDPPolicyWorkerBase pid=2663495, ip=10.128.32.46) [W607 17:03:10.575118569 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-46.jupiter.internal]:41111 (errno: 97 - Address family not supported by protocol). [repeated 2x across cluster]
(FSDPPolicyWorkerBase pid=2663495, ip=10.128.32.46) [W607 17:03:10.575405606 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 2x across cluster]
(skyrl_entrypoint pid=391889) [fd-monitor] Started monitoring (every 120s)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:02:41] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:02:41] OK: RSS 1.38 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(skyrl_entrypoint pid=391889) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=391889) [fd-monitor] [16:04:41] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:04:41] OK: RSS 1.50 GiB | node mem 161.4/858.0 GiB used (18.8%), avail 696.6 GiB
(skyrl_entrypoint pid=391889) INFO 06-07 16:06:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(skyrl_entrypoint pid=391889) [fd-monitor] [16:06:41] OK: 51 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:06:41] OK: RSS 1.59 GiB | node mem 161.5/858.0 GiB used (18.8%), avail 696.5 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [16:08:41] OK: 62 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:08:41] OK: RSS 1.63 GiB | node mem 161.0/858.0 GiB used (18.8%), avail 696.9 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [16:10:41] OK: 62 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:10:41] OK: RSS 1.63 GiB | node mem 161.0/858.0 GiB used (18.8%), avail 696.9 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [16:12:41] OK: 63 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:12:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.8 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [16:14:41] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:14:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(pid=3113603, ip=10.128.32.42) INFO 06-07 16:16:16 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(skyrl_entrypoint pid=391889) [fd-monitor] [16:16:41] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:16:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) WARNING 06-07 16:16:48 [arg_utils.py:1256] The global random seed is set to 42. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) INFO 06-07 16:16:48 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) INFO 06-07 16:16:48 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) INFO 06-07 16:16:48 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) INFO 06-07 16:16:48 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) INFO 06-07 16:16:48 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) WARNING 06-07 16:16:48 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) INFO 06-07 16:16:48 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) WARNING 06-07 16:16:48 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) WARNING 06-07 16:16:48 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) INFO 06-07 16:17:22 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:17:23 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=42, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:18 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:19 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.42:48887 backend=nccl
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:19 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:19 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:20 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:28 [default_loader.py:293] Loading weights took 8.20 seconds
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:28 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 8.778860 seconds
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:31 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:31 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:31 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:31 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:31 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.54 seconds
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) WARNING 06-07 16:18:32 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:32 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) WARNING 06-07 16:18:32 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) (EngineCore_DP0 pid=3114084) INFO 06-07 16:18:32 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3113603, ip=10.128.32.42) WARNING 06-07 16:18:32 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:35 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) WARNING 06-07 16:18:39 [arg_utils.py:1256] The global random seed is set to 56. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:39 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:39 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:39 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:39 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) WARNING 06-07 16:18:39 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:39 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:39 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) WARNING 06-07 16:18:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) WARNING 06-07 16:18:40 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) INFO 06-07 16:18:40 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) INFO 06-07 16:18:40 [vllm.py:846] Cudagraph is disabled under eager mode
(pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:35 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 2x across cluster]
(skyrl_entrypoint pid=391889) [fd-monitor] [16:18:41] OK: 68 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:18:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) WARNING 06-07 16:18:40 [arg_utils.py:1256] The global random seed is set to 57. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:40 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:40 [model.py:1549] Using max model len 32768 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:40 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:40 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) WARNING 06-07 16:18:40 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) WARNING 06-07 16:18:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) WARNING 06-07 16:18:40 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:40 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:40 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) INFO 06-07 16:18:48 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) INFO 06-07 16:18:48 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:48 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=56, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:51 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:52 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.42:44549 backend=nccl
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:52 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:53 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) INFO 06-07 16:18:48 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:53 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) INFO 06-07 16:18:49 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=55, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:58 [default_loader.py:293] Loading weights took 4.13 seconds
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) INFO 06-07 16:18:52 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) INFO 06-07 16:18:53 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.42:51575 backend=nccl [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) INFO 06-07 16:18:53 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) INFO 06-07 16:18:53 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:18:58 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 4.877260 seconds
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:19:00 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:19:00 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:19:00 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=3114112, ip=10.128.32.42) (EngineCore_DP0 pid=3114390) INFO 06-07 16:18:53 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:19:00 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:19:00 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.44 seconds
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) WARNING 06-07 16:19:01 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:19:01 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) WARNING 06-07 16:19:01 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) (EngineCore_DP0 pid=3114382) INFO 06-07 16:19:01 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3114113, ip=10.128.32.42) WARNING 06-07 16:19:01 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=391889) [fd-monitor] [16:20:41] OK: 68 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:20:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:18:58 [default_loader.py:293] Loading weights took 4.38 seconds [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:18:58 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 5.087965 seconds [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:19:01 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:19:01 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:19:01 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:19:01 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:19:01 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.46 seconds [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) WARNING 06-07 16:19:01 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:19:01 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) WARNING 06-07 16:19:01 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) (EngineCore_DP0 pid=3114394) INFO 06-07 16:19:01 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3114114, ip=10.128.32.42) WARNING 06-07 16:19:01 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 2x across cluster]
(skyrl_entrypoint pid=391889) [fd-monitor] [16:22:41] OK: 70 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:22:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.8 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [16:24:41] OK: 72 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:24:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [16:26:41] OK: 72 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:26:41] OK: RSS 1.63 GiB | node mem 161.0/858.0 GiB used (18.8%), avail 696.9 GiB
(pid=2697284, ip=10.128.32.43) INFO 06-07 16:28:12 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(skyrl_entrypoint pid=391889) [fd-monitor] [16:28:41] OK: 74 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:28:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(pid=2697413, ip=10.128.32.43) INFO 06-07 16:28:12 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) WARNING 06-07 16:29:10 [arg_utils.py:1256] The global random seed is set to 50. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) INFO 06-07 16:29:11 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) INFO 06-07 16:29:11 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) INFO 06-07 16:29:11 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) INFO 06-07 16:29:11 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) INFO 06-07 16:29:11 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) WARNING 06-07 16:29:11 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) INFO 06-07 16:29:11 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) WARNING 06-07 16:29:11 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) WARNING 06-07 16:29:11 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(skyrl_entrypoint pid=391889) [fd-monitor] [16:30:41] OK: 75 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:30:41] OK: RSS 1.63 GiB | node mem 161.0/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) WARNING 06-07 16:29:10 [arg_utils.py:1256] The global random seed is set to 53. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) INFO 06-07 16:29:11 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) INFO 06-07 16:29:11 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) INFO 06-07 16:29:11 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) INFO 06-07 16:29:11 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) INFO 06-07 16:29:11 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) WARNING 06-07 16:29:11 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) INFO 06-07 16:29:11 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) WARNING 06-07 16:29:11 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) WARNING 06-07 16:29:11 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(skyrl_entrypoint pid=391889) [fd-monitor] [16:32:41] OK: 80 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:32:41] OK: RSS 1.63 GiB | node mem 161.0/858.0 GiB used (18.8%), avail 697.0 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [16:34:41] OK: 80 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:34:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) INFO 06-07 16:35:54 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:35:55 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=50, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(skyrl_entrypoint pid=391889) [fd-monitor] [16:36:41] OK: 80 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:36:41] OK: RSS 1.63 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) INFO 06-07 16:35:54 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:35:55 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=53, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(pid=2576535, ip=10.128.32.40) INFO 06-07 16:37:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(pid=2576836, ip=10.128.32.40) INFO 06-07 16:37:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) WARNING 06-07 16:37:50 [arg_utils.py:1256] The global random seed is set to 63. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(pid=2576838, ip=10.128.32.40) INFO 06-07 16:37:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) INFO 06-07 16:37:50 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) INFO 06-07 16:37:50 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) INFO 06-07 16:37:50 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) INFO 06-07 16:37:50 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) INFO 06-07 16:37:50 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) WARNING 06-07 16:37:50 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) INFO 06-07 16:37:50 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) WARNING 06-07 16:37:50 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) WARNING 06-07 16:37:50 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(skyrl_entrypoint pid=391889) [fd-monitor] [16:38:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:38:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) WARNING 06-07 16:37:51 [arg_utils.py:1256] The global random seed is set to 62. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) INFO 06-07 16:37:51 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) INFO 06-07 16:37:51 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) INFO 06-07 16:37:51 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) INFO 06-07 16:37:51 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) INFO 06-07 16:37:51 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) WARNING 06-07 16:37:51 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) INFO 06-07 16:37:51 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) WARNING 06-07 16:37:51 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) WARNING 06-07 16:37:51 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(pid=2739368, ip=10.128.32.34) INFO 06-07 16:39:00 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:23 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(pid=2739604, ip=10.128.32.34) INFO 06-07 16:39:00 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) WARNING 06-07 16:40:24 [arg_utils.py:1256] The global random seed is set to 58. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:24 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.43:60183 backend=nccl
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) INFO 06-07 16:40:24 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) INFO 06-07 16:40:24 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:24 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) INFO 06-07 16:40:24 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) INFO 06-07 16:40:24 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) INFO 06-07 16:40:24 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) WARNING 06-07 16:40:24 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) INFO 06-07 16:40:24 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) WARNING 06-07 16:40:24 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) WARNING 06-07 16:40:24 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:25 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:27 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:41 [default_loader.py:293] Loading weights took 13.76 seconds
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:23 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) WARNING 06-07 16:40:24 [arg_utils.py:1256] The global random seed is set to 60. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:24 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.43:46619 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) INFO 06-07 16:40:24 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) INFO 06-07 16:40:24 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:24 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) INFO 06-07 16:40:24 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) INFO 06-07 16:40:24 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) INFO 06-07 16:40:24 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) WARNING 06-07 16:40:24 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) INFO 06-07 16:40:24 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) WARNING 06-07 16:40:25 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) WARNING 06-07 16:40:25 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:25 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:27 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(skyrl_entrypoint pid=391889) [fd-monitor] [16:40:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:40:41] OK: RSS 1.64 GiB | node mem 161.0/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:41 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.544833 seconds
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:44 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:44 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:44 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2697412, ip=10.128.32.43) (EngineCore_DP0 pid=2698009) INFO 06-07 16:40:44 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) INFO 06-07 16:40:44 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) (EngineCore_DP0 pid=2698000) WARNING 06-07 16:40:45 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2697284, ip=10.128.32.43) WARNING 06-07 16:40:45 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=391889) [fd-monitor] [16:42:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:42:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:41 [default_loader.py:293] Loading weights took 13.76 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:45 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:45 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) WARNING 06-07 16:40:45 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:41 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.540163 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:44 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:44 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:44 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:44 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) INFO 06-07 16:40:44 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.85 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) (EngineCore_DP0 pid=2698005) WARNING 06-07 16:40:45 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2697413, ip=10.128.32.43) WARNING 06-07 16:40:45 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(pid=2631427, ip=10.128.32.36) INFO 06-07 16:43:22 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) WARNING 06-07 16:43:53 [arg_utils.py:1256] The global random seed is set to 79. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(pid=2631820, ip=10.128.32.36) INFO 06-07 16:43:22 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) INFO 06-07 16:43:53 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) INFO 06-07 16:43:53 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) INFO 06-07 16:43:53 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) INFO 06-07 16:43:53 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) INFO 06-07 16:43:53 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) WARNING 06-07 16:43:53 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) INFO 06-07 16:43:53 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) WARNING 06-07 16:43:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2631819, ip=10.128.32.36) WARNING 06-07 16:43:54 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) WARNING 06-07 16:43:54 [arg_utils.py:1256] The global random seed is set to 81. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) INFO 06-07 16:44:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) INFO 06-07 16:43:54 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) INFO 06-07 16:43:54 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) INFO 06-07 16:43:54 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) INFO 06-07 16:43:54 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) INFO 06-07 16:43:54 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) WARNING 06-07 16:43:54 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) INFO 06-07 16:43:54 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) WARNING 06-07 16:43:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) WARNING 06-07 16:43:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) INFO 06-07 16:44:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:44:09 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=62, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(skyrl_entrypoint pid=391889) [fd-monitor] [16:44:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:44:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) INFO 06-07 16:44:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:44:09 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=65, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:46:25 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:46:26 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.40:39565 backend=nccl
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:46:26 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:46:26 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(skyrl_entrypoint pid=391889) [fd-monitor] [16:46:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:46:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:46:25 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:46:26 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.40:52891 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:46:26 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:46:26 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576837, ip=10.128.32.40) (EngineCore_DP0 pid=2577489) INFO 06-07 16:46:53 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:07 [default_loader.py:293] Loading weights took 13.09 seconds
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:46:54 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:07 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.482660 seconds
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:10 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:10 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:10 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2576837, ip=10.128.32.40) (EngineCore_DP0 pid=2577489) INFO 06-07 16:47:10 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:10 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.84 seconds
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) WARNING 06-07 16:47:11 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:11 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) WARNING 06-07 16:47:11 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) (EngineCore_DP0 pid=2577500) INFO 06-07 16:47:11 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2576836, ip=10.128.32.40) WARNING 06-07 16:47:11 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) INFO 06-07 16:47:33 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:07 [default_loader.py:293] Loading weights took 13.09 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:07 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.481976 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:10 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:10 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:10 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:10 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:10 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.86 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) WARNING 06-07 16:47:11 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:11 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) WARNING 06-07 16:47:11 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576838, ip=10.128.32.40) (EngineCore_DP0 pid=2577495) INFO 06-07 16:47:11 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2576535, ip=10.128.32.40) WARNING 06-07 16:47:11 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:47:34 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=58, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(pid=3010613, ip=10.128.32.44) INFO 06-07 16:48:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:47:34 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=61, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) INFO 06-07 16:48:26 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:48:27 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=66, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) WARNING 06-07 16:48:40 [arg_utils.py:1256] The global random seed is set to 68. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) INFO 06-07 16:48:26 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:48:27 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=81, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) INFO 06-07 16:48:40 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) INFO 06-07 16:48:40 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) INFO 06-07 16:48:40 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) INFO 06-07 16:48:40 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) INFO 06-07 16:48:40 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) WARNING 06-07 16:48:40 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) INFO 06-07 16:48:40 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) WARNING 06-07 16:48:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) WARNING 06-07 16:48:40 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(skyrl_entrypoint pid=391889) [fd-monitor] [16:48:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:48:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:49:22 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) WARNING 06-07 16:48:40 [arg_utils.py:1256] The global random seed is set to 69. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) INFO 06-07 16:48:40 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) INFO 06-07 16:48:40 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) INFO 06-07 16:48:40 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) INFO 06-07 16:48:40 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) INFO 06-07 16:48:40 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) WARNING 06-07 16:48:40 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) INFO 06-07 16:48:40 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) WARNING 06-07 16:48:40 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011432, ip=10.128.32.44) WARNING 06-07 16:48:40 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:49:22 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.34:58359 backend=nccl
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:49:22 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:15 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:49:22 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:49:22 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.34:58341 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:49:22 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:16 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:29 [default_loader.py:293] Loading weights took 13.02 seconds
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:15 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:16 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:29 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.171795 seconds
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:32 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:32 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:32 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:32 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:32 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) WARNING 06-07 16:50:33 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:33 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) WARNING 06-07 16:50:33 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) (EngineCore_DP0 pid=2740424) INFO 06-07 16:50:33 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2739368, ip=10.128.32.34) WARNING 06-07 16:50:33 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=391889) [fd-monitor] [16:50:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:50:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:29 [default_loader.py:293] Loading weights took 12.88 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:29 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.175118 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:32 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:32 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:32 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) (EngineCore_DP0 pid=2740428) INFO 06-07 16:50:32 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438) INFO 06-07 16:50:32 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438) WARNING 06-07 16:50:33 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438) INFO 06-07 16:50:33 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438) WARNING 06-07 16:50:33 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739603, ip=10.128.32.34) (EngineCore_DP0 pid=2740438) INFO 06-07 16:50:33 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2739604, ip=10.128.32.34) WARNING 06-07 16:50:33 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) INFO 06-07 16:50:42 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) INFO 06-07 16:50:43 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.36:48845 backend=nccl
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) INFO 06-07 16:50:43 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:51:09 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:50:43 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:50:43 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.36:42177 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:50:43 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(pid=2705575, ip=10.128.32.38) INFO 06-07 16:51:38 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:51:09 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) INFO 06-07 16:51:54 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:51:56 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=67, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:52:02 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) INFO 06-07 16:51:54 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:51:56 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=78, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) WARNING 06-07 16:52:09 [arg_utils.py:1256] The global random seed is set to 72. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:52:02 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) INFO 06-07 16:52:09 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) INFO 06-07 16:52:09 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) INFO 06-07 16:52:09 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) INFO 06-07 16:52:09 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) INFO 06-07 16:52:09 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) WARNING 06-07 16:52:09 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) INFO 06-07 16:52:09 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) WARNING 06-07 16:52:10 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2705804, ip=10.128.32.38) WARNING 06-07 16:52:10 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:52:15 [default_loader.py:293] Loading weights took 13.37 seconds
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) WARNING 06-07 16:52:10 [arg_utils.py:1256] The global random seed is set to 70. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) INFO 06-07 16:52:10 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) INFO 06-07 16:52:10 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) INFO 06-07 16:52:10 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) INFO 06-07 16:52:10 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) INFO 06-07 16:52:10 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) WARNING 06-07 16:52:10 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) INFO 06-07 16:52:10 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) WARNING 06-07 16:52:11 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) WARNING 06-07 16:52:11 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:52:16 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 65.938186 seconds
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) INFO 06-07 16:52:18 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) INFO 06-07 16:52:18 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) INFO 06-07 16:52:18 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2631821, ip=10.128.32.36) (EngineCore_DP0 pid=2632466) INFO 06-07 16:52:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:52:18 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) WARNING 06-07 16:52:19 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) WARNING 06-07 16:52:19 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=391889) [fd-monitor] [16:52:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:52:41] OK: RSS 1.64 GiB | node mem 161.0/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:52:15 [default_loader.py:293] Loading weights took 13.37 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:52:19 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:52:19 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) WARNING 06-07 16:52:19 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:52:16 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 65.945630 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:52:18 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:52:18 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631427, ip=10.128.32.36) (EngineCore_DP0 pid=2632455) INFO 06-07 16:52:18 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:52:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) INFO 06-07 16:52:18 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.79 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) (EngineCore_DP0 pid=2632471) WARNING 06-07 16:52:19 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2631820, ip=10.128.32.36) WARNING 06-07 16:52:19 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) INFO 06-07 16:53:15 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:53:17 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=70, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:54:12 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) INFO 06-07 16:53:15 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:53:17 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=73, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:54:12 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.44:48903 backend=nccl
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:54:12 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:54:12 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(pid=2831550, ip=10.128.32.45) INFO 06-07 16:54:18 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:54:12 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.44:53241 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:54:12 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(pid=2831684, ip=10.128.32.45) INFO 06-07 16:54:18 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(skyrl_entrypoint pid=391889) [fd-monitor] [16:54:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:54:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(pid=2831686, ip=10.128.32.45) INFO 06-07 16:54:18 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:04 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:06 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:19 [default_loader.py:293] Loading weights took 13.56 seconds
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:04 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:06 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:20 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.714728 seconds
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:22 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:22 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:22 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) WARNING 06-07 16:55:23 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:23 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) WARNING 06-07 16:55:23 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) (EngineCore_DP0 pid=3012340) INFO 06-07 16:55:23 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=3010613, ip=10.128.32.44) WARNING 06-07 16:55:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:55:32 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:19 [default_loader.py:293] Loading weights took 13.43 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:20 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.718624 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) (EngineCore_DP0 pid=3012342) INFO 06-07 16:55:22 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) (EngineCore_DP0 pid=3012342) INFO 06-07 16:55:22 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011433, ip=10.128.32.44) (EngineCore_DP0 pid=3012342) INFO 06-07 16:55:22 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) WARNING 06-07 16:55:23 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:23 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) WARNING 06-07 16:55:23 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) (EngineCore_DP0 pid=3012341) INFO 06-07 16:55:23 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3011671, ip=10.128.32.44) WARNING 06-07 16:55:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:55:33 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.38:39711 backend=nccl
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:55:33 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) WARNING 06-07 16:55:42 [arg_utils.py:1256] The global random seed is set to 82. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:55:32 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:55:33 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.38:41289 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:55:33 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:56:00 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2831685, ip=10.128.32.45) WARNING 06-07 16:55:42 [arg_utils.py:1256] The global random seed is set to 84. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) INFO 06-07 16:56:09 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) INFO 06-07 16:56:09 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) INFO 06-07 16:56:09 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) INFO 06-07 16:56:09 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) INFO 06-07 16:56:09 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) WARNING 06-07 16:56:09 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) INFO 06-07 16:56:09 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:56:00 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) WARNING 06-07 16:56:09 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) WARNING 06-07 16:56:09 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(skyrl_entrypoint pid=391889) [fd-monitor] [16:56:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:56:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) INFO 06-07 16:56:09 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) INFO 06-07 16:56:09 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) INFO 06-07 16:56:09 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) INFO 06-07 16:56:09 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) INFO 06-07 16:56:09 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) WARNING 06-07 16:56:09 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) INFO 06-07 16:56:09 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) WARNING 06-07 16:56:09 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) WARNING 06-07 16:56:09 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) INFO 06-07 16:56:50 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 16:56:52 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=82, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:56:54 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:08 [default_loader.py:293] Loading weights took 13.39 seconds
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) INFO 06-07 16:56:50 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 16:56:52 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=85, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:56:54 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 67.984083 seconds
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:11 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:11 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:11 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:11 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:11 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) WARNING 06-07 16:57:12 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:12 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) WARNING 06-07 16:57:12 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) (EngineCore_DP0 pid=2706759) INFO 06-07 16:57:12 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2705575, ip=10.128.32.38) WARNING 06-07 16:57:12 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=391889) [fd-monitor] [16:58:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [16:58:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:08 [default_loader.py:293] Loading weights took 13.39 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 67.996617 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:11 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:11 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:11 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:11 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:11 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.82 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) WARNING 06-07 16:57:12 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:12 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) WARNING 06-07 16:57:12 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) (EngineCore_DP0 pid=2706739) INFO 06-07 16:57:12 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2705803, ip=10.128.32.38) WARNING 06-07 16:57:12 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 16:59:59 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:00:00 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.45:41873 backend=nccl
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:00:00 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(skyrl_entrypoint pid=391889) [fd-monitor] [17:00:41] OK: 88 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [17:00:41] OK: RSS 1.64 GiB | node mem 161.1/858.0 GiB used (18.8%), avail 696.9 GiB
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 16:59:59 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:00:00 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.45:60113 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:00:00 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:00:52 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:00:53 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:07 [default_loader.py:293] Loading weights took 13.57 seconds
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:00:52 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:00:53 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.159333 seconds
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:10 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:10 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:10 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2831684, ip=10.128.32.45) (EngineCore_DP0 pid=2832749) INFO 06-07 17:01:10 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:10 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) WARNING 06-07 17:01:11 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:11 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) WARNING 06-07 17:01:11 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) (EngineCore_DP0 pid=2832747) INFO 06-07 17:01:11 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2831684, ip=10.128.32.45) WARNING 06-07 17:01:11 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(pid=2662625, ip=10.128.32.46) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:07 [default_loader.py:293] Loading weights took 13.57 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.161752 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:10 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:10 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:10 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:10 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:10 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.84 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) WARNING 06-07 17:01:11 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:11 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) WARNING 06-07 17:01:11 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831686, ip=10.128.32.45) (EngineCore_DP0 pid=2832741) INFO 06-07 17:01:11 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2831550, ip=10.128.32.45) WARNING 06-07 17:01:11 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(skyrl_entrypoint pid=391889) [fd-monitor] [17:02:41] OK: 86 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [17:02:41] OK: RSS 1.64 GiB | node mem 162.3/858.0 GiB used (18.9%), avail 695.7 GiB
(pid=2663493, ip=10.128.32.46) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=391889) [fd-monitor] [17:04:41] OK: 89 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [17:04:41] OK: RSS 1.64 GiB | node mem 162.4/858.0 GiB used (18.9%), avail 695.6 GiB
(pid=2663495, ip=10.128.32.46) ⚙️ Running in WANDB offline mode [repeated 2x across cluster]
(skyrl_entrypoint pid=391889) [fd-monitor] [17:06:41] OK: 89 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [17:06:41] OK: RSS 1.64 GiB | node mem 162.4/858.0 GiB used (18.9%), avail 695.5 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [17:08:41] OK: 83 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [17:08:41] OK: RSS 1.64 GiB | node mem 162.4/858.0 GiB used (18.9%), avail 695.6 GiB
(skyrl_entrypoint pid=391889) [fd-monitor] [17:10:41] OK: 83 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=391889) [fd-monitor] [17:10:41] OK: RSS 1.64 GiB | node mem 162.5/858.0 GiB used (18.9%), avail 695.5 GiB
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=sequence_mean', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqmean-arm0', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(DistStoreError): ray::skyrl_entrypoint() (pid=391889, ip=10.128.32.33)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 481, in run
trainer = self._setup_trainer()
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 450, in _setup_trainer
trainer.build_models(PolicyWorker, CriticWorker, RefWorker, policy_pg=self.policy_pg)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 768, in build_models
policy_model = PPORayActorGroup(
^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 511, in __init__
self._initiate_actors(pg, num_gpus_per_actor)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 637, in _initiate_actors
ray.get([actor.init_worker_process_group.remote() for actor in self._actor_handlers])
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(DistStoreError): ray::FSDPPolicyWorkerBase.init_worker_process_group() (pid=2662625, ip=10.128.32.46, actor_id=b5939e2b137362308272f3e202000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400df5e2cf50>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 149, in init_worker_process_group
torch.distributed.init_process_group(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
func_return = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
store, rank, world_size = next(rendezvous_iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 278, in _env_rendezvous_handler
store = _create_c10d_store(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
return TCPStore(
^^^^^^^^^
torch.distributed.DistStoreError: Timed out after 601 seconds waiting for clients. 4/8 clients joined.
Exception raised from waitForWorkers at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:396 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000bf09c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5e9c9c0 (0x400ae12fc9c0 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::waitForWorkers() + 0x350 (0x400ae1391410 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x468 (0x400ae13918c8 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x109a094 (0x4000be86a094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x113236c (0x4000be90236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x5d6d60 (0x4000bdda6d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x1b7a38 (0xaaaacb007a38 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #8: _PyObject_MakeTpCall + 0x98 (0xaaaacafb5db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #9: <unknown function> + 0x169f50 (0xaaaacafb9f50 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #10: <unknown function> + 0x1682e4 (0xaaaacafb82e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #11: <unknown function> + 0x1e0ce8 (0xaaaacb030ce8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #12: <unknown function> + 0x1d7ddc (0xaaaacb027ddc in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #13: <unknown function> + 0x646b0c (0x4000bde16b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #14: _PyObject_MakeTpCall + 0x98 (0xaaaacafb5db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #15: _PyEval_EvalFrameDefault + 0x280c (0xaaaacb0bae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #16: <unknown function> + 0x1808c0 (0xaaaacafd08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #17: <unknown function> + 0x182bf8 (0xaaaacafd2bf8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #18: <unknown function> + 0x25fd30 (0xaaaacb0afd30 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #19: <unknown function> + 0x1b7d20 (0xaaaacb007d20 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #20: PyObject_Vectorcall + 0x54 (0xaaaacafb60e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #21: _PyEval_EvalFrameDefault + 0x280c (0xaaaacb0bae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #22: <unknown function> + 0x1808c0 (0xaaaacafd08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #23: <unknown function> + 0x1835c8 (0xaaaacafd35c8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #24: <unknown function> + 0x84b708 (0x40000a80b708 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #25: <unknown function> + 0x865fe4 (0x40000a825fe4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #26: <unknown function> + 0x823470 (0x40000a7e3470 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #27: <unknown function> + 0x830110 (0x40000a7f0110 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #28: <unknown function> + 0x174a60 (0xaaaacafc4a60 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #29: PyObject_VectorcallMethod + 0xa4 (0xaaaacafb6270 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #30: PyIter_Send + 0xbc (0xaaaacaf9b62c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #31: <unknown function> + 0xaa04 (0x40000ca7aa04 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #32: <unknown function> + 0xbcc4 (0x40000ca7bcc4 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #33: _PyObject_MakeTpCall + 0x98 (0xaaaacafb5db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #34: <unknown function> + 0x28cfac (0xaaaacb0dcfac in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #35: <unknown function> + 0x1b7b68 (0xaaaacb007b68 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaacb0bd8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #37: <unknown function> + 0x83c564 (0x40000a7fc564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #38: _PyEval_EvalFrameDefault + 0x52ac (0xaaaacb0bd8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #39: <unknown function> + 0x169f88 (0xaaaacafb9f88 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #40: <unknown function> + 0x36002c (0xaaaacb1b002c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #41: <unknown function> + 0x2e4024 (0xaaaacb134024 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #42: <unknown function> + 0x80e00 (0x400009730e00 in /lib64/libc.so.6)
frame #43: <unknown function> + 0xeb49c (0x40000979b49c in /lib64/libc.so.6)
Stopping Ray cluster...
Warning: Failed to stop Ray on jpbo-041-36: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-36', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Ray cluster stopped
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
episodes: last
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
Collecting Ray logs from worker jpbo-041-34...
Collecting Ray logs from worker jpbo-041-35...
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink