laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0-tis_653584.out
ModelHub XC 9f22667c70 初始化项目，由ModelHub XC社区提供模型 
Model: laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B
Source: Original Platform
2026-08-09 05:27:18 +08:00

1893 lines
363 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_653584
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_653584
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_653584
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.16.35 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653584.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653584.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/configs/ablation-pymethods2test-seqmean-arm0-tis_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0-tis ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-001-35 (10.128.16.35)
Ray port: 6379
============================
Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-001-35
Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-001-36
Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-001-37
Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-001-38
Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-001-41
Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-001-42
Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-001-43
Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-001-44
Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-001-45
Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-001-46
Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-001-47
Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-001-48
Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-003-37
Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-003-39
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'GPU': 56.0, 'accelerator_type:GH200': 14.0, 'node:10.128.16.43': 1.0, 'memory': 10289339891712.0, 'CPU': 4032.0, 'object_store_memory': 601295421440.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.46': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.36': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.135': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.41': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
Address: 10.128.16.35:6379
Total GPUs: 56
=========================
Ray cluster ready at 10.128.16.35:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
Running SkyRL:
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
Args: 123 Hydra arguments
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=sequence_mean trainer.algorithm.use_tis=true trainer.algorithm.tis_imp_ratio_cap=2.0 trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0-tis ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqmean-arm0-tis trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.collect_rollout_details=true +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653584.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 15:37:07.080 | WARNING | skyrl_train.utils.utils:validate_cfg:479 - `generator.sampling_params.logprobs` is `None` but `trainer.algorithm.use_tis` is `True`. Setting `logprobs` to `True`.
2026-06-07 15:37:07.289 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:753 - Exporting wandb api key to ray runtime env
2026-06-07 15:37:07.289 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:772 - Exporting RAY_ADDRESS to ray runtime env
2026-06-07 15:37:07.289 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
2026-06-07 15:37:07.289 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
2026-06-07 15:37:07.289 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
2026-06-07 15:37:07,290 INFO worker.py:1680 -- Using address 10.128.16.35:6379 set in the environment variable RAY_ADDRESS
2026-06-07 15:37:07,324 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
2026-06-07 15:37:07,334 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.16.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 15:37:09.507 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
[2026-06-07 15:37:37,531 E 1617107 1617929] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(skyrl_entrypoint pid=1617955) 2026-06-07 15:37:18.206 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:212 - SkyRL log level set to: INFO
(skyrl_entrypoint pid=1617955) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=1617955) 2026-06-07 15:37:18.546 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=1617955) 2026-06-07 15:37:20.166 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 5000 valid task directories out of 5000 total directories
(skyrl_entrypoint pid=1617955) 2026-06-07 15:37:20.166 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 5000 task paths
(skyrl_entrypoint pid=1617955) 2026-06-07 15:37:20.178 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:405 - data:
(skyrl_entrypoint pid=1617955) train_data:
(skyrl_entrypoint pid=1617955) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=1617955) val_data: []
(skyrl_entrypoint pid=1617955) trainer:
(skyrl_entrypoint pid=1617955) placement:
(skyrl_entrypoint pid=1617955) colocate_all: false
(skyrl_entrypoint pid=1617955) colocate_policy_ref: true
(skyrl_entrypoint pid=1617955) policy_num_nodes: 2
(skyrl_entrypoint pid=1617955) policy_num_gpus_per_node: 4
(skyrl_entrypoint pid=1617955) critic_num_nodes: 1
(skyrl_entrypoint pid=1617955) critic_num_gpus_per_node: 4
(skyrl_entrypoint pid=1617955) ref_num_nodes: 2
(skyrl_entrypoint pid=1617955) ref_num_gpus_per_node: 4
(skyrl_entrypoint pid=1617955) policy_strict_spread_pg: false
(skyrl_entrypoint pid=1617955) policy_per_gpu_bundles: false
(skyrl_entrypoint pid=1617955) policy_force_cvd_mask: false
(skyrl_entrypoint pid=1617955) sequence_parallel_backend: ulysses
(skyrl_entrypoint pid=1617955) strategy: fsdp2
(skyrl_entrypoint pid=1617955) policy:
(skyrl_entrypoint pid=1617955) model:
(skyrl_entrypoint pid=1617955) path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=1617955) lora:
(skyrl_entrypoint pid=1617955) rank: 0
(skyrl_entrypoint pid=1617955) alpha: 16
(skyrl_entrypoint pid=1617955) dropout: 0
(skyrl_entrypoint pid=1617955) lora_sync_path: /tmp/skyrl_lora_sync
(skyrl_entrypoint pid=1617955) target_modules: all-linear
(skyrl_entrypoint pid=1617955) exclude_modules: null
(skyrl_entrypoint pid=1617955) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=1617955) optimizer_config:
(skyrl_entrypoint pid=1617955) optimizer: AdamW
(skyrl_entrypoint pid=1617955) lr: 8.0e-06
(skyrl_entrypoint pid=1617955) adam_betas:
(skyrl_entrypoint pid=1617955) - 0.9
(skyrl_entrypoint pid=1617955) - 0.999
(skyrl_entrypoint pid=1617955) weight_decay: 0.0
(skyrl_entrypoint pid=1617955) max_grad_norm: 0.9
(skyrl_entrypoint pid=1617955) offload_after_step: true
(skyrl_entrypoint pid=1617955) num_warmup_steps: 0
(skyrl_entrypoint pid=1617955) scheduler: constant_with_warmup
(skyrl_entrypoint pid=1617955) optimizer_kwargs: {}
(skyrl_entrypoint pid=1617955) fsdp_config:
(skyrl_entrypoint pid=1617955) cpu_offload: false
(skyrl_entrypoint pid=1617955) reshard_after_forward: true
(skyrl_entrypoint pid=1617955) fsdp_size: 4
(skyrl_entrypoint pid=1617955) expert_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=1617955) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=1617955) moe_router_replay: false
(skyrl_entrypoint pid=1617955) moe_grouped_gemm: false
(skyrl_entrypoint pid=1617955) ep_comm_backend: torch
(skyrl_entrypoint pid=1617955) deepep_num_sms: 20
(skyrl_entrypoint pid=1617955) deepep_token_chunk_size: null
(skyrl_entrypoint pid=1617955) sequence_parallel_size: 1
(skyrl_entrypoint pid=1617955) use_torch_compile: false
(skyrl_entrypoint pid=1617955) record_memory: false
(skyrl_entrypoint pid=1617955) megatron_config:
(skyrl_entrypoint pid=1617955) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) context_parallel_size: 1
(skyrl_entrypoint pid=1617955) expert_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) expert_tensor_parallel_size: null
(skyrl_entrypoint pid=1617955) ddp_config:
(skyrl_entrypoint pid=1617955) grad_reduce_in_fp32: true
(skyrl_entrypoint pid=1617955) overlap_grad_reduce: false
(skyrl_entrypoint pid=1617955) overlap_param_gather: false
(skyrl_entrypoint pid=1617955) average_in_collective: true
(skyrl_entrypoint pid=1617955) model_config_kwargs: {}
(skyrl_entrypoint pid=1617955) torch_profiler_config:
(skyrl_entrypoint pid=1617955) enable: false
(skyrl_entrypoint pid=1617955) ranks: []
(skyrl_entrypoint pid=1617955) save_path: null
(skyrl_entrypoint pid=1617955) optimizer_config_kwargs:
(skyrl_entrypoint pid=1617955) overlap_cpu_optimizer_d2h_h2d: false
(skyrl_entrypoint pid=1617955) use_precision_aware_optimizer: false
(skyrl_entrypoint pid=1617955) optimizer_cpu_offload: false
(skyrl_entrypoint pid=1617955) optimizer_offload_fraction: 0.0
(skyrl_entrypoint pid=1617955) transformer_config_kwargs:
(skyrl_entrypoint pid=1617955) recompute_granularity: full
(skyrl_entrypoint pid=1617955) recompute_modules:
(skyrl_entrypoint pid=1617955) - core_attn
(skyrl_entrypoint pid=1617955) recompute_method: uniform
(skyrl_entrypoint pid=1617955) recompute_num_layers: 1
(skyrl_entrypoint pid=1617955) empty_cuda_cache: true
(skyrl_entrypoint pid=1617955) ref:
(skyrl_entrypoint pid=1617955) model:
(skyrl_entrypoint pid=1617955) path: ${trainer.policy.model.path}
(skyrl_entrypoint pid=1617955) sequence_parallel_size: 1
(skyrl_entrypoint pid=1617955) deepspeed_config: ${deepspeed_config.eval}
(skyrl_entrypoint pid=1617955) fsdp_config:
(skyrl_entrypoint pid=1617955) cpu_offload: false
(skyrl_entrypoint pid=1617955) reshard_after_forward: true
(skyrl_entrypoint pid=1617955) fsdp_size: 4
(skyrl_entrypoint pid=1617955) expert_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=1617955) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=1617955) moe_router_replay: false
(skyrl_entrypoint pid=1617955) moe_grouped_gemm: false
(skyrl_entrypoint pid=1617955) ep_comm_backend: torch
(skyrl_entrypoint pid=1617955) deepep_num_sms: 20
(skyrl_entrypoint pid=1617955) deepep_token_chunk_size: null
(skyrl_entrypoint pid=1617955) megatron_config:
(skyrl_entrypoint pid=1617955) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) context_parallel_size: 1
(skyrl_entrypoint pid=1617955) expert_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=1617955) model_config_kwargs: {}
(skyrl_entrypoint pid=1617955) transformer_config_kwargs: {}
(skyrl_entrypoint pid=1617955) critic:
(skyrl_entrypoint pid=1617955) model:
(skyrl_entrypoint pid=1617955) path: null
(skyrl_entrypoint pid=1617955) lora:
(skyrl_entrypoint pid=1617955) rank: 0
(skyrl_entrypoint pid=1617955) alpha: 16
(skyrl_entrypoint pid=1617955) dropout: 0
(skyrl_entrypoint pid=1617955) target_modules: all-linear
(skyrl_entrypoint pid=1617955) exclude_modules: null
(skyrl_entrypoint pid=1617955) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=1617955) optimizer_config:
(skyrl_entrypoint pid=1617955) optimizer: AdamW
(skyrl_entrypoint pid=1617955) lr: 5.0e-06
(skyrl_entrypoint pid=1617955) adam_betas:
(skyrl_entrypoint pid=1617955) - 0.9
(skyrl_entrypoint pid=1617955) - 0.999
(skyrl_entrypoint pid=1617955) weight_decay: 0.01
(skyrl_entrypoint pid=1617955) max_grad_norm: 1.0
(skyrl_entrypoint pid=1617955) offload_after_step: true
(skyrl_entrypoint pid=1617955) num_warmup_steps: 0
(skyrl_entrypoint pid=1617955) scheduler: constant_with_warmup
(skyrl_entrypoint pid=1617955) optimizer_kwargs: {}
(skyrl_entrypoint pid=1617955) fsdp_config:
(skyrl_entrypoint pid=1617955) cpu_offload: false
(skyrl_entrypoint pid=1617955) reshard_after_forward: true
(skyrl_entrypoint pid=1617955) fsdp_size: -1
(skyrl_entrypoint pid=1617955) expert_model_parallel_size: 1
(skyrl_entrypoint pid=1617955) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=1617955) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=1617955) moe_router_replay: false
(skyrl_entrypoint pid=1617955) moe_grouped_gemm: false
(skyrl_entrypoint pid=1617955) ep_comm_backend: torch
(skyrl_entrypoint pid=1617955) deepep_num_sms: 20
(skyrl_entrypoint pid=1617955) deepep_token_chunk_size: null
(skyrl_entrypoint pid=1617955) sequence_parallel_size: 1
(skyrl_entrypoint pid=1617955) algorithm:
(skyrl_entrypoint pid=1617955) advantage_estimator: rloo_n
(skyrl_entrypoint pid=1617955) kl_ctrl:
(skyrl_entrypoint pid=1617955) type: fixed
(skyrl_entrypoint pid=1617955) kl_target: 0.1
(skyrl_entrypoint pid=1617955) horizon: 10000
(skyrl_entrypoint pid=1617955) kl_estimator_type: k3
(skyrl_entrypoint pid=1617955) use_kl_estimator_k3: false
(skyrl_entrypoint pid=1617955) use_abs_kl: false
(skyrl_entrypoint pid=1617955) use_kl_in_reward: false
(skyrl_entrypoint pid=1617955) use_kl_loss: false
(skyrl_entrypoint pid=1617955) kl_loss_coef: 0.0
(skyrl_entrypoint pid=1617955) use_entropy_loss: false
(skyrl_entrypoint pid=1617955) entropy_loss_coef: 0.01
(skyrl_entrypoint pid=1617955) advantage_batch_normalize: false
(skyrl_entrypoint pid=1617955) value_head_prefix: value_head
(skyrl_entrypoint pid=1617955) policy_loss_type: regular
(skyrl_entrypoint pid=1617955) loss_reduction: sequence_mean
(skyrl_entrypoint pid=1617955) global_loss_denom: null
(skyrl_entrypoint pid=1617955) grpo_norm_by_std: true
(skyrl_entrypoint pid=1617955) rloo_n_min_group_size: 4
(skyrl_entrypoint pid=1617955) rloo_n_filter_zero_reward_groups: true
(skyrl_entrypoint pid=1617955) lambd: 1.0
(skyrl_entrypoint pid=1617955) gamma: 1.0
(skyrl_entrypoint pid=1617955) eps_clip_low: 0.2
(skyrl_entrypoint pid=1617955) eps_clip_high: 0.05
(skyrl_entrypoint pid=1617955) clip_ratio_c: 3.0
(skyrl_entrypoint pid=1617955) tis_imp_ratio_cap: 2.0
(skyrl_entrypoint pid=1617955) use_tis: true
(skyrl_entrypoint pid=1617955) sapo:
(skyrl_entrypoint pid=1617955) tau_pos: 1.0
(skyrl_entrypoint pid=1617955) tau_neg: 1.05
(skyrl_entrypoint pid=1617955) value_clip: 0.2
(skyrl_entrypoint pid=1617955) dynamic_sampling:
(skyrl_entrypoint pid=1617955) type: null
(skyrl_entrypoint pid=1617955) max_sample_batches: 30
(skyrl_entrypoint pid=1617955) min_replace_ratio: 0.3
(skyrl_entrypoint pid=1617955) clip_cov:
(skyrl_entrypoint pid=1617955) clip_ratio: 0.0002
(skyrl_entrypoint pid=1617955) clip_cov_lb: 1.0
(skyrl_entrypoint pid=1617955) clip_cov_ub: 5.0
(skyrl_entrypoint pid=1617955) kl_cov:
(skyrl_entrypoint pid=1617955) kl_cov_frac: 0.2
(skyrl_entrypoint pid=1617955) ppo_kl_coef: 1.0
(skyrl_entrypoint pid=1617955) cispo:
(skyrl_entrypoint pid=1617955) cispo_eps_clip_low: 0
(skyrl_entrypoint pid=1617955) cispo_eps_clip_high: 5
(skyrl_entrypoint pid=1617955) z_clip:
(skyrl_entrypoint pid=1617955) enabled: false
(skyrl_entrypoint pid=1617955) alpha: 0.97
(skyrl_entrypoint pid=1617955) z_thresh: 2.5
(skyrl_entrypoint pid=1617955) warmup_steps: 25
(skyrl_entrypoint pid=1617955) mode: zscore
(skyrl_entrypoint pid=1617955) clip_option: adaptive_scaling
(skyrl_entrypoint pid=1617955) clip_factor: 1.0
(skyrl_entrypoint pid=1617955) skip_update_on_spike: false
(skyrl_entrypoint pid=1617955) stale_clip:
(skyrl_entrypoint pid=1617955) enabled: false
(skyrl_entrypoint pid=1617955) alpha: 0.3
(skyrl_entrypoint pid=1617955) entropy_threshold: 0.15
(skyrl_entrypoint pid=1617955) entropy_window: 10
(skyrl_entrypoint pid=1617955) min_lr_scale: 0.1
(skyrl_entrypoint pid=1617955) max_seq_len: 1004095
(skyrl_entrypoint pid=1617955) fully_async:
(skyrl_entrypoint pid=1617955) max_staleness_steps: 16
(skyrl_entrypoint pid=1617955) num_parallel_generation_workers: 338
(skyrl_entrypoint pid=1617955) gradient_checkpointing: true
(skyrl_entrypoint pid=1617955) gradient_checkpointing_use_reentrant: false
(skyrl_entrypoint pid=1617955) seed: 42
(skyrl_entrypoint pid=1617955) resume_mode: latest
(skyrl_entrypoint pid=1617955) resume_path: null
(skyrl_entrypoint pid=1617955) ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/checkpoints
(skyrl_entrypoint pid=1617955) max_ckpts_to_keep: -1
(skyrl_entrypoint pid=1617955) ckpt_interval: 2
(skyrl_entrypoint pid=1617955) hf_save_interval: 5
(skyrl_entrypoint pid=1617955) hf_upload_mode: latest
(skyrl_entrypoint pid=1617955) export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/exports
(skyrl_entrypoint pid=1617955) bf16: true
(skyrl_entrypoint pid=1617955) epochs: 2
(skyrl_entrypoint pid=1617955) max_steps: 80
(skyrl_entrypoint pid=1617955) update_epochs_per_batch: 1
(skyrl_entrypoint pid=1617955) train_batch_size: 64
(skyrl_entrypoint pid=1617955) policy_mini_batch_size: 64
(skyrl_entrypoint pid=1617955) critic_mini_batch_size: 256
(skyrl_entrypoint pid=1617955) micro_train_batch_size_per_gpu: 1
(skyrl_entrypoint pid=1617955) micro_forward_batch_size_per_gpu: 4
(skyrl_entrypoint pid=1617955) update_ref_every_epoch: false
(skyrl_entrypoint pid=1617955) use_sample_packing: true
(skyrl_entrypoint pid=1617955) eval_batch_size: 64
(skyrl_entrypoint pid=1617955) eval_before_train: false
(skyrl_entrypoint pid=1617955) eval_interval: 999999
(skyrl_entrypoint pid=1617955) max_prompt_length: 999999
(skyrl_entrypoint pid=1617955) flash_attn: true
(skyrl_entrypoint pid=1617955) disable_fast_tokenizer: false
(skyrl_entrypoint pid=1617955) target_modules: null
(skyrl_entrypoint pid=1617955) exclude_modules: null
(skyrl_entrypoint pid=1617955) project_name: OpenThoughts-Agent
(skyrl_entrypoint pid=1617955) run_name: ablation-pymethods2test-seqmean-arm0-tis
(skyrl_entrypoint pid=1617955) logger: console
(skyrl_entrypoint pid=1617955) tracker_commit_each_step: true
(skyrl_entrypoint pid=1617955) dump_data_batch: false
(skyrl_entrypoint pid=1617955) dump_eval_results: true
(skyrl_entrypoint pid=1617955) log_level: INFO
(skyrl_entrypoint pid=1617955) rope_scaling: null
(skyrl_entrypoint pid=1617955) rope_theta: null
(skyrl_entrypoint pid=1617955) step_wise_training: false
(skyrl_entrypoint pid=1617955) hf_hub_repo_id: laion/ablation-pymethods2test-seqmean-arm0-tis
(skyrl_entrypoint pid=1617955) hf_hub_private: false
(skyrl_entrypoint pid=1617955) hf_hub_revision: main
(skyrl_entrypoint pid=1617955) enable_db_registration: false
(skyrl_entrypoint pid=1617955) generator:
(skyrl_entrypoint pid=1617955) model_name: ${trainer.policy.model.path}
(skyrl_entrypoint pid=1617955) model_dtype: bfloat16
(skyrl_entrypoint pid=1617955) timeout_multiplier: 1.0
(skyrl_entrypoint pid=1617955) run_engines_locally: true
(skyrl_entrypoint pid=1617955) num_inference_engines: 48
(skyrl_entrypoint pid=1617955) backend: vllm
(skyrl_entrypoint pid=1617955) weight_sync_backend: nccl
(skyrl_entrypoint pid=1617955) fuse_weights: false
(skyrl_entrypoint pid=1617955) weight_transfer_threshold_cuda_ipc_GB: 1.0
(skyrl_entrypoint pid=1617955) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=1617955) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=1617955) inference_engine_expert_parallel_size: 1
(skyrl_entrypoint pid=1617955) inference_engine_data_parallel_size: 1
(skyrl_entrypoint pid=1617955) n_samples_per_prompt: 8
(skyrl_entrypoint pid=1617955) async_engine: true
(skyrl_entrypoint pid=1617955) batched: false
(skyrl_entrypoint pid=1617955) max_input_length: ${trainer.max_prompt_length}
(skyrl_entrypoint pid=1617955) vllm_v1_disable_multiproc: true
(skyrl_entrypoint pid=1617955) enable_prefix_caching: true
(skyrl_entrypoint pid=1617955) enable_chunked_prefill: true
(skyrl_entrypoint pid=1617955) max_num_batched_tokens: 65536
(skyrl_entrypoint pid=1617955) enforce_eager: true
(skyrl_entrypoint pid=1617955) fully_sharded_loras: false
(skyrl_entrypoint pid=1617955) enable_ray_prometheus_stats: false
(skyrl_entrypoint pid=1617955) vllm_stats_interval: 1
(skyrl_entrypoint pid=1617955) gpu_memory_utilization: 0.75
(skyrl_entrypoint pid=1617955) max_num_seqs: 24
(skyrl_entrypoint pid=1617955) remote_inference_engine_urls:
(skyrl_entrypoint pid=1617955) - 127.0.0.1:8001
(skyrl_entrypoint pid=1617955) enable_http_endpoint: true
(skyrl_entrypoint pid=1617955) http_endpoint_host: 127.0.0.1
(skyrl_entrypoint pid=1617955) http_endpoint_port: 8000
(skyrl_entrypoint pid=1617955) max_turns: 999999
(skyrl_entrypoint pid=1617955) chat_template:
(skyrl_entrypoint pid=1617955) source: name
(skyrl_entrypoint pid=1617955) name_or_path: null
(skyrl_entrypoint pid=1617955) chat_template_kwargs: {}
(skyrl_entrypoint pid=1617955) engine_init_kwargs:
(skyrl_entrypoint pid=1617955) max_model_len: 32768
(skyrl_entrypoint pid=1617955) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=1617955) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=1617955) override_existing_update_group: disable
(skyrl_entrypoint pid=1617955) sampling_params:
(skyrl_entrypoint pid=1617955) max_generate_length: 4096
(skyrl_entrypoint pid=1617955) repetition_penalty: 1.0
(skyrl_entrypoint pid=1617955) temperature: 0.7
(skyrl_entrypoint pid=1617955) top_p: 0.95
(skyrl_entrypoint pid=1617955) min_p: 0.0
(skyrl_entrypoint pid=1617955) top_k: 20
(skyrl_entrypoint pid=1617955) logprobs: 0
(skyrl_entrypoint pid=1617955) stop: null
(skyrl_entrypoint pid=1617955) use_conversation_multi_turn: true
(skyrl_entrypoint pid=1617955) append_eos_token_after_stop_str_in_multi_turn: true
(skyrl_entrypoint pid=1617955) eval_sampling_params:
(skyrl_entrypoint pid=1617955) max_generate_length: ${generator.sampling_params.max_generate_length}
(skyrl_entrypoint pid=1617955) repetition_penalty: 1.0
(skyrl_entrypoint pid=1617955) temperature: 0.0
(skyrl_entrypoint pid=1617955) top_p: 1.0
(skyrl_entrypoint pid=1617955) min_p: 0.0
(skyrl_entrypoint pid=1617955) top_k: -1
(skyrl_entrypoint pid=1617955) logprobs: null
(skyrl_entrypoint pid=1617955) stop: null
(skyrl_entrypoint pid=1617955) eval_n_samples_per_prompt: 8
(skyrl_entrypoint pid=1617955) zero_reward_on_non_stop: false
(skyrl_entrypoint pid=1617955) apply_overlong_filtering: false
(skyrl_entrypoint pid=1617955) rope_scaling: ${trainer.rope_scaling}
(skyrl_entrypoint pid=1617955) rope_theta: ${trainer.rope_theta}
(skyrl_entrypoint pid=1617955) teacher:
(skyrl_entrypoint pid=1617955) model_path: null
(skyrl_entrypoint pid=1617955) top_k_logprobs: 256
(skyrl_entrypoint pid=1617955) num_inference_engines: 1
(skyrl_entrypoint pid=1617955) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=1617955) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=1617955) gpu_memory_utilization: 0.9
(skyrl_entrypoint pid=1617955) enforce_eager: false
(skyrl_entrypoint pid=1617955) backend: vllm
(skyrl_entrypoint pid=1617955) engine_init_kwargs: {}
(skyrl_entrypoint pid=1617955) environment:
(skyrl_entrypoint pid=1617955) env_class: gsm8k
(skyrl_entrypoint pid=1617955) skyrl_gym:
(skyrl_entrypoint pid=1617955) max_env_workers: 32
(skyrl_entrypoint pid=1617955) text2sql:
(skyrl_entrypoint pid=1617955) db_path: /home/ray/default/sql_data
(skyrl_entrypoint pid=1617955) llm_as_a_judge:
(skyrl_entrypoint pid=1617955) model: gpt-4o-mini
(skyrl_entrypoint pid=1617955) base_url: null
(skyrl_entrypoint pid=1617955) search:
(skyrl_entrypoint pid=1617955) log_requests: false
(skyrl_entrypoint pid=1617955) search_url: http://127.0.0.1:8000/retrieve
(skyrl_entrypoint pid=1617955) topk: 3
(skyrl_entrypoint pid=1617955) timeout: 30
(skyrl_entrypoint pid=1617955) rollout:
(skyrl_entrypoint pid=1617955) fanout:
(skyrl_entrypoint pid=1617955) enabled: true
(skyrl_entrypoint pid=1617955) num_coordinators: 4
(skyrl_entrypoint pid=1617955) cpus_per_coordinator: 8
(skyrl_entrypoint pid=1617955) deepspeed_config:
(skyrl_entrypoint pid=1617955) train:
(skyrl_entrypoint pid=1617955) zero_optimization:
(skyrl_entrypoint pid=1617955) stage: 3
(skyrl_entrypoint pid=1617955) offload_param:
(skyrl_entrypoint pid=1617955) device: none
(skyrl_entrypoint pid=1617955) offload_optimizer:
(skyrl_entrypoint pid=1617955) device: none
(skyrl_entrypoint pid=1617955) pin_memory: true
(skyrl_entrypoint pid=1617955) sub_group_size: auto
(skyrl_entrypoint pid=1617955) reduce_bucket_size: auto
(skyrl_entrypoint pid=1617955) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=1617955) stage3_prefetch_bucket_size: auto
(skyrl_entrypoint pid=1617955) stage3_max_live_parameters: auto
(skyrl_entrypoint pid=1617955) stage3_max_reuse_distance: auto
(skyrl_entrypoint pid=1617955) round_robin_gradients: true
(skyrl_entrypoint pid=1617955) zero_hpz_partition_size: 1
(skyrl_entrypoint pid=1617955) zero_quantized_weights: false
(skyrl_entrypoint pid=1617955) zero_quantized_gradients: false
(skyrl_entrypoint pid=1617955) torch_autocast:
(skyrl_entrypoint pid=1617955) enabled: true
(skyrl_entrypoint pid=1617955) dtype: bfloat16
(skyrl_entrypoint pid=1617955) disable_trace_cache: false
(skyrl_entrypoint pid=1617955) data_types:
(skyrl_entrypoint pid=1617955) grad_accum_dtype: fp32
(skyrl_entrypoint pid=1617955) gradient_clipping: 1.0
(skyrl_entrypoint pid=1617955) wall_clock_breakdown: false
(skyrl_entrypoint pid=1617955) prescale_gradient: false
(skyrl_entrypoint pid=1617955) eval:
(skyrl_entrypoint pid=1617955) zero_optimization:
(skyrl_entrypoint pid=1617955) stage: 3
(skyrl_entrypoint pid=1617955) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=1617955) offload_param:
(skyrl_entrypoint pid=1617955) device: cpu
(skyrl_entrypoint pid=1617955) pin_memory: true
(skyrl_entrypoint pid=1617955) torch_autocast:
(skyrl_entrypoint pid=1617955) enabled: true
(skyrl_entrypoint pid=1617955) dtype: bfloat16
(skyrl_entrypoint pid=1617955) gradient_clipping: 1.0
(skyrl_entrypoint pid=1617955) prescale_gradient: false
(skyrl_entrypoint pid=1617955) wall_clock_breakdown: false
(skyrl_entrypoint pid=1617955) terminal_bench_config:
(skyrl_entrypoint pid=1617955) trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs
(skyrl_entrypoint pid=1617955) harbor:
(skyrl_entrypoint pid=1617955) name: terminus-2
(skyrl_entrypoint pid=1617955) max_episodes: 999999
(skyrl_entrypoint pid=1617955) enable_summarize: false
(skyrl_entrypoint pid=1617955) store_all_messages: true
(skyrl_entrypoint pid=1617955) trajectory_config:
(skyrl_entrypoint pid=1617955) raw_content: true
(skyrl_entrypoint pid=1617955) enable_episode_logging: false
(skyrl_entrypoint pid=1617955) record_terminal_session: false
(skyrl_entrypoint pid=1617955) enable_pane_logging: false
(skyrl_entrypoint pid=1617955) strict_json_parser: true
(skyrl_entrypoint pid=1617955) interleaved_thinking: true
(skyrl_entrypoint pid=1617955) extra_body:
(skyrl_entrypoint pid=1617955) chat_template_kwargs:
(skyrl_entrypoint pid=1617955) enable_thinking: true
(skyrl_entrypoint pid=1617955) override_timeout_sec: 900
(skyrl_entrypoint pid=1617955) override_cpus: 1
(skyrl_entrypoint pid=1617955) override_memory_mb: 2048
(skyrl_entrypoint pid=1617955) override_storage_mb: 2048
(skyrl_entrypoint pid=1617955) auto_snapshot: true
(skyrl_entrypoint pid=1617955) verifier_override_timeout_sec: 120
(skyrl_entrypoint pid=1617955) max_retries: 3
(skyrl_entrypoint pid=1617955) min_wait_sec: 60.0
(skyrl_entrypoint pid=1617955) max_wait_sec: 600.0
(skyrl_entrypoint pid=1617955) wait_multiplier: 2.0
(skyrl_entrypoint pid=1617955) exclude_exceptions:
(skyrl_entrypoint pid=1617955) - VerifierTimeoutError
(skyrl_entrypoint pid=1617955) - VerifierRuntimeError
(skyrl_entrypoint pid=1617955) - RewardFileNotFoundError
(skyrl_entrypoint pid=1617955) - RewardFileEmptyError
(skyrl_entrypoint pid=1617955) - VerifierOutputParseError
(skyrl_entrypoint pid=1617955) n_concurrent_trials: 675
(skyrl_entrypoint pid=1617955) log_level: INFO
(skyrl_entrypoint pid=1617955) enable_reward_shaping: false
(skyrl_entrypoint pid=1617955) collect_rollout_details: true
(skyrl_entrypoint pid=1617955) enable_error_classification: true
(skyrl_entrypoint pid=1617955) mask_exceptions:
(skyrl_entrypoint pid=1617955) - DaytonaError
(skyrl_entrypoint pid=1617955) - EnvironmentStartTimeoutError
(skyrl_entrypoint pid=1617955) - NetworkError
(skyrl_entrypoint pid=1617955) - ConnectionError
(skyrl_entrypoint pid=1617955) - RewardFileNotFoundError
(skyrl_entrypoint pid=1617955) - RewardFileEmptyError
(skyrl_entrypoint pid=1617955) - AgentEnvironmentTimeoutError
(skyrl_entrypoint pid=1617955) - ContextLengthExceededError
(skyrl_entrypoint pid=1617955) default_error_treatment: zero
(skyrl_entrypoint pid=1617955) passthrough_exceptions:
(skyrl_entrypoint pid=1617955) - AgentTimeoutError
(skyrl_entrypoint pid=1617955) zero_exceptions: []
(skyrl_entrypoint pid=1617955) model_info:
(skyrl_entrypoint pid=1617955) max_input_tokens: 32000
(skyrl_entrypoint pid=1617955) max_output_tokens: 4096
(skyrl_entrypoint pid=1617955) archiving:
(skyrl_entrypoint pid=1617955) enabled: false
(skyrl_entrypoint pid=1617955) trace_upload:
(skyrl_entrypoint pid=1617955) enabled: true
(skyrl_entrypoint pid=1617955) repo_org: DCAgent
(skyrl_entrypoint pid=1617955) episodes: last
(skyrl_entrypoint pid=1617955) dataset_type: SFT
(skyrl_entrypoint pid=1617955) cleanup: true
(skyrl_entrypoint pid=1617955) 
(RegistryActor pid=1354131, ip=10.128.16.45) [2026-06-07 15:37:38,501 E 1354131 1354171] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(skyrl_entrypoint pid=1617955) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(skyrl_entrypoint pid=1617955) No module named 'vllm._version'
(skyrl_entrypoint pid=1617955) from .version import __version__, __version_tuple__ # isort:skip
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 12x across cluster]
(skyrl_entrypoint pid=1617955) [2026-06-07 15:37:40,589 E 1617955 1617998] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster]
(skyrl_entrypoint pid=1617955) W0607 15:42:21.650000 1617955 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(get_all_env_variables pid=1618274) [2026-06-07 15:44:18,568 E 1618274 1618314] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 29x across cluster]
(pid=1354512, ip=10.128.16.45) [2026-06-07 15:44:20,995 E 1354512 1354615] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(raylet, ip=10.128.16.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 2x across cluster]
(raylet, ip=10.128.16.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 10x across cluster]
(pid=1354512, ip=10.128.16.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1354512, ip=10.128.16.45) No module named 'vllm._version'
(pid=1354512, ip=10.128.16.45) from .version import __version__, __version_tuple__ # isort:skip
(get_addr_port pid=1516023, ip=10.128.16.42) [2026-06-07 15:45:41,920 E 1516023 1516098] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 2x across cluster]
(pid=1516024, ip=10.128.16.42) [2026-06-07 15:45:41,966 E 1516024 1516126] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=1516170, ip=10.128.16.42) [2026-06-07 15:47:31,328 E 1516170 1516245] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 22x across cluster]
(pid=1496715, ip=10.128.16.37) [2026-06-07 15:47:31,482 E 1496715 1496817] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(raylet, ip=10.128.16.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
(pid=1497002, ip=10.128.16.37) [2026-06-07 15:52:22,486 E 1497002 1497139] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 15x across cluster]
(pid=1497001, ip=10.128.16.37) [2026-06-07 15:52:22,439 E 1497001 1497111] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:25.238 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:25.239 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:25.239 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=1497003, ip=10.128.16.37) [2026-06-07 15:52:22,695 E 1497003 1497167] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:25.845 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:26.289 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:26.289 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:26.289 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:26.289 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(raylet, ip=10.128.16.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(pid=1486338, ip=10.128.16.41) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1486338, ip=10.128.16.41) No module named 'vllm._version'
(pid=1486338, ip=10.128.16.41) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:50.076 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:50.077 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:50.077 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(raylet, ip=10.128.16.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 53x across cluster]
(pid=1354981, ip=10.128.16.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 6x across cluster]
(pid=1354981, ip=10.128.16.45) No module named 'vllm._version' [repeated 6x across cluster]
(pid=1354981, ip=10.128.16.45) from .version import __version__, __version_tuple__ # isort:skip [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:51.035 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:51.038 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:51.038 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:51.038 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) 2026-06-07 15:53:51.038 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:53:52.539 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.11s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:50.076 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:50.077 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:50.077 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:51.025 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:51.029 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:51.029 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:51.029 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:51.030 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) [2026-06-07 15:54:10,417 E 1354979 1355118] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:53:52.539 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.70s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) No module named 'vllm._version' [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) from .version import __version__, __version_tuple__ # isort:skip [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) _C._set_float32_matmul_precision(precision)
(pid=1486805, ip=10.128.16.41) [2026-06-07 15:54:10,667 E 1486805 1486970] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) (EngineCore_DP0 pid=1355199) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) [W607 15:55:02.757090169 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-45-interconnect-1.jupiter.internal]:51011 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) [W607 15:55:02.760273491 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) [rank0]:[W607 15:55:02.766688838 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) [W607 15:55:02.757164023 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-45.jupiter.internal]:44505 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) [W607 15:55:02.760282259 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) [rank0]:[W607 15:55:02.766374060 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.20s/it]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.26s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204)
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) (EngineCore_DP0 pid=1355199)
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215)
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219)
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00, 3.54s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) 2026-06-07 15:56:14.991 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(pid=1516024, ip=10.128.16.42) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1516024, ip=10.128.16.42) No module named 'vllm._version'
(pid=1516024, ip=10.128.16.42) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) 2026-06-07 15:56:15.026 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(raylet, ip=10.128.16.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(pid=1516547, ip=10.128.16.42) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=1516547, ip=10.128.16.42) No module named 'vllm._version' [repeated 3x across cluster]
(pid=1516547, ip=10.128.16.42) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(get_addr_port pid=1538389, ip=10.128.16.47) [2026-06-07 15:58:34,128 E 1538389 1538463] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 29x across cluster]
(pid=1561915, ip=10.128.16.46) [2026-06-07 15:58:34,380 E 1561915 1562081] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(raylet, ip=10.128.16.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
(pid=1538388, ip=10.128.16.47) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1538388, ip=10.128.16.47) No module named 'vllm._version'
(pid=1538388, ip=10.128.16.47) from .version import __version__, __version_tuple__ # isort:skip
(get_addr_port pid=2087413, ip=10.128.16.133) [2026-06-07 16:00:47,642 E 2087413 2087489] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.133) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 27x across cluster]
(pid=1538603, ip=10.128.16.47) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=1538603, ip=10.128.16.47) No module named 'vllm._version' [repeated 3x across cluster]
(pid=1538603, ip=10.128.16.47) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(pid=2087414, ip=10.128.16.133) [2026-06-07 16:00:47,689 E 2087414 2087517] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:12.660 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:12.661 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:12.661 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=1538603, ip=10.128.16.47) [2026-06-07 16:00:47,874 E 1538603 1538768] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:13.898 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:14.213 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:14.213 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:14.214 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:14.214 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:03:14.287 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.95s (attempt 1/5) to avoid port collisions
(raylet, ip=10.128.16.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:12.660 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:12.661 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:12.661 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:13.898 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:14.213 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:14.213 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:14.213 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:14.213 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) 2026-06-07 16:03:14.287 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.51s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(raylet, ip=10.128.16.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 3x across cluster]
(raylet, ip=10.128.16.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 2x across cluster]
(get_addr_port pid=1204669, ip=10.128.16.43) [2026-06-07 16:04:20,488 E 1204669 1204744] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(raylet, ip=10.128.16.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 22x across cluster]
(pid=1204670, ip=10.128.16.43) [2026-06-07 16:04:20,535 E 1204670 1204772] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(pid=1204670, ip=10.128.16.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1204670, ip=10.128.16.43) No module named 'vllm._version'
(pid=1204670, ip=10.128.16.43) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 2x across cluster]
(pid=1204805, ip=10.128.16.43) [2026-06-07 16:04:48,441 E 1204805 1204857] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(pid=1204805, ip=10.128.16.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 2x across cluster]
(pid=1204805, ip=10.128.16.43) No module named 'vllm._version' [repeated 2x across cluster]
(pid=1204805, ip=10.128.16.43) from .version import __version__, __version_tuple__ # isort:skip [repeated 2x across cluster]
(pid=1495134, ip=10.128.16.36) [2026-06-07 16:04:48,804 E 1495134 1495237] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) No module named 'vllm._version' [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41) (EngineCore_DP0 pid=1487383) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41) (EngineCore_DP0 pid=1487374) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) [W607 16:05:55.641875132 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-41-interconnect-1.jupiter.internal]:48789 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) [W607 16:05:55.643733206 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) [rank0]:[W607 16:05:55.649849626 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) [W607 16:05:55.641934203 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-41-interconnect-1.jupiter.internal]:48427 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) [W607 16:05:55.643746806 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) [rank0]:[W607 16:05:55.649674461 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(pid=1496715, ip=10.128.16.37) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1496715, ip=10.128.16.37) No module named 'vllm._version'
(pid=1496715, ip=10.128.16.37) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.16.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:05<00:17, 5.96s/it]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(pid=1495334, ip=10.128.16.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(pid=1495334, ip=10.128.16.36) No module named 'vllm._version' [repeated 7x across cluster]
(pid=1495334, ip=10.128.16.36) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(raylet, ip=10.128.16.48) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 29x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:13<00:04, 4.39s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375)
(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41) (EngineCore_DP0 pid=1487383)
(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41) (EngineCore_DP0 pid=1487374)
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379)
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:15<00:00, 3.82s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) 2026-06-07 16:06:23.164 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(pid=1495335, ip=10.128.16.36) [2026-06-07 16:06:35,975 E 1495335 1495447] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41) 2026-06-07 16:06:23.183 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(raylet, ip=10.128.16.48) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(pid=1495334, ip=10.128.16.36) [2026-06-07 16:06:36,202 E 1495334 1495503] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(pid=1568271, ip=10.128.16.48) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1568271, ip=10.128.16.48) No module named 'vllm._version'
(pid=1568271, ip=10.128.16.48) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.16.133) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 53x across cluster]
(pid=1205099, ip=10.128.16.43) [2026-06-07 16:07:58,673 E 1205099 1205139] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=1568417, ip=10.128.16.48) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(pid=1568417, ip=10.128.16.48) No module named 'vllm._version' [repeated 4x across cluster]
(pid=1568417, ip=10.128.16.48) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(pid=2087857, ip=10.128.16.133) [2026-06-07 16:07:58,681 E 2087857 2087966] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(pid=1561695, ip=10.128.16.46) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1561695, ip=10.128.16.46) No module named 'vllm._version'
(pid=1561695, ip=10.128.16.46) from .version import __version__, __version_tuple__ # isort:skip
(pid=2087855, ip=10.128.16.133) [2026-06-07 16:07:58,946 E 2087855 2088022] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 7x across cluster]
(pid=1561916, ip=10.128.16.46) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1561916, ip=10.128.16.46) No module named 'vllm._version'
(pid=1561916, ip=10.128.16.46) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:32.006 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:32.007 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:32.007 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=1561915, ip=10.128.16.46) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 2x across cluster]
(pid=1561915, ip=10.128.16.46) No module named 'vllm._version' [repeated 2x across cluster]
(pid=1561915, ip=10.128.16.46) from .version import __version__, __version_tuple__ # isort:skip [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:33.258 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:33.747 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:33.747 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:33.747 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:33.747 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:11:33.788 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.84s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:32.006 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:32.007 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:32.007 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:33.258 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:33.746 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:33.746 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:33.746 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:33.746 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) 2026-06-07 16:11:33.787 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.53s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(raylet, ip=10.128.16.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) No module named 'vllm._version' [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) from .version import __version__, __version_tuple__ # isort:skip [repeated 2x across cluster]
(get_addr_port pid=1446435, ip=10.128.16.44) [2026-06-07 16:12:25,922 E 1446435 1446509] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.16.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) _C._set_float32_matmul_precision(precision)
(pid=1211472, ip=10.128.16.38) [2026-06-07 16:12:26,041 E 1211472 1211634] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) (EngineCore_DP0 pid=1539228) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) [W607 16:13:11.271167533 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-47.jupiter.internal]:59713 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) [W607 16:13:11.272737868 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) [rank0]:[W607 16:13:11.277554377 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) [W607 16:13:11.615631887 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-47.jupiter.internal]:60611 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) [W607 16:13:11.616170660 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) [rank0]:[W607 16:13:11.618501524 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(raylet, ip=10.128.16.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(skyrl_entrypoint pid=1617955) [2026-06-07 16:13:41] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
(skyrl_entrypoint pid=1617955) [2026-06-07 16:13:41] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
(skyrl_entrypoint pid=1617955) [2026-06-07 16:13:42] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.555 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:969 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.556 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines.
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.558 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:242 - Harbor logging level set to INFO
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.559 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:142 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.559 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:158 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.560 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 156
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.560 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:357 - Length of train_dataloader: 5000
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.560 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:358 - Number of steps per epoch: 78
(skyrl_entrypoint pid=1617955) 2026-06-07 16:13:42.560 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:359 - Total training steps: 80
(pid=1446434, ip=10.128.16.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1446434, ip=10.128.16.44) No module named 'vllm._version'
(pid=1446434, ip=10.128.16.44) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.14s/it]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(raylet, ip=10.128.16.135) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 24x across cluster]
(pid=1446661, ip=10.128.16.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=1446661, ip=10.128.16.44) No module named 'vllm._version' [repeated 3x across cluster]
(pid=1446661, ip=10.128.16.44) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.17s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238)
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) (EngineCore_DP0 pid=1539228)
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233)
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220)
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) 2026-06-07 16:13:56.813 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.45s/it] [repeated 11x across cluster]
(pid=1446659, ip=10.128.16.44) [2026-06-07 16:14:12,531 E 1446659 1446769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) 2026-06-07 16:13:56.839 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(pid=2087414, ip=10.128.16.133) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2087414, ip=10.128.16.133) No module named 'vllm._version'
(pid=2087414, ip=10.128.16.133) from .version import __version__, __version_tuple__ # isort:skip
(pid=3137116, ip=10.128.16.135) [2026-06-07 16:14:13,514 E 3137116 3137156] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:20.561 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:20.562 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:20.562 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(pid=2087855, ip=10.128.16.133) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=2087855, ip=10.128.16.133) No module named 'vllm._version' [repeated 3x across cluster]
(pid=2087855, ip=10.128.16.133) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:21.512 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:21.808 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:21.808 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:21.808 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:21.808 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:16:21.855 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.69s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:20.562 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:20.563 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:20.563 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:21.988 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:21.992 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:21.992 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:21.992 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:21.992 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:16:22.018 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.57s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:15.622 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:15.624 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:15.624 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) 2026-06-07 16:17:15.622 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) 2026-06-07 16:17:15.624 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) 2026-06-07 16:17:15.624 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:16.902 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:17.271 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:17.271 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:17.271 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:17.271 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:17:17.325 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.27s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) 2026-06-07 16:17:58.363 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) 2026-06-07 16:17:58.364 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) 2026-06-07 16:17:58.364 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) 2026-06-07 16:17:16.902 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) 2026-06-07 16:17:17.270 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) 2026-06-07 16:17:17.271 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) 2026-06-07 16:17:17.271 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) 2026-06-07 16:17:17.271 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) 2026-06-07 16:17:17.325 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.88s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:58.363 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:58.364 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:58.364 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:59.597 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:59.625 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:59.625 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:59.625 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:59.626 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:17:59.672 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.04s (attempt 1/5) to avoid port collisions [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 9x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(pid=3137116, ip=10.128.16.135) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) No module named 'vllm._version' [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48) (EngineCore_DP0 pid=1568977) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) [W607 16:20:38.111142654 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-48-interconnect-1.jupiter.internal]:53891 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) [W607 16:20:38.111310683 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48) [rank0]:[W607 16:20:38.116523344 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.26s/it]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) [W607 16:20:38.456774049 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-48-interconnect-1.jupiter.internal]:34821 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) [W607 16:20:38.457232632 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) [rank0]:[W607 16:20:38.459531497 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.08s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980)
(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48) (EngineCore_DP0 pid=1568977)
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978)
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976)
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) 2026-06-07 16:20:57.866 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.43s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) 2026-06-07 16:20:57.786 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) (EngineCore_DP0 pid=1495936) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36) (EngineCore_DP0 pid=1495912) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) [W607 16:22:42.320690462 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-36.jupiter.internal]:36041 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) [W607 16:22:42.324212382 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) [rank0]:[W607 16:22:42.330336336 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(pid=1211190, ip=10.128.16.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=1211190, ip=10.128.16.38) No module named 'vllm._version'
(pid=1211190, ip=10.128.16.38) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) [W607 16:22:42.320619263 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-36-interconnect-1.jupiter.internal]:36069 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) [W607 16:22:42.324226270 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) [rank0]:[W607 16:22:42.330042997 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) (EngineCore_DP0 pid=1205537) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43) (EngineCore_DP0 pid=1205529) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(pid=1211472, ip=10.128.16.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(pid=1211472, ip=10.128.16.38) No module named 'vllm._version' [repeated 3x across cluster]
(pid=1211472, ip=10.128.16.38) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) [W607 16:22:51.540094544 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-43-interconnect-1.jupiter.internal]:50323 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) [W607 16:22:51.542060717 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) [rank0]:[W607 16:22:51.548624154 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:11, 4.00s/it]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.06s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916)
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) (EngineCore_DP0 pid=1495936)
(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36) (EngineCore_DP0 pid=1495912)
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920)
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) 2026-06-07 16:23:27.000 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.36s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) 2026-06-07 16:23:27.057 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.02s/it]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.21s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535)
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) (EngineCore_DP0 pid=1205537)
(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43) (EngineCore_DP0 pid=1205529)
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528)
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) 2026-06-07 16:24:03.034 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.47s/it] [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:40.671 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:40.672 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:40.672 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43) (EngineCore_DP0 pid=1205529) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) 2026-06-07 16:24:03.085 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:41.853 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:41.880 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:41.881 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:41.881 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:41.881 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) 2026-06-07 16:28:41.928 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.91s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:40.671 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:40.672 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:40.672 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:41.853 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:41.881 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:41.881 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:41.881 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:41.881 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:28:41.928 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.63s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44) (EngineCore_DP0 pid=1447391) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44) (EngineCore_DP0 pid=1447388) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) [W607 16:33:02.639468297 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-44-interconnect-1.jupiter.internal]:57389 (errno: 97 - Address family not supported by protocol).
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) [W607 16:33:02.642952955 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) [rank0]:[W607 16:33:02.648917654 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:04<00:12, 4.08s/it]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) [W607 16:33:02.639681508 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-44.jupiter.internal]:48017 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) [W607 16:33:02.642967739 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) [rank0]:[W607 16:33:02.649384620 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387)
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:12<00:04, 4.09s/it] [repeated 8x across cluster]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387)
(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44) (EngineCore_DP0 pid=1447391)
(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44) (EngineCore_DP0 pid=1447388)
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390)
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44) 2026-06-07 16:33:21.860 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.43s/it] [repeated 11x across cluster]
(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135) 2026-06-07 16:43:51 INFO [ipv4-debug] hostname=jpbo-003-39.jupiter.internal
(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135) 2026-06-07 16:43:51 INFO [ipv4-debug] _global_node.node_ip_address=10.128.16.135
(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135) 2026-06-07 16:43:51 INFO [ipv4-debug] get_node_ip_address()=10.128.16.135
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) 2026-06-07 16:33:22.083 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 3x across cluster]
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(skyrl_entrypoint pid=1617955) 2026-06-07 16:43:51.863 | INFO  | skyrl_train.workers.worker:_initiate_actors:636 - Initializing process group for RayActorGroup
(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135) [W607 16:43:51.249797609 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39-interconnect-1.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol).
(pid=3138189, ip=10.128.16.135) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(pid=1620364) [2026-06-07 16:44:22,856 E 1620364 1620497] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster]
(pid=1620365) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster]
(FSDPPolicyWorkerBase pid=3138189, ip=10.128.16.135) [W607 16:46:13.956081717 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39-interconnect-1.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol).
(FSDPPolicyWorkerBase pid=3138189, ip=10.128.16.135) [W607 16:46:13.956399759 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(pid=1620366) [2026-06-07 16:44:22,896 E 1620366 1620595] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(FSDPPolicyWorkerBase pid=1620366) [W607 16:54:00.377895659 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=3138187, ip=10.128.16.135) [W607 16:46:13.956394959 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 2x across cluster]
2026-06-07 16:54:19.008 | ERROR | __main__:main:134 - Training failed: ray::skyrl_entrypoint() (pid=1617955, ip=10.128.16.35)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 481, in run
trainer = self._setup_trainer()
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 450, in _setup_trainer
trainer.build_models(PolicyWorker, CriticWorker, RefWorker, policy_pg=self.policy_pg)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 757, in build_models
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 511, in __init__
self._initiate_actors(pg, num_gpus_per_actor)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 637, in _initiate_actors
ray.get([actor.init_worker_process_group.remote() for actor in self._actor_handlers])
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(DistStoreError): ray::FSDPPolicyWorkerBase.init_worker_process_group() (pid=3137116, ip=10.128.16.135, actor_id=538f00c3532d78f16f4d391d02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400e17779700>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 149, in init_worker_process_group
torch.distributed.init_process_group(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
func_return = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
store, rank, world_size = next(rendezvous_iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 278, in _env_rendezvous_handler
store = _create_c10d_store(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
return TCPStore(
^^^^^^^^^
torch.distributed.DistStoreError: Timed out after 601 seconds waiting for clients. 4/8 clients joined.
Exception raised from waitForWorkers at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:396 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000ab70c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5e9c9c0 (0x400b025fc9c0 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::waitForWorkers() + 0x350 (0x400b02691410 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x468 (0x400b026918c8 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x109a094 (0x400afc4fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x113236c (0x400afc59236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x5d6d60 (0x400afba36d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x1b7a38 (0xaaaad83d7a38 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #8: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #9: <unknown function> + 0x169f50 (0xaaaad8389f50 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #10: <unknown function> + 0x1682e4 (0xaaaad83882e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #11: <unknown function> + 0x1e0ce8 (0xaaaad8400ce8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #12: <unknown function> + 0x1d7ddc (0xaaaad83f7ddc in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #13: <unknown function> + 0x646b0c (0x400afbaa6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #14: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #15: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #16: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #17: <unknown function> + 0x182bf8 (0xaaaad83a2bf8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #18: <unknown function> + 0x25fd30 (0xaaaad847fd30 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #19: <unknown function> + 0x1b7d20 (0xaaaad83d7d20 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #20: PyObject_Vectorcall + 0x54 (0xaaaad83860e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #21: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #22: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #23: <unknown function> + 0x1835c8 (0xaaaad83a35c8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #24: <unknown function> + 0x84b708 (0x40002c15b708 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #25: <unknown function> + 0x865fe4 (0x40002c175fe4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #26: <unknown function> + 0x823470 (0x40002c133470 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #27: <unknown function> + 0x830110 (0x40002c140110 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #28: <unknown function> + 0x174a60 (0xaaaad8394a60 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #29: PyObject_VectorcallMethod + 0xa4 (0xaaaad8386270 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #30: PyIter_Send + 0xbc (0xaaaad836b62c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #31: <unknown function> + 0xaa04 (0x40002e3caa04 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #32: <unknown function> + 0xbcc4 (0x40002e3cbcc4 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #33: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #34: <unknown function> + 0x28cfac (0xaaaad84acfac in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #35: <unknown function> + 0x1b7b68 (0xaaaad83d7b68 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #37: <unknown function> + 0x83c564 (0x40002c14c564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #38: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #39: <unknown function> + 0x169f88 (0xaaaad8389f88 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #40: <unknown function> + 0x36002c (0xaaaad858002c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #41: <unknown function> + 0x2e4024 (0xaaaad8504024 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #42: <unknown function> + 0x80e00 (0x40002b080e00 in /lib64/libc.so.6)
frame #43: <unknown function> + 0xeb49c (0x40002b0eb49c in /lib64/libc.so.6)
2026-06-07 16:54:19.009 | INFO | __main__:main:137 - Shutting down Ray on head node...
(FSDPPolicyWorkerBase pid=1620365) [W607 16:54:00.377853387 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39-interconnect-1.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol). [repeated 3x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] Started monitoring (every 120s)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:37:18] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:37:18] OK: RSS 1.38 GiB | node mem 171.6/858.0 GiB used (20.0%), avail 686.3 GiB
(skyrl_entrypoint pid=1617955) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:39:18] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:39:18] OK: RSS 1.50 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.0 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:41:18] OK: 51 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:41:18] OK: RSS 1.58 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
(skyrl_entrypoint pid=1617955) INFO 06-07 15:42:21 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:43:18] OK: 51 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:43:18] OK: RSS 1.59 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:45:18] OK: 62 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:45:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:47:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:47:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.8 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:49:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:49:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:51:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:51:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:53:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:53:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(pid=1354512, ip=10.128.16.45) INFO 06-07 15:53:23 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(pid=1354979, ip=10.128.16.45) INFO 06-07 15:53:49 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(pid=1354980, ip=10.128.16.45) INFO 06-07 15:53:49 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) WARNING 06-07 15:53:54 [arg_utils.py:1256] The global random seed is set to 56. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) INFO 06-07 15:53:54 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) INFO 06-07 15:53:54 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) INFO 06-07 15:53:54 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) INFO 06-07 15:53:54 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) INFO 06-07 15:53:54 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) WARNING 06-07 15:53:54 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) INFO 06-07 15:53:54 [vllm.py:846] Cudagraph is disabled under eager mode
(pid=1354981, ip=10.128.16.45) INFO 06-07 15:53:49 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) WARNING 06-07 15:53:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) WARNING 06-07 15:53:54 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) INFO 06-07 15:54:01 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) WARNING 06-07 15:53:55 [arg_utils.py:1256] The global random seed is set to 57. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) INFO 06-07 15:53:55 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) INFO 06-07 15:53:55 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) INFO 06-07 15:53:55 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) INFO 06-07 15:53:55 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) INFO 06-07 15:53:55 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) WARNING 06-07 15:53:55 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) INFO 06-07 15:53:55 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) WARNING 06-07 15:53:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) WARNING 06-07 15:53:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) (EngineCore_DP0 pid=1355199) INFO 06-07 15:54:02 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=56, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) INFO 06-07 15:54:04 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) INFO 06-07 15:54:04 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=55, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.45:51011 backend=nccl
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:55:18] OK: 66 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:55:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.45:44505 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:56 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:10 [default_loader.py:293] Loading weights took 14.22 seconds
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:56 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:11 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 41.635916 seconds
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:13 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:13 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:13 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45) (EngineCore_DP0 pid=1355199) INFO 06-07 15:56:14 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:14 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.24 seconds
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) WARNING 06-07 15:56:14 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:14 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) WARNING 06-07 15:56:14 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:14 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45) WARNING 06-07 15:56:14 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:57:18] OK: 66 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:57:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:10 [default_loader.py:293] Loading weights took 14.19 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:11 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 41.644509 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:13 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:13 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:13 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:14 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:14 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.27 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) WARNING 06-07 15:56:14 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:14 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) WARNING 06-07 15:56:14 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45) (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:14 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45) WARNING 06-07 15:56:15 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:59:18] OK: 68 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [15:59:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:01:18] OK: 70 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:01:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
(pid=1486338, ip=10.128.16.41) INFO 06-07 16:03:10 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:03:18] OK: 70 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:03:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.8 GiB
(pid=1486805, ip=10.128.16.41) INFO 06-07 16:03:10 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) WARNING 06-07 16:04:07 [arg_utils.py:1256] The global random seed is set to 50. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) INFO 06-07 16:04:32 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) INFO 06-07 16:04:32 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) INFO 06-07 16:04:32 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) INFO 06-07 16:04:32 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) INFO 06-07 16:04:32 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) WARNING 06-07 16:04:32 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) INFO 06-07 16:04:32 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) WARNING 06-07 16:04:07 [arg_utils.py:1256] The global random seed is set to 53. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) WARNING 06-07 16:04:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) WARNING 06-07 16:04:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) INFO 06-07 16:04:32 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) INFO 06-07 16:04:32 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) INFO 06-07 16:04:32 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) INFO 06-07 16:04:32 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) INFO 06-07 16:04:32 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) WARNING 06-07 16:04:32 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) INFO 06-07 16:04:32 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) INFO 06-07 16:05:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) WARNING 06-07 16:04:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) WARNING 06-07 16:04:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:10 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=50, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:05:18] OK: 74 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:05:18] OK: RSS 1.63 GiB | node mem 172.0/858.0 GiB used (20.1%), avail 685.9 GiB
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) INFO 06-07 16:05:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:10 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=53, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:54 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:55 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.41:48789 backend=nccl
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:55 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:01 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:54 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:55 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.41:48427 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:55 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:03 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:19 [default_loader.py:293] Loading weights took 15.29 seconds
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:01 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:03 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:19 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.289778 seconds
(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41) (EngineCore_DP0 pid=1487374) INFO 06-07 16:06:21 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41) (EngineCore_DP0 pid=1487374) INFO 06-07 16:06:21 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41) (EngineCore_DP0 pid=1487374) INFO 06-07 16:06:21 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.01 seconds
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) WARNING 06-07 16:06:22 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) WARNING 06-07 16:06:22 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41) WARNING 06-07 16:06:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:07:18] OK: 76 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:07:18] OK: RSS 1.63 GiB | node mem 172.0/858.0 GiB used (20.1%), avail 685.9 GiB
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:19 [default_loader.py:293] Loading weights took 15.29 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:19 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.295963 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.04 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) WARNING 06-07 16:06:22 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) WARNING 06-07 16:06:22 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41) (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41) WARNING 06-07 16:06:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:09:18] OK: 78 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:09:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.9 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:11:18] OK: 78 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:11:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.9 GiB
(pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:30 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) WARNING 06-07 16:11:35 [arg_utils.py:1256] The global random seed is set to 65. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:35 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:35 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:35 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:35 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:35 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) WARNING 06-07 16:11:35 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:35 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) WARNING 06-07 16:11:35 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) WARNING 06-07 16:11:35 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:30 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) WARNING 06-07 16:11:36 [arg_utils.py:1256] The global random seed is set to 62. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:36 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:36 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:36 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:36 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:36 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) WARNING 06-07 16:11:36 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:36 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) WARNING 06-07 16:11:37 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) WARNING 06-07 16:11:37 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) INFO 06-07 16:11:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) INFO 06-07 16:11:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:11:46 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=62, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:10 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) INFO 06-07 16:11:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:11:46 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=65, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:11 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.47:59713 backend=nccl
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:11 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:13:18] OK: 80 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:13:18] OK: RSS 1.63 GiB | node mem 172.0/858.0 GiB used (20.1%), avail 685.9 GiB
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:11 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:11 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.47:60611 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:11 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:37 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:38 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) (EngineCore_DP0 pid=1539228) INFO 06-07 16:13:53 [default_loader.py:293] Loading weights took 13.80 seconds
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:37 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) (EngineCore_DP0 pid=1539228) INFO 06-07 16:13:38 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:53 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.105954 seconds
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:55 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:55 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:55 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.79 seconds
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) WARNING 06-07 16:13:56 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) WARNING 06-07 16:13:56 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47) WARNING 06-07 16:13:56 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:15:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:15:18] OK: RSS 1.64 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:53 [default_loader.py:293] Loading weights took 13.80 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:53 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.107068 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:55 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:55 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:55 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47) (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:56 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:56 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.79 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) WARNING 06-07 16:13:56 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:56 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) WARNING 06-07 16:13:56 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47) (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:56 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47) WARNING 06-07 16:13:56 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(pid=1568271, ip=10.128.16.48) INFO 06-07 16:15:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) WARNING 06-07 16:16:49 [arg_utils.py:1256] The global random seed is set to 74. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) INFO 06-07 16:16:49 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) INFO 06-07 16:16:49 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) INFO 06-07 16:16:49 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) INFO 06-07 16:16:49 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) INFO 06-07 16:16:49 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) WARNING 06-07 16:16:49 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) INFO 06-07 16:16:49 [vllm.py:846] Cudagraph is disabled under eager mode
(pid=1568417, ip=10.128.16.48) INFO 06-07 16:15:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) WARNING 06-07 16:16:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) WARNING 06-07 16:16:50 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) WARNING 06-07 16:16:49 [arg_utils.py:1256] The global random seed is set to 76. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) INFO 06-07 16:16:49 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) INFO 06-07 16:16:49 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) INFO 06-07 16:16:49 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) INFO 06-07 16:16:49 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) INFO 06-07 16:16:49 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) WARNING 06-07 16:16:49 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) INFO 06-07 16:16:49 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(pid=1204670, ip=10.128.16.43) INFO 06-07 16:17:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) WARNING 06-07 16:16:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) WARNING 06-07 16:16:49 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(pid=1204806, ip=10.128.16.43) INFO 06-07 16:17:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:17:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:17:18] OK: RSS 1.64 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
(pid=1495134, ip=10.128.16.36) INFO 06-07 16:17:56 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36) WARNING 06-07 16:18:01 [arg_utils.py:1256] The global random seed is set to 72. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) INFO 06-07 16:18:01 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) INFO 06-07 16:18:01 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) INFO 06-07 16:18:01 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) INFO 06-07 16:18:01 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) INFO 06-07 16:18:01 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) WARNING 06-07 16:18:01 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) INFO 06-07 16:18:01 [vllm.py:846] Cudagraph is disabled under eager mode
(pid=1495334, ip=10.128.16.36) INFO 06-07 16:17:56 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36) WARNING 06-07 16:18:01 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36) WARNING 06-07 16:18:01 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) WARNING 06-07 16:18:11 [arg_utils.py:1256] The global random seed is set to 67. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) INFO 06-07 16:18:02 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) INFO 06-07 16:18:02 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) INFO 06-07 16:18:02 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) INFO 06-07 16:18:02 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) INFO 06-07 16:18:02 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) WARNING 06-07 16:18:02 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) INFO 06-07 16:18:02 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) WARNING 06-07 16:18:02 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) WARNING 06-07 16:18:02 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) INFO 06-07 16:19:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) WARNING 06-07 16:18:11 [arg_utils.py:1256] The global random seed is set to 78. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) INFO 06-07 16:18:11 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) INFO 06-07 16:18:11 [model.py:1549] Using max model len 32768 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) INFO 06-07 16:18:11 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) INFO 06-07 16:18:11 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) INFO 06-07 16:18:11 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) WARNING 06-07 16:18:11 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) INFO 06-07 16:18:11 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) WARNING 06-07 16:18:12 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) WARNING 06-07 16:18:12 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) INFO 06-07 16:19:14 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=74, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:19:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:19:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.0 GiB
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) INFO 06-07 16:20:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:19:14 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=76, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:37 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) INFO 06-07 16:20:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:20:10 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=78, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48) (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:38 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.48:55611 backend=nccl
(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48) (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:38 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48) (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:38 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48) (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:40 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:38 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) INFO 06-07 16:20:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:38 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.48:34821 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:38 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:39 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:40 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) INFO 06-07 16:20:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:20:52 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=70, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:54 [default_loader.py:293] Loading weights took 13.69 seconds
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:54 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.966695 seconds
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:56 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:56 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:56 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) INFO 06-07 16:20:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) WARNING 06-07 16:20:57 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) WARNING 06-07 16:20:57 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:20:52 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=73, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48) WARNING 06-07 16:20:57 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:21:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:21:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:54 [default_loader.py:293] Loading weights took 13.74 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:54 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.958987 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:56 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:56 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:56 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) WARNING 06-07 16:20:57 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) WARNING 06-07 16:20:57 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48) (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48) WARNING 06-07 16:20:57 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:41 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:42 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.36:36041 backend=nccl
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) (EngineCore_DP0 pid=1495936) INFO 06-07 16:22:42 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:42 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) INFO 06-07 16:22:51 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:22:42 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.36:36069 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:42 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:22:42 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:09 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:22:51 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:22:51 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.43:50323 backend=nccl [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:22:51 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 4x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:23:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:23:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:09 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:18 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:23 [default_loader.py:293] Loading weights took 13.44 seconds
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:18 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:23 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.381533 seconds
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) (EngineCore_DP0 pid=1495936) INFO 06-07 16:23:26 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) WARNING 06-07 16:23:26 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) WARNING 06-07 16:23:26 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36) (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36) WARNING 06-07 16:23:27 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:45 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:23 [default_loader.py:293] Loading weights took 13.45 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:23 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.383910 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:25 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:25 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:25 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) WARNING 06-07 16:23:26 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) WARNING 06-07 16:23:26 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36) WARNING 06-07 16:23:27 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:45 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:59 [default_loader.py:293] Loading weights took 13.87 seconds
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) (EngineCore_DP0 pid=1205537) INFO 06-07 16:23:59 [default_loader.py:293] Loading weights took 13.88 seconds
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:59 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.839385 seconds
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:01 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:01 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:01 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43) (EngineCore_DP0 pid=1205529) INFO 06-07 16:24:02 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:02 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.71 seconds
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) WARNING 06-07 16:24:02 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) INFO 06-07 16:24:02 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) WARNING 06-07 16:24:02 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) (EngineCore_DP0 pid=1205535) INFO 06-07 16:24:02 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) WARNING 06-07 16:24:03 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:25:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:25:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:59 [default_loader.py:293] Loading weights took 13.87 seconds [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:59 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.851068 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43) (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43) (EngineCore_DP0 pid=1205529) INFO 06-07 16:24:02 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.76 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) WARNING 06-07 16:24:02 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:02 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) WARNING 06-07 16:24:02 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43) (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:02 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43) WARNING 06-07 16:24:03 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:27:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:27:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
(pid=1446434, ip=10.128.16.44) INFO 06-07 16:27:46 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) WARNING 06-07 16:29:09 [arg_utils.py:1256] The global random seed is set to 86. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(pid=1446661, ip=10.128.16.44) INFO 06-07 16:27:46 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) INFO 06-07 16:29:09 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) INFO 06-07 16:29:09 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) INFO 06-07 16:29:09 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) INFO 06-07 16:29:09 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) INFO 06-07 16:29:09 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) WARNING 06-07 16:29:09 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) INFO 06-07 16:29:09 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) WARNING 06-07 16:29:10 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) WARNING 06-07 16:29:10 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:29:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:29:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) WARNING 06-07 16:29:09 [arg_utils.py:1256] The global random seed is set to 88. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) INFO 06-07 16:29:09 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) INFO 06-07 16:29:09 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) INFO 06-07 16:29:09 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) INFO 06-07 16:29:09 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) INFO 06-07 16:29:09 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) WARNING 06-07 16:29:09 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) INFO 06-07 16:29:09 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) WARNING 06-07 16:29:10 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) WARNING 06-07 16:29:10 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:31:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:31:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) INFO 06-07 16:32:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:32:29 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=86, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) INFO 06-07 16:32:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:32:29 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=88, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.44:57389 backend=nccl
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44) (EngineCore_DP0 pid=1447391) INFO 06-07 16:33:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:18 [default_loader.py:293] Loading weights took 13.72 seconds
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.44:48017 backend=nccl [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 3x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:33:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:33:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.118078 seconds
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44) (EngineCore_DP0 pid=1447391) INFO 06-07 16:33:21 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:21 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) WARNING 06-07 16:33:21 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:21 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) WARNING 06-07 16:33:21 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44) (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:21 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44) WARNING 06-07 16:33:21 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:35:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:35:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:18 [default_loader.py:293] Loading weights took 13.71 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.118351 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44) (EngineCore_DP0 pid=1447388) INFO 06-07 16:33:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44) (EngineCore_DP0 pid=1447388) INFO 06-07 16:33:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44) (EngineCore_DP0 pid=1447388) INFO 06-07 16:33:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) WARNING 06-07 16:33:21 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) WARNING 06-07 16:33:21 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44) WARNING 06-07 16:33:22 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:37:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:37:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:39:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:39:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:41:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:41:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:43:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:43:18] OK: RSS 1.64 GiB | node mem 171.7/858.0 GiB used (20.0%), avail 686.3 GiB
(pid=3137116, ip=10.128.16.135) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:45:18] OK: 80 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:45:18] OK: RSS 1.64 GiB | node mem 173.1/858.0 GiB used (20.2%), avail 684.9 GiB
(pid=3138189, ip=10.128.16.135) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:47:18] OK: 81 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:47:18] OK: RSS 1.64 GiB | node mem 173.1/858.0 GiB used (20.2%), avail 684.9 GiB
(pid=3138187, ip=10.128.16.135) ⚙️ Running in WANDB offline mode [repeated 2x across cluster]
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:49:18] OK: 81 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:49:18] OK: RSS 1.64 GiB | node mem 173.1/858.0 GiB used (20.2%), avail 684.8 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:51:18] OK: 82 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:51:18] OK: RSS 1.64 GiB | node mem 173.2/858.0 GiB used (20.2%), avail 684.8 GiB
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:53:18] OK: 81 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=1617955) [fd-monitor] [16:53:18] OK: RSS 1.64 GiB | node mem 173.2/858.0 GiB used (20.2%), avail 684.7 GiB
(pid=1620366) ⚙️ Running in WANDB offline mode
(pid=1620365) ⚙️ Running in WANDB offline mode [repeated 3x across cluster]
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=sequence_mean', 'trainer.algorithm.use_tis=true', 'trainer.algorithm.tis_imp_ratio_cap=2.0', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0-tis', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqmean-arm0-tis', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.collect_rollout_details=true', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(DistStoreError): ray::skyrl_entrypoint() (pid=1617955, ip=10.128.16.35)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 481, in run
trainer = self._setup_trainer()
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 450, in _setup_trainer
trainer.build_models(PolicyWorker, CriticWorker, RefWorker, policy_pg=self.policy_pg)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 757, in build_models
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 511, in __init__
self._initiate_actors(pg, num_gpus_per_actor)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 637, in _initiate_actors
ray.get([actor.init_worker_process_group.remote() for actor in self._actor_handlers])
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(DistStoreError): ray::FSDPPolicyWorkerBase.init_worker_process_group() (pid=3137116, ip=10.128.16.135, actor_id=538f00c3532d78f16f4d391d02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400e17779700>)
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 149, in init_worker_process_group
torch.distributed.init_process_group(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
func_return = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
store, rank, world_size = next(rendezvous_iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 278, in _env_rendezvous_handler
store = _create_c10d_store(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
return TCPStore(
^^^^^^^^^
torch.distributed.DistStoreError: Timed out after 601 seconds waiting for clients. 4/8 clients joined.
Exception raised from waitForWorkers at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:396 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000ab70c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5e9c9c0 (0x400b025fc9c0 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::waitForWorkers() + 0x350 (0x400b02691410 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x468 (0x400b026918c8 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x109a094 (0x400afc4fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x113236c (0x400afc59236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x5d6d60 (0x400afba36d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x1b7a38 (0xaaaad83d7a38 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #8: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #9: <unknown function> + 0x169f50 (0xaaaad8389f50 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #10: <unknown function> + 0x1682e4 (0xaaaad83882e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #11: <unknown function> + 0x1e0ce8 (0xaaaad8400ce8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #12: <unknown function> + 0x1d7ddc (0xaaaad83f7ddc in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #13: <unknown function> + 0x646b0c (0x400afbaa6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #14: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #15: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #16: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #17: <unknown function> + 0x182bf8 (0xaaaad83a2bf8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #18: <unknown function> + 0x25fd30 (0xaaaad847fd30 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #19: <unknown function> + 0x1b7d20 (0xaaaad83d7d20 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #20: PyObject_Vectorcall + 0x54 (0xaaaad83860e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #21: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #22: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #23: <unknown function> + 0x1835c8 (0xaaaad83a35c8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #24: <unknown function> + 0x84b708 (0x40002c15b708 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #25: <unknown function> + 0x865fe4 (0x40002c175fe4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #26: <unknown function> + 0x823470 (0x40002c133470 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #27: <unknown function> + 0x830110 (0x40002c140110 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #28: <unknown function> + 0x174a60 (0xaaaad8394a60 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #29: PyObject_VectorcallMethod + 0xa4 (0xaaaad8386270 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #30: PyIter_Send + 0xbc (0xaaaad836b62c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #31: <unknown function> + 0xaa04 (0x40002e3caa04 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #32: <unknown function> + 0xbcc4 (0x40002e3cbcc4 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #33: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #34: <unknown function> + 0x28cfac (0xaaaad84acfac in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #35: <unknown function> + 0x1b7b68 (0xaaaad83d7b68 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #37: <unknown function> + 0x83c564 (0x40002c14c564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #38: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #39: <unknown function> + 0x169f88 (0xaaaad8389f88 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #40: <unknown function> + 0x36002c (0xaaaad858002c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #41: <unknown function> + 0x2e4024 (0xaaaad8504024 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #42: <unknown function> + 0x80e00 (0x40002b080e00 in /lib64/libc.so.6)
frame #43: <unknown function> + 0xeb49c (0x40002b0eb49c in /lib64/libc.so.6)
Stopping Ray cluster...
Warning: Failed to stop Ray on jpbo-001-42: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-42', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Warning: Failed to stop Ray on jpbo-001-43: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-43', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Warning: Failed to stop Ray on jpbo-001-45: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-45', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Warning: Failed to stop Ray on jpbo-001-46: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-46', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Warning: Failed to stop Ray on jpbo-001-47: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-47', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Warning: Failed to stop Ray on jpbo-003-39: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-003-39', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Ray cluster stopped
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs, skipping upload.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ray_logs/
Collecting Ray logs from worker jpbo-001-36...
Collecting Ray logs from worker jpbo-001-37...
Collecting Ray logs from worker jpbo-001-38...
Collecting Ray logs from worker jpbo-001-41...
Collecting Ray logs from worker jpbo-001-42...
Collecting Ray logs from worker jpbo-001-43...
Collecting Ray logs from worker jpbo-001-44...
Collecting Ray logs from worker jpbo-001-45...
Collecting Ray logs from worker jpbo-001-46...
Collecting Ray logs from worker jpbo-001-47...
Collecting Ray logs from worker jpbo-001-48...
Collecting Ray logs from worker jpbo-003-37...
Collecting Ray logs from worker jpbo-003-39...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink