
The following have been reloaded with a version change:
  1) GCCcore/.14.3.0 => GCCcore/14.3.0


Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".

Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_653584
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_653584
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_653584
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.16.35 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653584.conf
[proxy]   - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy]   - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653584.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/configs/ablation-pymethods2test-seqmean-arm0-tis_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0-tis ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
  HF_TOKEN=****pDbg
  HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
  SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
  TENSOR_PARALLEL_SIZE=1
  NUM_INFERENCE_ENGINES=56
  POLICY_NUM_NODES=14
  WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
  Nodes: 14
  GPUs per node: 4
  CPUs per node: 288
  Head node: jpbo-001-35 (10.128.16.35)
  Ray port: 6379
============================
  Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray head on jpbo-001-35
  Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 1 on jpbo-001-36
  Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 2 on jpbo-001-37
  Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 3 on jpbo-001-38
  Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 4 on jpbo-001-41
  Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 5 on jpbo-001-42
  Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 6 on jpbo-001-43
  Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 7 on jpbo-001-44
  Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 8 on jpbo-001-45
  Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 9 on jpbo-001-46
  Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 10 on jpbo-001-47
  Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 11 on jpbo-001-48
  Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 12 on jpbo-003-37
  Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
  Started Ray worker 13 on jpbo-003-39
  Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'GPU': 56.0, 'accelerator_type:GH200': 14.0, 'node:10.128.16.43': 1.0, 'memory': 10289339891712.0, 'CPU': 4032.0, 'object_store_memory': 601295421440.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.46': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.36': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.35': 1.0, 'node:10.128.16.48': 1.0, 'node:10.128.16.135': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.41': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
  Address: 10.128.16.35:6379
  Total GPUs: 56
=========================
Ray cluster ready at 10.128.16.35:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM

Running SkyRL:
  Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
  Entrypoint: examples.terminal_bench.entrypoints.main_tbench
  Args: 123 Hydra arguments
  Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4

Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=sequence_mean trainer.algorithm.use_tis=true trainer.algorithm.tis_imp_ratio_cap=2.0 trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0-tis ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqmean-arm0-tis trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.collect_rollout_details=true +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_653584.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 15:37:07.080 | WARNING  | skyrl_train.utils.utils:validate_cfg:479 - `generator.sampling_params.logprobs` is `None` but `trainer.algorithm.use_tis` is `True`. Setting `logprobs` to `True`.
2026-06-07 15:37:07.289 | INFO     | skyrl_train.utils.utils:prepare_runtime_environment:753 - Exporting wandb api key to ray runtime env
2026-06-07 15:37:07.289 | INFO     | skyrl_train.utils.utils:prepare_runtime_environment:772 - Exporting RAY_ADDRESS to ray runtime env
2026-06-07 15:37:07.289 | INFO     | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
2026-06-07 15:37:07.289 | INFO     | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
2026-06-07 15:37:07.289 | INFO     | skyrl_train.utils.utils:prepare_runtime_environment:797 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
2026-06-07 15:37:07,290	INFO worker.py:1680 -- Using address 10.128.16.35:6379 set in the environment variable RAY_ADDRESS
2026-06-07 15:37:07,324	INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
2026-06-07 15:37:07,334	INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
  warnings.warn(
[33m(raylet, ip=10.128.16.45)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 15:37:09.507 | INFO     | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
[2026-06-07 15:37:37,531 E 1617107 1617929] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.45)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 5x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 15:37:18.206[0m | [1mINFO    [0m | [36mskyrl_train.entrypoints.main_base[0m:[36m_configure_log_level[0m:[36m212[0m - [1mSkyRL log level set to: INFO[0m
[36m(skyrl_entrypoint pid=1617955)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 15:37:18.546[0m | [1mINFO    [0m | [36mexamples.terminal_bench.dataset[0m:[36m_load_data_files[0m:[36m40[0m - [1mLoading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 15:37:20.166[0m | [1mINFO    [0m | [36mexamples.terminal_bench.dataset[0m:[36m_load_data_files[0m:[36m50[0m - [1mFound 5000 valid task directories out of 5000 total directories[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 15:37:20.166[0m | [1mINFO    [0m | [36mexamples.terminal_bench.dataset[0m:[36m__init__[0m:[36m27[0m - [1mTerminalBenchTaskDataset initialized with 5000 task paths[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 15:37:20.178[0m | [1mINFO    [0m | [36mskyrl_train.entrypoints.main_base[0m:[36m_setup_trainer[0m:[36m405[0m - [1mdata:
[36m(skyrl_entrypoint pid=1617955)[0m   train_data:
[36m(skyrl_entrypoint pid=1617955)[0m   - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
[36m(skyrl_entrypoint pid=1617955)[0m   val_data: []
[36m(skyrl_entrypoint pid=1617955)[0m trainer:
[36m(skyrl_entrypoint pid=1617955)[0m   placement:
[36m(skyrl_entrypoint pid=1617955)[0m     colocate_all: false
[36m(skyrl_entrypoint pid=1617955)[0m     colocate_policy_ref: true
[36m(skyrl_entrypoint pid=1617955)[0m     policy_num_nodes: 2
[36m(skyrl_entrypoint pid=1617955)[0m     policy_num_gpus_per_node: 4
[36m(skyrl_entrypoint pid=1617955)[0m     critic_num_nodes: 1
[36m(skyrl_entrypoint pid=1617955)[0m     critic_num_gpus_per_node: 4
[36m(skyrl_entrypoint pid=1617955)[0m     ref_num_nodes: 2
[36m(skyrl_entrypoint pid=1617955)[0m     ref_num_gpus_per_node: 4
[36m(skyrl_entrypoint pid=1617955)[0m     policy_strict_spread_pg: false
[36m(skyrl_entrypoint pid=1617955)[0m     policy_per_gpu_bundles: false
[36m(skyrl_entrypoint pid=1617955)[0m     policy_force_cvd_mask: false
[36m(skyrl_entrypoint pid=1617955)[0m   sequence_parallel_backend: ulysses
[36m(skyrl_entrypoint pid=1617955)[0m   strategy: fsdp2
[36m(skyrl_entrypoint pid=1617955)[0m   policy:
[36m(skyrl_entrypoint pid=1617955)[0m     model:
[36m(skyrl_entrypoint pid=1617955)[0m       path: /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
[36m(skyrl_entrypoint pid=1617955)[0m       lora:
[36m(skyrl_entrypoint pid=1617955)[0m         rank: 0
[36m(skyrl_entrypoint pid=1617955)[0m         alpha: 16
[36m(skyrl_entrypoint pid=1617955)[0m         dropout: 0
[36m(skyrl_entrypoint pid=1617955)[0m         lora_sync_path: /tmp/skyrl_lora_sync
[36m(skyrl_entrypoint pid=1617955)[0m         target_modules: all-linear
[36m(skyrl_entrypoint pid=1617955)[0m         exclude_modules: null
[36m(skyrl_entrypoint pid=1617955)[0m     deepspeed_config: ${deepspeed_config.train}
[36m(skyrl_entrypoint pid=1617955)[0m     optimizer_config:
[36m(skyrl_entrypoint pid=1617955)[0m       optimizer: AdamW
[36m(skyrl_entrypoint pid=1617955)[0m       lr: 8.0e-06
[36m(skyrl_entrypoint pid=1617955)[0m       adam_betas:
[36m(skyrl_entrypoint pid=1617955)[0m       - 0.9
[36m(skyrl_entrypoint pid=1617955)[0m       - 0.999
[36m(skyrl_entrypoint pid=1617955)[0m       weight_decay: 0.0
[36m(skyrl_entrypoint pid=1617955)[0m       max_grad_norm: 0.9
[36m(skyrl_entrypoint pid=1617955)[0m       offload_after_step: true
[36m(skyrl_entrypoint pid=1617955)[0m       num_warmup_steps: 0
[36m(skyrl_entrypoint pid=1617955)[0m       scheduler: constant_with_warmup
[36m(skyrl_entrypoint pid=1617955)[0m       optimizer_kwargs: {}
[36m(skyrl_entrypoint pid=1617955)[0m     fsdp_config:
[36m(skyrl_entrypoint pid=1617955)[0m       cpu_offload: false
[36m(skyrl_entrypoint pid=1617955)[0m       reshard_after_forward: true
[36m(skyrl_entrypoint pid=1617955)[0m       fsdp_size: 4
[36m(skyrl_entrypoint pid=1617955)[0m       expert_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_tensor_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       moe_token_dispatcher_type: alltoall
[36m(skyrl_entrypoint pid=1617955)[0m       moe_router_replay: false
[36m(skyrl_entrypoint pid=1617955)[0m       moe_grouped_gemm: false
[36m(skyrl_entrypoint pid=1617955)[0m       ep_comm_backend: torch
[36m(skyrl_entrypoint pid=1617955)[0m       deepep_num_sms: 20
[36m(skyrl_entrypoint pid=1617955)[0m       deepep_token_chunk_size: null
[36m(skyrl_entrypoint pid=1617955)[0m     sequence_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m     use_torch_compile: false
[36m(skyrl_entrypoint pid=1617955)[0m     record_memory: false
[36m(skyrl_entrypoint pid=1617955)[0m     megatron_config:
[36m(skyrl_entrypoint pid=1617955)[0m       tensor_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       pipeline_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       context_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_tensor_parallel_size: null
[36m(skyrl_entrypoint pid=1617955)[0m       ddp_config:
[36m(skyrl_entrypoint pid=1617955)[0m         grad_reduce_in_fp32: true
[36m(skyrl_entrypoint pid=1617955)[0m         overlap_grad_reduce: false
[36m(skyrl_entrypoint pid=1617955)[0m         overlap_param_gather: false
[36m(skyrl_entrypoint pid=1617955)[0m         average_in_collective: true
[36m(skyrl_entrypoint pid=1617955)[0m       model_config_kwargs: {}
[36m(skyrl_entrypoint pid=1617955)[0m       torch_profiler_config:
[36m(skyrl_entrypoint pid=1617955)[0m         enable: false
[36m(skyrl_entrypoint pid=1617955)[0m         ranks: []
[36m(skyrl_entrypoint pid=1617955)[0m         save_path: null
[36m(skyrl_entrypoint pid=1617955)[0m       optimizer_config_kwargs:
[36m(skyrl_entrypoint pid=1617955)[0m         overlap_cpu_optimizer_d2h_h2d: false
[36m(skyrl_entrypoint pid=1617955)[0m         use_precision_aware_optimizer: false
[36m(skyrl_entrypoint pid=1617955)[0m         optimizer_cpu_offload: false
[36m(skyrl_entrypoint pid=1617955)[0m         optimizer_offload_fraction: 0.0
[36m(skyrl_entrypoint pid=1617955)[0m       transformer_config_kwargs:
[36m(skyrl_entrypoint pid=1617955)[0m         recompute_granularity: full
[36m(skyrl_entrypoint pid=1617955)[0m         recompute_modules:
[36m(skyrl_entrypoint pid=1617955)[0m         - core_attn
[36m(skyrl_entrypoint pid=1617955)[0m         recompute_method: uniform
[36m(skyrl_entrypoint pid=1617955)[0m         recompute_num_layers: 1
[36m(skyrl_entrypoint pid=1617955)[0m       empty_cuda_cache: true
[36m(skyrl_entrypoint pid=1617955)[0m   ref:
[36m(skyrl_entrypoint pid=1617955)[0m     model:
[36m(skyrl_entrypoint pid=1617955)[0m       path: ${trainer.policy.model.path}
[36m(skyrl_entrypoint pid=1617955)[0m     sequence_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m     deepspeed_config: ${deepspeed_config.eval}
[36m(skyrl_entrypoint pid=1617955)[0m     fsdp_config:
[36m(skyrl_entrypoint pid=1617955)[0m       cpu_offload: false
[36m(skyrl_entrypoint pid=1617955)[0m       reshard_after_forward: true
[36m(skyrl_entrypoint pid=1617955)[0m       fsdp_size: 4
[36m(skyrl_entrypoint pid=1617955)[0m       expert_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_tensor_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       moe_token_dispatcher_type: alltoall
[36m(skyrl_entrypoint pid=1617955)[0m       moe_router_replay: false
[36m(skyrl_entrypoint pid=1617955)[0m       moe_grouped_gemm: false
[36m(skyrl_entrypoint pid=1617955)[0m       ep_comm_backend: torch
[36m(skyrl_entrypoint pid=1617955)[0m       deepep_num_sms: 20
[36m(skyrl_entrypoint pid=1617955)[0m       deepep_token_chunk_size: null
[36m(skyrl_entrypoint pid=1617955)[0m     megatron_config:
[36m(skyrl_entrypoint pid=1617955)[0m       tensor_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       pipeline_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       context_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_tensor_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       model_config_kwargs: {}
[36m(skyrl_entrypoint pid=1617955)[0m       transformer_config_kwargs: {}
[36m(skyrl_entrypoint pid=1617955)[0m   critic:
[36m(skyrl_entrypoint pid=1617955)[0m     model:
[36m(skyrl_entrypoint pid=1617955)[0m       path: null
[36m(skyrl_entrypoint pid=1617955)[0m       lora:
[36m(skyrl_entrypoint pid=1617955)[0m         rank: 0
[36m(skyrl_entrypoint pid=1617955)[0m         alpha: 16
[36m(skyrl_entrypoint pid=1617955)[0m         dropout: 0
[36m(skyrl_entrypoint pid=1617955)[0m         target_modules: all-linear
[36m(skyrl_entrypoint pid=1617955)[0m         exclude_modules: null
[36m(skyrl_entrypoint pid=1617955)[0m     deepspeed_config: ${deepspeed_config.train}
[36m(skyrl_entrypoint pid=1617955)[0m     optimizer_config:
[36m(skyrl_entrypoint pid=1617955)[0m       optimizer: AdamW
[36m(skyrl_entrypoint pid=1617955)[0m       lr: 5.0e-06
[36m(skyrl_entrypoint pid=1617955)[0m       adam_betas:
[36m(skyrl_entrypoint pid=1617955)[0m       - 0.9
[36m(skyrl_entrypoint pid=1617955)[0m       - 0.999
[36m(skyrl_entrypoint pid=1617955)[0m       weight_decay: 0.01
[36m(skyrl_entrypoint pid=1617955)[0m       max_grad_norm: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m       offload_after_step: true
[36m(skyrl_entrypoint pid=1617955)[0m       num_warmup_steps: 0
[36m(skyrl_entrypoint pid=1617955)[0m       scheduler: constant_with_warmup
[36m(skyrl_entrypoint pid=1617955)[0m       optimizer_kwargs: {}
[36m(skyrl_entrypoint pid=1617955)[0m     fsdp_config:
[36m(skyrl_entrypoint pid=1617955)[0m       cpu_offload: false
[36m(skyrl_entrypoint pid=1617955)[0m       reshard_after_forward: true
[36m(skyrl_entrypoint pid=1617955)[0m       fsdp_size: -1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_model_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       expert_tensor_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       moe_token_dispatcher_type: alltoall
[36m(skyrl_entrypoint pid=1617955)[0m       moe_router_replay: false
[36m(skyrl_entrypoint pid=1617955)[0m       moe_grouped_gemm: false
[36m(skyrl_entrypoint pid=1617955)[0m       ep_comm_backend: torch
[36m(skyrl_entrypoint pid=1617955)[0m       deepep_num_sms: 20
[36m(skyrl_entrypoint pid=1617955)[0m       deepep_token_chunk_size: null
[36m(skyrl_entrypoint pid=1617955)[0m     sequence_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m   algorithm:
[36m(skyrl_entrypoint pid=1617955)[0m     advantage_estimator: rloo_n
[36m(skyrl_entrypoint pid=1617955)[0m     kl_ctrl:
[36m(skyrl_entrypoint pid=1617955)[0m       type: fixed
[36m(skyrl_entrypoint pid=1617955)[0m       kl_target: 0.1
[36m(skyrl_entrypoint pid=1617955)[0m       horizon: 10000
[36m(skyrl_entrypoint pid=1617955)[0m     kl_estimator_type: k3
[36m(skyrl_entrypoint pid=1617955)[0m     use_kl_estimator_k3: false
[36m(skyrl_entrypoint pid=1617955)[0m     use_abs_kl: false
[36m(skyrl_entrypoint pid=1617955)[0m     use_kl_in_reward: false
[36m(skyrl_entrypoint pid=1617955)[0m     use_kl_loss: false
[36m(skyrl_entrypoint pid=1617955)[0m     kl_loss_coef: 0.0
[36m(skyrl_entrypoint pid=1617955)[0m     use_entropy_loss: false
[36m(skyrl_entrypoint pid=1617955)[0m     entropy_loss_coef: 0.01
[36m(skyrl_entrypoint pid=1617955)[0m     advantage_batch_normalize: false
[36m(skyrl_entrypoint pid=1617955)[0m     value_head_prefix: value_head
[36m(skyrl_entrypoint pid=1617955)[0m     policy_loss_type: regular
[36m(skyrl_entrypoint pid=1617955)[0m     loss_reduction: sequence_mean
[36m(skyrl_entrypoint pid=1617955)[0m     global_loss_denom: null
[36m(skyrl_entrypoint pid=1617955)[0m     grpo_norm_by_std: true
[36m(skyrl_entrypoint pid=1617955)[0m     rloo_n_min_group_size: 4
[36m(skyrl_entrypoint pid=1617955)[0m     rloo_n_filter_zero_reward_groups: true
[36m(skyrl_entrypoint pid=1617955)[0m     lambd: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     gamma: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     eps_clip_low: 0.2
[36m(skyrl_entrypoint pid=1617955)[0m     eps_clip_high: 0.05
[36m(skyrl_entrypoint pid=1617955)[0m     clip_ratio_c: 3.0
[36m(skyrl_entrypoint pid=1617955)[0m     tis_imp_ratio_cap: 2.0
[36m(skyrl_entrypoint pid=1617955)[0m     use_tis: true
[36m(skyrl_entrypoint pid=1617955)[0m     sapo:
[36m(skyrl_entrypoint pid=1617955)[0m       tau_pos: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m       tau_neg: 1.05
[36m(skyrl_entrypoint pid=1617955)[0m     value_clip: 0.2
[36m(skyrl_entrypoint pid=1617955)[0m     dynamic_sampling:
[36m(skyrl_entrypoint pid=1617955)[0m       type: null
[36m(skyrl_entrypoint pid=1617955)[0m       max_sample_batches: 30
[36m(skyrl_entrypoint pid=1617955)[0m       min_replace_ratio: 0.3
[36m(skyrl_entrypoint pid=1617955)[0m     clip_cov:
[36m(skyrl_entrypoint pid=1617955)[0m       clip_ratio: 0.0002
[36m(skyrl_entrypoint pid=1617955)[0m       clip_cov_lb: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m       clip_cov_ub: 5.0
[36m(skyrl_entrypoint pid=1617955)[0m     kl_cov:
[36m(skyrl_entrypoint pid=1617955)[0m       kl_cov_frac: 0.2
[36m(skyrl_entrypoint pid=1617955)[0m       ppo_kl_coef: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     cispo:
[36m(skyrl_entrypoint pid=1617955)[0m       cispo_eps_clip_low: 0
[36m(skyrl_entrypoint pid=1617955)[0m       cispo_eps_clip_high: 5
[36m(skyrl_entrypoint pid=1617955)[0m     z_clip:
[36m(skyrl_entrypoint pid=1617955)[0m       enabled: false
[36m(skyrl_entrypoint pid=1617955)[0m       alpha: 0.97
[36m(skyrl_entrypoint pid=1617955)[0m       z_thresh: 2.5
[36m(skyrl_entrypoint pid=1617955)[0m       warmup_steps: 25
[36m(skyrl_entrypoint pid=1617955)[0m       mode: zscore
[36m(skyrl_entrypoint pid=1617955)[0m       clip_option: adaptive_scaling
[36m(skyrl_entrypoint pid=1617955)[0m       clip_factor: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m       skip_update_on_spike: false
[36m(skyrl_entrypoint pid=1617955)[0m     stale_clip:
[36m(skyrl_entrypoint pid=1617955)[0m       enabled: false
[36m(skyrl_entrypoint pid=1617955)[0m       alpha: 0.3
[36m(skyrl_entrypoint pid=1617955)[0m       entropy_threshold: 0.15
[36m(skyrl_entrypoint pid=1617955)[0m       entropy_window: 10
[36m(skyrl_entrypoint pid=1617955)[0m       min_lr_scale: 0.1
[36m(skyrl_entrypoint pid=1617955)[0m     max_seq_len: 1004095
[36m(skyrl_entrypoint pid=1617955)[0m   fully_async:
[36m(skyrl_entrypoint pid=1617955)[0m     max_staleness_steps: 16
[36m(skyrl_entrypoint pid=1617955)[0m     num_parallel_generation_workers: 338
[36m(skyrl_entrypoint pid=1617955)[0m   gradient_checkpointing: true
[36m(skyrl_entrypoint pid=1617955)[0m   gradient_checkpointing_use_reentrant: false
[36m(skyrl_entrypoint pid=1617955)[0m   seed: 42
[36m(skyrl_entrypoint pid=1617955)[0m   resume_mode: latest
[36m(skyrl_entrypoint pid=1617955)[0m   resume_path: null
[36m(skyrl_entrypoint pid=1617955)[0m   ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/checkpoints
[36m(skyrl_entrypoint pid=1617955)[0m   max_ckpts_to_keep: -1
[36m(skyrl_entrypoint pid=1617955)[0m   ckpt_interval: 2
[36m(skyrl_entrypoint pid=1617955)[0m   hf_save_interval: 5
[36m(skyrl_entrypoint pid=1617955)[0m   hf_upload_mode: latest
[36m(skyrl_entrypoint pid=1617955)[0m   export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/exports
[36m(skyrl_entrypoint pid=1617955)[0m   bf16: true
[36m(skyrl_entrypoint pid=1617955)[0m   epochs: 2
[36m(skyrl_entrypoint pid=1617955)[0m   max_steps: 80
[36m(skyrl_entrypoint pid=1617955)[0m   update_epochs_per_batch: 1
[36m(skyrl_entrypoint pid=1617955)[0m   train_batch_size: 64
[36m(skyrl_entrypoint pid=1617955)[0m   policy_mini_batch_size: 64
[36m(skyrl_entrypoint pid=1617955)[0m   critic_mini_batch_size: 256
[36m(skyrl_entrypoint pid=1617955)[0m   micro_train_batch_size_per_gpu: 1
[36m(skyrl_entrypoint pid=1617955)[0m   micro_forward_batch_size_per_gpu: 4
[36m(skyrl_entrypoint pid=1617955)[0m   update_ref_every_epoch: false
[36m(skyrl_entrypoint pid=1617955)[0m   use_sample_packing: true
[36m(skyrl_entrypoint pid=1617955)[0m   eval_batch_size: 64
[36m(skyrl_entrypoint pid=1617955)[0m   eval_before_train: false
[36m(skyrl_entrypoint pid=1617955)[0m   eval_interval: 999999
[36m(skyrl_entrypoint pid=1617955)[0m   max_prompt_length: 999999
[36m(skyrl_entrypoint pid=1617955)[0m   flash_attn: true
[36m(skyrl_entrypoint pid=1617955)[0m   disable_fast_tokenizer: false
[36m(skyrl_entrypoint pid=1617955)[0m   target_modules: null
[36m(skyrl_entrypoint pid=1617955)[0m   exclude_modules: null
[36m(skyrl_entrypoint pid=1617955)[0m   project_name: OpenThoughts-Agent
[36m(skyrl_entrypoint pid=1617955)[0m   run_name: ablation-pymethods2test-seqmean-arm0-tis
[36m(skyrl_entrypoint pid=1617955)[0m   logger: console
[36m(skyrl_entrypoint pid=1617955)[0m   tracker_commit_each_step: true
[36m(skyrl_entrypoint pid=1617955)[0m   dump_data_batch: false
[36m(skyrl_entrypoint pid=1617955)[0m   dump_eval_results: true
[36m(skyrl_entrypoint pid=1617955)[0m   log_level: INFO
[36m(skyrl_entrypoint pid=1617955)[0m   rope_scaling: null
[36m(skyrl_entrypoint pid=1617955)[0m   rope_theta: null
[36m(skyrl_entrypoint pid=1617955)[0m   step_wise_training: false
[36m(skyrl_entrypoint pid=1617955)[0m   hf_hub_repo_id: laion/ablation-pymethods2test-seqmean-arm0-tis
[36m(skyrl_entrypoint pid=1617955)[0m   hf_hub_private: false
[36m(skyrl_entrypoint pid=1617955)[0m   hf_hub_revision: main
[36m(skyrl_entrypoint pid=1617955)[0m   enable_db_registration: false
[36m(skyrl_entrypoint pid=1617955)[0m generator:
[36m(skyrl_entrypoint pid=1617955)[0m   model_name: ${trainer.policy.model.path}
[36m(skyrl_entrypoint pid=1617955)[0m   model_dtype: bfloat16
[36m(skyrl_entrypoint pid=1617955)[0m   timeout_multiplier: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m   run_engines_locally: true
[36m(skyrl_entrypoint pid=1617955)[0m   num_inference_engines: 48
[36m(skyrl_entrypoint pid=1617955)[0m   backend: vllm
[36m(skyrl_entrypoint pid=1617955)[0m   weight_sync_backend: nccl
[36m(skyrl_entrypoint pid=1617955)[0m   fuse_weights: false
[36m(skyrl_entrypoint pid=1617955)[0m   weight_transfer_threshold_cuda_ipc_GB: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m   inference_engine_tensor_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m   inference_engine_pipeline_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m   inference_engine_expert_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m   inference_engine_data_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m   n_samples_per_prompt: 8
[36m(skyrl_entrypoint pid=1617955)[0m   async_engine: true
[36m(skyrl_entrypoint pid=1617955)[0m   batched: false
[36m(skyrl_entrypoint pid=1617955)[0m   max_input_length: ${trainer.max_prompt_length}
[36m(skyrl_entrypoint pid=1617955)[0m   vllm_v1_disable_multiproc: true
[36m(skyrl_entrypoint pid=1617955)[0m   enable_prefix_caching: true
[36m(skyrl_entrypoint pid=1617955)[0m   enable_chunked_prefill: true
[36m(skyrl_entrypoint pid=1617955)[0m   max_num_batched_tokens: 65536
[36m(skyrl_entrypoint pid=1617955)[0m   enforce_eager: true
[36m(skyrl_entrypoint pid=1617955)[0m   fully_sharded_loras: false
[36m(skyrl_entrypoint pid=1617955)[0m   enable_ray_prometheus_stats: false
[36m(skyrl_entrypoint pid=1617955)[0m   vllm_stats_interval: 1
[36m(skyrl_entrypoint pid=1617955)[0m   gpu_memory_utilization: 0.75
[36m(skyrl_entrypoint pid=1617955)[0m   max_num_seqs: 24
[36m(skyrl_entrypoint pid=1617955)[0m   remote_inference_engine_urls:
[36m(skyrl_entrypoint pid=1617955)[0m   - 127.0.0.1:8001
[36m(skyrl_entrypoint pid=1617955)[0m   enable_http_endpoint: true
[36m(skyrl_entrypoint pid=1617955)[0m   http_endpoint_host: 127.0.0.1
[36m(skyrl_entrypoint pid=1617955)[0m   http_endpoint_port: 8000
[36m(skyrl_entrypoint pid=1617955)[0m   max_turns: 999999
[36m(skyrl_entrypoint pid=1617955)[0m   chat_template:
[36m(skyrl_entrypoint pid=1617955)[0m     source: name
[36m(skyrl_entrypoint pid=1617955)[0m     name_or_path: null
[36m(skyrl_entrypoint pid=1617955)[0m   chat_template_kwargs: {}
[36m(skyrl_entrypoint pid=1617955)[0m   engine_init_kwargs:
[36m(skyrl_entrypoint pid=1617955)[0m     max_model_len: 32768
[36m(skyrl_entrypoint pid=1617955)[0m     custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
[36m(skyrl_entrypoint pid=1617955)[0m     served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
[36m(skyrl_entrypoint pid=1617955)[0m   override_existing_update_group: disable
[36m(skyrl_entrypoint pid=1617955)[0m   sampling_params:
[36m(skyrl_entrypoint pid=1617955)[0m     max_generate_length: 4096
[36m(skyrl_entrypoint pid=1617955)[0m     repetition_penalty: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     temperature: 0.7
[36m(skyrl_entrypoint pid=1617955)[0m     top_p: 0.95
[36m(skyrl_entrypoint pid=1617955)[0m     min_p: 0.0
[36m(skyrl_entrypoint pid=1617955)[0m     top_k: 20
[36m(skyrl_entrypoint pid=1617955)[0m     logprobs: 0
[36m(skyrl_entrypoint pid=1617955)[0m     stop: null
[36m(skyrl_entrypoint pid=1617955)[0m   use_conversation_multi_turn: true
[36m(skyrl_entrypoint pid=1617955)[0m   append_eos_token_after_stop_str_in_multi_turn: true
[36m(skyrl_entrypoint pid=1617955)[0m   eval_sampling_params:
[36m(skyrl_entrypoint pid=1617955)[0m     max_generate_length: ${generator.sampling_params.max_generate_length}
[36m(skyrl_entrypoint pid=1617955)[0m     repetition_penalty: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     temperature: 0.0
[36m(skyrl_entrypoint pid=1617955)[0m     top_p: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     min_p: 0.0
[36m(skyrl_entrypoint pid=1617955)[0m     top_k: -1
[36m(skyrl_entrypoint pid=1617955)[0m     logprobs: null
[36m(skyrl_entrypoint pid=1617955)[0m     stop: null
[36m(skyrl_entrypoint pid=1617955)[0m   eval_n_samples_per_prompt: 8
[36m(skyrl_entrypoint pid=1617955)[0m   zero_reward_on_non_stop: false
[36m(skyrl_entrypoint pid=1617955)[0m   apply_overlong_filtering: false
[36m(skyrl_entrypoint pid=1617955)[0m   rope_scaling: ${trainer.rope_scaling}
[36m(skyrl_entrypoint pid=1617955)[0m   rope_theta: ${trainer.rope_theta}
[36m(skyrl_entrypoint pid=1617955)[0m teacher:
[36m(skyrl_entrypoint pid=1617955)[0m   model_path: null
[36m(skyrl_entrypoint pid=1617955)[0m   top_k_logprobs: 256
[36m(skyrl_entrypoint pid=1617955)[0m   num_inference_engines: 1
[36m(skyrl_entrypoint pid=1617955)[0m   inference_engine_tensor_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m   inference_engine_pipeline_parallel_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m   gpu_memory_utilization: 0.9
[36m(skyrl_entrypoint pid=1617955)[0m   enforce_eager: false
[36m(skyrl_entrypoint pid=1617955)[0m   backend: vllm
[36m(skyrl_entrypoint pid=1617955)[0m   engine_init_kwargs: {}
[36m(skyrl_entrypoint pid=1617955)[0m environment:
[36m(skyrl_entrypoint pid=1617955)[0m   env_class: gsm8k
[36m(skyrl_entrypoint pid=1617955)[0m   skyrl_gym:
[36m(skyrl_entrypoint pid=1617955)[0m     max_env_workers: 32
[36m(skyrl_entrypoint pid=1617955)[0m     text2sql:
[36m(skyrl_entrypoint pid=1617955)[0m       db_path: /home/ray/default/sql_data
[36m(skyrl_entrypoint pid=1617955)[0m     llm_as_a_judge:
[36m(skyrl_entrypoint pid=1617955)[0m       model: gpt-4o-mini
[36m(skyrl_entrypoint pid=1617955)[0m       base_url: null
[36m(skyrl_entrypoint pid=1617955)[0m     search:
[36m(skyrl_entrypoint pid=1617955)[0m       log_requests: false
[36m(skyrl_entrypoint pid=1617955)[0m       search_url: http://127.0.0.1:8000/retrieve
[36m(skyrl_entrypoint pid=1617955)[0m       topk: 3
[36m(skyrl_entrypoint pid=1617955)[0m       timeout: 30
[36m(skyrl_entrypoint pid=1617955)[0m rollout:
[36m(skyrl_entrypoint pid=1617955)[0m   fanout:
[36m(skyrl_entrypoint pid=1617955)[0m     enabled: true
[36m(skyrl_entrypoint pid=1617955)[0m     num_coordinators: 4
[36m(skyrl_entrypoint pid=1617955)[0m     cpus_per_coordinator: 8
[36m(skyrl_entrypoint pid=1617955)[0m deepspeed_config:
[36m(skyrl_entrypoint pid=1617955)[0m   train:
[36m(skyrl_entrypoint pid=1617955)[0m     zero_optimization:
[36m(skyrl_entrypoint pid=1617955)[0m       stage: 3
[36m(skyrl_entrypoint pid=1617955)[0m       offload_param:
[36m(skyrl_entrypoint pid=1617955)[0m         device: none
[36m(skyrl_entrypoint pid=1617955)[0m       offload_optimizer:
[36m(skyrl_entrypoint pid=1617955)[0m         device: none
[36m(skyrl_entrypoint pid=1617955)[0m         pin_memory: true
[36m(skyrl_entrypoint pid=1617955)[0m       sub_group_size: auto
[36m(skyrl_entrypoint pid=1617955)[0m       reduce_bucket_size: auto
[36m(skyrl_entrypoint pid=1617955)[0m       stage3_param_persistence_threshold: auto
[36m(skyrl_entrypoint pid=1617955)[0m       stage3_prefetch_bucket_size: auto
[36m(skyrl_entrypoint pid=1617955)[0m       stage3_max_live_parameters: auto
[36m(skyrl_entrypoint pid=1617955)[0m       stage3_max_reuse_distance: auto
[36m(skyrl_entrypoint pid=1617955)[0m       round_robin_gradients: true
[36m(skyrl_entrypoint pid=1617955)[0m       zero_hpz_partition_size: 1
[36m(skyrl_entrypoint pid=1617955)[0m       zero_quantized_weights: false
[36m(skyrl_entrypoint pid=1617955)[0m       zero_quantized_gradients: false
[36m(skyrl_entrypoint pid=1617955)[0m     torch_autocast:
[36m(skyrl_entrypoint pid=1617955)[0m       enabled: true
[36m(skyrl_entrypoint pid=1617955)[0m       dtype: bfloat16
[36m(skyrl_entrypoint pid=1617955)[0m     disable_trace_cache: false
[36m(skyrl_entrypoint pid=1617955)[0m     data_types:
[36m(skyrl_entrypoint pid=1617955)[0m       grad_accum_dtype: fp32
[36m(skyrl_entrypoint pid=1617955)[0m     gradient_clipping: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     wall_clock_breakdown: false
[36m(skyrl_entrypoint pid=1617955)[0m     prescale_gradient: false
[36m(skyrl_entrypoint pid=1617955)[0m   eval:
[36m(skyrl_entrypoint pid=1617955)[0m     zero_optimization:
[36m(skyrl_entrypoint pid=1617955)[0m       stage: 3
[36m(skyrl_entrypoint pid=1617955)[0m       stage3_param_persistence_threshold: auto
[36m(skyrl_entrypoint pid=1617955)[0m       offload_param:
[36m(skyrl_entrypoint pid=1617955)[0m         device: cpu
[36m(skyrl_entrypoint pid=1617955)[0m         pin_memory: true
[36m(skyrl_entrypoint pid=1617955)[0m     torch_autocast:
[36m(skyrl_entrypoint pid=1617955)[0m       enabled: true
[36m(skyrl_entrypoint pid=1617955)[0m       dtype: bfloat16
[36m(skyrl_entrypoint pid=1617955)[0m     gradient_clipping: 1.0
[36m(skyrl_entrypoint pid=1617955)[0m     prescale_gradient: false
[36m(skyrl_entrypoint pid=1617955)[0m     wall_clock_breakdown: false
[36m(skyrl_entrypoint pid=1617955)[0m terminal_bench_config:
[36m(skyrl_entrypoint pid=1617955)[0m   trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs
[36m(skyrl_entrypoint pid=1617955)[0m   harbor:
[36m(skyrl_entrypoint pid=1617955)[0m     name: terminus-2
[36m(skyrl_entrypoint pid=1617955)[0m     max_episodes: 999999
[36m(skyrl_entrypoint pid=1617955)[0m     enable_summarize: false
[36m(skyrl_entrypoint pid=1617955)[0m     store_all_messages: true
[36m(skyrl_entrypoint pid=1617955)[0m     trajectory_config:
[36m(skyrl_entrypoint pid=1617955)[0m       raw_content: true
[36m(skyrl_entrypoint pid=1617955)[0m     enable_episode_logging: false
[36m(skyrl_entrypoint pid=1617955)[0m     record_terminal_session: false
[36m(skyrl_entrypoint pid=1617955)[0m     enable_pane_logging: false
[36m(skyrl_entrypoint pid=1617955)[0m     strict_json_parser: true
[36m(skyrl_entrypoint pid=1617955)[0m     interleaved_thinking: true
[36m(skyrl_entrypoint pid=1617955)[0m     extra_body:
[36m(skyrl_entrypoint pid=1617955)[0m       chat_template_kwargs:
[36m(skyrl_entrypoint pid=1617955)[0m         enable_thinking: true
[36m(skyrl_entrypoint pid=1617955)[0m     override_timeout_sec: 900
[36m(skyrl_entrypoint pid=1617955)[0m     override_cpus: 1
[36m(skyrl_entrypoint pid=1617955)[0m     override_memory_mb: 2048
[36m(skyrl_entrypoint pid=1617955)[0m     override_storage_mb: 2048
[36m(skyrl_entrypoint pid=1617955)[0m     auto_snapshot: true
[36m(skyrl_entrypoint pid=1617955)[0m     verifier_override_timeout_sec: 120
[36m(skyrl_entrypoint pid=1617955)[0m     max_retries: 3
[36m(skyrl_entrypoint pid=1617955)[0m     min_wait_sec: 60.0
[36m(skyrl_entrypoint pid=1617955)[0m     max_wait_sec: 600.0
[36m(skyrl_entrypoint pid=1617955)[0m     wait_multiplier: 2.0
[36m(skyrl_entrypoint pid=1617955)[0m     exclude_exceptions:
[36m(skyrl_entrypoint pid=1617955)[0m     - VerifierTimeoutError
[36m(skyrl_entrypoint pid=1617955)[0m     - VerifierRuntimeError
[36m(skyrl_entrypoint pid=1617955)[0m     - RewardFileNotFoundError
[36m(skyrl_entrypoint pid=1617955)[0m     - RewardFileEmptyError
[36m(skyrl_entrypoint pid=1617955)[0m     - VerifierOutputParseError
[36m(skyrl_entrypoint pid=1617955)[0m     n_concurrent_trials: 675
[36m(skyrl_entrypoint pid=1617955)[0m     log_level: INFO
[36m(skyrl_entrypoint pid=1617955)[0m     enable_reward_shaping: false
[36m(skyrl_entrypoint pid=1617955)[0m     collect_rollout_details: true
[36m(skyrl_entrypoint pid=1617955)[0m     enable_error_classification: true
[36m(skyrl_entrypoint pid=1617955)[0m     mask_exceptions:
[36m(skyrl_entrypoint pid=1617955)[0m     - DaytonaError
[36m(skyrl_entrypoint pid=1617955)[0m     - EnvironmentStartTimeoutError
[36m(skyrl_entrypoint pid=1617955)[0m     - NetworkError
[36m(skyrl_entrypoint pid=1617955)[0m     - ConnectionError
[36m(skyrl_entrypoint pid=1617955)[0m     - RewardFileNotFoundError
[36m(skyrl_entrypoint pid=1617955)[0m     - RewardFileEmptyError
[36m(skyrl_entrypoint pid=1617955)[0m     - AgentEnvironmentTimeoutError
[36m(skyrl_entrypoint pid=1617955)[0m     - ContextLengthExceededError
[36m(skyrl_entrypoint pid=1617955)[0m     default_error_treatment: zero
[36m(skyrl_entrypoint pid=1617955)[0m     passthrough_exceptions:
[36m(skyrl_entrypoint pid=1617955)[0m     - AgentTimeoutError
[36m(skyrl_entrypoint pid=1617955)[0m     zero_exceptions: []
[36m(skyrl_entrypoint pid=1617955)[0m   model_info:
[36m(skyrl_entrypoint pid=1617955)[0m     max_input_tokens: 32000
[36m(skyrl_entrypoint pid=1617955)[0m     max_output_tokens: 4096
[36m(skyrl_entrypoint pid=1617955)[0m   archiving:
[36m(skyrl_entrypoint pid=1617955)[0m     enabled: false
[36m(skyrl_entrypoint pid=1617955)[0m   trace_upload:
[36m(skyrl_entrypoint pid=1617955)[0m     enabled: true
[36m(skyrl_entrypoint pid=1617955)[0m     repo_org: DCAgent
[36m(skyrl_entrypoint pid=1617955)[0m     episodes: last
[36m(skyrl_entrypoint pid=1617955)[0m     dataset_type: SFT
[36m(skyrl_entrypoint pid=1617955)[0m     cleanup: true
[36m(skyrl_entrypoint pid=1617955)[0m [0m
[36m(RegistryActor pid=1354131, ip=10.128.16.45)[0m [2026-06-07 15:37:38,501 E 1354131 1354171] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(skyrl_entrypoint pid=1617955)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(skyrl_entrypoint pid=1617955)[0m No module named 'vllm._version'
[36m(skyrl_entrypoint pid=1617955)[0m   from .version import __version__, __version_tuple__  # isort:skip
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 12x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [2026-06-07 15:37:40,589 E 1617955 1617998] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 2x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m W0607 15:42:21.650000 1617955 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(get_all_env_variables pid=1618274)[0m [2026-06-07 15:44:18,568 E 1618274 1618314] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.45)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 29x across cluster][0m
[36m(pid=1354512, ip=10.128.16.45)[0m [2026-06-07 15:44:20,995 E 1354512 1354615] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 4x across cluster][0m
[33m(raylet, ip=10.128.16.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 2x across cluster][0m
[33m(raylet, ip=10.128.16.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 10x across cluster][0m
[36m(pid=1354512, ip=10.128.16.45)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1354512, ip=10.128.16.45)[0m No module named 'vllm._version'
[36m(pid=1354512, ip=10.128.16.45)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(get_addr_port pid=1516023, ip=10.128.16.42)[0m [2026-06-07 15:45:41,920 E 1516023 1516098] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 2x across cluster][0m
[36m(pid=1516024, ip=10.128.16.42)[0m [2026-06-07 15:45:41,966 E 1516024 1516126] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(pid=1516170, ip=10.128.16.42)[0m [2026-06-07 15:47:31,328 E 1516170 1516245] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.37)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 22x across cluster][0m
[36m(pid=1496715, ip=10.128.16.37)[0m [2026-06-07 15:47:31,482 E 1496715 1496817] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 3x across cluster][0m
[33m(raylet, ip=10.128.16.37)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 3x across cluster][0m
[36m(pid=1497002, ip=10.128.16.37)[0m [2026-06-07 15:52:22,486 E 1497002 1497139] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.37)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 15x across cluster][0m
[36m(pid=1497001, ip=10.128.16.37)[0m [2026-06-07 15:52:22,439 E 1497001 1497111] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:25.238 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:25.239 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:25.239 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[36m(pid=1497003, ip=10.128.16.37)[0m [2026-06-07 15:52:22,695 E 1497003 1497167] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:25.845 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:26.289 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:26.289 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:26.289 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:26.289 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
[33m(raylet, ip=10.128.16.41)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(pid=1486338, ip=10.128.16.41)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1486338, ip=10.128.16.41)[0m No module named 'vllm._version'
[36m(pid=1486338, ip=10.128.16.41)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:50.076 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:50.077 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:50.077 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[33m(raylet, ip=10.128.16.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 53x across cluster][0m
[36m(pid=1354981, ip=10.128.16.45)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 6x across cluster][0m
[36m(pid=1354981, ip=10.128.16.45)[0m No module named 'vllm._version'[32m [repeated 6x across cluster][0m
[36m(pid=1354981, ip=10.128.16.45)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:51.035 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:51.038 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:51.038 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:51.038 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m 2026-06-07 15:53:51.038 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:53:52.539 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.11s (attempt 1/5) to avoid port collisions
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:50.076 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:50.077 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:50.077 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m No module named 'vllm._version'
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m No module named 'vllm._version'
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:51.025 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:51.029 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:51.029 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:51.029 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:51.030 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m [2026-06-07 15:54:10,417 E 1354979 1355118] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:53:52.539 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.70s (attempt 1/5) to avoid port collisions[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m No module named 'vllm._version'[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204)   _C._set_float32_matmul_precision(precision)
[36m(pid=1486805, ip=10.128.16.41)[0m [2026-06-07 15:54:10,667 E 1486805 1486970] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355199)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m [W607 15:55:02.757090169 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-45-interconnect-1.jupiter.internal]:51011 (errno: 97 - Address family not supported by protocol).
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m [W607 15:55:02.760273491 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m [rank0]:[W607 15:55:02.766688838 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m [W607 15:55:02.757164023 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-45.jupiter.internal]:44505 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m [W607 15:55:02.760282259 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m [rank0]:[W607 15:55:02.766374060 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:04<00:12,  4.20s/it]
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.26s/it][32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) 
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355199) 
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) 
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) 
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00,  3.54s/it][32m [repeated 11x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m 2026-06-07 15:56:14.991 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
[36m(pid=1516024, ip=10.128.16.42)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1516024, ip=10.128.16.42)[0m No module named 'vllm._version'
[36m(pid=1516024, ip=10.128.16.42)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m 2026-06-07 15:56:15.026 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
[33m(raylet, ip=10.128.16.47)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(pid=1516547, ip=10.128.16.42)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(pid=1516547, ip=10.128.16.42)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(pid=1516547, ip=10.128.16.42)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(get_addr_port pid=1538389, ip=10.128.16.47)[0m [2026-06-07 15:58:34,128 E 1538389 1538463] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 29x across cluster][0m
[36m(pid=1561915, ip=10.128.16.46)[0m [2026-06-07 15:58:34,380 E 1561915 1562081] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 4x across cluster][0m
[33m(raylet, ip=10.128.16.47)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 3x across cluster][0m
[36m(pid=1538388, ip=10.128.16.47)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1538388, ip=10.128.16.47)[0m No module named 'vllm._version'
[36m(pid=1538388, ip=10.128.16.47)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(get_addr_port pid=2087413, ip=10.128.16.133)[0m [2026-06-07 16:00:47,642 E 2087413 2087489] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.133)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 27x across cluster][0m
[36m(pid=1538603, ip=10.128.16.47)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(pid=1538603, ip=10.128.16.47)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(pid=1538603, ip=10.128.16.47)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(pid=2087414, ip=10.128.16.133)[0m [2026-06-07 16:00:47,689 E 2087414 2087517] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:12.660 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:12.661 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:12.661 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[36m(pid=1538603, ip=10.128.16.47)[0m [2026-06-07 16:00:47,874 E 1538603 1538768] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:13.898 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:14.213 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:14.213 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:14.214 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:14.214 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:03:14.287 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.95s (attempt 1/5) to avoid port collisions
[33m(raylet, ip=10.128.16.43)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:12.660 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:12.661 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:12.661 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:13.898 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:14.213 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:14.213 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:14.213 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:14.213 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m 2026-06-07 16:03:14.287 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.51s (attempt 1/5) to avoid port collisions[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[33m(raylet, ip=10.128.16.43)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 11x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 3x across cluster][0m
[33m(raylet, ip=10.128.16.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 2x across cluster][0m
[36m(get_addr_port pid=1204669, ip=10.128.16.43)[0m [2026-06-07 16:04:20,488 E 1204669 1204744] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[33m(raylet, ip=10.128.16.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 22x across cluster][0m
[36m(pid=1204670, ip=10.128.16.43)[0m [2026-06-07 16:04:20,535 E 1204670 1204772] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(pid=1204670, ip=10.128.16.43)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1204670, ip=10.128.16.43)[0m No module named 'vllm._version'
[36m(pid=1204670, ip=10.128.16.43)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 2x across cluster][0m
[36m(pid=1204805, ip=10.128.16.43)[0m [2026-06-07 16:04:48,441 E 1204805 1204857] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(pid=1204805, ip=10.128.16.43)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 2x across cluster][0m
[36m(pid=1204805, ip=10.128.16.43)[0m No module named 'vllm._version'[32m [repeated 2x across cluster][0m
[36m(pid=1204805, ip=10.128.16.43)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 2x across cluster][0m
[36m(pid=1495134, ip=10.128.16.36)[0m [2026-06-07 16:04:48,804 E 1495134 1495237] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m No module named 'vllm._version'[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487383)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487374)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m [W607 16:05:55.641875132 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-41-interconnect-1.jupiter.internal]:48789 (errno: 97 - Address family not supported by protocol).
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m [W607 16:05:55.643733206 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m [rank0]:[W607 16:05:55.649849626 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m [W607 16:05:55.641934203 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-41-interconnect-1.jupiter.internal]:48427 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m [W607 16:05:55.643746806 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m [rank0]:[W607 16:05:55.649674461 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
[36m(pid=1496715, ip=10.128.16.37)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1496715, ip=10.128.16.37)[0m No module named 'vllm._version'
[36m(pid=1496715, ip=10.128.16.37)[0m   from .version import __version__, __version_tuple__  # isort:skip
[33m(raylet, ip=10.128.16.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:05<00:17,  5.96s/it]
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
[36m(pid=1495334, ip=10.128.16.36)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 7x across cluster][0m
[36m(pid=1495334, ip=10.128.16.36)[0m No module named 'vllm._version'[32m [repeated 7x across cluster][0m
[36m(pid=1495334, ip=10.128.16.36)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 7x across cluster][0m
[33m(raylet, ip=10.128.16.48)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 29x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:13<00:04,  4.39s/it][32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) 
[36m(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487383) 
[36m(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487374) 
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) 
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:15<00:00,  3.82s/it][32m [repeated 11x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m 2026-06-07 16:06:23.164 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
[36m(pid=1495335, ip=10.128.16.36)[0m [2026-06-07 16:06:35,975 E 1495335 1495447] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41)[0m 2026-06-07 16:06:23.183 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
[33m(raylet, ip=10.128.16.48)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(pid=1495334, ip=10.128.16.36)[0m [2026-06-07 16:06:36,202 E 1495334 1495503] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 4x across cluster][0m
[36m(pid=1568271, ip=10.128.16.48)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1568271, ip=10.128.16.48)[0m No module named 'vllm._version'
[36m(pid=1568271, ip=10.128.16.48)[0m   from .version import __version__, __version_tuple__  # isort:skip
[33m(raylet, ip=10.128.16.133)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 53x across cluster][0m
[36m(pid=1205099, ip=10.128.16.43)[0m [2026-06-07 16:07:58,673 E 1205099 1205139] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(pid=1568417, ip=10.128.16.48)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 4x across cluster][0m
[36m(pid=1568417, ip=10.128.16.48)[0m No module named 'vllm._version'[32m [repeated 4x across cluster][0m
[36m(pid=1568417, ip=10.128.16.48)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 4x across cluster][0m
[36m(pid=2087857, ip=10.128.16.133)[0m [2026-06-07 16:07:58,681 E 2087857 2087966] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(pid=1561695, ip=10.128.16.46)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1561695, ip=10.128.16.46)[0m No module named 'vllm._version'
[36m(pid=1561695, ip=10.128.16.46)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(pid=2087855, ip=10.128.16.133)[0m [2026-06-07 16:07:58,946 E 2087855 2088022] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 7x across cluster][0m
[36m(pid=1561916, ip=10.128.16.46)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1561916, ip=10.128.16.46)[0m No module named 'vllm._version'
[36m(pid=1561916, ip=10.128.16.46)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:32.006 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:32.007 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:32.007 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[36m(pid=1561915, ip=10.128.16.46)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 2x across cluster][0m
[36m(pid=1561915, ip=10.128.16.46)[0m No module named 'vllm._version'[32m [repeated 2x across cluster][0m
[36m(pid=1561915, ip=10.128.16.46)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:33.258 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:33.747 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:33.747 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:33.747 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:33.747 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:11:33.788 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.84s (attempt 1/5) to avoid port collisions
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:32.006 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:32.007 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:32.007 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m No module named 'vllm._version'
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m No module named 'vllm._version'
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:33.258 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:33.746 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:33.746 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:33.746 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:33.746 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m 2026-06-07 16:11:33.787 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.53s (attempt 1/5) to avoid port collisions[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[33m(raylet, ip=10.128.16.38)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 9x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m No module named 'vllm._version'[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 2x across cluster][0m
[36m(get_addr_port pid=1446435, ip=10.128.16.44)[0m [2026-06-07 16:12:25,922 E 1446435 1446509] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet, ip=10.128.16.38)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 27x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238)   _C._set_float32_matmul_precision(precision)
[36m(pid=1211472, ip=10.128.16.38)[0m [2026-06-07 16:12:26,041 E 1211472 1211634] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539228)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m [W607 16:13:11.271167533 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-47.jupiter.internal]:59713 (errno: 97 - Address family not supported by protocol).
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m [W607 16:13:11.272737868 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m [rank0]:[W607 16:13:11.277554377 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m [W607 16:13:11.615631887 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-47.jupiter.internal]:60611 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m [W607 16:13:11.616170660 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m [rank0]:[W607 16:13:11.618501524 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
[33m(raylet, ip=10.128.16.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(skyrl_entrypoint pid=1617955)[0m [2026-06-07 16:13:41] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
[36m(skyrl_entrypoint pid=1617955)[0m [2026-06-07 16:13:41] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
[36m(skyrl_entrypoint pid=1617955)[0m [2026-06-07 16:13:42] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.555[0m | [1mINFO    [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m_spin_up_http_endpoint[0m:[36m969[0m - [1mInferenceEngineClient HTTP endpoint started on 127.0.0.1:8000[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.556[0m | [1mINFO    [0m | [36mskyrl_train.inference_engines.inference_engine_client[0m:[36m__init__[0m:[36m61[0m - [1mInferenceEngineClient initialized with 48 engines.[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.558[0m | [1mINFO    [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m_configure_harbor_logging[0m:[36m242[0m - [1mHarbor logging level set to INFO[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.559[0m | [1mINFO    [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m__init__[0m:[36m142[0m - [1mTerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'collect_rollout_details', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.559[0m | [1mINFO    [0m | [36mexamples.terminal_bench.terminal_bench_generator[0m:[36m__init__[0m:[36m158[0m - [1mTerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.560[0m | [1mINFO    [0m | [36mskyrl_train.utils.trainer_utils[0m:[36mbuild_dataloader[0m:[36m656[0m - [1mTotal steps: 156[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.560[0m | [1mINFO    [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m357[0m - [1mLength of train_dataloader: 5000[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.560[0m | [1mINFO    [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m358[0m - [1mNumber of steps per epoch: 78[0m
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:13:42.560[0m | [1mINFO    [0m | [36mskyrl_train.fully_async_trainer[0m:[36m_build_train_dataloader_and_compute_training_steps[0m:[36m359[0m - [1mTotal training steps: 80[0m
[36m(pid=1446434, ip=10.128.16.44)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1446434, ip=10.128.16.44)[0m No module named 'vllm._version'
[36m(pid=1446434, ip=10.128.16.44)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:04<00:12,  4.14s/it]
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
[33m(raylet, ip=10.128.16.135)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 24x across cluster][0m
[36m(pid=1446661, ip=10.128.16.44)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(pid=1446661, ip=10.128.16.44)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(pid=1446661, ip=10.128.16.44)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.17s/it][32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) 
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539228) 
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) 
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) 
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m 2026-06-07 16:13:56.813 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.45s/it][32m [repeated 11x across cluster][0m
[36m(pid=1446659, ip=10.128.16.44)[0m [2026-06-07 16:14:12,531 E 1446659 1446769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m 2026-06-07 16:13:56.839 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
[36m(pid=2087414, ip=10.128.16.133)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=2087414, ip=10.128.16.133)[0m No module named 'vllm._version'
[36m(pid=2087414, ip=10.128.16.133)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(pid=3137116, ip=10.128.16.135)[0m [2026-06-07 16:14:13,514 E 3137116 3137156] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:20.561 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:20.562 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:20.562 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[36m(pid=2087855, ip=10.128.16.133)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(pid=2087855, ip=10.128.16.133)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(pid=2087855, ip=10.128.16.133)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:21.512 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:21.808 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:21.808 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:21.808 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:21.808 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:16:21.855 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.69s (attempt 1/5) to avoid port collisions
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:20.562 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:20.563 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:20.563 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:21.988 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:21.992 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:21.992 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:21.992 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:21.992 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:16:22.018 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.57s (attempt 1/5) to avoid port collisions[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m No module named 'vllm._version'
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:15.622 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:15.624 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:15.624 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m 2026-06-07 16:17:15.622 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m 2026-06-07 16:17:15.624 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m 2026-06-07 16:17:15.624 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:16.902 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:17.271 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:17.271 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:17.271 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:17.271 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:17:17.325 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.27s (attempt 1/5) to avoid port collisions
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m 2026-06-07 16:17:58.363 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m 2026-06-07 16:17:58.364 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m 2026-06-07 16:17:58.364 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m 2026-06-07 16:17:16.902 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m 2026-06-07 16:17:17.270 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m 2026-06-07 16:17:17.271 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m 2026-06-07 16:17:17.271 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m 2026-06-07 16:17:17.271 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m 2026-06-07 16:17:17.325 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.88s (attempt 1/5) to avoid port collisions[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m No module named 'vllm._version'
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:58.363 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:58.364 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:58.364 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:59.597 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:59.625 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:59.625 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:59.625 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:59.626 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:17:59.672 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 2.04s (attempt 1/5) to avoid port collisions[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 9x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(pid=3137116, ip=10.128.16.135)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m No module named 'vllm._version'[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568977)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m [W607 16:20:38.111142654 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-48-interconnect-1.jupiter.internal]:53891 (errno: 97 - Address family not supported by protocol).
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m [W607 16:20:38.111310683 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48)[0m [rank0]:[W607 16:20:38.116523344 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:04<00:12,  4.26s/it]
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m [W607 16:20:38.456774049 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-48-interconnect-1.jupiter.internal]:34821 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m [W607 16:20:38.457232632 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m [rank0]:[W607 16:20:38.459531497 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.08s/it][32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) 
[36m(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568977) 
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) 
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) 
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m 2026-06-07 16:20:57.866 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.43s/it][32m [repeated 11x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m 2026-06-07 16:20:57.786 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495936)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495912)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m [W607 16:22:42.320690462 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-36.jupiter.internal]:36041 (errno: 97 - Address family not supported by protocol).
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m [W607 16:22:42.324212382 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m [rank0]:[W607 16:22:42.330336336 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
[36m(pid=1211190, ip=10.128.16.38)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(pid=1211190, ip=10.128.16.38)[0m No module named 'vllm._version'
[36m(pid=1211190, ip=10.128.16.38)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m [W607 16:22:42.320619263 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-36-interconnect-1.jupiter.internal]:36069 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m [W607 16:22:42.324226270 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m [rank0]:[W607 16:22:42.330042997 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205537)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205529)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[36m(pid=1211472, ip=10.128.16.38)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(pid=1211472, ip=10.128.16.38)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(pid=1211472, ip=10.128.16.38)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m [W607 16:22:51.540094544 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-43-interconnect-1.jupiter.internal]:50323 (errno: 97 - Address family not supported by protocol).[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m [W607 16:22:51.542060717 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m [rank0]:[W607 16:22:51.548624154 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:03<00:11,  4.00s/it]
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.06s/it][32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) 
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495936) 
[36m(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495912) 
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) 
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m 2026-06-07 16:23:27.000 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.36s/it][32m [repeated 11x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m 2026-06-07 16:23:27.057 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:04<00:12,  4.02s/it]
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.21s/it][32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) 
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205537) 
[36m(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205529) 
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) 
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m 2026-06-07 16:24:03.034 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.47s/it][32m [repeated 11x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:40.671 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:40.672 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:40.672 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
[36m(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205529) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m 2026-06-07 16:24:03.085 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:41.853 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:41.880 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:41.881 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:41.881 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:41.881 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m 2026-06-07 16:28:41.928 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.91s (attempt 1/5) to avoid port collisions
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:40.671 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:152 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=2, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:40.672 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:162 - setup_envvars_for_vllm: numa_enabled=True[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:40.672 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:165 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:41.853 | INFO     | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 144-215 (NUMA node 2) for GPU 2[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:41.881 | DEBUG    | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 2[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:41.881 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:613 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:41.881 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:__init__:621 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:41.881 | WARNING  | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1187 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:28:41.928 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1239 - Engine startup stagger: sleeping 1.63s (attempt 1/5) to avoid port collisions[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m No module named 'vllm._version'
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m   from .version import __version__, __version_tuple__  # isort:skip
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 6x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m No module named 'vllm._version'[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m   from .version import __version__, __version_tuple__  # isort:skip[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447391)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447388)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390)   _C._set_float32_matmul_precision(precision)
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m [W607 16:33:02.639468297 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-44-interconnect-1.jupiter.internal]:57389 (errno: 97 - Address family not supported by protocol).
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m [W607 16:33:02.642952955 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m [rank0]:[W607 16:33:02.648917654 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:04<00:12,  4.08s/it]
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m [W607 16:33:02.639681508 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-001-44.jupiter.internal]:48017 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m [W607 16:33:02.642967739 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m [rank0]:[W607 16:33:02.649384620 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.09s/it][32m [repeated 8x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) 
[36m(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447391) 
[36m(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447388) 
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) 
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[36m(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44)[0m 2026-06-07 16:33:21.860 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.43s/it][32m [repeated 11x across cluster][0m
[36m(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135)[0m 2026-06-07 16:43:51 INFO     [ipv4-debug] hostname=jpbo-003-39.jupiter.internal
[36m(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135)[0m 2026-06-07 16:43:51 INFO     [ipv4-debug] _global_node.node_ip_address=10.128.16.135
[36m(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135)[0m 2026-06-07 16:43:51 INFO     [ipv4-debug] get_node_ip_address()=10.128.16.135
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) The tokenizer you are loading from '/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m 2026-06-07 16:33:22.083 | INFO     | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1296 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2[32m [repeated 3x across cluster][0m
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[36m(skyrl_entrypoint pid=1617955)[0m [32m2026-06-07 16:43:51.863[0m | [1m[32mINFO    [0m | [36mskyrl_train.workers.worker[0m:[36m_initiate_actors[0m:[36m636[0m - [1m[32mInitializing process group for RayActorGroup[0m
[36m(FSDPPolicyWorkerBase pid=3137116, ip=10.128.16.135)[0m [W607 16:43:51.249797609 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39-interconnect-1.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol).
[36m(pid=3138189, ip=10.128.16.135)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
[36m(pid=1620364)[0m [2026-06-07 16:44:22,856 E 1620364 1620497] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
[33m(raylet)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 41x across cluster][0m
[36m(pid=1620365)[0m Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.[32m [repeated 6x across cluster][0m
[36m(FSDPPolicyWorkerBase pid=3138189, ip=10.128.16.135)[0m [W607 16:46:13.956081717 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39-interconnect-1.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol).
[36m(FSDPPolicyWorkerBase pid=3138189, ip=10.128.16.135)[0m [W607 16:46:13.956399759 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
[36m(pid=1620366)[0m [2026-06-07 16:44:22,896 E 1620366 1620595] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 6x across cluster][0m
[36m(FSDPPolicyWorkerBase pid=1620366)[0m [W607 16:54:00.377895659 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(FSDPPolicyWorkerBase pid=3138187, ip=10.128.16.135)[0m [W607 16:46:13.956394959 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())[32m [repeated 2x across cluster][0m
2026-06-07 16:54:19.008 | ERROR    | __main__:main:134 - Training failed: [36mray::skyrl_entrypoint()[39m (pid=1617955, ip=10.128.16.35)
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
    exp.run()
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 481, in run
    trainer = self._setup_trainer()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 450, in _setup_trainer
    trainer.build_models(PolicyWorker, CriticWorker, RefWorker, policy_pg=self.policy_pg)
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 757, in build_models
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 511, in __init__
    self._initiate_actors(pg, num_gpus_per_actor)
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 637, in _initiate_actors
    ray.get([actor.init_worker_process_group.remote() for actor in self._actor_handlers])
           ^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^
                                  ^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(DistStoreError): [36mray::FSDPPolicyWorkerBase.init_worker_process_group()[39m (pid=3137116, ip=10.128.16.135, actor_id=538f00c3532d78f16f4d391d02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400e17779700>)
  File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
           ^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 149, in init_worker_process_group
    torch.distributed.init_process_group(
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
    func_return = func(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 278, in _env_rendezvous_handler
    store = _create_c10d_store(
            ^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
    return TCPStore(
           ^^^^^^^^^
torch.distributed.DistStoreError: Timed out after 601 seconds waiting for clients. 4/8 clients joined.
Exception raised from waitForWorkers at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:396 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000ab70c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5e9c9c0 (0x400b025fc9c0 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::waitForWorkers() + 0x350 (0x400b02691410 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x468 (0x400b026918c8 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x109a094 (0x400afc4fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x113236c (0x400afc59236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x5d6d60 (0x400afba36d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x1b7a38 (0xaaaad83d7a38 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #8: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #9: <unknown function> + 0x169f50 (0xaaaad8389f50 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #10: <unknown function> + 0x1682e4 (0xaaaad83882e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #11: <unknown function> + 0x1e0ce8 (0xaaaad8400ce8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #12: <unknown function> + 0x1d7ddc (0xaaaad83f7ddc in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #13: <unknown function> + 0x646b0c (0x400afbaa6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #14: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #15: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #16: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #17: <unknown function> + 0x182bf8 (0xaaaad83a2bf8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #18: <unknown function> + 0x25fd30 (0xaaaad847fd30 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #19: <unknown function> + 0x1b7d20 (0xaaaad83d7d20 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #20: PyObject_Vectorcall + 0x54 (0xaaaad83860e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #21: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #22: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #23: <unknown function> + 0x1835c8 (0xaaaad83a35c8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #24: <unknown function> + 0x84b708 (0x40002c15b708 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #25: <unknown function> + 0x865fe4 (0x40002c175fe4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #26: <unknown function> + 0x823470 (0x40002c133470 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #27: <unknown function> + 0x830110 (0x40002c140110 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #28: <unknown function> + 0x174a60 (0xaaaad8394a60 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #29: PyObject_VectorcallMethod + 0xa4 (0xaaaad8386270 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #30: PyIter_Send + 0xbc (0xaaaad836b62c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #31: <unknown function> + 0xaa04 (0x40002e3caa04 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #32: <unknown function> + 0xbcc4 (0x40002e3cbcc4 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #33: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #34: <unknown function> + 0x28cfac (0xaaaad84acfac in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #35: <unknown function> + 0x1b7b68 (0xaaaad83d7b68 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #37: <unknown function> + 0x83c564 (0x40002c14c564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #38: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #39: <unknown function> + 0x169f88 (0xaaaad8389f88 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #40: <unknown function> + 0x36002c (0xaaaad858002c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #41: <unknown function> + 0x2e4024 (0xaaaad8504024 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #42: <unknown function> + 0x80e00 (0x40002b080e00 in /lib64/libc.so.6)
frame #43: <unknown function> + 0xeb49c (0x40002b0eb49c in /lib64/libc.so.6)
2026-06-07 16:54:19.009 | INFO     | __main__:main:137 - Shutting down Ray on head node...
[36m(FSDPPolicyWorkerBase pid=1620365)[0m [W607 16:54:00.377853387 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-003-39-interconnect-1.jupiter.internal]:54749 (errno: 97 - Address family not supported by protocol).[32m [repeated 3x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] Started monitoring (every 120s)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:37:18] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:37:18] OK: RSS 1.38 GiB | node mem 171.6/858.0 GiB used (20.0%), avail 686.3 GiB
[36m(skyrl_entrypoint pid=1617955)[0m ⚙️  Running in WANDB offline mode
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:39:18] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:39:18] OK: RSS 1.50 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.0 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:41:18] OK: 51 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:41:18] OK: RSS 1.58 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
[36m(skyrl_entrypoint pid=1617955)[0m INFO 06-07 15:42:21 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:43:18] OK: 51 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:43:18] OK: RSS 1.59 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:45:18] OK: 62 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:45:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:47:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:47:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:49:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:49:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:51:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:51:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:53:18] OK: 64 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:53:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(pid=1354512, ip=10.128.16.45)[0m INFO 06-07 15:53:23 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:53:49 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(pid=1354980, ip=10.128.16.45)[0m INFO 06-07 15:53:49 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m WARNING 06-07 15:53:54 [arg_utils.py:1256] The global random seed is set to 56. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:53:54 [model.py:529] Resolved architecture: Qwen3ForCausalLM
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:53:54 [model.py:1549] Using max model len 32768
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:53:54 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:53:54 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:53:54 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m WARNING 06-07 15:53:54 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:53:54 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(pid=1354981, ip=10.128.16.45)[0m INFO 06-07 15:53:49 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m WARNING 06-07 15:53:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m WARNING 06-07 15:53:54 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m INFO 06-07 15:54:01 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m WARNING 06-07 15:53:55 [arg_utils.py:1256] The global random seed is set to 57. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m INFO 06-07 15:53:55 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m INFO 06-07 15:53:55 [model.py:1549] Using max model len 32768[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m INFO 06-07 15:53:55 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m INFO 06-07 15:53:55 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m INFO 06-07 15:53:55 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m WARNING 06-07 15:53:55 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m INFO 06-07 15:53:55 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m WARNING 06-07 15:53:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m WARNING 06-07 15:53:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355199) INFO 06-07 15:54:02 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=56, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m INFO 06-07 15:54:04 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) INFO 06-07 15:54:04 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=55, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.45:51011 backend=nccl
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:55:18] OK: 66 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:55:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.45:44505 backend=nccl[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:55:56 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:28 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:10 [default_loader.py:293] Loading weights took 14.22 seconds
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:55:56 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:11 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 41.635916 seconds
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:13 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:13 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:13 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
[36m(AsyncVLLMInferenceEngine pid=1354979, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355199) INFO 06-07 15:56:14 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:14 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.24 seconds
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) WARNING 06-07 15:56:14 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:14 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) WARNING 06-07 15:56:14 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355204) INFO 06-07 15:56:14 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1354512, ip=10.128.16.45)[0m WARNING 06-07 15:56:14 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:57:18] OK: 66 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:57:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:10 [default_loader.py:293] Loading weights took 14.19 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:11 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 41.644509 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:13 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:13 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:13 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:14 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355219) INFO 06-07 15:56:14 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.27 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) WARNING 06-07 15:56:14 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:14 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) WARNING 06-07 15:56:14 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354980, ip=10.128.16.45)[0m (EngineCore_DP0 pid=1355215) INFO 06-07 15:56:14 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1354981, ip=10.128.16.45)[0m WARNING 06-07 15:56:15 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:59:18] OK: 68 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [15:59:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:01:18] OK: 70 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:01:18] OK: RSS 1.63 GiB | node mem 172.2/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:03:10 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:03:18] OK: 70 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:03:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.8 GiB
[36m(pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:03:10 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m WARNING 06-07 16:04:07 [arg_utils.py:1256] The global random seed is set to 50. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [model.py:529] Resolved architecture: Qwen3ForCausalLM
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [model.py:1549] Using max model len 32768
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m WARNING 06-07 16:04:32 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m WARNING 06-07 16:04:07 [arg_utils.py:1256] The global random seed is set to 53. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m WARNING 06-07 16:04:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m WARNING 06-07 16:04:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [model.py:1549] Using max model len 32768[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m WARNING 06-07 16:04:32 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:04:32 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m INFO 06-07 16:05:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m WARNING 06-07 16:04:55 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m WARNING 06-07 16:04:55 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:10 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=50, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:05:18] OK: 74 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:05:18] OK: RSS 1.63 GiB | node mem 172.0/858.0 GiB used (20.1%), avail 685.9 GiB
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m INFO 06-07 16:05:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:10 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=53, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:54 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:55 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.41:48789 backend=nccl
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:05:55 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:01 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:54 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:55 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.41:48427 backend=nccl[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:05:55 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:03 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:19 [default_loader.py:293] Loading weights took 15.29 seconds
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:01 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:03 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:19 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.289778 seconds
[36m(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487374) INFO 06-07 16:06:21 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
[36m(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487374) INFO 06-07 16:06:21 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
[36m(AsyncVLLMInferenceEngine pid=1486804, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487374) INFO 06-07 16:06:21 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.01 seconds
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) WARNING 06-07 16:06:22 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) WARNING 06-07 16:06:22 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487375) INFO 06-07 16:06:22 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1486338, ip=10.128.16.41)[0m WARNING 06-07 16:06:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:07:18] OK: 76 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:07:18] OK: RSS 1.63 GiB | node mem 172.0/858.0 GiB used (20.1%), avail 685.9 GiB
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:19 [default_loader.py:293] Loading weights took 15.29 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:19 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 17.295963 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.04 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) WARNING 06-07 16:06:22 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) WARNING 06-07 16:06:22 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486805, ip=10.128.16.41)[0m (EngineCore_DP0 pid=1487379) INFO 06-07 16:06:22 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1486803, ip=10.128.16.41)[0m WARNING 06-07 16:06:23 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:09:18] OK: 78 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:09:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.9 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:11:18] OK: 78 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:11:18] OK: RSS 1.63 GiB | node mem 172.1/858.0 GiB used (20.1%), avail 685.9 GiB
[36m(pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:30 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m WARNING 06-07 16:11:35 [arg_utils.py:1256] The global random seed is set to 65. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:35 [model.py:529] Resolved architecture: Qwen3ForCausalLM
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:35 [model.py:1549] Using max model len 32768
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:35 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:35 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:35 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m WARNING 06-07 16:11:35 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:35 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m WARNING 06-07 16:11:35 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m WARNING 06-07 16:11:35 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
[36m(pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:30 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m WARNING 06-07 16:11:36 [arg_utils.py:1256] The global random seed is set to 62. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:36 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:36 [model.py:1549] Using max model len 32768[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:36 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:36 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:36 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m WARNING 06-07 16:11:36 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:36 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m WARNING 06-07 16:11:37 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m WARNING 06-07 16:11:37 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m INFO 06-07 16:11:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m INFO 06-07 16:11:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:11:46 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=62, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:10 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m INFO 06-07 16:11:45 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:11:46 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=65, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:11 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.47:59713 backend=nccl
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:11 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:13:18] OK: 80 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:13:18] OK: RSS 1.63 GiB | node mem 172.0/858.0 GiB used (20.1%), avail 685.9 GiB
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:11 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:11 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.47:60611 backend=nccl[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:11 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:37 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:38 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539228) INFO 06-07 16:13:53 [default_loader.py:293] Loading weights took 13.80 seconds
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:37 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539228) INFO 06-07 16:13:38 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:53 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.105954 seconds
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:55 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:55 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:55 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.79 seconds
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) WARNING 06-07 16:13:56 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) WARNING 06-07 16:13:56 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:56 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1538601, ip=10.128.16.47)[0m WARNING 06-07 16:13:56 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:15:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:15:18] OK: RSS 1.64 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539238) INFO 06-07 16:13:53 [default_loader.py:293] Loading weights took 13.80 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:53 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.107068 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:55 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:55 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:55 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538602, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539233) INFO 06-07 16:13:56 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:56 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.79 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) WARNING 06-07 16:13:56 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:56 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) WARNING 06-07 16:13:56 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538603, ip=10.128.16.47)[0m (EngineCore_DP0 pid=1539220) INFO 06-07 16:13:56 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1538388, ip=10.128.16.47)[0m WARNING 06-07 16:13:56 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
[36m(pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:15:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m WARNING 06-07 16:16:49 [arg_utils.py:1256] The global random seed is set to 74. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [model.py:529] Resolved architecture: Qwen3ForCausalLM
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [model.py:1549] Using max model len 32768
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m WARNING 06-07 16:16:49 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(pid=1568417, ip=10.128.16.48)[0m INFO 06-07 16:15:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m WARNING 06-07 16:16:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m WARNING 06-07 16:16:50 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m WARNING 06-07 16:16:49 [arg_utils.py:1256] The global random seed is set to 76. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [model.py:1549] Using max model len 32768[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m WARNING 06-07 16:16:49 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m INFO 06-07 16:16:49 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(pid=1204670, ip=10.128.16.43)[0m INFO 06-07 16:17:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m WARNING 06-07 16:16:49 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m WARNING 06-07 16:16:49 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 3x across cluster][0m
[36m(pid=1204806, ip=10.128.16.43)[0m INFO 06-07 16:17:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:17:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:17:18] OK: RSS 1.64 GiB | node mem 172.0/858.0 GiB used (20.0%), avail 686.0 GiB
[36m(pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:17:56 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36)[0m WARNING 06-07 16:18:01 [arg_utils.py:1256] The global random seed is set to 72. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:18:01 [model.py:529] Resolved architecture: Qwen3ForCausalLM
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:18:01 [model.py:1549] Using max model len 32768
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:18:01 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:18:01 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:18:01 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m WARNING 06-07 16:18:01 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:18:01 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(pid=1495334, ip=10.128.16.36)[0m INFO 06-07 16:17:56 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36)[0m WARNING 06-07 16:18:01 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1495336, ip=10.128.16.36)[0m WARNING 06-07 16:18:01 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m WARNING 06-07 16:18:11 [arg_utils.py:1256] The global random seed is set to 67. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m INFO 06-07 16:18:02 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m INFO 06-07 16:18:02 [model.py:1549] Using max model len 32768[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m INFO 06-07 16:18:02 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m INFO 06-07 16:18:02 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m INFO 06-07 16:18:02 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m WARNING 06-07 16:18:02 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m INFO 06-07 16:18:02 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m WARNING 06-07 16:18:02 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m WARNING 06-07 16:18:02 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m INFO 06-07 16:19:13 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m WARNING 06-07 16:18:11 [arg_utils.py:1256] The global random seed is set to 78. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m INFO 06-07 16:18:11 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m INFO 06-07 16:18:11 [model.py:1549] Using max model len 32768[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m INFO 06-07 16:18:11 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m INFO 06-07 16:18:11 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m INFO 06-07 16:18:11 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m WARNING 06-07 16:18:11 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m INFO 06-07 16:18:11 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m WARNING 06-07 16:18:12 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m WARNING 06-07 16:18:12 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) INFO 06-07 16:19:14 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=74, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:19:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:19:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.0 GiB
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m INFO 06-07 16:20:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:19:14 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=76, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:37 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m INFO 06-07 16:20:08 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:20:10 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=78, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:38 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.48:55611 backend=nccl
[36m(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:38 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
[36m(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:38 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
[36m(AsyncVLLMInferenceEngine pid=1568418, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568977) INFO 06-07 16:20:40 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:38 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m INFO 06-07 16:20:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:38 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.48:34821 backend=nccl[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:38 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:39 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:40 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m INFO 06-07 16:20:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:20:52 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=70, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568980) INFO 06-07 16:20:54 [default_loader.py:293] Loading weights took 13.69 seconds
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:54 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.966695 seconds
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:56 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:56 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:56 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m INFO 06-07 16:20:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) WARNING 06-07 16:20:57 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) WARNING 06-07 16:20:57 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:57 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:20:52 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=73, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568271, ip=10.128.16.48)[0m WARNING 06-07 16:20:57 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:21:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:21:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568976) INFO 06-07 16:20:54 [default_loader.py:293] Loading weights took 13.74 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:54 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.958987 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:56 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:56 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:56 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.83 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) WARNING 06-07 16:20:57 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) WARNING 06-07 16:20:57 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568416, ip=10.128.16.48)[0m (EngineCore_DP0 pid=1568978) INFO 06-07 16:20:57 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1568417, ip=10.128.16.48)[0m WARNING 06-07 16:20:57 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:41 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:42 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.36:36041 backend=nccl
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495936) INFO 06-07 16:22:42 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:42 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) INFO 06-07 16:22:51 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:22:42 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.36:36069 backend=nccl[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:22:42 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:22:42 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:09 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:22:51 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:22:51 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.43:50323 backend=nccl[32m [repeated 4x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:22:51 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 4x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:23:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:23:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:09 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:18 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:23 [default_loader.py:293] Loading weights took 13.44 seconds
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:18 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:23 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.381533 seconds
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495936) INFO 06-07 16:23:26 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) WARNING 06-07 16:23:26 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) WARNING 06-07 16:23:26 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
[36m(AsyncVLLMInferenceEngine pid=1495134, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495916) INFO 06-07 16:23:26 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1495335, ip=10.128.16.36)[0m WARNING 06-07 16:23:27 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:45 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:23 [default_loader.py:293] Loading weights took 13.45 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:23 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.383910 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:25 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:25 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:25 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) WARNING 06-07 16:23:26 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) WARNING 06-07 16:23:26 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m (EngineCore_DP0 pid=1495920) INFO 06-07 16:23:26 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1495334, ip=10.128.16.36)[0m WARNING 06-07 16:23:27 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:45 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:59 [default_loader.py:293] Loading weights took 13.87 seconds
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205537) INFO 06-07 16:23:59 [default_loader.py:293] Loading weights took 13.88 seconds
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) INFO 06-07 16:23:59 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.839385 seconds
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:01 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:01 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:01 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
[36m(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205529) INFO 06-07 16:24:02 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:02 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.71 seconds
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) WARNING 06-07 16:24:02 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) INFO 06-07 16:24:02 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) WARNING 06-07 16:24:02 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205535) INFO 06-07 16:24:02 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m WARNING 06-07 16:24:03 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:25:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:25:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:59 [default_loader.py:293] Loading weights took 13.87 seconds[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:23:59 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 40.851068 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204806, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205537) INFO 06-07 16:24:02 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204805, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205529) INFO 06-07 16:24:02 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.76 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) WARNING 06-07 16:24:02 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:02 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) WARNING 06-07 16:24:02 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1205099, ip=10.128.16.43)[0m (EngineCore_DP0 pid=1205528) INFO 06-07 16:24:02 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1204670, ip=10.128.16.43)[0m WARNING 06-07 16:24:03 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:27:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:27:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
[36m(pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:27:46 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m WARNING 06-07 16:29:09 [arg_utils.py:1256] The global random seed is set to 86. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
[36m(pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:27:46 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [model.py:529] Resolved architecture: Qwen3ForCausalLM
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [model.py:1549] Using max model len 32768
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m WARNING 06-07 16:29:09 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m WARNING 06-07 16:29:10 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m WARNING 06-07 16:29:10 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:29:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:29:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m WARNING 06-07 16:29:09 [arg_utils.py:1256] The global random seed is set to 88. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [model.py:529] Resolved architecture: Qwen3ForCausalLM[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [model.py:1549] Using max model len 32768[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m WARNING 06-07 16:29:09 [vllm.py:728] Enforce eager set, overriding optimization level to -O0[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:29:09 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m WARNING 06-07 16:29:10 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m WARNING 06-07 16:29:10 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned[32m [repeated 3x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:31:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:31:18] OK: RSS 1.64 GiB | node mem 171.9/858.0 GiB used (20.0%), avail 686.1 GiB
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m INFO 06-07 16:32:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:32:29 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=86, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m INFO 06-07 16:32:28 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:32:29 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=88, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.44:57389 backend=nccl
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
[36m(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447391) INFO 06-07 16:33:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:18 [default_loader.py:293] Loading weights took 13.72 seconds
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:01 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'][32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:02 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.16.44:48017 backend=nccl[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:02 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:02 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:04 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].[32m [repeated 3x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:33:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:33:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.118078 seconds
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
[36m(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447391) INFO 06-07 16:33:21 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:21 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) WARNING 06-07 16:33:21 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:21 [vllm.py:690] Asynchronous scheduling is enabled.
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) WARNING 06-07 16:33:21 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
[36m(AsyncVLLMInferenceEngine pid=1446434, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447387) INFO 06-07 16:33:21 [vllm.py:846] Cudagraph is disabled under eager mode
[36m(AsyncVLLMInferenceEngine pid=1446659, ip=10.128.16.44)[0m WARNING 06-07 16:33:21 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:35:18] OK: 87 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:35:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:18 [default_loader.py:293] Loading weights took 13.71 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 15.118351 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447388) INFO 06-07 16:33:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447388) INFO 06-07 16:33:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446660, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447388) INFO 06-07 16:33:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) WARNING 06-07 16:33:21 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [vllm.py:690] Asynchronous scheduling is enabled.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) WARNING 06-07 16:33:21 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m (EngineCore_DP0 pid=1447390) INFO 06-07 16:33:21 [vllm.py:846] Cudagraph is disabled under eager mode[32m [repeated 3x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=1446661, ip=10.128.16.44)[0m WARNING 06-07 16:33:22 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.[32m [repeated 3x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:37:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:37:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:39:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:39:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:41:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:41:18] OK: RSS 1.64 GiB | node mem 171.8/858.0 GiB used (20.0%), avail 686.2 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:43:18] OK: 85 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:43:18] OK: RSS 1.64 GiB | node mem 171.7/858.0 GiB used (20.0%), avail 686.3 GiB
[36m(pid=3137116, ip=10.128.16.135)[0m ⚙️  Running in WANDB offline mode
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:45:18] OK: 80 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:45:18] OK: RSS 1.64 GiB | node mem 173.1/858.0 GiB used (20.2%), avail 684.9 GiB
[36m(pid=3138189, ip=10.128.16.135)[0m ⚙️  Running in WANDB offline mode
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:47:18] OK: 81 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:47:18] OK: RSS 1.64 GiB | node mem 173.1/858.0 GiB used (20.2%), avail 684.9 GiB
[36m(pid=3138187, ip=10.128.16.135)[0m ⚙️  Running in WANDB offline mode[32m [repeated 2x across cluster][0m
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:49:18] OK: 81 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:49:18] OK: RSS 1.64 GiB | node mem 173.1/858.0 GiB used (20.2%), avail 684.8 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:51:18] OK: 82 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:51:18] OK: RSS 1.64 GiB | node mem 173.2/858.0 GiB used (20.2%), avail 684.8 GiB
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:53:18] OK: 81 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
[36m(skyrl_entrypoint pid=1617955)[0m [fd-monitor] [16:53:18] OK: RSS 1.64 GiB | node mem 173.2/858.0 GiB used (20.2%), avail 684.7 GiB
[36m(pid=1620366)[0m ⚙️  Running in WANDB offline mode
[36m(pid=1620365)[0m ⚙️  Running in WANDB offline mode[32m [repeated 3x across cluster][0m
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=sequence_mean', 'trainer.algorithm.use_tis=true', 'trainer.algorithm.tis_imp_ratio_cap=2.0', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0-tis', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqmean-arm0-tis', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.collect_rollout_details=true', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
    main()
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
           ^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
            ^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
        ^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
    ray.get(skyrl_entrypoint.remote(cfg))
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(DistStoreError): [36mray::skyrl_entrypoint()[39m (pid=1617955, ip=10.128.16.35)
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
    exp.run()
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 481, in run
    trainer = self._setup_trainer()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 450, in _setup_trainer
    trainer.build_models(PolicyWorker, CriticWorker, RefWorker, policy_pg=self.policy_pg)
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 757, in build_models
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 511, in __init__
    self._initiate_actors(pg, num_gpus_per_actor)
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 637, in _initiate_actors
    ray.get([actor.init_worker_process_group.remote() for actor in self._actor_handlers])
           ^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^
                                  ^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(DistStoreError): [36mray::FSDPPolicyWorkerBase.init_worker_process_group()[39m (pid=3137116, ip=10.128.16.135, actor_id=538f00c3532d78f16f4d391d02000000, repr=<skyrl_train.workers.fsdp.fsdp_worker.FSDPPolicyWorkerBase object at 0x400e17779700>)
  File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
           ^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/workers/worker.py", line 149, in init_worker_process_group
    torch.distributed.init_process_group(
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
    func_return = func(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 278, in _env_rendezvous_handler
    store = _create_c10d_store(
            ^^^^^^^^^^^^^^^^^^^
  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
    return TCPStore(
           ^^^^^^^^^
torch.distributed.DistStoreError: Timed out after 601 seconds waiting for clients. 4/8 clients joined.
Exception raised from waitForWorkers at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:396 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000ab70c700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5e9c9c0 (0x400b025fc9c0 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::waitForWorkers() + 0x350 (0x400b02691410 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x468 (0x400b026918c8 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x109a094 (0x400afc4fa094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x113236c (0x400afc59236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x5d6d60 (0x400afba36d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x1b7a38 (0xaaaad83d7a38 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #8: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #9: <unknown function> + 0x169f50 (0xaaaad8389f50 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #10: <unknown function> + 0x1682e4 (0xaaaad83882e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #11: <unknown function> + 0x1e0ce8 (0xaaaad8400ce8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #12: <unknown function> + 0x1d7ddc (0xaaaad83f7ddc in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #13: <unknown function> + 0x646b0c (0x400afbaa6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #14: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #15: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #16: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #17: <unknown function> + 0x182bf8 (0xaaaad83a2bf8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #18: <unknown function> + 0x25fd30 (0xaaaad847fd30 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #19: <unknown function> + 0x1b7d20 (0xaaaad83d7d20 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #20: PyObject_Vectorcall + 0x54 (0xaaaad83860e4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #21: _PyEval_EvalFrameDefault + 0x280c (0xaaaad848ae54 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #22: <unknown function> + 0x1808c0 (0xaaaad83a08c0 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #23: <unknown function> + 0x1835c8 (0xaaaad83a35c8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #24: <unknown function> + 0x84b708 (0x40002c15b708 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #25: <unknown function> + 0x865fe4 (0x40002c175fe4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #26: <unknown function> + 0x823470 (0x40002c133470 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #27: <unknown function> + 0x830110 (0x40002c140110 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #28: <unknown function> + 0x174a60 (0xaaaad8394a60 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #29: PyObject_VectorcallMethod + 0xa4 (0xaaaad8386270 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #30: PyIter_Send + 0xbc (0xaaaad836b62c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #31: <unknown function> + 0xaa04 (0x40002e3caa04 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #32: <unknown function> + 0xbcc4 (0x40002e3cbcc4 in /e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/lib-dynload/_asyncio.cpython-312-aarch64-linux-gnu.so)
frame #33: _PyObject_MakeTpCall + 0x98 (0xaaaad8385db8 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #34: <unknown function> + 0x28cfac (0xaaaad84acfac in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #35: <unknown function> + 0x1b7b68 (0xaaaad83d7b68 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #37: <unknown function> + 0x83c564 (0x40002c14c564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_raylet.so)
frame #38: _PyEval_EvalFrameDefault + 0x52ac (0xaaaad848d8f4 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #39: <unknown function> + 0x169f88 (0xaaaad8389f88 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #40: <unknown function> + 0x36002c (0xaaaad858002c in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #41: <unknown function> + 0x2e4024 (0xaaaad8504024 in ray::FSDPPolicyWorkerBase.init_worker_process_group)
frame #42: <unknown function> + 0x80e00 (0x40002b080e00 in /lib64/libc.so.6)
frame #43: <unknown function> + 0xeb49c (0x40002b0eb49c in /lib64/libc.so.6)
Stopping Ray cluster...
  Warning: Failed to stop Ray on jpbo-001-42: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-42', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
  Warning: Failed to stop Ray on jpbo-001-43: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-43', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
  Warning: Failed to stop Ray on jpbo-001-45: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-45', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
  Warning: Failed to stop Ray on jpbo-001-46: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-46', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
  Warning: Failed to stop Ray on jpbo-001-47: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-47', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
  Warning: Failed to stop Ray on jpbo-003-39: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-003-39', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; ray stop --force']' timed out after 30 seconds
Ray cluster stopped
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ablation-pymethods2test-seqmean-arm0-tis/trace_jobs, skipping upload.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0-tis/ray_logs/
  Collecting Ray logs from worker jpbo-001-36...
  Collecting Ray logs from worker jpbo-001-37...
  Collecting Ray logs from worker jpbo-001-38...
  Collecting Ray logs from worker jpbo-001-41...
  Collecting Ray logs from worker jpbo-001-42...
  Collecting Ray logs from worker jpbo-001-43...
  Collecting Ray logs from worker jpbo-001-44...
  Collecting Ray logs from worker jpbo-001-45...
  Collecting Ray logs from worker jpbo-001-46...
  Collecting Ray logs from worker jpbo-001-47...
  Collecting Ray logs from worker jpbo-001-48...
  Collecting Ray logs from worker jpbo-003-37...
  Collecting Ray logs from worker jpbo-003-39...
Ray log preservation complete
