laion/ablation-pymethods2test-seqmean-arm0-30-8B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
ablation-pymethods2test-seq…/training_logs/ablation-pymethods2test-seqmean-arm0_630125.out
ModelHub XC 0c293fc953 初始化项目，由ModelHub XC社区提供模型 
Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B
Source: Original Platform
2026-07-18 18:02:02 +08:00

1553 lines
280 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The following have been reloaded with a version change:
1) GCCcore/.14.3.0 => GCCcore/14.3.0
Lmod is automatically replacing "GCC/14.3.0" with
"nvidia-compilers/25.9-CUDA-13".
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
[ray] RAY_TMPDIR=/tmp/ray/ray_630125
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630125
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630125
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
[proxy] Setting up SSH tunnel to jpbl-s01-01
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
[proxy] Tunnel port: 7003
[proxy] Node IP: 10.128.32.34 (workers will connect here)
[proxy] ✓ SSH tunnel started successfully
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630125.conf
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
[proxy] - External traffic (internet) → PROXY via tunnel
[proxy] ✓ Daytona timeout settings configured
[proxy] Testing proxy connectivity...
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630125.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
[proxy] ⚠ Tunnel not accessible at 10.128.32.34:7003 (workers may fail)
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
[container_runtime] Using cloud backend: daytona (no local container setup)
=== Universal RL Training Runner ===
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Python version: Python 3.12.12
UV_USE_IO_URING: 0
Proxy: DISABLED (direct internet or not configured)
========================================
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
HF_TOKEN=****pDbg
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
Environment configured:
TENSOR_PARALLEL_SIZE=1
NUM_INFERENCE_ENGINES=56
POLICY_NUM_NODES=14
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
Starting Ray cluster with 14 nodes, 4 GPUs/node
Cleaning up existing Ray instances...
=== Starting Ray Cluster ===
Nodes: 14
GPUs per node: 4
CPUs per node: 288
Head node: jpbo-041-34 (10.128.32.34)
Ray port: 6379
============================
Starting Ray head on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-34.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.34 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray head on jpbo-041-34
Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 1 on jpbo-041-36
Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 2 on jpbo-041-37
Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 3 on jpbo-041-38
Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 4 on jpbo-041-39
Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 5 on jpbo-041-40
Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 6 on jpbo-041-41
Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 7 on jpbo-041-42
Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 8 on jpbo-041-43
Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 9 on jpbo-041-44
Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 10 on jpbo-041-45
Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 11 on jpbo-041-46
Starting Ray worker on jpbo-041-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-47.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 12 on jpbo-041-47
Starting Ray worker on jpbo-041-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-48.log)...
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.34:6379 --node-ip-address=10.128.32.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
Started Ray worker 13 on jpbo-041-48
Waiting for cluster (56 GPUs, 14 nodes)...
Connecting to Ray at 10.128.32.34:6379 (expecting 14 nodes, 56.0 GPUs)
Ray connection established, polling for resources...
[Ray wait] nodes=14/14 GPUs=56.0/56.0 resources={'object_store_memory': 601295421440.0, 'memory': 10695797243904.0, 'GPU': 56.0, 'CPU': 4032.0, 'accelerator_type:GH200': 14.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.39': 1.0, 'node:10.128.32.37': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.40': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.36': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.41': 1.0, 'node:10.128.32.48': 1.0, 'node:10.128.32.47': 1.0}
✓ Ray cluster ready
=== Ray Cluster Ready ===
Address: 10.128.32.34:6379
Total GPUs: 56
=========================
Ray cluster ready at 10.128.32.34:6379
Total GPUs available: 56
[RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona)
[RLJobRunner] No Pinggy tunnel needed, using local vLLM
Running SkyRL:
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
Entrypoint: examples.terminal_bench.entrypoints.main_tbench
Args: 120 Hydra arguments
Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train
Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.05 trainer.algorithm.loss_reduction=sequence_mean trainer.epochs=2 trainer.max_steps=80 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=4 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=2 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0 ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=false trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=ablation-pymethods2test-seqmean-arm0 trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports trainer.policy.optimizer_config.lr=8e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=0.9 trainer.policy.fsdp_config.cpu_offload=false trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 trainer.ref.fsdp_config.cpu_offload=false trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=2 trainer.placement.ref_num_nodes=2 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=338 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=1 generator.num_inference_engines=48 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.75 generator.max_num_seqs=24 generator.max_num_batched_tokens=65536 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=4096 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"] data.val_data=[] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=900 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=675 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"] +terminal_bench_config.harbor.zero_exceptions=[] +terminal_bench_config.model_info.max_input_tokens=32000 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630125.conf
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 02:11:35.483 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:686 - Exporting wandb api key to ray runtime env
2026-06-07 02:11:35.483 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:705 - Exporting RAY_ADDRESS to ray runtime env
2026-06-07 02:11:35.483 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_SOCKET_IFNAME` to ray runtime env: ib0
2026-06-07 02:11:35.484 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_SOCKET_FAMILY` to ray runtime env: AF_INET
2026-06-07 02:11:35.484 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:730 - Exporting `NCCL_DEBUG` to ray runtime env: WARN
2026-06-07 02:11:35,484 INFO worker.py:1680 -- Using address 10.128.32.34:6379 set in the environment variable RAY_ADDRESS
2026-06-07 02:11:35,519 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.34:6379...
2026-06-07 02:11:35,529 INFO worker.py:2007 -- Connected to Ray cluster.
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(raylet, ip=10.128.32.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
2026-06-07 02:11:37.864 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor
(skyrl_entrypoint pid=2701319) 2026-06-07 02:11:44.748 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:212 - SkyRL log level set to: INFO
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(skyrl_entrypoint pid=2701319) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(skyrl_entrypoint pid=2701319) 2026-06-07 02:11:45.133 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=2701319) 2026-06-07 02:11:48.174 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 5000 valid task directories out of 5000 total directories
(skyrl_entrypoint pid=2701319) 2026-06-07 02:11:48.174 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 5000 task paths
(skyrl_entrypoint pid=2701319) 2026-06-07 02:11:48.187 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:405 - data:
(skyrl_entrypoint pid=2701319) train_data:
(skyrl_entrypoint pid=2701319) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large
(skyrl_entrypoint pid=2701319) val_data: []
(skyrl_entrypoint pid=2701319) trainer:
(skyrl_entrypoint pid=2701319) placement:
(skyrl_entrypoint pid=2701319) colocate_all: false
(skyrl_entrypoint pid=2701319) colocate_policy_ref: true
(skyrl_entrypoint pid=2701319) policy_num_nodes: 2
(skyrl_entrypoint pid=2701319) policy_num_gpus_per_node: 4
(skyrl_entrypoint pid=2701319) critic_num_nodes: 1
(skyrl_entrypoint pid=2701319) critic_num_gpus_per_node: 4
(skyrl_entrypoint pid=2701319) ref_num_nodes: 2
(skyrl_entrypoint pid=2701319) ref_num_gpus_per_node: 4
(skyrl_entrypoint pid=2701319) policy_strict_spread_pg: false
(skyrl_entrypoint pid=2701319) policy_per_gpu_bundles: false
(skyrl_entrypoint pid=2701319) sequence_parallel_backend: ulysses
(skyrl_entrypoint pid=2701319) strategy: fsdp2
(skyrl_entrypoint pid=2701319) policy:
(skyrl_entrypoint pid=2701319) model:
(skyrl_entrypoint pid=2701319) path: /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=2701319) lora:
(skyrl_entrypoint pid=2701319) rank: 0
(skyrl_entrypoint pid=2701319) alpha: 16
(skyrl_entrypoint pid=2701319) dropout: 0
(skyrl_entrypoint pid=2701319) lora_sync_path: /tmp/skyrl_lora_sync
(skyrl_entrypoint pid=2701319) target_modules: all-linear
(skyrl_entrypoint pid=2701319) exclude_modules: null
(skyrl_entrypoint pid=2701319) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=2701319) optimizer_config:
(skyrl_entrypoint pid=2701319) optimizer: AdamW
(skyrl_entrypoint pid=2701319) lr: 8.0e-06
(skyrl_entrypoint pid=2701319) adam_betas:
(skyrl_entrypoint pid=2701319) - 0.9
(skyrl_entrypoint pid=2701319) - 0.999
(skyrl_entrypoint pid=2701319) weight_decay: 0.0
(skyrl_entrypoint pid=2701319) max_grad_norm: 0.9
(skyrl_entrypoint pid=2701319) offload_after_step: true
(skyrl_entrypoint pid=2701319) num_warmup_steps: 0
(skyrl_entrypoint pid=2701319) scheduler: constant_with_warmup
(skyrl_entrypoint pid=2701319) optimizer_kwargs: {}
(skyrl_entrypoint pid=2701319) fsdp_config:
(skyrl_entrypoint pid=2701319) cpu_offload: false
(skyrl_entrypoint pid=2701319) reshard_after_forward: true
(skyrl_entrypoint pid=2701319) fsdp_size: 4
(skyrl_entrypoint pid=2701319) expert_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=2701319) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=2701319) moe_router_replay: false
(skyrl_entrypoint pid=2701319) moe_grouped_gemm: false
(skyrl_entrypoint pid=2701319) ep_comm_backend: torch
(skyrl_entrypoint pid=2701319) deepep_num_sms: 20
(skyrl_entrypoint pid=2701319) deepep_token_chunk_size: null
(skyrl_entrypoint pid=2701319) sequence_parallel_size: 1
(skyrl_entrypoint pid=2701319) use_torch_compile: false
(skyrl_entrypoint pid=2701319) record_memory: false
(skyrl_entrypoint pid=2701319) megatron_config:
(skyrl_entrypoint pid=2701319) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) context_parallel_size: 1
(skyrl_entrypoint pid=2701319) expert_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) expert_tensor_parallel_size: null
(skyrl_entrypoint pid=2701319) ddp_config:
(skyrl_entrypoint pid=2701319) grad_reduce_in_fp32: true
(skyrl_entrypoint pid=2701319) overlap_grad_reduce: false
(skyrl_entrypoint pid=2701319) overlap_param_gather: false
(skyrl_entrypoint pid=2701319) average_in_collective: true
(skyrl_entrypoint pid=2701319) model_config_kwargs: {}
(skyrl_entrypoint pid=2701319) torch_profiler_config:
(skyrl_entrypoint pid=2701319) enable: false
(skyrl_entrypoint pid=2701319) ranks: []
(skyrl_entrypoint pid=2701319) save_path: null
(skyrl_entrypoint pid=2701319) optimizer_config_kwargs:
(skyrl_entrypoint pid=2701319) overlap_cpu_optimizer_d2h_h2d: false
(skyrl_entrypoint pid=2701319) use_precision_aware_optimizer: false
(skyrl_entrypoint pid=2701319) optimizer_cpu_offload: false
(skyrl_entrypoint pid=2701319) optimizer_offload_fraction: 0.0
(skyrl_entrypoint pid=2701319) transformer_config_kwargs:
(skyrl_entrypoint pid=2701319) recompute_granularity: full
(skyrl_entrypoint pid=2701319) recompute_modules:
(skyrl_entrypoint pid=2701319) - core_attn
(skyrl_entrypoint pid=2701319) recompute_method: uniform
(skyrl_entrypoint pid=2701319) recompute_num_layers: 1
(skyrl_entrypoint pid=2701319) empty_cuda_cache: true
(skyrl_entrypoint pid=2701319) ref:
(skyrl_entrypoint pid=2701319) model:
(skyrl_entrypoint pid=2701319) path: ${trainer.policy.model.path}
(skyrl_entrypoint pid=2701319) sequence_parallel_size: 1
(skyrl_entrypoint pid=2701319) deepspeed_config: ${deepspeed_config.eval}
(skyrl_entrypoint pid=2701319) fsdp_config:
(skyrl_entrypoint pid=2701319) cpu_offload: false
(skyrl_entrypoint pid=2701319) reshard_after_forward: true
(skyrl_entrypoint pid=2701319) fsdp_size: 4
(skyrl_entrypoint pid=2701319) expert_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=2701319) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=2701319) moe_router_replay: false
(skyrl_entrypoint pid=2701319) moe_grouped_gemm: false
(skyrl_entrypoint pid=2701319) ep_comm_backend: torch
(skyrl_entrypoint pid=2701319) deepep_num_sms: 20
(skyrl_entrypoint pid=2701319) deepep_token_chunk_size: null
(skyrl_entrypoint pid=2701319) megatron_config:
(skyrl_entrypoint pid=2701319) tensor_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) pipeline_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) context_parallel_size: 1
(skyrl_entrypoint pid=2701319) expert_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=2701319) model_config_kwargs: {}
(skyrl_entrypoint pid=2701319) transformer_config_kwargs: {}
(skyrl_entrypoint pid=2701319) critic:
(skyrl_entrypoint pid=2701319) model:
(skyrl_entrypoint pid=2701319) path: null
(skyrl_entrypoint pid=2701319) lora:
(skyrl_entrypoint pid=2701319) rank: 0
(skyrl_entrypoint pid=2701319) alpha: 16
(skyrl_entrypoint pid=2701319) dropout: 0
(skyrl_entrypoint pid=2701319) target_modules: all-linear
(skyrl_entrypoint pid=2701319) exclude_modules: null
(skyrl_entrypoint pid=2701319) deepspeed_config: ${deepspeed_config.train}
(skyrl_entrypoint pid=2701319) optimizer_config:
(skyrl_entrypoint pid=2701319) optimizer: AdamW
(skyrl_entrypoint pid=2701319) lr: 5.0e-06
(skyrl_entrypoint pid=2701319) adam_betas:
(skyrl_entrypoint pid=2701319) - 0.9
(skyrl_entrypoint pid=2701319) - 0.999
(skyrl_entrypoint pid=2701319) weight_decay: 0.01
(skyrl_entrypoint pid=2701319) max_grad_norm: 1.0
(skyrl_entrypoint pid=2701319) offload_after_step: true
(skyrl_entrypoint pid=2701319) num_warmup_steps: 0
(skyrl_entrypoint pid=2701319) scheduler: constant_with_warmup
(skyrl_entrypoint pid=2701319) optimizer_kwargs: {}
(skyrl_entrypoint pid=2701319) fsdp_config:
(skyrl_entrypoint pid=2701319) cpu_offload: false
(skyrl_entrypoint pid=2701319) reshard_after_forward: true
(skyrl_entrypoint pid=2701319) fsdp_size: -1
(skyrl_entrypoint pid=2701319) expert_model_parallel_size: 1
(skyrl_entrypoint pid=2701319) expert_tensor_parallel_size: 1
(skyrl_entrypoint pid=2701319) moe_token_dispatcher_type: alltoall
(skyrl_entrypoint pid=2701319) moe_router_replay: false
(skyrl_entrypoint pid=2701319) moe_grouped_gemm: false
(skyrl_entrypoint pid=2701319) ep_comm_backend: torch
(skyrl_entrypoint pid=2701319) deepep_num_sms: 20
(skyrl_entrypoint pid=2701319) deepep_token_chunk_size: null
(skyrl_entrypoint pid=2701319) sequence_parallel_size: 1
(skyrl_entrypoint pid=2701319) algorithm:
(skyrl_entrypoint pid=2701319) advantage_estimator: rloo_n
(skyrl_entrypoint pid=2701319) kl_ctrl:
(skyrl_entrypoint pid=2701319) type: fixed
(skyrl_entrypoint pid=2701319) kl_target: 0.1
(skyrl_entrypoint pid=2701319) horizon: 10000
(skyrl_entrypoint pid=2701319) kl_estimator_type: k3
(skyrl_entrypoint pid=2701319) use_kl_estimator_k3: false
(skyrl_entrypoint pid=2701319) use_abs_kl: false
(skyrl_entrypoint pid=2701319) use_kl_in_reward: false
(skyrl_entrypoint pid=2701319) use_kl_loss: false
(skyrl_entrypoint pid=2701319) kl_loss_coef: 0.0
(skyrl_entrypoint pid=2701319) use_entropy_loss: false
(skyrl_entrypoint pid=2701319) entropy_loss_coef: 0.01
(skyrl_entrypoint pid=2701319) advantage_batch_normalize: false
(skyrl_entrypoint pid=2701319) value_head_prefix: value_head
(skyrl_entrypoint pid=2701319) policy_loss_type: regular
(skyrl_entrypoint pid=2701319) loss_reduction: sequence_mean
(skyrl_entrypoint pid=2701319) global_loss_denom: null
(skyrl_entrypoint pid=2701319) grpo_norm_by_std: true
(skyrl_entrypoint pid=2701319) rloo_n_min_group_size: 4
(skyrl_entrypoint pid=2701319) rloo_n_filter_zero_reward_groups: true
(skyrl_entrypoint pid=2701319) lambd: 1.0
(skyrl_entrypoint pid=2701319) gamma: 1.0
(skyrl_entrypoint pid=2701319) eps_clip_low: 0.2
(skyrl_entrypoint pid=2701319) eps_clip_high: 0.05
(skyrl_entrypoint pid=2701319) clip_ratio_c: 3.0
(skyrl_entrypoint pid=2701319) tis_imp_ratio_cap: -1.0
(skyrl_entrypoint pid=2701319) use_tis: false
(skyrl_entrypoint pid=2701319) sapo:
(skyrl_entrypoint pid=2701319) tau_pos: 1.0
(skyrl_entrypoint pid=2701319) tau_neg: 1.05
(skyrl_entrypoint pid=2701319) value_clip: 0.2
(skyrl_entrypoint pid=2701319) dynamic_sampling:
(skyrl_entrypoint pid=2701319) type: null
(skyrl_entrypoint pid=2701319) max_sample_batches: 30
(skyrl_entrypoint pid=2701319) min_replace_ratio: 0.3
(skyrl_entrypoint pid=2701319) clip_cov:
(skyrl_entrypoint pid=2701319) clip_ratio: 0.0002
(skyrl_entrypoint pid=2701319) clip_cov_lb: 1.0
(skyrl_entrypoint pid=2701319) clip_cov_ub: 5.0
(skyrl_entrypoint pid=2701319) kl_cov:
(skyrl_entrypoint pid=2701319) kl_cov_frac: 0.2
(skyrl_entrypoint pid=2701319) ppo_kl_coef: 1.0
(skyrl_entrypoint pid=2701319) cispo:
(skyrl_entrypoint pid=2701319) cispo_eps_clip_low: 0
(skyrl_entrypoint pid=2701319) cispo_eps_clip_high: 5
(skyrl_entrypoint pid=2701319) z_clip:
(skyrl_entrypoint pid=2701319) enabled: false
(skyrl_entrypoint pid=2701319) alpha: 0.97
(skyrl_entrypoint pid=2701319) z_thresh: 2.5
(skyrl_entrypoint pid=2701319) warmup_steps: 25
(skyrl_entrypoint pid=2701319) mode: zscore
(skyrl_entrypoint pid=2701319) clip_option: adaptive_scaling
(skyrl_entrypoint pid=2701319) clip_factor: 1.0
(skyrl_entrypoint pid=2701319) skip_update_on_spike: false
(skyrl_entrypoint pid=2701319) stale_clip:
(skyrl_entrypoint pid=2701319) enabled: false
(skyrl_entrypoint pid=2701319) alpha: 0.3
(skyrl_entrypoint pid=2701319) entropy_threshold: 0.15
(skyrl_entrypoint pid=2701319) entropy_window: 10
(skyrl_entrypoint pid=2701319) min_lr_scale: 0.1
(skyrl_entrypoint pid=2701319) max_seq_len: 1004095
(skyrl_entrypoint pid=2701319) fully_async:
(skyrl_entrypoint pid=2701319) max_staleness_steps: 16
(skyrl_entrypoint pid=2701319) num_parallel_generation_workers: 338
(skyrl_entrypoint pid=2701319) gradient_checkpointing: true
(skyrl_entrypoint pid=2701319) gradient_checkpointing_use_reentrant: false
(skyrl_entrypoint pid=2701319) seed: 42
(skyrl_entrypoint pid=2701319) resume_mode: latest
(skyrl_entrypoint pid=2701319) resume_path: null
(skyrl_entrypoint pid=2701319) ckpt_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints
(skyrl_entrypoint pid=2701319) max_ckpts_to_keep: -1
(skyrl_entrypoint pid=2701319) ckpt_interval: 2
(skyrl_entrypoint pid=2701319) hf_save_interval: 5
(skyrl_entrypoint pid=2701319) hf_upload_mode: latest
(skyrl_entrypoint pid=2701319) export_path: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports
(skyrl_entrypoint pid=2701319) bf16: true
(skyrl_entrypoint pid=2701319) epochs: 2
(skyrl_entrypoint pid=2701319) max_steps: 80
(skyrl_entrypoint pid=2701319) update_epochs_per_batch: 1
(skyrl_entrypoint pid=2701319) train_batch_size: 64
(skyrl_entrypoint pid=2701319) policy_mini_batch_size: 64
(skyrl_entrypoint pid=2701319) critic_mini_batch_size: 256
(skyrl_entrypoint pid=2701319) micro_train_batch_size_per_gpu: 1
(skyrl_entrypoint pid=2701319) micro_forward_batch_size_per_gpu: 4
(skyrl_entrypoint pid=2701319) update_ref_every_epoch: false
(skyrl_entrypoint pid=2701319) use_sample_packing: true
(skyrl_entrypoint pid=2701319) eval_batch_size: 64
(skyrl_entrypoint pid=2701319) eval_before_train: false
(skyrl_entrypoint pid=2701319) eval_interval: 999999
(skyrl_entrypoint pid=2701319) max_prompt_length: 999999
(skyrl_entrypoint pid=2701319) flash_attn: true
(skyrl_entrypoint pid=2701319) disable_fast_tokenizer: false
(skyrl_entrypoint pid=2701319) target_modules: null
(skyrl_entrypoint pid=2701319) exclude_modules: null
(skyrl_entrypoint pid=2701319) project_name: OpenThoughts-Agent
(skyrl_entrypoint pid=2701319) run_name: ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=2701319) logger: console
(skyrl_entrypoint pid=2701319) tracker_commit_each_step: true
(skyrl_entrypoint pid=2701319) dump_data_batch: false
(skyrl_entrypoint pid=2701319) dump_eval_results: true
(skyrl_entrypoint pid=2701319) log_level: INFO
(skyrl_entrypoint pid=2701319) rope_scaling: null
(skyrl_entrypoint pid=2701319) rope_theta: null
(skyrl_entrypoint pid=2701319) step_wise_training: false
(skyrl_entrypoint pid=2701319) hf_hub_repo_id: laion/ablation-pymethods2test-seqmean-arm0
(skyrl_entrypoint pid=2701319) hf_hub_private: false
(skyrl_entrypoint pid=2701319) hf_hub_revision: main
(skyrl_entrypoint pid=2701319) enable_db_registration: false
(skyrl_entrypoint pid=2701319) generator:
(skyrl_entrypoint pid=2701319) model_name: ${trainer.policy.model.path}
(skyrl_entrypoint pid=2701319) model_dtype: bfloat16
(skyrl_entrypoint pid=2701319) timeout_multiplier: 1.0
(skyrl_entrypoint pid=2701319) run_engines_locally: true
(skyrl_entrypoint pid=2701319) num_inference_engines: 48
(skyrl_entrypoint pid=2701319) backend: vllm
(skyrl_entrypoint pid=2701319) weight_sync_backend: nccl
(skyrl_entrypoint pid=2701319) fuse_weights: false
(skyrl_entrypoint pid=2701319) weight_transfer_threshold_cuda_ipc_GB: 1.0
(skyrl_entrypoint pid=2701319) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=2701319) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=2701319) inference_engine_expert_parallel_size: 1
(skyrl_entrypoint pid=2701319) inference_engine_data_parallel_size: 1
(skyrl_entrypoint pid=2701319) n_samples_per_prompt: 8
(skyrl_entrypoint pid=2701319) async_engine: true
(skyrl_entrypoint pid=2701319) batched: false
(skyrl_entrypoint pid=2701319) max_input_length: ${trainer.max_prompt_length}
(skyrl_entrypoint pid=2701319) vllm_v1_disable_multiproc: true
(skyrl_entrypoint pid=2701319) enable_prefix_caching: true
(skyrl_entrypoint pid=2701319) enable_chunked_prefill: true
(skyrl_entrypoint pid=2701319) max_num_batched_tokens: 65536
(skyrl_entrypoint pid=2701319) enforce_eager: true
(skyrl_entrypoint pid=2701319) fully_sharded_loras: false
(skyrl_entrypoint pid=2701319) enable_ray_prometheus_stats: false
(skyrl_entrypoint pid=2701319) vllm_stats_interval: 1
(skyrl_entrypoint pid=2701319) gpu_memory_utilization: 0.75
(skyrl_entrypoint pid=2701319) max_num_seqs: 24
(skyrl_entrypoint pid=2701319) remote_inference_engine_urls:
(skyrl_entrypoint pid=2701319) - 127.0.0.1:8001
(skyrl_entrypoint pid=2701319) enable_http_endpoint: true
(skyrl_entrypoint pid=2701319) http_endpoint_host: 127.0.0.1
(skyrl_entrypoint pid=2701319) http_endpoint_port: 8000
(skyrl_entrypoint pid=2701319) max_turns: 999999
(skyrl_entrypoint pid=2701319) chat_template:
(skyrl_entrypoint pid=2701319) source: name
(skyrl_entrypoint pid=2701319) name_or_path: null
(skyrl_entrypoint pid=2701319) chat_template_kwargs: {}
(skyrl_entrypoint pid=2701319) engine_init_kwargs:
(skyrl_entrypoint pid=2701319) max_model_len: 32768
(skyrl_entrypoint pid=2701319) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=2701319) served_model_name: 0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6
(skyrl_entrypoint pid=2701319) override_existing_update_group: disable
(skyrl_entrypoint pid=2701319) sampling_params:
(skyrl_entrypoint pid=2701319) max_generate_length: 4096
(skyrl_entrypoint pid=2701319) repetition_penalty: 1.0
(skyrl_entrypoint pid=2701319) temperature: 0.7
(skyrl_entrypoint pid=2701319) top_p: 0.95
(skyrl_entrypoint pid=2701319) min_p: 0.0
(skyrl_entrypoint pid=2701319) top_k: 20
(skyrl_entrypoint pid=2701319) logprobs: null
(skyrl_entrypoint pid=2701319) stop: null
(skyrl_entrypoint pid=2701319) use_conversation_multi_turn: true
(skyrl_entrypoint pid=2701319) append_eos_token_after_stop_str_in_multi_turn: true
(skyrl_entrypoint pid=2701319) eval_sampling_params:
(skyrl_entrypoint pid=2701319) max_generate_length: ${generator.sampling_params.max_generate_length}
(skyrl_entrypoint pid=2701319) repetition_penalty: 1.0
(skyrl_entrypoint pid=2701319) temperature: 0.0
(skyrl_entrypoint pid=2701319) top_p: 1.0
(skyrl_entrypoint pid=2701319) min_p: 0.0
(skyrl_entrypoint pid=2701319) top_k: -1
(skyrl_entrypoint pid=2701319) logprobs: null
(skyrl_entrypoint pid=2701319) stop: null
(skyrl_entrypoint pid=2701319) eval_n_samples_per_prompt: 8
(skyrl_entrypoint pid=2701319) zero_reward_on_non_stop: false
(skyrl_entrypoint pid=2701319) apply_overlong_filtering: false
(skyrl_entrypoint pid=2701319) rope_scaling: ${trainer.rope_scaling}
(skyrl_entrypoint pid=2701319) rope_theta: ${trainer.rope_theta}
(skyrl_entrypoint pid=2701319) teacher:
(skyrl_entrypoint pid=2701319) model_path: null
(skyrl_entrypoint pid=2701319) top_k_logprobs: 256
(skyrl_entrypoint pid=2701319) num_inference_engines: 1
(skyrl_entrypoint pid=2701319) inference_engine_tensor_parallel_size: 1
(skyrl_entrypoint pid=2701319) inference_engine_pipeline_parallel_size: 1
(skyrl_entrypoint pid=2701319) gpu_memory_utilization: 0.9
(skyrl_entrypoint pid=2701319) enforce_eager: false
(skyrl_entrypoint pid=2701319) backend: vllm
(skyrl_entrypoint pid=2701319) engine_init_kwargs: {}
(skyrl_entrypoint pid=2701319) environment:
(skyrl_entrypoint pid=2701319) env_class: gsm8k
(skyrl_entrypoint pid=2701319) skyrl_gym:
(skyrl_entrypoint pid=2701319) max_env_workers: 32
(skyrl_entrypoint pid=2701319) text2sql:
(skyrl_entrypoint pid=2701319) db_path: /home/ray/default/sql_data
(skyrl_entrypoint pid=2701319) llm_as_a_judge:
(skyrl_entrypoint pid=2701319) model: gpt-4o-mini
(skyrl_entrypoint pid=2701319) base_url: null
(skyrl_entrypoint pid=2701319) search:
(skyrl_entrypoint pid=2701319) log_requests: false
(skyrl_entrypoint pid=2701319) search_url: http://127.0.0.1:8000/retrieve
(skyrl_entrypoint pid=2701319) topk: 3
(skyrl_entrypoint pid=2701319) timeout: 30
(skyrl_entrypoint pid=2701319) rollout:
(skyrl_entrypoint pid=2701319) fanout:
(skyrl_entrypoint pid=2701319) enabled: true
(skyrl_entrypoint pid=2701319) num_coordinators: 4
(skyrl_entrypoint pid=2701319) cpus_per_coordinator: 8
(skyrl_entrypoint pid=2701319) deepspeed_config:
(skyrl_entrypoint pid=2701319) train:
(skyrl_entrypoint pid=2701319) zero_optimization:
(skyrl_entrypoint pid=2701319) stage: 3
(skyrl_entrypoint pid=2701319) offload_param:
(skyrl_entrypoint pid=2701319) device: none
(skyrl_entrypoint pid=2701319) offload_optimizer:
(skyrl_entrypoint pid=2701319) device: none
(skyrl_entrypoint pid=2701319) pin_memory: true
(skyrl_entrypoint pid=2701319) sub_group_size: auto
(skyrl_entrypoint pid=2701319) reduce_bucket_size: auto
(skyrl_entrypoint pid=2701319) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=2701319) stage3_prefetch_bucket_size: auto
(skyrl_entrypoint pid=2701319) stage3_max_live_parameters: auto
(skyrl_entrypoint pid=2701319) stage3_max_reuse_distance: auto
(skyrl_entrypoint pid=2701319) round_robin_gradients: true
(skyrl_entrypoint pid=2701319) zero_hpz_partition_size: 1
(skyrl_entrypoint pid=2701319) zero_quantized_weights: false
(skyrl_entrypoint pid=2701319) zero_quantized_gradients: false
(skyrl_entrypoint pid=2701319) torch_autocast:
(skyrl_entrypoint pid=2701319) enabled: true
(skyrl_entrypoint pid=2701319) dtype: bfloat16
(skyrl_entrypoint pid=2701319) disable_trace_cache: false
(skyrl_entrypoint pid=2701319) data_types:
(skyrl_entrypoint pid=2701319) grad_accum_dtype: fp32
(skyrl_entrypoint pid=2701319) gradient_clipping: 1.0
(skyrl_entrypoint pid=2701319) wall_clock_breakdown: false
(skyrl_entrypoint pid=2701319) prescale_gradient: false
(skyrl_entrypoint pid=2701319) eval:
(skyrl_entrypoint pid=2701319) zero_optimization:
(skyrl_entrypoint pid=2701319) stage: 3
(skyrl_entrypoint pid=2701319) stage3_param_persistence_threshold: auto
(skyrl_entrypoint pid=2701319) offload_param:
(skyrl_entrypoint pid=2701319) device: cpu
(skyrl_entrypoint pid=2701319) pin_memory: true
(skyrl_entrypoint pid=2701319) torch_autocast:
(skyrl_entrypoint pid=2701319) enabled: true
(skyrl_entrypoint pid=2701319) dtype: bfloat16
(skyrl_entrypoint pid=2701319) gradient_clipping: 1.0
(skyrl_entrypoint pid=2701319) prescale_gradient: false
(skyrl_entrypoint pid=2701319) wall_clock_breakdown: false
(skyrl_entrypoint pid=2701319) terminal_bench_config:
(skyrl_entrypoint pid=2701319) trials_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs
(skyrl_entrypoint pid=2701319) harbor:
(skyrl_entrypoint pid=2701319) name: terminus-2
(skyrl_entrypoint pid=2701319) max_episodes: 999999
(skyrl_entrypoint pid=2701319) enable_summarize: false
(skyrl_entrypoint pid=2701319) store_all_messages: true
(skyrl_entrypoint pid=2701319) trajectory_config:
(skyrl_entrypoint pid=2701319) raw_content: true
(skyrl_entrypoint pid=2701319) enable_episode_logging: false
(skyrl_entrypoint pid=2701319) record_terminal_session: false
(skyrl_entrypoint pid=2701319) enable_pane_logging: false
(skyrl_entrypoint pid=2701319) strict_json_parser: true
(skyrl_entrypoint pid=2701319) interleaved_thinking: true
(skyrl_entrypoint pid=2701319) extra_body:
(skyrl_entrypoint pid=2701319) chat_template_kwargs:
(skyrl_entrypoint pid=2701319) enable_thinking: true
(skyrl_entrypoint pid=2701319) override_timeout_sec: 900
(skyrl_entrypoint pid=2701319) override_cpus: 1
(skyrl_entrypoint pid=2701319) override_memory_mb: 2048
(skyrl_entrypoint pid=2701319) override_storage_mb: 2048
(skyrl_entrypoint pid=2701319) auto_snapshot: true
(skyrl_entrypoint pid=2701319) verifier_override_timeout_sec: 120
(skyrl_entrypoint pid=2701319) max_retries: 3
(skyrl_entrypoint pid=2701319) min_wait_sec: 60.0
(skyrl_entrypoint pid=2701319) max_wait_sec: 600.0
(skyrl_entrypoint pid=2701319) wait_multiplier: 2.0
(skyrl_entrypoint pid=2701319) exclude_exceptions:
(skyrl_entrypoint pid=2701319) - VerifierTimeoutError
(skyrl_entrypoint pid=2701319) - VerifierRuntimeError
(skyrl_entrypoint pid=2701319) - RewardFileNotFoundError
(skyrl_entrypoint pid=2701319) - RewardFileEmptyError
(skyrl_entrypoint pid=2701319) - VerifierOutputParseError
(skyrl_entrypoint pid=2701319) n_concurrent_trials: 675
(skyrl_entrypoint pid=2701319) log_level: INFO
(skyrl_entrypoint pid=2701319) enable_reward_shaping: false
(skyrl_entrypoint pid=2701319) enable_error_classification: true
(skyrl_entrypoint pid=2701319) mask_exceptions:
(skyrl_entrypoint pid=2701319) - DaytonaError
(skyrl_entrypoint pid=2701319) - EnvironmentStartTimeoutError
(skyrl_entrypoint pid=2701319) - NetworkError
(skyrl_entrypoint pid=2701319) - ConnectionError
(skyrl_entrypoint pid=2701319) - RewardFileNotFoundError
(skyrl_entrypoint pid=2701319) - RewardFileEmptyError
(skyrl_entrypoint pid=2701319) - AgentEnvironmentTimeoutError
(skyrl_entrypoint pid=2701319) - ContextLengthExceededError
(skyrl_entrypoint pid=2701319) default_error_treatment: zero
(skyrl_entrypoint pid=2701319) passthrough_exceptions:
(skyrl_entrypoint pid=2701319) - AgentTimeoutError
(skyrl_entrypoint pid=2701319) zero_exceptions: []
(skyrl_entrypoint pid=2701319) model_info:
(skyrl_entrypoint pid=2701319) max_input_tokens: 32000
(skyrl_entrypoint pid=2701319) max_output_tokens: 4096
(skyrl_entrypoint pid=2701319) archiving:
(skyrl_entrypoint pid=2701319) enabled: false
(skyrl_entrypoint pid=2701319) trace_upload:
(skyrl_entrypoint pid=2701319) enabled: true
(skyrl_entrypoint pid=2701319) repo_org: DCAgent
(skyrl_entrypoint pid=2701319) episodes: last
(skyrl_entrypoint pid=2701319) dataset_type: SFT
(skyrl_entrypoint pid=2701319) cleanup: true
(skyrl_entrypoint pid=2701319) 
(skyrl_entrypoint pid=2701319) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(skyrl_entrypoint pid=2701319) No module named 'vllm._version'
(skyrl_entrypoint pid=2701319) from .version import __version__, __version_tuple__ # isort:skip
(skyrl_entrypoint pid=2701319) W0607 02:12:04.505000 2701319 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13'
[2026-06-07 02:12:05,742 E 2700889 2701293] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RegistryActor pid=2575022, ip=10.128.32.41) [2026-06-07 02:12:06,659 E 2575022 2575062] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(skyrl_entrypoint pid=2701319) [2026-06-07 02:12:08,855 E 2701319 2701362] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster]
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 31x across cluster]
(pid=2666524, ip=10.128.32.38) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(pid=2666524, ip=10.128.32.38) No module named 'vllm._version'
(pid=2666524, ip=10.128.32.38) from .version import __version__, __version_tuple__ # isort:skip
(raylet, ip=10.128.32.48) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 64x across cluster]
(pid=2539107, ip=10.128.32.40) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(pid=2539107, ip=10.128.32.40) No module named 'vllm._version' [repeated 7x across cluster]
(pid=2539107, ip=10.128.32.40) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(raylet, ip=10.128.32.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 116x across cluster]
(pid=2593412, ip=10.128.32.36) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 16x across cluster]
(pid=2593412, ip=10.128.32.36) No module named 'vllm._version' [repeated 16x across cluster]
(pid=2593412, ip=10.128.32.36) from .version import __version__, __version_tuple__ # isort:skip [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:28.309 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset>
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:28.334 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:28.335 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:28.982 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:29.008 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:29.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:29.008 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:29.008 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) 2026-06-07 02:12:29.059 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.94s (attempt 1/5) to avoid port collisions
(raylet, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 106x across cluster]
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(pid=2659898, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 13x across cluster]
(pid=2659898, ip=10.128.32.43) No module named 'vllm._version' [repeated 13x across cluster]
(pid=2659898, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:31.812 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=3, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:31.812 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:31.812 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 3x across cluster]
(skyrl_entrypoint pid=2701319) [2026-06-07 02:12:33] INFO inference_engine_client_http_endpoint.py:350: Starting server on 0.0.0.0:8000
(skyrl_entrypoint pid=2701319) [2026-06-07 02:12:34] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint...
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:32.760 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 216-287 (NUMA node 3) for GPU 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:32.763 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 3 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:32.763 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:32.763 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:32.763 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) 2026-06-07 02:12:32.784 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.51s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(skyrl_entrypoint pid=2701319) [2026-06-07 02:12:34] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds)
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.803 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:969 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.803 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 48 engines.
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.805 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.806 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 675. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.817 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.818 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 156
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.818 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:357 - Length of train_dataloader: 5000
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.818 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:358 - Number of steps per epoch: 78
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:34.818 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:359 - Total training steps: 80
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 63x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(pid=2955859, ip=10.128.32.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 15x across cluster]
(pid=2955859, ip=10.128.32.39) No module named 'vllm._version' [repeated 15x across cluster]
(pid=2955859, ip=10.128.32.39) from .version import __version__, __version_tuple__ # isort:skip [repeated 15x across cluster]
(pid=2623829, ip=10.128.32.46) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:39.814 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:39.815 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:39.815 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) 2026-06-07 02:12:38.211 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) 2026-06-07 02:12:38.234 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) 2026-06-07 02:12:38.234 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) 2026-06-07 02:12:38.234 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) 2026-06-07 02:12:38.234 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) 2026-06-07 02:12:38.253 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.33s (attempt 1/5) to avoid port collisions
(bundle_reservation_check_func pid=2701396) [2026-06-07 02:12:39,951 E 2701396 2701436] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:40.760 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:40.764 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:40.764 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:40.765 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:40.765 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) 2026-06-07 02:12:40.793 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.14s (attempt 1/5) to avoid port collisions
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) No module named 'vllm._version'
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) from .version import __version__, __version_tuple__ # isort:skip
(AsyncVLLMInferenceEngine pid=3076043, ip=10.128.32.42) 2026-06-07 02:12:45.362 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=0, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076043, ip=10.128.32.42) 2026-06-07 02:12:45.364 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076043, ip=10.128.32.42) 2026-06-07 02:12:45.364 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) [2026-06-07 02:12:41,839 E 2666524 2666626] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) 2026-06-07 02:12:45.860 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) 2026-06-07 02:12:45.883 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) 2026-06-07 02:12:45.883 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) 2026-06-07 02:12:45.883 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) 2026-06-07 02:12:45.883 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) 2026-06-07 02:12:45.916 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.38s (attempt 1/5) to avoid port collisions [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) 2026-06-07 02:12:45 INFO [ipv4-debug] hostname=jpbo-041-46.jupiter.internal
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) 2026-06-07 02:12:45 INFO [ipv4-debug] _global_node.node_ip_address=10.128.32.46
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) 2026-06-07 02:12:45 INFO [ipv4-debug] get_node_ip_address()=10.128.32.46
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:46.122 | INFO  | skyrl_train.workers.worker:_initiate_actors:563 - Initializing process group for RayActorGroup
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [W607 02:12:46.969120626 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-46.jupiter.internal]:46781 (errno: 97 - Address family not supported by protocol).
(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2793258, ip=10.128.32.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2557696, ip=10.128.32.48) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2557696, ip=10.128.32.48) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2557696, ip=10.128.32.48) No module named 'vllm._version' [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2557696, ip=10.128.32.48) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster]
(pid=2623903, ip=10.128.32.46) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) (EngineCore_DP0 pid=2666891) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) (EngineCore_DP0 pid=2666891) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2666657, ip=10.128.32.38) (EngineCore_DP0 pid=2666896) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) (EngineCore_DP0 pid=2666900) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:49.608 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=uni, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=1, VLLM_ENABLE_V1_MULTIPROCESSING=<unset> [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:49.610 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:133 - setup_envvars_for_vllm: numa_enabled=True [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:49.610 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:136 - setup_envvars_for_vllm: set VLLM_ENABLE_V1_MULTIPROCESSING=0 for NUMA affinity [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2557696, ip=10.128.32.48) [2026-06-07 02:12:49,438 E 2557696 2557862] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) [W607 02:12:50.707628733 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator())
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) [rank0]:[W607 02:12:50.713854317 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) 2026-06-07 02:12:50.870 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 0-71 (NUMA node 0) for GPU 0 [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) 2026-06-07 02:12:50.889 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 0 [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) 2026-06-07 02:12:50.889 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) 2026-06-07 02:12:50.889 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) 2026-06-07 02:12:50.889 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) 2026-06-07 02:12:50.920 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 2.99s (attempt 1/5) to avoid port collisions [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) [W607 02:12:50.703217538 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-38-interconnect-1.jupiter.internal]:38761 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 107x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2955860, ip=10.128.32.39) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 71x across cluster]
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) (EngineCore_DP0 pid=2660131) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) No module named 'vllm._version' [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) from .version import __version__, __version_tuple__ # isort:skip [repeated 35x across cluster]
(pid=2701488) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) (EngineCore_DP0 pid=2660139) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2593412, ip=10.128.32.36) [2026-06-07 02:12:55,179 E 2593412 2593580] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 19x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) [W607 02:12:55.527964802 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 10x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) [rank0]:[W607 02:12:55.530401044 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:50.861 | INFO | skyrl_train.utils.numa:set_numa_affinity_for_gpu:340 - NUMA affinity: bound process to CPUs 72-143 (NUMA node 1) for GPU 1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:50.889 | DEBUG | skyrl_train.utils.numa:_set_membind_via_libnuma:375 - NUMA affinity: set memory preferred to NUMA node 1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:50.889 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:584 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:50.889 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:592 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:50.889 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1158 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:12:50.917 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1210 - Engine startup stagger: sleeping 1.98s (attempt 1/5) to avoid port collisions [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) [W607 02:12:55.527535851 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-43.jupiter.internal]:50889 (errno: 97 - Address family not supported by protocol). [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 8x across cluster]
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:57.653 | INFO  | skyrl_train.workers.worker:_initiate_actors:565 - Initialized process group for RayActorGroup
(skyrl_entrypoint pid=2701319) 2026-06-07 02:12:57.664 | INFO  | skyrl_train.workers.worker:_initiate_actors:567 - Mesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=8, dp_size=8, pp_size=1)]
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) `torch_dtype` is deprecated! Use `dtype` instead!
(FSDPPolicyWorkerBase pid=2701489)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46)
Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:00, 3.50it/s]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) No module named 'vllm._version' [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) from .version import __version__, __version_tuple__ # isort:skip [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:10, 3.51s/it]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2793258, ip=10.128.32.45) (EngineCore_DP0 pid=2793487) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2593412, ip=10.128.32.36) (EngineCore_DP0 pid=2593646) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2593411, ip=10.128.32.36) (EngineCore_DP0 pid=2593642) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2593411, ip=10.128.32.36) (EngineCore_DP0 pid=2593642) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=3076043, ip=10.128.32.42) [2026-06-07 02:12:58,732 E 3076043 3076146] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=290611, ip=10.128.32.47) (EngineCore_DP0 pid=290849) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2575228, ip=10.128.32.41) (EngineCore_DP0 pid=2575453) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=290737, ip=10.128.32.47) (EngineCore_DP0 pid=290838) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2557696, ip=10.128.32.48) (EngineCore_DP0 pid=2558131) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [W607 02:12:57.986763802 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2557898, ip=10.128.32.48) (EngineCore_DP0 pid=2558139) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2701488) [W607 02:12:57.876876876 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-46.jupiter.internal]:46781 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2793257, ip=10.128.32.45) (EngineCore_DP0 pid=2793503) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2575228, ip=10.128.32.41) [rank0]:[W607 02:13:01.642740161 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator())
(AsyncVLLMInferenceEngine pid=2557899, ip=10.128.32.48) (EngineCore_DP0 pid=2558147) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46)
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.05it/s]
(AsyncVLLMInferenceEngine pid=2593413, ip=10.128.32.36) (EngineCore_DP0 pid=2593666) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2701489)
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.06it/s]
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) (EngineCore_DP0 pid=2575477) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) (EngineCore_DP0 pid=2558151) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=290612, ip=10.128.32.47) (EngineCore_DP0 pid=290842) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2793126, ip=10.128.32.45) (EngineCore_DP0 pid=2793507) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) Process EngineCore_DP0:
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) self.run()
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/multiprocessing/process.py", line 108, in run
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) self._target(*self._args, **self._kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1010, in run_engine_core
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) raise e
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) super().__init__(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) self._init_executor()
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) init_distributed_environment(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) return TCPStore(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 37945, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000730cc700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #1: <unknown function> + 0x5f29220 (0x400053269220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #2: <unknown function> + 0x5f4326c (0x40005328326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #3: <unknown function> + 0x5f49074 (0x400053289074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #4: <unknown function> + 0x5f49138 (0x400053289138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #5: <unknown function> + 0x5f2ccc4 (0x40005326ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x400053271564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #7: <unknown function> + 0x109a094 (0x40004d01a094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #8: <unknown function> + 0x113236c (0x40004d0b236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #9: <unknown function> + 0x5d6d60 (0x40004c556d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #10: <unknown function> + 0x1b7a38 (0xaaaabc307a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaabc2b5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #12: <unknown function> + 0x169f50 (0xaaaabc2b9f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #13: <unknown function> + 0x1682e4 (0xaaaabc2b82e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #14: <unknown function> + 0x1e0ce8 (0xaaaabc330ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #15: <unknown function> + 0x1d7ddc (0xaaaabc327ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #16: <unknown function> + 0x646b0c (0x40004c5c6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaabc2b5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaabc3bae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #19: <unknown function> + 0x1808c0 (0xaaaabc2d08c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #20: <unknown function> + 0x182bf8 (0xaaaabc2d2bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #21: <unknown function> + 0x25fd30 (0xaaaabc3afd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #22: <unknown function> + 0x1b7d20 (0xaaaabc307d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #23: PyObject_Vectorcall + 0x54 (0xaaaabc2b60e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaabc3bae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaabc2b7f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaabc2b81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #27: <unknown function> + 0x1e0d80 (0xaaaabc330d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #28: <unknown function> + 0x1d7ddc (0xaaaabc327ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaabc2b5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaabc3bae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaabc2b7fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaabc2b81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #33: <unknown function> + 0x1e0d80 (0xaaaabc330d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #34: <unknown function> + 0x1d7ddc (0xaaaabc327ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #35: _PyObject_Call + 0x68 (0xaaaabc2b8488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaabc3bd8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #37: PyEval_EvalCode + 0xb4 (0xaaaabc3c2eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #38: <unknown function> + 0x2ccdcc (0xaaaabc41cdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #39: <unknown function> + 0x2ccef4 (0xaaaabc41cef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #40: PyRun_StringFlags + 0x90 (0xaaaabc421050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaabc4210f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #42: Py_RunMain + 0x2c8 (0xaaaabc449190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #43: Py_BytesMain + 0x64 (0xaaaabc449fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #44: <unknown function> + 0x27540 (0x40003b5c7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #45: __libc_start_main + 0x98 (0x40003b5c7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) frame #46: <unknown function> + 0x10e0c0 (0xaaaabc25e0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) [pynccl] dumped 0 entries to /e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace_pynccl_pid2593652 (reason=atexit)
(FSDPPolicyWorkerBase pid=2701488) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=2623904, ip=10.128.32.46)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) (EngineCore_DP0 pid=3076408) _C._set_float32_matmul_precision(precision)
(FSDPPolicyWorkerBase pid=2701488)
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:03<00:01, 1.38s/it] [repeated 23x across cluster]
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) (EngineCore_DP0 pid=290850) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2575093, ip=10.128.32.41) (EngineCore_DP0 pid=2575462) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) (EngineCore_DP0 pid=2666891)
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:08<00:07, 4.00s/it] [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=3076172, ip=10.128.32.42) (EngineCore_DP0 pid=3076398) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2972527, ip=10.128.32.44) (EngineCore_DP0 pid=2972899) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2539108, ip=10.128.32.40) (EngineCore_DP0 pid=2539361) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2538980, ip=10.128.32.40) (EngineCore_DP0 pid=2539337) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2972656, ip=10.128.32.44) (EngineCore_DP0 pid=2972886) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3076173, ip=10.128.32.42) (EngineCore_DP0 pid=3076402) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2972657, ip=10.128.32.44) (EngineCore_DP0 pid=2972891) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2539109, ip=10.128.32.40) (EngineCore_DP0 pid=2539345) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=2593284, ip=10.128.32.36, actor_id=a9ee7a598f6c230f2b02b9e202000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400c0f140ec0>)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) return self.__get_result()
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) raise self._exception
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) super().__init__(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) self.llm = self._create_engine(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) return cls(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) self.engine_core = EngineCoreClient.make_async_mp_client(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) return AsyncMPClient(*client_args)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) super().__init__(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) next(self.gen)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) wait_for_engine_startup(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) raise RuntimeError(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) [pynccl] dumped 0 entries to /e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace_pynccl_pid2593284 (reason=atexit)
(AsyncVLLMInferenceEngine pid=984183, ip=10.128.32.37) (EngineCore_DP0 pid=984423) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=3076043, ip=10.128.32.42) (EngineCore_DP0 pid=3076412) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=290611, ip=10.128.32.47) (EngineCore_DP0 pid=290849)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 14x across cluster]
(AsyncVLLMInferenceEngine pid=2955861, ip=10.128.32.39) (EngineCore_DP0 pid=2956093) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=984182, ip=10.128.32.37) (EngineCore_DP0 pid=984418) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2539107, ip=10.128.32.40) (EngineCore_DP0 pid=2539341) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2972655, ip=10.128.32.44) (EngineCore_DP0 pid=2972907) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=984055, ip=10.128.32.37) (EngineCore_DP0 pid=984427) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=984055, ip=10.128.32.37) (EngineCore_DP0 pid=984427) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) [2026-06-07 02:13:04,817 E 2955859 2955971] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955860, ip=10.128.32.39) (EngineCore_DP0 pid=2956089) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=984310, ip=10.128.32.37) (EngineCore_DP0 pid=984419) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=984183, ip=10.128.32.37) [W607 02:13:05.754325616 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 34x across cluster]
(AsyncVLLMInferenceEngine pid=984310, ip=10.128.32.37) [W607 02:13:06.082184402 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-37.jupiter.internal]:34951 (errno: 97 - Address family not supported by protocol). [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=984310, ip=10.128.32.37) [rank0]:[W607 02:13:06.084447273 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 34x across cluster]
(FSDPPolicyWorkerBase pid=2623904, ip=10.128.32.46)
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.05it/s] [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=2701488)
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00, 1.06it/s] [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113) _C._set_float32_matmul_precision(precision)
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123)
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) (EngineCore_DP0 pid=2660139)
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) (EngineCore_DP0 pid=2660131)
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148)
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870)
(AsyncVLLMInferenceEngine pid=2666657, ip=10.128.32.38) (EngineCore_DP0 pid=2666896)
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) (EngineCore_DP0 pid=2666900)
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) (EngineCore_DP0 pid=2666891)
(AsyncVLLMInferenceEngine pid=290737, ip=10.128.32.47) (EngineCore_DP0 pid=290838)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:09, 3.22s/it] [repeated 47x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097)
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] [repeated 25x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) [repeated 4x across cluster]
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [2026-06-07 02:13:05,834 E 2623829 2623869] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) [W607 02:13:07.834305247 Utils.hpp:166] Warning: Environment variable NCCL_BLOCKING_WAIT is deprecated; use TORCH_NCCL_BLOCKING_WAIT instead (function operator()) [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) [W607 02:13:07.833825704 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-39-interconnect-1.jupiter.internal]:38597 (errno: 97 - Address family not supported by protocol). [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) [rank0]:[W607 02:13:07.836433304 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) 2026-06-07 02:13:12.381 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2
(AsyncVLLMInferenceEngine pid=2575093, ip=10.128.32.41) (EngineCore_DP0 pid=2575462)
(AsyncVLLMInferenceEngine pid=2575228, ip=10.128.32.41) (EngineCore_DP0 pid=2575453)
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) (EngineCore_DP0 pid=2575477)
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458)
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:14.347 | INFO  | skyrl_train.trainer:build_models:831 - init policy/ref/critic models done
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:14.366 | INFO  | skyrl_train.fully_async_trainer:_maybe_enable_rollout_fanout:456 - Rollout fan-out ENABLED: replacing single-process generator with RolloutDispatcher (K=4, cpus_per_coordinator=8).
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:14.388 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:387 - [RolloutDispatcher] fan-out path: overriding inference host 127.0.0.1 -> 10.128.32.34 (routable head IP) for coordinator litellm base_url connectivity
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:14.388 | INFO  | examples.terminal_bench.rollout_coordinator:__init__:405 - [RolloutDispatcher] configured num_coordinators=4, cpus_per_coordinator=8
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:14.409 | INFO  | examples.terminal_bench.rollout_coordinator:startup:434 - [RolloutDispatcher] PlacementGroup ready: 4 bundles x 8 CPU (SPREAD)
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097)
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:03<00:10, 3.57s/it] [repeated 47x across cluster]
(raylet, ip=10.128.32.39) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(AsyncVLLMInferenceEngine pid=2593411, ip=10.128.32.36) (EngineCore_DP0 pid=2593642)
(AsyncVLLMInferenceEngine pid=2593413, ip=10.128.32.36) (EngineCore_DP0 pid=2593666)
(AsyncVLLMInferenceEngine pid=2593412, ip=10.128.32.36) (EngineCore_DP0 pid=2593646)
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 11x across cluster]
(FSDPPolicyWorkerBase pid=2623904, ip=10.128.32.46) [rank3]:[W607 02:13:12.044597514 Utils.hpp:137] Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead (function operator()) [repeated 8x across cluster]
(FSDPPolicyWorkerBase pid=2623903, ip=10.128.32.46) [2026-06-07 02:13:16,931 E 2623903 2624043] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(AsyncVLLMInferenceEngine pid=290484, ip=10.128.32.47) (EngineCore_DP0 pid=290850)
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) 2026-06-07 02:13:16.277 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=290611, ip=10.128.32.47) (EngineCore_DP0 pid=290849)
(AsyncVLLMInferenceEngine pid=290612, ip=10.128.32.47) (EngineCore_DP0 pid=290842)
(AsyncVLLMInferenceEngine pid=290737, ip=10.128.32.47) (EngineCore_DP0 pid=290838)
(AsyncVLLMInferenceEngine pid=2557696, ip=10.128.32.48) (EngineCore_DP0 pid=2558131)
(AsyncVLLMInferenceEngine pid=2557898, ip=10.128.32.48) (EngineCore_DP0 pid=2558139)
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) (EngineCore_DP0 pid=2558151)
(AsyncVLLMInferenceEngine pid=2557899, ip=10.128.32.48) (EngineCore_DP0 pid=2558147)
(AsyncVLLMInferenceEngine pid=2793126, ip=10.128.32.45) (EngineCore_DP0 pid=2793507)
(AsyncVLLMInferenceEngine pid=2793257, ip=10.128.32.45) (EngineCore_DP0 pid=2793503)
(AsyncVLLMInferenceEngine pid=2793258, ip=10.128.32.45) (EngineCore_DP0 pid=2793487)
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492)
(AsyncVLLMInferenceEngine pid=2972527, ip=10.128.32.44) (EngineCore_DP0 pid=2972899)
(AsyncVLLMInferenceEngine pid=2972527, ip=10.128.32.44) (EngineCore_DP0 pid=2972899)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.30s/it] [repeated 79x across cluster]
(raylet, ip=10.128.32.39) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2972655, ip=10.128.32.44) (EngineCore_DP0 pid=2972907)
(AsyncVLLMInferenceEngine pid=2972656, ip=10.128.32.44) (EngineCore_DP0 pid=2972886)
(AsyncVLLMInferenceEngine pid=2972657, ip=10.128.32.44) (EngineCore_DP0 pid=2972891)
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.159 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(AsyncVLLMInferenceEngine pid=2538980, ip=10.128.32.40) (EngineCore_DP0 pid=2539337)
(AsyncVLLMInferenceEngine pid=2539109, ip=10.128.32.40) (EngineCore_DP0 pid=2539345)
(AsyncVLLMInferenceEngine pid=2539108, ip=10.128.32.40) (EngineCore_DP0 pid=2539361)
(AsyncVLLMInferenceEngine pid=2539107, ip=10.128.32.40) (EngineCore_DP0 pid=2539341)
(AsyncVLLMInferenceEngine pid=3076043, ip=10.128.32.42) (EngineCore_DP0 pid=3076412)
(AsyncVLLMInferenceEngine pid=3076173, ip=10.128.32.42) (EngineCore_DP0 pid=3076402)
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) (EngineCore_DP0 pid=3076408)
(AsyncVLLMInferenceEngine pid=3076172, ip=10.128.32.42) (EngineCore_DP0 pid=3076398)
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.385 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.385 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.387 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.387 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 0/4] constructed (http=10.128.32.34:8000)
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.390 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.390 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:13:20.390 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 0] startup complete
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:20.391 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 1/4 started
(AsyncVLLMInferenceEngine pid=984055, ip=10.128.32.37) (EngineCore_DP0 pid=984427)
(AsyncVLLMInferenceEngine pid=984182, ip=10.128.32.37) (EngineCore_DP0 pid=984418)
(AsyncVLLMInferenceEngine pid=984183, ip=10.128.32.37) (EngineCore_DP0 pid=984423)
(AsyncVLLMInferenceEngine pid=984310, ip=10.128.32.37) (EngineCore_DP0 pid=984419)
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113)
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) (EngineCore_DP0 pid=2558151) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 16x across cluster]
(FSDPPolicyWorkerBase pid=2701488) [2026-06-07 02:13:17,107 E 2701488 2701649] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2955860, ip=10.128.32.39) (EngineCore_DP0 pid=2956089)
(AsyncVLLMInferenceEngine pid=2955861, ip=10.128.32.39) (EngineCore_DP0 pid=2956093)
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097)
(AsyncVLLMInferenceEngine pid=2793126, ip=10.128.32.45) 2026-06-07 02:13:21.517 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097)
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00, 3.30s/it] [repeated 46x across cluster]
(raylet, ip=10.128.32.37) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.158 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) 2026-06-07 02:13:26.137 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1267 - Initializing OpenAIServingChat with custom_chat_template read from: chat_templates/qwen3_thinking_acc.jinja2 [repeated 20x across cluster]
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.376 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.376 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.376 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.376 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 1/4] constructed (http=10.128.32.34:8000)
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.379 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.379 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=984614, ip=10.128.32.37) 2026-06-07 02:13:28.379 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 1] startup complete
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:28.381 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 2/4 started
(raylet, ip=10.128.32.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(raylet, ip=10.128.32.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.536 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(RolloutCoordinator pid=984614, ip=10.128.32.37) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(raylet, ip=10.128.32.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 4x across cluster]
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:36.763 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 3/4 started
(RolloutCoordinator pid=2660382, ip=10.128.32.43) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.758 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.758 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.758 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.758 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 2/4] constructed (http=10.128.32.34:8000)
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.761 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.761 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2660382, ip=10.128.32.43) 2026-06-07 02:13:36.761 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 2] startup complete
(RolloutCoordinator pid=2956288, ip=10.128.32.39) [2026-06-07 02:13:45,184 E 2956288 2956328] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(raylet, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 6x across cluster]
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.382 | INFO | examples.terminal_bench.rollout_coordinator:_scale_terminal_bench_cfg:121 - [RolloutCoordinator] scaled n_concurrent_trials 675 -> 168 (// 4)
(RolloutCoordinator pid=2593907, ip=10.128.32.36) The tokenizer you are loading from '/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.604 | INFO | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:236 - Harbor logging level set to INFO
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.604 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:136 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions', 'zero_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 168. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.604 | INFO | examples.terminal_bench.terminal_bench_generator:__init__:152 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.604 | INFO | examples.terminal_bench.rollout_coordinator:__init__:246 - [RolloutCoordinator 3/4] constructed (http=10.128.32.34:8000)
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.607 | INFO | examples.terminal_bench.terminal_bench_generator:_create_orchestrator:292 - QueueOrchestrator created and started with n_concurrent_trials=168, rollback_hook registered for ContextLengthExceededError/AgentTimeoutError
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.607 | INFO | examples.terminal_bench.terminal_bench_generator:startup:249 - TerminalBenchGenerator startup complete. Shared orchestrator ready with n_concurrent_trials=168
(RolloutCoordinator pid=2593907, ip=10.128.32.36) 2026-06-07 02:13:45.607 | INFO | examples.terminal_bench.rollout_coordinator:startup:254 - [RolloutCoordinator 3] startup complete
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.608 | INFO  | examples.terminal_bench.rollout_coordinator:startup:469 - [RolloutDispatcher] coordinator 4/4 started
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.608 | INFO  | examples.terminal_bench.rollout_coordinator:startup:477 - [RolloutDispatcher] 4 coordinators started
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.608 | INFO  | skyrl_train.fully_async_trainer:train:485 - Generator startup complete
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.608 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Started: 'load_checkpoints'
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.617 | INFO  | skyrl_train.trainer:load_checkpoints:1664 - Loading checkpoint from: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.617 | INFO  | skyrl_train.trainer:load_checkpoints:1670 - Resuming from global_step: 28
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.646 | INFO  | skyrl_train.trainer:load_checkpoints:1686 - Successfully loaded trainer state
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.660 | INFO  | skyrl_train.trainer:load_checkpoints:1696 - Successfully loaded dataloader state
(skyrl_entrypoint pid=2701319) 2026-06-07 02:13:45.660 | INFO  | skyrl_train.trainer:load_checkpoints:1705 - Loading policy checkpoint from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy
(RolloutCoordinator pid=984614, ip=10.128.32.37) [2026-06-07 02:13:53,284 E 984614 984654] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(RolloutCoordinator pid=2660382, ip=10.128.32.43) [2026-06-07 02:14:01,250 E 2660382 2660422] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) warnings.warn( # warn only once
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:03.113 | INFO  | skyrl_train.trainer:load_checkpoints:1715 - Successfully loaded policy checkpoint
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:03.114 | INFO  | skyrl_train.trainer:load_checkpoints:1731 - Successfully loaded complete checkpoint state from global_step_28
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:03.114 | INFO  | skyrl_train.fully_async_trainer:_train_loop:512 - Resumed training from global_step 28
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:03.131 | INFO  | skyrl_train.utils.data_tracker:load_state:97 - Loaded data tracker state: epoch=0, consumed_in_epoch=1792, total_consumed=1792
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:03.132 | INFO  | skyrl_train.fully_async_trainer:_train_loop:510 - Finished: 'load_checkpoints', time cost: 17.52s
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:03.132 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Started: 'init_weight_sync_state'
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) No module named 'vllm._version'
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) from .version import __version__, __version_tuple__ # isort:skip
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) 2026-06-07 02:14:08.866 | INFO  | logging:info:2216 - [weight-sync] Using master_addr=10.128.32.46, master_port=59681
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) 2026-06-07 02:14:08.865 | INFO  | logging:info:2216 - [weight-sync] get_node_ip_address()=10.128.32.46 [repeated 3x across cluster]
(FSDPPolicyWorkerBase pid=2701490) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=2701490) No module named 'vllm._version' [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=2701490) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster]
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank0]:[W607 02:14:08.715725173 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-041-46-interconnect-1.jupiter.internal]:59681 (errno: 97 - Address family not supported by protocol).
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:08.938 | INFO  | skyrl_train.fully_async_trainer:_train_loop:544 - Finished: 'init_weight_sync_state', time cost: 5.81s
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:08.938 | ERROR  | skyrl_train.fully_async_trainer:train:493 - Train loop failed at global_step 28: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
(skyrl_entrypoint pid=2701319) Traceback (most recent call last):
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
(skyrl_entrypoint pid=2701319) await self._train_loop()
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
(skyrl_entrypoint pid=2701319) self.init_weight_sync_state()
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
(skyrl_entrypoint pid=2701319) raise RuntimeError(
(skyrl_entrypoint pid=2701319) RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
(RolloutCoordinator pid=2593907, ip=10.128.32.36) [2026-06-07 02:14:09,830 E 2593907 2593949] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:09.964 | INFO  | skyrl_train.inference_engines.inference_engine_client_http_endpoint:shutdown_server:203 - Server shut down after 2 seconds
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:09.964 | INFO  | skyrl_train.trainer:_guarded_sync:226 - HTTP endpoint shutdown complete
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:09.968 | INFO  | skyrl_train.trainer:_guarded_async:215 - Generator shutdown complete
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:09.980 | WARNING  | skyrl_train.trainer:_guarded_async:219 - Inference engine teardown error (non-fatal): The actor died because of an error raised in its creation task, ray::AsyncVLLMInferenceEngine.__init__() (pid=2593284, ip=10.128.32.36, actor_id=a9ee7a598f6c230f2b02b9e202000000, repr=<skyrl_train.inference_engines.vllm.vllm_engine.AsyncVLLMInferenceEngine object at 0x400c0f140ec0>)
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(skyrl_entrypoint pid=2701319) return self.__get_result()
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(skyrl_entrypoint pid=2701319) raise self._exception
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__
(skyrl_entrypoint pid=2701319) super().__init__(*args, **kwargs)
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__
(skyrl_entrypoint pid=2701319) self.llm = self._create_engine(*args, **kwargs)
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine
(skyrl_entrypoint pid=2701319) engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args
(skyrl_entrypoint pid=2701319) return cls(
(skyrl_entrypoint pid=2701319) ^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
(skyrl_entrypoint pid=2701319) self.engine_core = EngineCoreClient.make_async_mp_client(
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client
(skyrl_entrypoint pid=2701319) return AsyncMPClient(*client_args)
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__
(skyrl_entrypoint pid=2701319) super().__init__(
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__
(skyrl_entrypoint pid=2701319) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(skyrl_entrypoint pid=2701319) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__
(skyrl_entrypoint pid=2701319) next(self.gen)
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines
(skyrl_entrypoint pid=2701319) wait_for_engine_startup(
(skyrl_entrypoint pid=2701319) File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup
(skyrl_entrypoint pid=2701319) raise RuntimeError(
(skyrl_entrypoint pid=2701319) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(skyrl_entrypoint pid=2701319) 2026-06-07 02:14:09.981 | INFO  | skyrl_train.trainer:_kill_ray_actors:248 - Killing policy_model actors...
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:14:09.965 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:363 - Shutting down shared QueueOrchestrator...
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:14:09.965 | INFO | examples.terminal_bench.terminal_bench_generator:shutdown:365 - QueueOrchestrator shutdown complete
(RolloutCoordinator pid=2956288, ip=10.128.32.39) 2026-06-07 02:14:09.966 | INFO | examples.terminal_bench.rollout_coordinator:shutdown:258 - [RolloutCoordinator 0] shutdown complete
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) warnings.warn("No model update group to destroy")
2026-06-07 02:14:10.027 | ERROR | __main__:main:134 - Training failed: ray::skyrl_entrypoint() (pid=2701319, ip=10.128.32.34)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 483, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
raise RuntimeError(
RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
2026-06-07 02:14:10.027 | INFO | __main__:main:137 - Shutting down Ray on head node...
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py:470: UserWarning: No model update group to destroy [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) warnings.warn("No model update group to destroy") [repeated 27x across cluster]
(skyrl_entrypoint pid=2701319) [fd-monitor] Started monitoring (every 120s)
(skyrl_entrypoint pid=2701319) [fd-monitor] [02:11:44] OK: 50 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=2701319) [fd-monitor] [02:11:44] OK: RSS 1.38 GiB | node mem 126.9/858.0 GiB used (14.8%), avail 731.1 GiB
(skyrl_entrypoint pid=2701319) ⚙️ Running in WANDB offline mode
(skyrl_entrypoint pid=2701319) INFO 06-07 02:12:04 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:26 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(pid=2666657, ip=10.128.32.38) INFO 06-07 02:12:30 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) WARNING 06-07 02:12:31 [arg_utils.py:1256] The global random seed is set to 42. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:31 [model.py:529] Resolved architecture: Qwen3ForCausalLM
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:31 [model.py:1549] Using max model len 32768
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:31 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}}
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:31 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536.
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:31 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) WARNING 06-07 02:12:31 [vllm.py:728] Enforce eager set, overriding optimization level to -O0
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:31 [vllm.py:846] Cudagraph is disabled under eager mode
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) WARNING 06-07 02:12:31 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) WARNING 06-07 02:12:31 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(pid=2659764, ip=10.128.32.43) INFO 06-07 02:12:35 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) WARNING 06-07 02:12:34 [arg_utils.py:1256] The global random seed is set to 55. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) INFO 06-07 02:12:34 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) INFO 06-07 02:12:34 [model.py:1549] Using max model len 32768 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) INFO 06-07 02:12:34 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) INFO 06-07 02:12:34 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) INFO 06-07 02:12:34 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) WARNING 06-07 02:12:34 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) INFO 06-07 02:12:34 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) WARNING 06-07 02:12:34 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666658, ip=10.128.32.38) WARNING 06-07 02:12:34 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) INFO 06-07 02:12:39 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=42, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) INFO 06-07 02:12:39 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) WARNING 06-07 02:12:43 [arg_utils.py:1256] The global random seed is set to 81. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) INFO 06-07 02:12:43 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) INFO 06-07 02:12:43 [model.py:1549] Using max model len 32768 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) INFO 06-07 02:12:43 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) INFO 06-07 02:12:43 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) INFO 06-07 02:12:43 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) WARNING 06-07 02:12:43 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) INFO 06-07 02:12:43 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) WARNING 06-07 02:12:43 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=2659897, ip=10.128.32.43) WARNING 06-07 02:12:43 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(pid=2623829, ip=10.128.32.46) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=2666657, ip=10.128.32.38) (EngineCore_DP0 pid=2666896) INFO 06-07 02:12:44 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=56, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster]
(pid=2972657, ip=10.128.32.44) INFO 06-07 02:12:44 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 39x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) WARNING 06-07 02:12:48 [arg_utils.py:1256] The global random seed is set to 71. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) INFO 06-07 02:12:48 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) INFO 06-07 02:12:48 [model.py:1549] Using max model len 32768 [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) INFO 06-07 02:12:48 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) INFO 06-07 02:12:48 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) INFO 06-07 02:12:48 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) WARNING 06-07 02:12:48 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=3076171, ip=10.128.32.42) INFO 06-07 02:12:48 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) WARNING 06-07 02:12:48 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) WARNING 06-07 02:12:48 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 18x across cluster]
(AsyncVLLMInferenceEngine pid=2666656, ip=10.128.32.38) (EngineCore_DP0 pid=2666891) INFO 06-07 02:12:50 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc']
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) INFO 06-07 02:12:50 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.38:51279 backend=nccl
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) INFO 06-07 02:12:50 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:12:48 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=66, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=2659899, ip=10.128.32.43) (EngineCore_DP0 pid=2660131) INFO 06-07 02:12:51 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=80, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) INFO 06-07 02:12:52 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) INFO 06-07 02:12:51 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 8x across cluster]
(pid=2623902, ip=10.128.32.46) ⚙️ Running in WANDB offline mode
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) WARNING 06-07 02:12:53 [arg_utils.py:1256] The global random seed is set to 86. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) INFO 06-07 02:12:53 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) INFO 06-07 02:12:53 [model.py:1549] Using max model len 32768 [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) INFO 06-07 02:12:53 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'NCCL_DEBUG': '***', 'NCCL_SOCKET_FAMILY': '***', 'NCCL_SOCKET_IFNAME': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) INFO 06-07 02:12:53 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=65536. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) INFO 06-07 02:12:53 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) WARNING 06-07 02:12:53 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) INFO 06-07 02:12:53 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 13x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) WARNING 06-07 02:12:53 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) WARNING 06-07 02:12:53 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 22x across cluster]
(AsyncVLLMInferenceEngine pid=2666524, ip=10.128.32.38) (EngineCore_DP0 pid=2666870) INFO 06-07 02:12:54 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) INFO 06-07 02:12:55 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) INFO 06-07 02:12:55 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.43:50889 backend=nccl [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) INFO 06-07 02:12:55 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) INFO 06-07 02:12:52 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=81, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) INFO 06-07 02:12:55 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2593411, ip=10.128.32.36) INFO 06-07 02:12:57 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 12x across cluster]
(pid=2701488) ⚙️ Running in WANDB offline mode [repeated 6x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) WARNING 06-07 02:12:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) WARNING 06-07 02:12:54 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) (EngineCore_DP0 pid=2660148) INFO 06-07 02:12:56 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 7x across cluster]
(AsyncVLLMInferenceEngine pid=2593412, ip=10.128.32.36) (EngineCore_DP0 pid=2593646) INFO 06-07 02:13:00 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 2x across cluster]
(AsyncVLLMInferenceEngine pid=2575228, ip=10.128.32.41) (EngineCore_DP0 pid=2575453) INFO 06-07 02:13:01 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.41:58449 backend=nccl
(AsyncVLLMInferenceEngine pid=2575228, ip=10.128.32.41) (EngineCore_DP0 pid=2575453) INFO 06-07 02:13:01 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:00 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=75, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 36x across cluster]
(AsyncVLLMInferenceEngine pid=2575228, ip=10.128.32.41) (EngineCore_DP0 pid=2575453) INFO 06-07 02:13:02 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2955861, ip=10.128.32.39) INFO 06-07 02:13:01 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 25x across cluster]
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] EngineCore failed to start.
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] Traceback (most recent call last):
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 996, in run_engine_core
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 740, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] super().__init__(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 106, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] self.model_executor = executor_class(vllm_config)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] self._init_executor()
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] self.driver_worker.init_device()
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in init_device
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] self.worker.init_device() # type: ignore
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 232, in init_device
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] init_worker_distributed_environment(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1040, in init_worker_distributed_environment
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] init_distributed_environment(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1256, in init_distributed_environment
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] torch.distributed.init_process_group(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] return func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] func_return = func(*args, **kwargs)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1762, in init_process_group
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] store, rank, world_size = next(rendezvous_iterator)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 230, in _tcp_rendezvous_handler
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] store = _create_c10d_store(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^^^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 198, in _create_c10d_store
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] return TCPStore(
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] ^^^^^^^^^
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 37945, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] Exception raised from makeWithPort at /pytorch/torch/csrc/distributed/c10d/TCPStoreLibUvBackend.cpp:307 (most recent call first):
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0x4000730cc700 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libc10.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #1: <unknown function> + 0x5f29220 (0x400053269220 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #2: <unknown function> + 0x5f4326c (0x40005328326c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #3: <unknown function> + 0x5f49074 (0x400053289074 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #4: <unknown function> + 0x5f49138 (0x400053289138 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #5: <unknown function> + 0x5f2ccc4 (0x40005326ccc4 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #6: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x104 (0x400053271564 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #7: <unknown function> + 0x109a094 (0x40004d01a094 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #8: <unknown function> + 0x113236c (0x40004d0b236c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #9: <unknown function> + 0x5d6d60 (0x40004c556d60 in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #10: <unknown function> + 0x1b7a38 (0xaaaabc307a38 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #11: _PyObject_MakeTpCall + 0x98 (0xaaaabc2b5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #12: <unknown function> + 0x169f50 (0xaaaabc2b9f50 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #13: <unknown function> + 0x1682e4 (0xaaaabc2b82e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #14: <unknown function> + 0x1e0ce8 (0xaaaabc330ce8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #15: <unknown function> + 0x1d7ddc (0xaaaabc327ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #16: <unknown function> + 0x646b0c (0x40004c5c6b0c in /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #17: _PyObject_MakeTpCall + 0x98 (0xaaaabc2b5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #18: _PyEval_EvalFrameDefault + 0x280c (0xaaaabc3bae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #19: <unknown function> + 0x1808c0 (0xaaaabc2d08c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #20: <unknown function> + 0x182bf8 (0xaaaabc2d2bf8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #21: <unknown function> + 0x25fd30 (0xaaaabc3afd30 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #22: <unknown function> + 0x1b7d20 (0xaaaabc307d20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #23: PyObject_Vectorcall + 0x54 (0xaaaabc2b60e4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #24: _PyEval_EvalFrameDefault + 0x280c (0xaaaabc3bae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #25: _PyObject_FastCallDictTstate + 0x80 (0xaaaabc2b7f20 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #26: _PyObject_Call_Prepend + 0x140 (0xaaaabc2b81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #27: <unknown function> + 0x1e0d80 (0xaaaabc330d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #28: <unknown function> + 0x1d7ddc (0xaaaabc327ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #29: _PyObject_MakeTpCall + 0x98 (0xaaaabc2b5db8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #30: _PyEval_EvalFrameDefault + 0x280c (0xaaaabc3bae54 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #31: _PyObject_FastCallDictTstate + 0x10c (0xaaaabc2b7fac in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #32: _PyObject_Call_Prepend + 0x140 (0xaaaabc2b81ec in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #33: <unknown function> + 0x1e0d80 (0xaaaabc330d80 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #34: <unknown function> + 0x1d7ddc (0xaaaabc327ddc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #35: _PyObject_Call + 0x68 (0xaaaabc2b8488 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #36: _PyEval_EvalFrameDefault + 0x52ac (0xaaaabc3bd8f4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #37: PyEval_EvalCode + 0xb4 (0xaaaabc3c2eb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #38: <unknown function> + 0x2ccdcc (0xaaaabc41cdcc in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #39: <unknown function> + 0x2ccef4 (0xaaaabc41cef4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #40: PyRun_StringFlags + 0x90 (0xaaaabc421050 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #41: PyRun_SimpleStringFlags + 0x58 (0xaaaabc4210f8 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #42: Py_RunMain + 0x2c8 (0xaaaabc449190 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #43: Py_BytesMain + 0x64 (0xaaaabc449fb4 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #44: <unknown function> + 0x27540 (0x40003b5c7540 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #45: __libc_start_main + 0x98 (0x40003b5c7618 in /lib64/libc.so.6)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006] frame #46: <unknown function> + 0x10e0c0 (0xaaaabc25e0c0 in VLLM::EngineCore)
(AsyncVLLMInferenceEngine pid=2593284, ip=10.128.32.36) (EngineCore_DP0 pid=2593652) ERROR 06-07 02:13:02 [core.py:1006]
(AsyncVLLMInferenceEngine pid=2593411, ip=10.128.32.36) (EngineCore_DP0 pid=2593642) INFO 06-07 02:13:02 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6...
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) INFO 06-07 02:13:03 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 12x across cluster]
(AsyncVLLMInferenceEngine pid=2972655, ip=10.128.32.44) (EngineCore_DP0 pid=2972907) INFO 06-07 02:13:05 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=984310, ip=10.128.32.37) (EngineCore_DP0 pid=984419) INFO 06-07 02:13:06 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.37:34951 backend=nccl [repeated 35x across cluster]
(AsyncVLLMInferenceEngine pid=984310, ip=10.128.32.37) (EngineCore_DP0 pid=984419) INFO 06-07 02:13:06 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 34x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) INFO 06-07 02:13:03 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', speculative_config=None, tokenizer='/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=87, served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [65536], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) INFO 06-07 02:13:02 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 3x across cluster]
(AsyncVLLMInferenceEngine pid=984310, ip=10.128.32.37) (EngineCore_DP0 pid=984419) INFO 06-07 02:13:06 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 33x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:08 [default_loader.py:293] Loading weights took 13.01 seconds
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:08 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 13.474666 seconds
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:11 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:11 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:11 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113) INFO 06-07 02:13:08 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 27x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113) INFO 06-07 02:13:07 [worker_base.py:289] Injected <class 'skyrl_train.inference_engines.vllm.vllm_engine.WorkerWrap'> into <class 'vllm.v1.worker.gpu_worker.Worker'> for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'read_named_weights', 'set_numa_affinity', 'test_rpc'] [repeated 5x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:11 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113) INFO 06-07 02:13:07 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.128.32.39:38597 backend=nccl [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113) INFO 06-07 02:13:07 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:11 [core.py:278] init engine (profile, create kv cache, warmup model) took 3.23 seconds
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) WARNING 06-07 02:13:12 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:12 [vllm.py:690] Asynchronous scheduling is enabled.
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) WARNING 06-07 02:13:12 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) (EngineCore_DP0 pid=2660123) INFO 06-07 02:13:12 [vllm.py:846] Cudagraph is disabled under eager mode
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) NCCL version 2.27.7+cuda13.0
(AsyncVLLMInferenceEngine pid=2659764, ip=10.128.32.43) WARNING 06-07 02:13:12 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(AsyncVLLMInferenceEngine pid=2955732, ip=10.128.32.39) (EngineCore_DP0 pid=2956113) INFO 06-07 02:13:08 [gpu_model_runner.py:4125] Starting to load model /e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6... [repeated 4x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:12 [default_loader.py:293] Loading weights took 7.01 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:12 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 7.348132 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:15 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:15 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:15 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) (EngineCore_DP0 pid=2575477) INFO 06-07 02:13:15 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:15 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.79 seconds [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) WARNING 06-07 02:13:16 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:16 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) WARNING 06-07 02:13:16 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575227, ip=10.128.32.41) (EngineCore_DP0 pid=2575458) INFO 06-07 02:13:16 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2575229, ip=10.128.32.41) WARNING 06-07 02:13:16 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 11x across cluster]
(AsyncVLLMInferenceEngine pid=2972657, ip=10.128.32.44) (EngineCore_DP0 pid=2972891) INFO 06-07 02:13:20 [default_loader.py:293] Loading weights took 13.29 seconds [repeated 19x across cluster]
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) INFO 06-07 02:13:18 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.779174 seconds [repeated 15x across cluster]
(RolloutCoordinator pid=2956288, ip=10.128.32.39) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2956288, ip=10.128.32.39) [fd-monitor] [02:13:20] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2956288, ip=10.128.32.39) [fd-monitor] [02:13:20] OK: RSS 0.78 GiB | node mem 203.7/858.0 GiB used (23.7%), avail 654.3 GiB
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) INFO 06-07 02:13:20 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) INFO 06-07 02:13:20 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) INFO 06-07 02:13:20 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) INFO 06-07 02:13:20 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) INFO 06-07 02:13:20 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.81 seconds [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2793256, ip=10.128.32.45) (EngineCore_DP0 pid=2793492) WARNING 06-07 02:13:21 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) (EngineCore_DP0 pid=2558151) INFO 06-07 02:13:21 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) (EngineCore_DP0 pid=2558151) WARNING 06-07 02:13:21 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2557897, ip=10.128.32.48) (EngineCore_DP0 pid=2558151) INFO 06-07 02:13:21 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2793126, ip=10.128.32.45) WARNING 06-07 02:13:21 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 15x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) INFO 06-07 02:13:22 [default_loader.py:293] Loading weights took 13.19 seconds [repeated 16x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) INFO 06-07 02:13:22 [gpu_model_runner.py:4222] Model loading took 15.27 GiB memory and 14.293649 seconds [repeated 20x across cluster]
(RolloutCoordinator pid=984614, ip=10.128.32.37) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=984614, ip=10.128.32.37) [fd-monitor] [02:13:28] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=984614, ip=10.128.32.37) [fd-monitor] [02:13:28] OK: RSS 0.78 GiB | node mem 389.3/858.0 GiB used (45.4%), avail 468.6 GiB
(AsyncVLLMInferenceEngine pid=2955861, ip=10.128.32.39) (EngineCore_DP0 pid=2956093) INFO 06-07 02:13:25 [gpu_worker.py:373] Available KV cache memory: 49.32 GiB [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955861, ip=10.128.32.39) (EngineCore_DP0 pid=2956093) INFO 06-07 02:13:25 [kv_cache_utils.py:1307] GPU KV cache size: 359,104 tokens [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955861, ip=10.128.32.39) (EngineCore_DP0 pid=2956093) INFO 06-07 02:13:25 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 10.96x [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) INFO 06-07 02:13:25 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) INFO 06-07 02:13:25 [core.py:278] init engine (profile, create kv cache, warmup model) took 2.80 seconds [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) WARNING 06-07 02:13:25 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) INFO 06-07 02:13:25 [vllm.py:690] Asynchronous scheduling is enabled. [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) WARNING 06-07 02:13:25 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) (EngineCore_DP0 pid=2956097) INFO 06-07 02:13:25 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 20x across cluster]
(AsyncVLLMInferenceEngine pid=2955859, ip=10.128.32.39) WARNING 06-07 02:13:26 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 20x across cluster]
(RolloutCoordinator pid=2660382, ip=10.128.32.43) [fd-monitor] Started monitoring (every 120s)
(RolloutCoordinator pid=2660382, ip=10.128.32.43) [fd-monitor] [02:13:36] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2660382, ip=10.128.32.43) [fd-monitor] [02:13:36] OK: RSS 0.78 GiB | node mem 390.8/858.0 GiB used (45.5%), avail 467.2 GiB
(skyrl_entrypoint pid=2701319) [fd-monitor] [02:13:44] OK: 101 / 131,072 FDs open (0.1% of soft limit, hard limit: 131,072)
(skyrl_entrypoint pid=2701319) [fd-monitor] [02:13:44] OK: RSS 1.64 GiB | node mem 213.7/858.0 GiB used (24.9%), avail 644.2 GiB
(RolloutCoordinator pid=2593907, ip=10.128.32.36) [fd-monitor] Started monitoring (every 120s)
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/model_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/extra_state_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy/optim_world_size_8_rank_0.pt
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank-0]: Successfully loaded model state dict
(RolloutCoordinator pid=2593907, ip=10.128.32.36) [fd-monitor] [02:13:45] OK: 46 / 131,072 FDs open (0.0% of soft limit, hard limit: 131,072)
(RolloutCoordinator pid=2593907, ip=10.128.32.36) [fd-monitor] [02:13:45] OK: RSS 0.78 GiB | node mem 324.8/858.0 GiB used (37.9%), avail 533.2 GiB
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank-0]: Successfully loaded optimizer state
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank-0]: Successfully loaded scheduler state
(FSDPPolicyWorkerBase pid=2623829, ip=10.128.32.46) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints/global_step_28/policy
(FSDPPolicyWorkerBase pid=2701489) INFO 06-07 02:14:06 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s
(AsyncVLLMInferenceEngine pid=2659898, ip=10.128.32.43) ERROR 06-07 02:14:10 [core_client.py:616] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
(FSDPPolicyWorkerBase pid=2623904, ip=10.128.32.46) INFO 06-07 02:14:07 [pynccl.py:178] pynccl trace buffer enabled: size=10000 dump_dir=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps flush_interval=0 s [repeated 7x across cluster]
Error executing job with overrides: ['+terminal_bench_config=terminal_bench', 'trainer.strategy=fsdp2', 'trainer.algorithm.advantage_estimator=rloo_n', 'trainer.algorithm.use_kl_loss=false', 'trainer.algorithm.kl_loss_coef=0.0', 'trainer.algorithm.eps_clip_low=0.2', 'trainer.algorithm.eps_clip_high=0.05', 'trainer.algorithm.loss_reduction=sequence_mean', 'trainer.epochs=2', 'trainer.max_steps=80', 'trainer.update_epochs_per_batch=1', 'trainer.train_batch_size=64', 'trainer.policy_mini_batch_size=64', 'trainer.eval_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=999999', 'trainer.eval_interval=999999', 'trainer.eval_before_train=false', 'trainer.ckpt_interval=2', 'trainer.resume_mode=latest', 'trainer.hf_save_interval=5', '++trainer.hf_hub_repo_id=laion/ablation-pymethods2test-seqmean-arm0', '++trainer.hf_hub_private=false', '++trainer.hf_hub_revision=main', '++trainer.enable_db_registration=false', 'trainer.project_name=OpenThoughts-Agent', 'trainer.log_level=INFO', 'trainer.tracker_commit_each_step=true', 'trainer.logger=console', 'trainer.run_name=ablation-pymethods2test-seqmean-arm0', 'trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/checkpoints', 'trainer.export_path=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/exports', 'trainer.policy.optimizer_config.lr=8e-6', 'trainer.policy.optimizer_config.weight_decay=0.0', 'trainer.policy.optimizer_config.adam_betas=[0.9,0.999]', 'trainer.policy.optimizer_config.max_grad_norm=0.9', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.policy.fsdp_config.reshard_after_forward=true', 'trainer.policy.fsdp_config.fsdp_size=4', 'trainer.policy.model.path=/e/home/jusers/feuer1/jupiter/.cache/huggingface/hub/models--laion--GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink/snapshots/0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'trainer.ref.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.reshard_after_forward=true', 'trainer.ref.fsdp_config.fsdp_size=4', 'trainer.placement.colocate_all=false', 'trainer.placement.policy_num_nodes=2', 'trainer.placement.ref_num_nodes=2', 'trainer.placement.policy_num_gpus_per_node=4', 'trainer.placement.ref_num_gpus_per_node=4', 'trainer.fully_async.max_staleness_steps=16', 'trainer.fully_async.num_parallel_generation_workers=338', 'generator.backend=vllm', 'generator.timeout_multiplier=1.0', 'generator.model_dtype=bfloat16', 'generator.inference_engine_tensor_parallel_size=1', 'generator.num_inference_engines=48', 'generator.n_samples_per_prompt=8', 'generator.eval_n_samples_per_prompt=8', 'generator.gpu_memory_utilization=0.75', 'generator.max_num_seqs=24', 'generator.max_num_batched_tokens=65536', 'generator.enable_prefix_caching=true', 'generator.enable_chunked_prefill=true', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'generator.enable_http_endpoint=true', 'generator.enable_ray_prometheus_stats=false', 'generator.vllm_stats_interval=1', 'generator.append_eos_token_after_stop_str_in_multi_turn=true', 'generator.max_turns=999999', 'generator.sampling_params.max_generate_length=4096', 'generator.sampling_params.temperature=0.7', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.top_k=20', '++generator.engine_init_kwargs.max_model_len=32768', '++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2', '++generator.engine_init_kwargs.served_model_name=0e3bff0c4e51f6b9ec0713b98b9eec36efb91cc6', 'data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_pymethods2test-large"]', 'data.val_data=[]', '+terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/trace_jobs', '+terminal_bench_config.harbor.name=terminus-2', '+terminal_bench_config.harbor.max_episodes=999999', '+terminal_bench_config.harbor.enable_summarize=false', '+terminal_bench_config.harbor.store_all_messages=true', '+terminal_bench_config.harbor.trajectory_config.raw_content=true', '+terminal_bench_config.harbor.enable_episode_logging=false', '+terminal_bench_config.harbor.record_terminal_session=false', '+terminal_bench_config.harbor.enable_pane_logging=false', '+terminal_bench_config.harbor.strict_json_parser=true', '+terminal_bench_config.harbor.interleaved_thinking=true', '+terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true', '+terminal_bench_config.harbor.override_timeout_sec=900', '+terminal_bench_config.harbor.override_cpus=1', '+terminal_bench_config.harbor.override_memory_mb=2048', '+terminal_bench_config.harbor.override_storage_mb=2048', '+terminal_bench_config.harbor.auto_snapshot=true', '+terminal_bench_config.harbor.verifier_override_timeout_sec=120', '+terminal_bench_config.harbor.max_retries=3', '+terminal_bench_config.harbor.min_wait_sec=60.0', '+terminal_bench_config.harbor.max_wait_sec=600.0', '+terminal_bench_config.harbor.wait_multiplier=2.0', '+terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"]', '+terminal_bench_config.harbor.n_concurrent_trials=675', '+terminal_bench_config.harbor.log_level=INFO', '+terminal_bench_config.harbor.enable_reward_shaping=false', '+terminal_bench_config.harbor.enable_error_classification=true', '+terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError","ContextLengthExceededError"]', '+terminal_bench_config.harbor.default_error_treatment=zero', '+terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError"]', '+terminal_bench_config.harbor.zero_exceptions=[]', '+terminal_bench_config.model_info.max_input_tokens=32000', '+terminal_bench_config.model_info.max_output_tokens=4096', '+terminal_bench_config.archiving.enabled=false', '+terminal_bench_config.trace_upload.enabled=true', '+terminal_bench_config.trace_upload.repo_org=DCAgent', '+terminal_bench_config.trace_upload.episodes=last', '+terminal_bench_config.trace_upload.dataset_type=SFT', '+terminal_bench_config.trace_upload.cleanup=true']
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 142, in <module>
main()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 132, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
values, debugger_breakpoint = worker.get_objects(
^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::skyrl_entrypoint() (pid=2701319, ip=10.128.32.34)
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/examples/terminal_bench/entrypoints/main_tbench.py", line 105, in skyrl_entrypoint
exp.run()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 483, in run
asyncio.run(trainer.train())
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 491, in train
await self._train_loop()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/fully_async_trainer.py", line 545, in _train_loop
self.init_weight_sync_state()
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/trainer.py", line 851, in init_weight_sync_state
raise RuntimeError(
RuntimeError: init_weight_sync_state failed at Ray boundary: RayTaskError(RuntimeError)(RuntimeError('tp_size() failed at Ray boundary: ActorDiedError(RayTaskError(\'__init__\', \'Traceback (most recent call last):\\n File "python/ray/_raylet.pyx", line 1722, in ray._raylet.execute_task\\n File "python/ray/_raylet.pyx", line 1659, in ray._raylet.execute_task.function_executor\\n File "python/ray/_raylet.pyx", line 4342, in ray._raylet.CoreWorker.run_async_func_or_coro_in_event_loop\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 449, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n File "python/ray/_raylet.pyx", line 4329, in async_func\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/async_compat.py", line 52, in wrapper\\n return func(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/function_manager.py", line 693, in actor_method_executor\\n return method(__ray_actor, *args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1139, in __init__\\n super().__init__(*args, **kwargs)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 603, in __init__\\n self.llm = self._create_engine(*args, **kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span\\n return method(self, *_args, **_kwargs)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train/skyrl_train/inference_engines/vllm/vllm_engine.py", line 1216, in _create_engine\\n engine = vllm.AsyncLLMEngine.from_engine_args(engine_args, stat_loggers=stat_loggers)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 251, in from_engine_args\\n return cls(\\n ^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__\\n self.engine_core = EngineCoreClient.make_async_mp_client(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client\\n return AsyncMPClient(*client_args)\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__\\n super().__init__(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__\\n with launch_core_engines(vllm_config, executor_class, log_stats) as (\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/contextlib.py", line 144, in __exit__\\n next(self.gen)\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 936, in launch_core_engines\\n wait_for_engine_startup(\\n File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 995, in wait_for_engine_startup\\n raise RuntimeError(\\nRuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\\n\', RuntimeError(\'Engine core initialization failed. See root cause above. Failed core proc(s): {}\'), \'ray::AsyncVLLMInferenceEngine.__init__\', None, None))'))
Stopping Ray cluster...
Ray cluster stopped
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
Collecting Ray logs from worker jpbo-041-47...
Collecting Ray logs from worker jpbo-041-48...
[RLJobRunner] Crash-time Ray log preservation complete.
[RLJobRunner] Launching trace upload (training exit code: 1):
repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
episodes: last
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
[RLJobRunner] Waiting for trace upload to complete...
[RLJobRunner] Trace upload failed with exit code 1.
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
Collecting Ray logs from worker jpbo-041-36...
Collecting Ray logs from worker jpbo-041-37...
Collecting Ray logs from worker jpbo-041-38...
Collecting Ray logs from worker jpbo-041-39...
Collecting Ray logs from worker jpbo-041-40...
Collecting Ray logs from worker jpbo-041-41...
Collecting Ray logs from worker jpbo-041-42...
Collecting Ray logs from worker jpbo-041-43...
Collecting Ray logs from worker jpbo-041-44...
Collecting Ray logs from worker jpbo-041-45...
Collecting Ray logs from worker jpbo-041-46...
Collecting Ray logs from worker jpbo-041-47...
Collecting Ray logs from worker jpbo-041-48...
Ray log preservation complete
Reference in New Issue View Git Blame Copy Permalink